Building Data Pipelines Part 2: How To Create A Task In Luigi

How do you build your first data pipeline?

First, you need to pick which framework will work best for you.

In this video we will discuss how you can use the Python library Luigi to start building your first task.

Now, if you want a full course, you can check out the top data engineering courses video I have: https://www.youtube.com/watch?v=lVj0RlSxTXk

The code for this video is below.

Code From Video class StockTwitScrape(luigi.Task):

def output(self):
    return luigi.LocalTarget("raw_data.json")

def run(self):
    url = "https://api.stocktwits.com/api/2/streams/symbol/wy.json"
    content = requests.get(url).text
    with self.output().open("w") as f:
        json.dump(json.JSONDecoder().decode(content),f)

If you need data consulting help, then reach out to our team here: https://www.theseattledataguy.com/​​​​

Also, if you'd like to read up on my updates about the data field, then you can sign up for our newsletter here. https://seattledataguy.substack.com/​​​​

Check out my Medium here: https://medium.com/@SeattleDataGuy​​​ ... https://www.youtube.com/watch?v=vtZba4pnGuQ