Data Engineering Portfolio Project Part (2/5) - Setting Up Airflow Tasks To Connect Postgres And S3

Data engineering projects can be a great way to show off your skills.

But they can be hard to put together.

The purpose of this series is to help you put together a data engineering project.

For this video you will need to have an EC2 instance set up as well as an RDS Postgres instance and S3 on AWS.

In addition we will use Airflow for managing our tasks.

For the commands to set-up a connection between Postgres and S3. https://gist.github.com/bAcheron/e5def0429d79c9312fd1806912b3cbcc

Video Breakdown

0:00 Intro 1:52 Setting Up Postgres to S3 Connection 5:20 Airflow Set Up 8:50 Airflow - Writing Your First Pipeline 13:26 Airflow - Setting Up A Postgres Connection

If you enjoyed this video, check out some of my other top videos.

What Skills Do Data Engineers Need? https://www.youtube.com/watch?v=LgSHaOvNodA

Data Engineering Project Ideas https://youtu.be/LJkVvNWlO0g

If you want to learn more about machine learning, check out DataCamps Machine Learining Course

https://bit.ly/3BeLEml

If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.

https://seattledataguy.substack.com/​​

Or check out my blog https://www.theseattledataguy.com/

Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio

Subscribe: https://www.youtube.com/channel/UCmLGJ3VYBcfRaWbP6JLJcpA?sub_confirmation=1

About me: I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consults on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.

I do participate in affiliate programs, if a link has an "" by it, then I may receive a small portion of the proceeds at no extra cost to you. ... https://www.youtube.com/watch?v=30VDVVSNLcc