Python - Data Engineering with Python and AI/LLMs - Data Loading

Master data ingestion for data engineering with Python. Learn to tackle common pipeline failures like schema changes and API limits by adopting the mindset and practices of a senior platform engineer. This course covers essential techniques including extracting data from APIs, automatic schema management, incremental loading, and orchestrating scalable, automated workflows using modern tools.

Course developed by Alexey Grigorev & Adrian Brudaru. 💻 Code: https://github.com/dlt-hub/dlthub-education/tree/main/courses/freecodecamp/de_with_dlt_2025 🏗️ dlthub.com provided a grant to make this course possible.

⭐️ Contents ⭐️ Alexey's part 0:00:00 1. Introduction 0:08:02 2. What is data ingestion 0:10:04 3. Extracting data: Data Streaming & Batching 0:14:00 4. Extracting data: Working with RestAPI 0:29:36 5. Normalizing data 0:43:41 6. Loading data into DuckDB 0:48:39 7. Dynamic schema management 0:56:26 8. What is next?

Adrian's part 0:56:36 1. Introduction 0:59:29 2. Overview 1:02:08 3. Extracting data with dlt: dlt RestAPI Client 1:08:05 4. dlt Resources 1:10:42 5. How to configure secrets 1:15:12 6. Normalizing data with dlt 1:24:09 7. Data Contracts 1:31:05 8. Alerting schema changes 1:33:56 9. Loading data with dlt 1:33:56 10. Write dispositions 1:37:34 11. Incremental loading 1:43:46 12. Loading data from SQL database to SQL database 1:47:46 13. Backfilling 1:50:42 14. SCD2 1:54:29 15. Performance tuning 2:03:12 16. Loading data to Data Lakes & Lakehouses & Catalogs 2:12:17 17. Loading data to Warehouses/MPPs,Staging 2:18:15 18. Deployment & orchestration 2:18:15 19. Deployment with Git Actions 2:29:04 20. Deployment with Crontab 2:40:05 21. Deployment with Dagster 2:49:47 22. Deployment with Airflow 3:07:00 23. Create pipelines with LLMs: Understanding the challenge 3:10:35 24. Create pipelines with LLMs: Creating prompts and LLM friendly documentation 3:31:38 25. Create pipelines with LLMs: Demo

🎉 Thanks to our Champion and Sponsor supporters: 👾 Drake Milly 👾 Ulises Moralez 👾 Goddard Tan 👾 David MG 👾 Matthew Springman 👾 Claudio 👾 Oscar R. 👾 jedi-or-sith 👾 Nattira Maneerat 👾 Justin Hual

Learn to code for free and get a developer job: https://www.freecodecamp.org Read hundreds of articles on programming: https://freecodecamp.org/news