End-to-End Data Engineering System on Real Data with Kafka, Spark, Airflow, Postgres, and Docker

This article is part of a project that’s split into two main phases. The first phase focuses on building a data pipeline. This involves getting data from an API and storing it in a PostgreSQL database. In the second phase, we’ll develop an application that uses a language model to interact with this database.

Ideal for those new to data systems or language model applications, this project is structured into two segments:

• This initial article guides you through constructing a …

airflow data engineering docker kafka spark