Deploy Long-Running ETL Pipelines to ECS with Fargate

To keep things simple and costs to a minimum

ETL stands for Extract, Transform, and Load. An ETL pipeline is essentially just a data transformation process — extracting data from one place, doing something with it, and then loading it back to the same or a different place.

If you are working with natural language processing via APIs, which I’m guessing most will start doing, you can easily hit the timeout …

data deep-dives devops etl programming