Automatically Managing Data Pipeline Infrastructures With Terraform

I know the manual work you did last summer

Introduction

A few weeks ago, I wrote a post about developing a data pipeline using both on-premise and AWS tools. This post is part of my recent effort in bringing more cloud-oriented data engineering posts.

However, when mentally reviewing this post, I noticed a big problem: the manual work.

Whenever I develop a new project, whether real or fictional, I always try to …

data engineering data science devops terraform