Data engineering is more popular than DS. Unfortunately, ML (over)engineering is a reason for this

I remember when I automated models with crontab and 15 lines of shell

Now you need a huge pile of Airflow, Kafka, Snowflake, Spark, Stitch, Grafana, Presto, Amazon Athena, Redshift etc. behind your XGBoost model.

\>90% of the ML models I've seen weren't even good enough to justify any kind of complex automation. The stupidest models are the clustering ones, the same old k-means fitted 30 times to match with the business knowledge about the client. After 2 years, the …

data data engineering datascience engineering ml popular