Data Validation with Great Expectations and Argo Workflows

Great Expectations goes Kubernetes native

Recently at work we’ve been discussing what our dream machine learning pipeline would look like. As background research we came across the Great Expectations package, which looks really promising as a data validation step in our pipeline. In this article I’m going to explore using Great Expectations as part of an Argo Workflow to run data validation during data preprocessing.

Disclaimer: I’m not an expert! This is a learning process for me, so what I …

argo data data engineering data validation great expectations machine learning mlops validation workflows