Data Quality Comparison on AWS Glue and Great Expectations

In my previous articles (post one and post two), I described how you can handle homogeneous data sources stored as Apache Parquet files of moderate size (~500 MB). But what if you need to deal with Big Data? How can you test it by using Great Expectations on AWS? How can you compare two non-homogeneous datasets? In this article, I will explore one way to do just that.

Challenges

The Provectus Data Engineering …

aws aws glue comparison data data quality data science great expectations machine learning quality