[D] Tips for ML workflow on raw data

I am an ML researcher and work on real-world problems. Unlike core ML research where datasets are pretty much defined (in most cases), I usually deal with the raw data. We have live data (time-series) recorded and available at the servers.

We have different objects and each object can have different sensors. Experiments usually compare different sensors among the same objects and the other way around. And it can be more than 100k records for one sensor/object.

Currently, my workflow …

data machinelearning ml tips workflow