Considerations for Automation of a Batch Process

Hi everyone,

I am designing an architecture to automate a batch data science process. At a high level, the system has the following steps:

1a-) Data update (geospatial data, so it is a new satellite image, which is large.)

1b-) Step 1a, runs once a week. If there is an update, the system continue to run the steps below. If there is not update, it turns back to Step 1a.

2-) Relevant metrics and indices are estimated

3-) New metrics …

automation datascience process