Web: https://www.reddit.com/r/datascience/comments/schi6m/considerations_for_automation_of_a_batch_process/

Jan. 25, 2022, 4:23 p.m. | /u/Miriel18

Data Science reddit.com

Hi everyone,

I am designing an architecture to automate a batch data science process. At a high level, the system has the following steps:

1a-) Data update (geospatial data, so it is a new satellite image, which is large.)

1b-) Step 1a, runs once a week. If there is an update, the system continue to run the steps below. If there is not update, it turns back to Step 1a.

2-) Relevant metrics and indices are estimated

3-) New metrics …

automation datascience process

Senior Data Analyst

@ Fanatics Inc | Remote - New York

Data Engineer - Search

@ Cytora | United Kingdom - Remote

Product Manager, Technical - Data Infrastructure and Streaming

@ Nubank | Berlin

Postdoctoral Fellow: ML for autonomous materials discovery

@ Lawrence Berkeley National Lab | Berkeley, CA

Principal Data Scientist

@ Zuora | Remote

Data Engineer

@ Veeva Systems | Pennsylvania - Fort Washington