Designing Data Systems: Complexity & Modular Design

Web: https://towardsdatascience.com/designing-data-systems-complexity-modular-design-384b28fec672?source=rss----7f60cf5620c9---4

Design thinking approach for data scientists and engineers.

Motivation

Going from notebooks to creating machine learning systems that work in the real-world means shifting context from writing simple scripts, notebooks and visualizing data in the lab environment. Now it's the time to think about building a system and good systems have key characteristics namely resilience, performance and reliability. Often, as we design systems the frictional drag of complexity starts to become apparent due …

complexity data data engineering data science design software development software engineering systems