flâneur

From Data Engineer to YAML Engineer

juhache.substack.com · 1,013 words · saved by 1 readers

Over the years, the industry has constructed layers upon layers to develop increasingly complex software. The same trend is happening in the data world. An increasing number of tools are emerging to standardize the construction of data pipelines, pushing towards a declarative paradigm. Engineers spend less and less time coding and more and more parametrizing and coordinating functional building blocks. In this cross-post with Benoît Pimpaud, we wanted to examine this trend from various angles within the stack: Cloud processing: config-only event enrichment pattern with AWS Pipes Data Warehouse: config-only table sync with Snowflake Dynamic Tables Orchestrator: data pipeline templates (“Blueprints”) with Kestra Thanks for reading Ju Data Engineering Newsletter! Subscribe for free to receive new posts and support my work. The evolution of cloud infrastructure towards managed services has facilitated a shift toward a declarative paradigm. Consider a basic AWS workflow: A customer places a

From Data Engineer to YAML Engineer Ju Data Engineering Weekly - Ep 40 Julien Hurault and Benoit Pimpaud Nov 22, 2023 59 3 6 Share Software has always been a matter of abstraction. Over the years, the industry has constructed layers upon layers to develop increasingly complex software. The same trend is happening in the data world. An increasing number of tools are emerging to standardize the construction of data pipelines, pushing towards a declarative paradigm. Engineers spend less and less time coding and more and more parametrizing and coordinating functional building blocks. In this cross

saved by

related reading