✳flâneur — a map of the web's best reading

A guide on Airflow best practices

ayc-data.com · 5,267 words · saved by 1 readers

Analytics Engineering - creating, calculating or building analytics or data warehouse models using SQL Machine Learning - training, evaluating and deploying a ML model. Data workflows range from simple to very complex, with hundreds of steps and branching/dependencies. Due to its complex nature and inter-related steps, managing data workflows is often a daunting task. Fortunately, we have Airflow! So in this blog I’ll explore some best practices to hopefully make the Airflow experience less daunting for newcomers. Apache Airflow Airflow (https://airflow.apache.org/) is a leading Apache open-source data orchestration tool for running data workflows - used in many tech companies. For example, Uber and Airflow’s original creator, Airbnb both use Airflow extensively. Airflow provides an easy-to-use web UI, with integrated logging, monitoring, workflow run management, dependency handling. This includes managing dependencies and task monitoring/management through its web UI: monitor the stat

What is a Data Workflow and why it is hard to manage A data workflow is a well-defined series of steps to accomplish a particular data-related task . Common examples include: Data pipelines - Extracting, Loading and Transforming (ELT) data from one data source to another Analytics Engineering - creating, calculating or building analytics or data warehouse models using SQL Machine Learning - training, evaluating and deploying a ML model. Data workflows range from simple to very complex, with hundreds of steps and branching/dependencies. Due to its complex nature and inter-related steps, managin

Explore this link on the map →

related reading