flâneur

Prefect - The New Standard in Dataflow Automation - Prefect

prefect.io · 488 words · saved by 1 readers

Prefect is a new data workflow management system, designed for modern infrastructure. Prefect provides a flexible open-source Python framework to easily combine tasks into workflows, then deploy, schedule, and monitor their execution through the Prefect UI or API. Create powerful data pipelines and let Prefect take care of scheduling, infrastructure, error handling, retries, logs, triggers, data serialization, parameterization, dynamic mapping, caching, concurrency, and more.

Trusted in Production by: One platformfrom ML to AI Data you can depend on Replace brittle cron jobs with pipelines that heal themselves, so your data always arrives on time. from prefect import flow, task @task(retries=3) def extract(source): ... @flow def daily_revenue_elt(): load(transform(extract("warehouse"))) Deployments / daily_revenue_elt · k8s-prodRunning · 0:00 Success rate 99.2%▲ +0.3% Avg duration 1.4s▼ −0.2s Runs · 24h 2,182▲ +8.4% p95 latency 4.1s▲ +0.4s Task timeline6 tasks ingest3s normalize6s validate5s enrich13s score5s load3s 0:000:100:200:30…

saved by

related reading