Prefect - The New Standard in Dataflow Automation - Prefect
Prefect is a new data workflow management system, designed for modern infrastructure. Prefect provides a flexible open-source Python framework to easily combine tasks into workflows, then deploy, schedule, and monitor their execution through the Prefect UI or API. Create powerful data pipelines and let Prefect take care of scheduling, infrastructure, error handling, retries, logs, triggers, data serialization, parameterization, dynamic mapping, caching, concurrency, and more.
Trusted in Production by: One platformfrom ML to AI Data you can depend on Replace brittle cron jobs with pipelines that heal themselves, so your data always arrives on time. from prefect import flow, task @task(retries=3) def extract(source): ... @flow def daily_revenue_elt(): load(transform(extract("warehouse"))) Deployments / daily_revenue_elt · k8s-prodRunning · 0:00 Success rate 99.2%▲ +0.3% Avg duration 1.4s▼ −0.2s Runs · 24h 2,182▲ +8.4% p95 latency 4.1s▲ +0.4s Task timeline6 tasks ingest3s normalize6s validate5s enrich13s score5s load3s 0:000:100:200:30…
saved by
related reading
- Welcome to Prefect - Prefect Docsorion-docs.prefect.io
- Prefect Cloudapp.prefect.cloud
- Cookbookcookbook.openai.com
- Durable Execution Solutionstemporal.io
- Airflow Architecture: Key Components & Best Practiceshevodata.com
- Streamlit • A faster way to build and share data appsstreamlit.io
- Advanced AI Workflow Automation Software & Tools - n8nn8n.io
- A guide on Airflow best practicesayc-data.com
- Durable Execution Solutions | Temporaltemporal.io
- Using Apache Airflow to monitor data pipelinesbiomadeira.github.io
- Why data scientists shouldn’t need to know Kuberneteshuyenchip.com
- Agent Ontologies - by Philippe Page - Mainframeblog.mainfra.me