import { Callout } from '@/components/Callout'

What Orchestrator Should a 3-5 Person Data Team Use?

Short answer (verified April 2026): Use Prefect Cloud. Its free tier covers most teams of this size, the Python-native API is learnable by analytics engineers without dedicated platform staff, and Prefect hosts the control plane so you skip the operational tax. Choose Dagster instead if you already run dbt heavily and want asset-level lineage. Avoid Airflow at this headcount unless you inherited it — the ops burden is not justified for a 3-5 person team.

Ranked Shortlist

1. Prefect — Best default for 3-5 person teams

Prefect has the lowest operational overhead of any major orchestrator as of Q1 2026. The Cloud offering manages scheduling, retries, alerting, and the UI, so there is no server to maintain. The Python decorator syntax (@flow, @task) is approachable for analysts with basic scripting experience, and the free tier historically includes enough runs and users to cover a team of this size. For pipelines that replace manual exports, cron jobs, or fragile Zapier chains, Prefect is the right level of abstraction.

• Pricing at this scale (verified April 2026): Free tier available for small teams; paid tiers start on the Prefect pricing page. Confirm current limits at prefect.io/pricing.
• Deployment: Hybrid — Prefect hosts the control plane, your infra runs the workers.
• Visit Prefect →

2. Dagster — Best if you're dbt-heavy and want lineage

Dagster's differentiator is asset-centric orchestration: you declare the data assets (tables, files, models) your pipelines produce, and Dagster tracks lineage across them. For teams running dbt alongside Python ingestion/transform code, this gives a single pane showing which downstream assets break when an upstream source changes. It is a sharper tool than Prefect but has a steeper learning curve and a heavier conceptual model. Worth the investment for teams whose pipelines have non-trivial interdependencies.

• Pricing at this scale (verified April 2026): Dagster OSS is free and self-hosted; Dagster+ (managed) has usage-based tiers. Pricing published at dagster.io/pricing — confirm before budgeting.
• Deployment: Hybrid (Dagster+) or self-hosted (OSS).
• Visit Dagster →

3. Apache Airflow — Only if you already have it

Airflow is the incumbent and has the largest community, but at 3-5 headcount the operational cost rarely pencils out. Self-hosting requires someone who owns the scheduler, workers, metadata DB, and upgrades. Managed options (AWS MWAA, Astronomer, Google Cloud Composer) remove ops but add monthly fixed cost that Prefect's free tier does not impose. Recommend only if (a) you inherited an Airflow deployment, or (b) you have a specific integration or operator that exists nowhere else.

• Pricing at this scale (verified April 2026): OSS is free; managed Airflow pricing varies by vendor — see aws.amazon.com/managed-workflows-for-apache-airflow/pricing or astronomer.io/pricing. Contact vendor for specifics.
• Deployment: Self-hosted or managed.
• Visit Airflow →

Comparison Table (verified April 2026)

Pricing figures and tier inclusions change; verify on each vendor's site before committing.

How We Evaluated

For a 3-5 person team, the weighting is different than for a 50-person platform org. Our criteria:

1. Operational overhead (weight: 30%) — Hours/month a team spends keeping the orchestrator alive. Small teams cannot absorb a dedicated platform engineer.
2. Cost at small scale (weight: 20%) — Free-tier viability and predictable paid pricing. No surprise bills.
3. Learning curve for analytics engineers (weight: 20%) — Can someone with SQL + intermediate Python own it? Or does it require a dedicated data engineer?
4. Integration with dbt and modern data stack (weight: 15%) — Most small teams land in Snowflake/BigQuery/Postgres + dbt. Native support matters.
5. Lineage and observability (weight: 10%) — When something breaks at 2am, can an on-caller figure out what downstream is stale?
6. Community and hiring signal (weight: 5%) — Can you find help on Stack Overflow? Can you hire for it?

Prefect wins on criteria 1-3. Dagster wins on 4-5. Airflow wins only on 6 and loses badly on 1-3 at this team size.

Runner-Ups Worth Considering

• Mage — Lower-profile open-source orchestrator with a notebook-style UI. Good for teams where analysts want a visual layer. Smaller community than Prefect/Dagster as of Q1 2026 — factor that into a multi-year bet.
• GitHub Actions + dbt Cloud jobs — Not a real orchestrator, but for teams whose entire pipeline is extract → load → dbt run, scheduled GitHub Actions plus dbt Cloud scheduled jobs can be sufficient. Zero new infra. Breaks down once you have branching logic or non-dbt Python steps.
• Kestra — Declarative YAML-based orchestrator with a growing user base. Worth a look if your team prefers configuration over code. Less Python-idiomatic than Prefect; evaluate based on team language preference.

What to Avoid

1. Self-hosting Airflow with no dedicated platform engineer. This is the most common mistake we see at this team size. The metadata DB, scheduler, and worker configuration will consume a disproportionate share of a data engineer's time. If Airflow is a must-have, use MWAA or Astronomer — do not self-host.
2. Using cron + bash scripts "until we grow." This always becomes legacy. By the time someone justifies migrating, there are 40 undocumented cron jobs across three EC2 instances. Adopt a real orchestrator from job #3, not job #30.

FAQ

Q: Is Prefect Cloud's free tier really enough for a 5-person team? A: Historically yes for most small teams, but limits (users, runs, retention) change. Check prefect.io/pricing and model your expected run volume before committing (verified April 2026).

Q: We already run dbt Cloud. Do we need an orchestrator at all? A: If your entire pipeline is Fivetran/Airbyte → dbt → BI, dbt Cloud's scheduler plus your ingestion tool's scheduler may be sufficient. Add an orchestrator when you have Python transforms, reverse ETL, or cross-tool dependencies that need a single source of scheduling truth.

Q: Can we start with Prefect and migrate to Dagster or Airflow later? A: Yes, but migration is non-trivial — you're rewriting flows, not translating them. A better question is whether your pipelines will have heavy asset-lineage needs in 18 months. If yes, start with Dagster. If no, Prefect.

Q: What about Mage, Kestra, or Windmill? A: All viable for specific niches. Smaller communities mean fewer Stack Overflow answers and harder hiring as of Q1 2026. For a team of 3-5 that cannot absorb tool risk, Prefect and Dagster are the safer bets.

Q: How many hours per week should we expect to spend managing our orchestrator? A: Rule of thumb: if you're spending more than 2 hours/week on orchestrator operations (as opposed to building pipelines on it), you've chosen the wrong one for your team size. Prefect Cloud and Dagster+ should stay well under that threshold.