Problem guide
Silent pipeline failures
Airflow is green. Slack is quiet. The CFO asks why revenue didn't move. Revoke the Passport and pause the sync before the stale mart keeps moving.
What orchestrators miss
Batch orchestrators report task success. Stakeholders consume tables. Monitoring has to attach to dataset freshness and volume, not only exit codes.
| Signal | Orchestrator only | + Dataset checks |
|---|---|---|
| Task exception | Caught | Caught |
| Zero-row load | Missed | Volume check |
| Stale partition / lag | Missed | Freshness check |
| KPI drifted but rows present | Missed | KPI / distribution drift |
How DataXPipe catches them
- Step 1
Import what you already run
Paste a dbt manifest or Airflow DAG — bootstrap lineage and default freshness checks.
- Step 2
Watch mart tables, not just tasks
Put error-severity freshness on executive KPIs and billing tables first.
- Step 3
Share a Passport, not a wiki
When checks pass, issue a Pipeline Passport. Revoke automatically when data goes stale.
- Step 4
Pause the next hop
The revoked Passport pauses the DAG and the reverse-ETL sync so the stale mart does not keep shipping.
Silent failures FAQ
Green orchestration vs trustworthy data.
What is a silent pipeline failure?+
A silent failure is when the orchestrator reports success (green DAG, exit code 0) but the data product is stale, empty, or drifted. Stakeholders discover it in a dashboard — not in your monitoring.
Why doesn't Airflow catch stale data?+
Airflow marks tasks successful when they finish without exceptions. It does not know whether a table's MAX(updated_at) is within your SLA, whether zero rows loaded, or whether KPIs drifted from baseline.
How does DataXPipe detect silent failures?+
Attach freshness, volume, and KPI-drift checks to pipeline targets. A failed check revokes the Passport and pauses armed circuit breakers: the Airflow DAG, in-progress dbt Cloud runs, and Hightouch or Census syncs.
Does detection stop the sync?+
Yes, when a circuit breaker is armed on that dataset. The pause happens because the Passport was revoked, including when Airflow already reported success. Rerun stays locked until the Passport is active again.
Catch the next silent failure before finance does
Free tier includes 2 pipelines. Guided demo shows a stale mart with Passport revoke.