Reliability
SLA data engineering that survives a green DAG
An SLA means the mart is fresh and usable — not that Airflow exited zero. A breach revokes the Passport, pauses the next hop, and shows up on the SLA report.
Why “SLA data engineering” fails in practice
Teams write SLAs in Notion: “executive_kpis by 7 a.m.” Orchestration stays green while the table is 38 hours stale. Nobody revoked the dashboard link. That is an SLA failure with perfect Airflow hygiene.
Real SLA data engineering ties thresholds to datasets, alerts on breach, and withdraws trust artifacts automatically.
SLA building blocks
Freshness thresholds
Warn/error windows per table matched to business hours.
Consumer contracts
Dashboards declare max staleness; producers must stay compatible.
Passport expiry
Shareable warrants that die when the SLA does.
Circuit breakers
A revoked Passport pauses Airflow, dbt Cloud, Hightouch, and Census.
SLA report
Uptime, time to detect, and freshness compliance for the budget review.
Volume floors
Catch empty or anemic loads that still “succeed.”
KPI drift
Wrong-but-green numbers that violate implicit accuracy SLAs.
CI preview
See which SLAs break before the merge.
Stand up SLAs in a day
- 01
Pick three critical marts
Revenue, ops, and exec KPIs — not every table.
- 02
Set freshness + alerts
Import dbt/Airflow and attach checks.
- 03
Issue Passports
Share trust; revoke automatically on breach.
Data engineering SLA FAQ
Promises stakeholders can verify.
What is an SLA in data engineering?+
A data engineering SLA is a measurable promise — usually freshness (data no older than X), completeness (row floors), or availability windows for critical tables. Unlike uptime SLAs for APIs, pipeline SLAs are about whether the data product is usable on time.
How do you monitor a data pipeline SLA?+
Attach freshness and volume checks to the datasets consumers care about, alert on failure, and revoke trust artifacts (like Passports) so stale share links cannot pretend the SLA still holds.
How does DataXPipe enforce SLAs?+
Producer freshness checks, consumer-declared max staleness, and Pipeline Passports. A failed check revokes the Passport and pauses armed Airflow, dbt Cloud, Hightouch, and Census breakers. The SLA report shows data uptime, time to detect, and the share of freshness checks that passed.
Make the SLA visible
Attach freshness to critical marts, declare consumer expectations, and share a Passport that expires with the SLA.