Airflow integration

Overview

DataXPipe does not connect to Airflow automatically. Your DAGs POST run and check events to the Catalog API when tasks execute.

Works with Airflow 2.x on Astronomer, MWAA, self-hosted Docker, or Kubernetes.

• How execution works →

Setup checklist

• Register your pipeline in DataXPipe (YAML upload or Airflow DAG import)
• Store Catalog URL and API key in Airflow Variables or a secrets backend
• Align warehouse connection ids with connection_ref in your spec
• Add HTTP callbacks at DAG start/end to POST /api/v1/runs/
• Run check tasks that POST results to /api/v1/checks/ or /api/v1/checks/execute

Variables and connections

Configure on your Airflow workers:

• Never commit API keys in DAG source files
• Use the same connection ids in Airflow and DataXPipe for traceability

# Example Airflow Variables
dataxpipe_url = "https://api.dataxpipe.com"
dataxpipe_api_key = "dxp_your_org_key"

# Warehouse — match connection_ref in your pipeline spec
# e.g. warehouse_pg → Postgres connection in Airflow UI

Run event flow

Each DAG execution should produce Catalog records:

DAG start
  → POST /api/v1/runs/  { pipeline_id, status: "running", started_at }
  → transform tasks (your warehouse SQL)
  → check tasks → POST check results
DAG end
  → POST /api/v1/runs/  { pipeline_id, status: "success"|"failed", ended_at }

View in app: https://app.dataxpipe.com/runs

Import an existing DAG

Paste DAG Python source in Get started or call the import API to build catalog metadata without rewriting your orchestration:

curl -X POST https://api.dataxpipe.com/api/v1/import/airflow-dag \
  -H "X-API-KEY: dxp_your_key" \
  -H "Content-Type: application/json" \
  -d '{"source": "...your dag code...", "register": true}'

• Import guide →

Troubleshooting

• Runs page empty after Airflow success — DAG is not POSTing to the Catalog API; verify URL and API key on workers
• 401 on run POST — expired or wrong API key; copy a fresh key from Settings
• Checks missing — check tasks must POST results; verify connection ids match your spec
• Pipeline not found — register the spec first; pipeline_id must match pipeline.name in YAML