DataXPipe
← Back to app

Product

How execution works

DataXPipe catalogs and observes pipelines. Your orchestrator runs ETL and reports back over the API.

Catalog vs executor

DataXPipe stores pipeline specs, lineage, run history, and check results. It does not run your warehouse jobs.

Your orchestrator (Airflow, dbt, Prefect, cron, CI, etc.) executes transforms and checks in your environment, then POSTs lifecycle events to the Catalog API.

You (app.dataxpipe.com)
  → register specs, view runs & lineage

DataXPipe Catalog API (api.dataxpipe.com)
  → pipelines, runs, checks, lineage

Your orchestrator + warehouse
  → runs SQL, moves data
  → POST run + check events to the Catalog API

Run lifecycle

Every production run follows the same contract:

  • Start — POST /api/v1/runs/ with pipeline_id and status running
  • Execute — your orchestrator runs transforms against your warehouse
  • Check — POST check results (or use /api/v1/checks/execute for SQL checks)
  • Finish — POST /api/v1/runs/ with status success or failed and ended_at

Warehouse connections

Pipeline specs use connection_ref to link datasets to a connection id. Configure the same ids in your orchestrator for ETL, and register them in DataXPipe for check execution.

App Run button

The Run button on the Pipelines page creates a catalog run record so you can verify the API and see activity on the Runs dashboard.

It does not trigger Airflow or execute warehouse SQL. Production runs always come from your orchestrator.

Other orchestrators

Any system that can make HTTPS requests can integrate. There is no proprietary agent.

  • dbt — import manifest.json, run dbt in CI, POST run status when jobs finish
  • Prefect / Dagster / Temporal — call /api/v1/runs/ at flow start and end
  • GitHub Actions — validate spec in CI, run warehouse SQL, POST run status from a final step
  • Custom scripts — same REST contract as Airflow callbacks
curl -X POST https://api.dataxpipe.com/api/v1/runs/ \
  -H "X-API-KEY: dxp_your_key" \
  -H "Content-Type: application/json" \
  -d '{"pipeline_id":"orders_sync","status":"running"}'

Airflow configuration

Set your Catalog URL and organization API key on Airflow workers (Variables or env vars). See the Airflow integration guide for details.

  • Catalog URL — https://api.dataxpipe.com
  • API key — from Settings after signup (dxp_...)
← All documentation