Pipeline specs

Spec structure

Every pipeline is a YAML document with pipeline metadata, optional connections, sources, transforms, targets, checks, and lineage.

pipeline:
  name: orders_sync
  description: "Sync orders to warehouse"
  owner: data-platform
  environment: dev
  schedule: "0 6 * * *"
  tags: [finance]

sources:
  - id: raw_orders
    type: postgres
    connection_ref: warehouse_pg
    object: public.orders
    load_mode: full

targets:
  - id: mart_orders
    type: postgres
    connection_ref: warehouse_pg
    object: public.mart_orders
    write_mode: append

checks:
  - id: freshness_mart
    type: freshness
    target: mart_orders
    max_delay_minutes: 1440

Supported check types

Checks reference a target dataset id from sources, transforms, or targets.

• schema_match — column types match expectation
• not_null — required columns present
• freshness — data updated within max_delay_minutes
• row_count — volume bounds
• uniqueness — key uniqueness
• accepted_values — enum validation
• kpi_baseline — metric vs rolling baseline (wrong-but-green)
• row_count_drift — volume vs prior successful runs
• distribution_drift — null % / uniqueness / mean bands

Connections

Declare connections in the spec and reference them via connection_ref on sources and targets.

Use your secrets manager or environment-specific values for credentials — do not commit production passwords.

connections:
  - id: warehouse_pg
    type: postgres
    config:
      host: your-db.example.com
      database: analytics
      sslmode: require

Lineage

Add optional lineage.edges to link datasets. Edges use from, to, and via (pipeline or transform name).

Lineage powers blast-radius analysis in failure explanations.

lineage:
  edges:
    - from: raw_orders
      to: mart_orders
      via: orders_sync

Validate a spec

Validate in the app before registering (Pipelines → Create pipeline → Validate), or call the validate API:

curl -X POST https://api.dataxpipe.com/api/v1/specs/validate \
  -H "Content-Type: application/json" \
  -d '{"spec_yaml": "pipeline:\n  name: orders_sync\n  ..."}'