DataXPipe for ML teams

Put a quality gate in front of batch scoring

Keep training, evaluation, and orchestration in the tools you already use. DataXPipe checks that evaluation data is fresh and that a metric such as ROC AUC meets your release bound before downstream batch scoring proceeds.

1. Evaluate as usual

Your existing Airflow, CI, or other orchestrator runs training and model evaluation, then asks DataXPipe to execute the SQL check.

2. Check a real metric

Use a SQL query that returns one scalar as metric and set a minimum, maximum, or both. Invalid or missing metrics fail closed.

3. Protect the next hop

A failed check can revoke the target Passport. Configure lineage and a circuit breaker for the downstream batch scoring workflow to pause it.

Pipeline spec example

pipeline:
  name: daily_risk_model_release
  description: "Validate evaluation metrics before batch scoring"
  owner: ml-platform
  environment: production
  schedule: "0 6 * * *"
  tags: [ml, batch]

sources:
  - id: model_eval_metrics
    type: warehouse
    connection_ref: analytics
    object: ML.MODEL_EVAL_METRICS
    load_mode: full

targets:
  - id: batch_scores
    type: warehouse
    connection_ref: analytics
    object: ML.BATCH_SCORES
    write_mode: append
    passport:
      enabled: true
      sla_minutes: 1440

checks:
  - id: evaluation_freshness
    type: freshness
    target: model_eval_metrics
    watermark_column: evaluated_at
    max_delay_minutes: 1440
  - id: roc_auc_release_gate
    type: ml_metric_threshold
    target: model_eval_metrics
    sql: "SELECT ROC_AUC AS metric FROM ML.MODEL_EVAL_METRICS ORDER BY EVALUATED_AT DESC LIMIT 1"
    min_value: 0.82

lineage:
  edges:
    - from: model_eval_metrics
      to: batch_scores
      via: daily_risk_model_release

DataXPipe validates and gates workflows; it does not train models, select acceptance thresholds, or replace your model registry. Choose thresholds from your product risk and evaluation practice.