CI/CD for models

Application CI/CD ships code; model CI/CD ships a bundle of code, data, weights and configuration whose “tests” are statistical, whose builds take hours on GPUs, and whose rollback has to restore the preprocessing along with the weights. The deployment mechanics are the ordinary ones (Deployment Strategies); this note is what changes when the artifact is a model.

What is different

The consequence is two pipelines, not one. CI runs on every commit and stays cheap: unit tests for the feature and preprocessing code, schema checks on sample data, a smoke run of training on a tiny slice. The training pipeline runs on a trigger — schedule, data arrival, a drift alarm — and ends in a candidate in the registry, not in production.

The training pipeline as code

data snapshot ─▶ validate ─▶ train ─▶ evaluate ─▶ register
 (DVC / lakeFS   (schema,     (GPU     (vs champion  (alias:
  / Delta)        drift)      job)      + slices)     challenger)

Each stage is a step in an orchestrator — Airflow, Prefect, Dagster, Kubeflow, SageMaker or Vertex Pipelines — and each produces a versioned artifact. The data snapshot is a hash the run records; the code is a commit; the config is a file. That triple is what “reproducible” means (Experiment tracking and the model registry), and it is the first thing missing when a model cannot be rebuilt.

The eval gate

The step that makes it CD rather than a cron job:

import mlflow

def promote_if_better(candidate_uri: str, name: str, eval_set) -> bool:
    client = mlflow.MlflowClient()
    cand = evaluate(candidate_uri, eval_set)          # dict of metrics
    base = evaluate(f"models:/{name}@champion", eval_set)
    gates = [
        cand["f1"] >= base["f1"] - 0.002,             # no regression
        cand["f1_worst_slice"] >= 0.80,               # floor per segment
        cand["p95_latency_ms"] <= 1.2 * base["p95_latency_ms"],
    ]
    if all(gates):
        v = client.create_model_version(name, candidate_uri, run_id=...)
        client.set_registered_model_alias(name, "challenger", v.version)
    return all(gates)

Three properties of a gate worth stating. It compares to the champion on the same held-out set, not to an absolute number from last quarter. It checks slices — the worst segment, not the average — because the average hides the customer who is about to churn.

And it is noise-aware: a 0.1-point improvement on a 500-row set is not an improvement, so the threshold has a tolerance and the set is large enough that the tolerance means something.

For an LLM system the “model” is the prompt, the retrieval config, the engine version and the quantisation as much as the weights, and every one of those changes runs the same golden-set gate (Building eval sets).

Promotion and rollout

The gate promotes to challenger; production still points at champion. Getting from one to the other is progressive delivery with a model-shaped twist:

• Shadow first: the challenger receives mirrored traffic, its outputs are logged and compared offline, and it affects nobody. This is the only way to see the model on real traffic before it can do harm.
• Canary at 1–5% with the online metric watched — not accuracy, which needs labels, but the proxies: validation-failure rate, latency, fallback rate, the business KPI where it is fast.
• Promote by moving the alias; roll back by moving it back. Serve by alias, never by version, so the rollback is a registry write and not a deployment.

Argo Rollouts and Flagger drive the traffic split (Progressive delivery on Kubernetes — Argo Rollouts & Flagger); the model registry drives what the pods load.

The serving artifact

Two ways to get weights into a pod, and the trade-off is a real one:

• Bake the weights into the image. Immutable, reproducible, slow to build, and a 16 GB image per model version.
• Pull by alias at startup. Small image, instant promotion, and a pod that starts differently depending on when it started. Pin the resolved version into the pod’s logs and metrics, or the first incident is “which model was this”.

Whichever you choose, the preprocessing code and the model travel together — a feature transform that changed between training and serving is the classic silent failure, and it is why the feature pipeline is versioned in the same run.

Closing the loop

Monitoring (Monitoring and drift) feeds the trigger: drift past a threshold, or labels arriving that show accuracy fell, starts the training pipeline; the gate decides whether the result is better; the rollout puts it in front of users.

When every step is automatic the system retrains itself — and the gate is the only thing standing between a bad data day and a bad model in production, which is why it is reviewed harder than the training code.

Where it breaks in production

• Training in a notebook, deploying from a laptop. No data hash, no commit; the model works and cannot be rebuilt.
• A gate on the average. The worst slice regressed; the average rose; the promotion went through.
• Preprocessing drift. Feature code updated in the training repo, not in the serving image. Same weights, different inputs.
• Rollback of weights only. The old model with the new tokenizer, or the old model with the new schema. Roll back the bundle.
• Manual promotion by Slack. “Deploy v14” with no record of what v14 was evaluated against. The alias move is the audit trail; keep it in the pipeline.

Related

• Experiment tracking and the model registry - MLflow and W&B, the registry, aliases.
• Monitoring and drift - what triggers the loop.
• GitHub Actions - the runner side, including self-hosted GPU runners.
• Progressive delivery on Kubernetes — Argo Rollouts & Flagger - canary and shadow mechanics.
• Evaluating document extraction - a concrete gate for an extraction model.