CI/CD for LLM quality

Catch the regression before it merges, not after a customer files a ticket.

EvalCI runs your evals on every change, compares the result against a baseline you set explicitly, and fails the build when quality drops — self-hosted, in your own infrastructure, from day one.

Apply as a design partner → View evalci-core on GitHub

python examples/regression_demo.py

The gap

Every eval tool tells you the score changed. Almost none of them block the merge.

Code gets a test suite and a CI gate. Prompt and model changes usually get someone eyeballing five outputs before shipping — until the regression that slips through becomes a support ticket.

✕

No explicit baseline

Most platforms compare against "recent runs," which drifts every time your dataset or config changes. You want to know: did it get worse than the version you approved.

✕

No real CI gate

A dashboard that shows a dropped score after the fact isn't a gate. A severity-aware exit code that fails the check before merge is.

✕

Self-hosting locked behind Enterprise

If your eval traffic can't leave your infrastructure, "contact sales" isn't an onboarding flow. It should run in your cluster from the first `docker compose up`.

Where this fits

An honest comparison, not a hero chart

The eval-tooling space is real and getting crowded. Here's where EvalCI is actually different — and where it isn't.

	EvalCI	Braintrust	LangSmith	Langfuse
Self-hosted from day one	Yes	Enterprise only	Enterprise only	Yes
Explicit regression baseline	Core primitive	Score-based	Score-based	Score-based
Severity-aware CI exit codes	Yes	Via Action	Via integration	Manual
Open-source core engine	Apache-2.0	Closed	Closed	MIT
Full observability suite (tracing, agents)	Roadmap	Yes	Yes	Yes

How it works

Three steps. The third one is the one that matters.

This is a real sequence — each step depends on the one before it — so it's numbered because the order actually carries information.

01

Set an explicit baseline

Pick the last run you actually approved. Every future comparison is against that run — never an inferred "recent average" that drifts.

baseline_run_id = "run_8f2a1c"

02

Run and score on every change

LLM-as-judge and lexical metrics run against your dataset. Triggered by a push, a schedule, or a manual call.

evals run --wait --git-branch=$BRANCH

03

Gate the merge

A regression at or above your configured severity fails the check. A warning-level dip doesn't block a critical-gated pipeline — you choose the bar.

--fail-on-regression critical

Infrastructure

Runs where your data already lives

Local dev on Docker Compose. Production on your own Kubernetes cluster. Nothing about your eval traffic has to leave your network.

docker-compose

Local, in one command

API, worker, scheduler, dashboard, Postgres, Redis, object storage — one stack, one command.

# clone, set your model API key, go
docker compose up -d
make migrate

kubernetes

Production, autoscaled

API tier scales under an HPA. Scheduling runs as a dedicated single-replica deployment — never duplicated across replicas.

# apply the manifests
kubectl apply -f infra/kubernetes/

Open core

The detection engine is yours to read, run, and audit

Not a watered-down teaser — the actual regression detector, trend analysis, and scoring engine, with the real test suite ported alongside it.

Zero-infra demo — no Docker, no database, no API key
48 tests ported directly from the production suite
Apache-2.0, importable as a real dependency

$ pip install evalci-core
$ python examples/regression_demo.py
[CRITICAL] overall: 0.910 → 0.820
regression_detected=True