Regression QA

Test saved AI outputs without invoking an AI.

Define deterministic assertions, keep a baseline output, paste a current output and surface real regressions: schema failures, missing text, forbidden text, length limits and JSON Pointer values.

JSON / JSONL suitesBaseline → current0 inference cost

Regression suite

JSON array or JSONL

Regression report

Current output is never generated here

T✓

Ready to test

This runner evaluates outputs you provide. It does not call a model or benchmark semantic quality.

Suite format

Assertions supported in v0.4.3

Each case can include prompt for traceability, baseline, current (or output) and assert. Supported assertions include format:"json", contains, notContains, regular expressions, min/max characters, exact values, JSON Schema and JSON Pointer equality.

What counts as a regression?

A regression is deterministic: the baseline passes the same assertions and the current output fails them. Changed output that still passes is reported separately rather than called a failure.