How the seismograph works
An independent record of when the model APIs that agent fleets depend on quietly change behaviour. It measures each model against its own past, and never ranks models.
What it measures
Every day a fixed, private probe battery runs against the model APIs in the roster. Responses are graded by deterministic, versioned code - no LLM judge. The public board shows per-model, per-dimension pass rates, token and latency distributions for the active dimensions. No raw model text and no per-probe detail is ever published.
Two batteries
A daily canary (a fast, shallow tripwire) runs against the daily-cadence models; a weekly deep battery runs against all models for more statistical power. They are different rulers and their series are never pooled.
Movements, watches, and the decision rules
A finding compares one model to its own rolling, witnessed baseline. The rule depends on the metric:
| metric | rule |
|---|---|
| probe pass rate | Two-proportion test with Benjamini-Hochberg FDR control. Movement survives at α=0.01; watch is nominally significant but unconfirmed. |
| thinking-token use | Relative-shift threshold - a serving-configuration tripwire. Not a significance test. |
| latency | Relative-shift threshold, treated as operational context, never a standalone drift claim. |
Under-alerting is deliberate policy: the feed publishes raw movement, and alert-grade announcements require editor sign-off.
Witnessing
Every run is signed into the Rekor public transparency log at measurement time. A baseline provably predates the event it is later cited against, so the record cannot be back-dated.
Readings, not ratings
The seismograph never grades, ranks, scores, or certifies a model. It answers "did this model change?", never "which model is best?". Cause is a separate question it does not settle.
What it is not, yet
The instrument is early. Its strongest asset is accumulated measurement that predates a disputed change. What it still needs to establish is its detection power: how large a change it reliably catches, how quickly, and its false-alarm rate.
Full detail
Methodology charter · README · code, readings, and witness bundles.