deepseek-chat@deepseek: instruction following pass rate fell
Movement on the daily battery. This model measured against its own witnessed baseline.
| model | deepseek-chat@deepseek |
|---|---|
| identity | alias · a floating name the provider may re-point; a change here means what is served under this name changed |
| dimension | Instruction following |
| metric | pass_rate |
| decision rule | Survives Benjamini-Hochberg FDR at α=0.01 |
| first flagged | 2026-10-17 · latest reading |
| coverage | 0 call errors on this reading |
Dated records alongside this reading
An annotation attaches a dated external record (a provider changelog entry, a documented deprecation, a known configuration artefact) to a reading the instrument already produced. It changes no reading and establishes no cause; the instrument detects, attribution stays a human judgment.
Provider record on the onset date · 2026-10-17
DeepSeek's API updates page carries an entry dated 2026-10-17: "DeepSeek V4.1 Flash has been upgraded to DeepSeek-V4.1-Flash-1017", described as a minor update with shorter outputs, served behind the deepseek-flash alias. The same page (2026-04-24 entry) routes deepseek-chat to V4.1 Flash's non-thinking mode. The first reading taken on or after that entry is the one that flagged, and the pass rate has held at the new level since. Same reading, same provider: deepseek-reasoner's mean thinking tokens on the verbosity probes fell from about 1,000 to about 200 and stayed there (visible in series.json; the detector logged it as a watch on 2026-10-21). The instrument flagged the change blind; the provider's dated record is the attribution candidate, not an established cause.
What changed, probe by probe (follow-up) · 2026-10-17
Regrading the witnessed raw sessions probe by probe: two of the five instruction-following probes flipped on 2026-10-17 and have failed identically on every reading since, three samples each at temperature 0. One probe asks for an answer that must not contain a named term; through 2026-10-16 the answer omitted it, from 2026-10-17 the answer contains it, at unchanged length. One probe sets a word-count range; through 2026-10-16 the answer sat inside it, from 2026-10-17 the answer is one word short. The other three probes (an exact-match answer, a format constraint, a longer structured answer) are unchanged. deepseek-reasoner, the thinking mode of the same model behind a sibling alias, still passes all five. On 2026-10-21 the dated snapshot deepseek-ai/DeepSeek-V4.1-Flash-1017 on an independent host (DeepInfra, separate key) failed the same two probes the same way on its first reading. Probe text is withheld per the charter; the raw sessions are witnessed and the grader is public.
Trace the evidence
The witnessed reading for this date and battery, and the Rekor witness bundle. The full series is in series.json.
This entry records an observation under the stated decision rule. Watches remain unconfirmed; latency is operational context. The instrument does not establish the cause of a change.