You shipped the release, the migration guide, maybe an llms.txt or a skill. Then a user asks their agent to "use the latest", and it writes the API you removed. The code often still runs, so nobody tells you.
The short llms.txt index was read 4 times, each time because an AGENTS.md pointed to it.
A vendor shipped a skill for exactly this upgrade. Not installed, it was invisible.
The tests passed, so users never noticed. They will at the next migration.
fig. 2: how often agents opened each channel when it was set up for them, over the 27 runs per channel across all three libraries.
A short brief in AGENTS.md moved the current-client pick from 4/9 to 9/9 on one library and cut tokens by a third on another. No other channel changed the result measurably.
Codex listed the docs MCP tools in every run and called them in none of its 9 runs. It read the package and searched the web instead.
A vendor's hosted docs MCP advertised OAuth without client registration. Codex marked it "not logged in" and hid every tool.
Package, what changed, the channels you ship or plan.
@acme/sdk 3.0Built around your change, with a hidden test for the result and the API used.
grade.py · hiddenFresh project, no memory, one channel on. 3 runs per agent per channel.
5 × 3 × 3 = 45 runsPer agent and channel, every log, and the lines to change.
report + fix listProcess automation SDK, release 8.10. Task: "use the official Node SDK, the current version from npm" to deploy a process and start an instance.
| Channel | Claude | Codex | Gemini | Current client | $/run |
|---|---|---|---|---|---|
| nothing | 3/3 | 1/3 | 0/3 | 4/9 | 0.34 |
| llms.txt | 3/3 | 1/3 | 0/3 | 4/9 | 0.33 |
| skill | 3/3 | 0/3 | 1/3 | 4/9 | 0.48 |
| docs MCP | 3/3 | 1/3 | 2/3 | 6/9 | 0.40 |
| AGENTS.md | 3/3 | 3/3 | 3/3 | 9/9 | 0.29 |
Claude Code cost per run. p = 0.03 for AGENTS.md against nothing; other gaps are noise, and the report says so.
"Should we build a docs MCP?"
Find out whether agents will call it, and whether it beats what you already ship, before you spend a quarter on it.
"We just shipped v3."
See how many agents still write v2, and which single change moves them to the new API.
"Is llms.txt worth it?"
Get numbers for the roadmap discussion instead of opinions.
Claude Code (Opus 5.5 or Sonnet 5.5), Codex (GPT-6.1-Sol) and Gemini (3.8 Flash in Antigravity). You choose in the form. Tell us if your users rely on another agent.
No. We test your published package and public docs, the way your users' agents meet them. For an unreleased package we can work from a tarball.
1 to 3 days. Most of it goes into building a task where agents actually go wrong; the runs take a few hours.
For large differences, yes: 4 of 9 against 9 of 9 is real. For small ones, no, and the report says so instead of ranking channels whose counts are close.
No. The report is yours. We ask before quoting any number, and name the library only with your yes.
The first checks are free while we're in early access. We ask for 20 minutes of feedback on the report.
Free in early access. Results in 1 to 3 days. Private unless you say otherwise.
We'll start by building a task for your package and email you when the report is ready, usually within 1 to 3 days.
Your check, as we'll run it. Every channel you pick is compared with "nothing" and with a short brief in AGENTS.md.