The Model Had Not Heard of It Yet¶
On September 22, 2026, Anthropic released a new artificial intelligence (AI) model, Claude Opus 5.5. That evening, a video card on this site's home page introduced a hands-on review of it as "Hands-on testing of Claude Opus 3.5".
The video was about Opus 5.5, and its own title said so. The model that writes this site's news summaries could not have heard of Opus 5.5, which arrived nineteen months after its knowledge ends, and it changed the name to one it had. The card ran on the home page, the Videos page and This Week, and I corrected it that evening.
One video, two names
The video's own title: Claude Opus 5.5
The card the model wrote: Claude Opus 3.5
Three other items about Opus 5.5 that day came through with the name intact, which is the unsettling part. The error was intermittent, so a spot check could easily have looked clean.
That card was the visible failure. This piece is mostly about the invisible one behind it, which had been running for at least a month.
How the News Gets Chosen¶
Every day, this site's news pipeline (the automated process behind its news pages) collects about sixty candidates from news sites, journals, podcasts and video channels, and a language model, which I will call the curator, decides which to keep. It writes a short summary of each item it keeps and a one-line reason for each item it drops.
The pipeline records every decision in a ledger, a running log of what it has already judged, and does not offer the same item twice. That is deliberate, because it stops yesterday's story from reappearing every morning, but it means a wrong drop stays dropped unless someone reopens it by hand. And readers never see what the curator drops. A dropped story leaves no gap that anyone would notice. It simply never appears.
As with the rest of this site, an AI coding assistant wrote the code. My part was the rules for what counts as news, the questions in this piece, and the decisions about what to trust and what to pay for.
Until September 29 the curator was Claude Haiku 4.5, a small, inexpensive model whose knowledge ends in February 2025. By September 2026, AI news was full of things released after that: new model families, new versions, new products. To the curator, every one of them was a name it had never seen.
The Failure Nobody Sees¶
The ledger keeps the curator's reason for every drop, so the failure was there to be read. Read back on September 29, the month's reasons showed it plainly: since September 1 the curator had dropped 87 items with reasons that blamed speculation or an unreleased subject. Two of them:
- "rumor or unconfirmed model", for an Ars Technica report that OpenAI had said its planned GPT-6.1 was too insecure to release
- "speculative fiction presented as current event", for an IEEE Spectrum explainer on how to stop AI agents from secretly working together
Neither was a rumor, and neither was fiction. Both were ordinary journalism about things that had happened. The curator could not tell "I have never heard of this" from "this did not happen", and it wrote down the first as if it were the second.
Telling It Was Not Enough¶
The obvious fix was to tell the curator about its own limits, and its instructions gained a rule that is still there:
An unfamiliar name is never evidence that something is unreleased, rumored, or fictional, and never a drop reason by itself.
It is a sensible rule. I could have added it, watched a few days of output, seen fewer bad drops, and called the problem solved. But the daily news varies so much that a few good days would have proved nothing, so instead of trusting the rule, I tested it against the month that had just gone wrong.
Testing the Fix on Last Month¶
A backtest replays real past cases through each proposed fix and scores the answers against reference judgments made in advance. This one took a single evening, September 29. Its cases were 278 of September's own, in three groups: drops blamed on speculation, drops blamed on marketing (the curator's other common reason), and a random sample of earlier keeps and drops, to show how often each version agreed with the decisions already made.
The reference judgments were the hard part. For each speculation drop, an AI agent looked up on the web when the model, product or event it named had become public, and a second agent tried to prove the first one wrong. Where they disagreed, eight times, the case was settled individually and the reasoning written down. Nine items labeled twice, independently, agreed on whether the drop reason was false nine times out of nine, and on keep or drop seven times out of nine. The marketing cases were judged by hand. Then a model from a different company, GPT-6 Astra, labeled 30 of the items without seeing the answers. It matched 26 exactly, came within one step on 3 (keep against borderline, say), and disagreed outright on 1.
By those answers, roughly half of the speculation drops were real news that the site's rules say to keep, and about two-thirds rested on something false: a released model called unreleased, or an on-the-record report called a rumor.
Then five versions of the curator judged every case three times each:
- Haiku 4.5, with its instructions as they stood before September 29
- Haiku 4.5, with the new rule added
- Haiku 4.5, with the rule and longer descriptions of each video
- Haiku 4.5, with the rule, the date, and the names of the models and tools this site already tracks
- Claude Sonnet 5.5, Anthropic's mid-sized model, whose knowledge runs to June 2026, with the rule and its effort set to low
What the Test Found¶
Real news kept, among September's drops blamed on speculation
Haiku 4.5, as it was33%
Haiku 4.5, with the rule35%
Haiku 4.5, longer video text37%
Haiku 4.5, told the date and names39%
Haiku 5.5, tested October 781%
Sonnet 5.592%
the share kept of the 45 that should have been kept, each judged three times
Every fix to Haiku 4.5 moved it a few points, and none moved it far. Given the speculation cases again, Haiku 4.5 as it stood kept a third of the real news among them. With the rule it kept 35 percent, with the longer descriptions 37, and with the date and names 39. And wherever its original reason had been false, it gave the same kind of reason again a third of the time or more.
Sonnet 5.5 kept 92 percent. Was it simply more lenient? Mostly not. Among the same cases it also dropped 94 percent of the items that should have been dropped, against Haiku 4.5's 92. Among the marketing cases it was a little more lenient: of those that should have been dropped, it dropped 85 percent, against Haiku 4.5's 90. In 75 calls each, neither model refused a request or ran out of room for its answer.
The lesson I took from this is uncomfortable for anyone who likes to fix things with better instructions, which includes me. The rule told Haiku 4.5 that its knowledge had an end. It could not tell it what had happened since, and that, as far as this test can tell, was what it lacked.
The switch to Sonnet 5.5 went live the same night.
What Happened Next¶
Six of September's wrong drops were news and podcast items still recent enough to be offered again, so I removed their ledger entries and let the new curator judge them again. That night's refresh kept four. The other two have not come through the pipeline again, so they were never judged again. Sixteen wrongly dropped videos stayed dropped: offering them all at once would have run into the site's limit of twelve videos per refresh, and the ledger would have recorded the overflow as dropped for good.
The renamed card had its own fix, written the evening it appeared: a check that rejects a summary or title when it gives a product a different version number from the one its source uses. It is narrow on purpose, looking only for a name followed by a number. Before it went live it was replayed over all 1,247 summaries and titles the curator had written. It first flagged 17. Some of those were its own false alarms, of four kinds (an unusual hyphen in one company's spelling of a version was one), and fixing them left 9, each a summary that gave a version number its title did not, though most of those numbers were probably in the fuller source text the curator also sees. When the check rejects a summary, the card falls back to the source's own words, so a wrong rejection costs a less tidy card, not a new error. It stays, because in the backtest even Sonnet 5.5 wrote two summaries it would have rejected.
The ledger shows what changed. Counted with the same wording pattern the test used to pick its cases, 77 of the curator's 1,171 drops in the 28 days before the switch blamed speculation, rumor, or an unreleased or unconfirmed subject: about one drop in fifteen. In the 34 days after, 36 of 1,446 did, about one in forty. The share of candidates kept barely moved, from 31 percent to 29. Not every one of those later drops is wrong, since the rules do ask for real rumors to be dropped, and I have not checked them one by one.
Haiku 5.5 arrived on October 7, with knowledge to June 2026, and went through the same test that day. With its effort set to low it kept 81 percent: far better than Haiku 4.5, below Sonnet 5.5, and at under a tenth of Sonnet's cost by that day's projections ($0.29 a month against $3.69). Curation stayed on Sonnet 5.5. On this test, the extra $3.40 a month keeps about one more in every nine of the items that should be kept.
What It Cost¶
The five September versions cost $3.58 in model calls, three runs each, and the two Haiku 5.5 settings tested in October about $0.26 more. I have no separate record of what the fact-checking agents cost.
At September's volume, Sonnet 5.5 curates for about $3.76 a month, against $1.31 for Haiku 4.5. That is 2.9 times the price, rather than the 2 times its price list suggests, because it splits the same text into about 40 percent more tokens, the small pieces of text that models read and that AI companies charge by.
What Transfers¶
- Every model's knowledge has an end date, and the major AI companies publish theirs. Anything newer is something the model never saw in training, and it may treat the unfamiliar as the impossible: "correcting" a name, or calling a real report a rumor. Here the subject was AI products. In medicine the same gap could hold a drug approved last month, a revised guideline, or a trial published after the model was trained. I have not tested that, and it would need its own test.
- Telling the model not to treat an unfamiliar name as a rumor, and to copy names and numbers exactly, was not enough here. The rule says both, and the older model still dropped most of the real news, even when it was also given the date and a list of current names. When the subject is newer than the model, check its names, versions and dates against the source yourself.
- Check what a model leaves out, not only what it says. A wrong summary can be seen; a wrong omission cannot.
- Before believing a fix, test it on past cases where the right answer is already known. A few good days prove very little.
What This Does Not Show¶
- It covers one task, choosing AI news, on one month of one site's candidates. It does not show how often the same models make this mistake with medical names; that would need its own test.
- It does not separate how recent a model's knowledge is from what class of model it is. Sonnet 5.5 differs from Haiku 4.5 in both. Haiku 5.5, as recent as Sonnet 5.5 but a smaller class, landed in between, which hints that both matter.
- The reference judgments for the speculation cases were made by AI agents checking each other, with disagreements settled one by one and an outside check on 30 items. They are careful, not perfect, and about a quarter of those cases were close calls that the scoring leaves out.
- The test picked its cases on September 29 and counted 87 speculation drops; today's ledger gives 77 for nearly the same days. Five of the ten are items I reopened, which no longer count as drops. I have not traced the other five.
- The before-and-after counts from the ledger were not checked against right answers. They show that the reasons changed, not that every later decision was correct.
- Prices and knowledge dates are as the companies published them on the dates given, and both change.
Content last reviewed November 2, 2026. Review dates are maintained as described on the About page.