what I found
The workflow was cutting articles short before the AI saw them.
The test used full articles, but the live workflow cut each one to its first 600 characters. On the same article, the digest scored 1.00 in the test and about 0.17 in the email I actually received.
Running the same test twice gave different results.
One run listed five of the games mentioned in an article, the next listed three. That alone moved the score from 1.00 to 0.83 with nothing changed.
The score moves by about ±0.04 on its own.
Four identical runs returned 0.96, 0.93, 0.93 and 0.92. A prompt change that moves the score by less than that can't be told apart from chance.
what I took from it
An eval is only useful if it tests the same thing that runs in production. Mine didn't at first, and comparing it with the real emails is what showed me the bug. Knowing how much the score moves on its own also tells me which future changes are real improvements and which are just noise.
Read the detailsHide the details
where this comes from
The digest looked fine. "Looked fine" is not a quality claim - so what number would I defend it with, and how would I know that number wasn't lying to me?
Back in 2023 I had already run a version of this in client work, without calling it an eval: scoring a legal AI chatbot's answers against the client's historical responses and averaging across a sample. This is the same discipline, with the loop closed and the judge itself checked.
what gets measured
Coverage - of the facts I hand-labelled as essential to a story, how many did the digest actually convey?
Two design decisions I'd defend:
Coverage is a checklist, not a rating. I don't ask the judge to score quality 1-5; that number drifts between runs and can't be checked against a human. It gets 3-6 pre-labelled atomic facts per article and returns present or absent for each, with a quote from the digest as evidence. If it can't quote it, it's absent. Binary, auditable, and directly comparable to my own judgement.
Partial counts as absent. If the label reads "~8,000 victims" and the digest says "thousands of victims", that's a miss - the number was the news. Without this rule the judge waves through exactly the failure mode I care about.
The judge returns a verdict per labelled fact, each backed by an exact quote from the digest.
the golden set
Six articles, 24 atomic must-include facts, labelled by hand. Small - see limitations. Three rules I only arrived at by getting them wrong first:
One fact per item. My first label read Zyaire Wilkins, 21, Florida student, arrested by FBI - four facts in one checkbox. The judge marked it present on the strength of the name and age; "student" appears nowhere in the digest. A compound item can't be half-ticked, so the judge defaults to lenient.
Label from the full source, not the snippet. One label was written against a truncated excerpt, which made a correctly-reported figure look invented. A golden set is only as good as the text it was written from.
Don't label what isn't there. Twice I caught myself writing a plausible fact the source never stated - the same failure I'm asking the judge to catch in the model.
The dataset in Google Sheets. n8n reads it row by row and writes the digest, coverage and full judge verdict back into the same sheet.
validating the judge
Before trusting a single number it produced, I graded all 24 items myself - blind, with the judge's column hidden in the sheet until mine was finished - and compared.
| Article | Items | My verdicts | Judge | Agreement |
|---|---|---|---|---|
| nl_001 | 3 | 1 1 1 | 1 1 1 | 3 / 3 |
| nl_002 | 3 | 1 1 1 | 1 1 1 | 3 / 3 |
| nl_003 | 4 | 1 1 0 1 | 1 1 0 1 | 4 / 4 |
| nl_004 | 4 | 1 1 1 1 | 1 1 1 1 | 4 / 4 |
| nl_005 | 4 | 1 1 1 1 | 1 1 1 1 | 4 / 4 |
| nl_006 | 6 | 1 0 1 1 1 1 | 1 0 1 1 1 1 | 6 / 6 |
Raw agreement 24/24 · Cohen's κ = 1.00
This result is weaker than it looks. 22 of 24 items are "present", so a judge that answered "yes" to everything would still score 91.7% agreement. The test barely pressed on it. What it does establish is that we agree on both negatives - the two real misses - which is the direction that matters. To make this result mean something I need a golden set with more failure cases in it.
finding 1 - the test and the live workflow saw different text
Baseline came back at 0.96. That felt too high, so I checked it against the digest that had actually been emailed that morning, on the same article.
| Labelled fact | Shipped email | Eval run |
|---|---|---|
| ~8,000 victims | "thousands of victims" | present |
| at least $220,000 stolen | "drained cryptocurrency" | present |
| ~80 crypto wallets | absent | present |
| five named games | absent | present |
| coverage | ~0.17 | 1.00 |
Same article, same prompt, same model - scored twice.
The cause was one line of preprocessing in the production workflow:
Production truncated each article to 600 characters of RSS snippet. The figures live further down the article - the model never received them. Meanwhile the golden set held the full text I'd pasted in by hand.
The AI wasn't leaving the numbers out. They were never in the text it received.
The fix
Production now fetches each article URL, strips the page to text, and passes 4,000 characters instead of 600. The next morning's digest - a different story, this one on US EV sales - carried "5.8% market share, 247,226 units, -20.5%, $7,500 tax credit": figures that sit well beyond the 600-character cutoff the old pipeline used.
What I take from it: an offline eval is only as valid as its resemblance to the production path. Mine diverged at one line, and that one line made the score look about six times better than the real output.
Production after the fix - the article fetch and text extraction now sit between the feed merge and the model.
finding 2 - the same input doesn't give the same output
Re-running the eval unchanged, one digest named five games; the next named three. Coverage on that article moved 1.00 → 0.83. Nothing had changed - not the prompt, not the data, not the model.
This is normal for AI models, and it means a single run is one sample, not a measurement.
finding 3 - the noise floor
Four identical runs. No changes of any kind between them.
Working rule: a difference below ~0.035 (2 sd) is not a result.
It took four runs to measure, and it tells me which prompt changes are real improvements and which are just noise.
Runs tracked in n8n's Evaluations tab. Metrics are set by the workflow itself, so score history comes for free - no separate store.
how it's wired
The eval is a second workflow, deliberately separate from production. It reads the golden set row by row, so every run scores the same frozen sample - otherwise a prompt change and a news cycle would move at the same time and neither could be blamed.
If the judge returns a broken answer, the workflow stops instead of recording a zero, because a zero would look like a bad digest when the problem is actually the judge.
Built on n8n's native evaluation nodes rather than a bespoke harness.
why six articles were enough
The sample is small, but the three things it found don't depend on sample size.
A truncation bug, a non-deterministic model and a noise floor look the same at n=6 as at n=600. The 600-character cutoff was silently costing the product every verifiable number in a story, and it had been doing that in my inbox every weekday morning without me noticing. Six labelled articles found it in an evening. Six hundred would have found the same thing, later and at more cost.
The mechanism now exists and has been checked. Dataset, judge, scoring, write-back, tracked metrics - and, importantly, a judge whose verdicts I've compared against my own rather than assumed. Growing this is labelling work now, not engineering work.
limitations
n = 6 articles, 24 items. Enough to build and validate the mechanism; not enough for confident deltas.
Class imbalance 22:2 makes κ = 1.00 far less impressive than the number suggests.
Single labeller. No second annotator, so no measure of how much my own labels drift.
Coverage only. It checks that required facts are present and is structurally blind to invented ones - a digest carrying every labelled fact plus a fabricated sentence would still score 1.00.
Four runs is a thin basis for a noise estimate. The ±0.04 is indicative, not precise.
No A/B yet - and at this sample size, against this noise floor, a small prompt delta would be unresolvable anyway.