RedHub Teardown: What Our AI Reviewer Caught
RedHub AI Editorial4 min read
Jump to a section10
RedHub Teardown: What Our AI Reviewer Caught
TL;DR
- The process: AI agents build fixes to our workbooks in groups, and a separate AI agent, told to attack, reviews the files before anything ships.
- What the reviewer caught: in five fix batches between October 30 and November 3, it sent back work four times. Two of its blockers were crashes the fix itself introduced.
- The catch that mattered most: our guards covered only the rows our sample data filled. Every automated test passed, because the tests used the same sample.
- The miss: no reviewer compared a product's zip to its page. A writer found that 16 kits were missing their lessons.
Does a second AI reviewer catch what the first AI missed?
In RedHub's November 2026 teardown, yes, but only for what it was asked to read. Across five batches of AI-built fixes to our workbooks, a separate reviewer agent told to find faults sent work back four times, including two crashes the fixes created and guards that covered only sample rows. It did not catch a problem outside its brief: 16 kits whose pages promised lessons their downloads did not contain.
The Teardown runs once a month. We take one process apart, show where it won and where it lost, and give the numbers behind both. This month: how we use AI agents to fix our own products, and what happened when another AI was told to find their mistakes.
The process
- Find a class of defect, not one bug. We test every product the same way at once, for example by pasting an error value into every filled cell, and rank what moved.
- Split the fixes into groups. One AI agent builds the fixes for each group of similar products.
- Review with a separate agent. The reviewer did not build anything. Its instruction is to find what is wrong, and it reads the actual files, not the builders' reports. A blocker goes back to the builder.
- Check in real Excel. Our fast test library and Excel can disagree, and Excel is what buyers open.
- Ship and read it back. Upload, then download every file from storage and compare it before calling it shipped.
The scorecard
| Batch | Scope | Reviewer result | What it found |
|---|---|---|---|
| Oct 30 | 23 zips, 6 builder agents | 2 blockers | Both were crashes the fix introduced: one on an oversized decimal, one on a file saved in an older Windows text encoding. |
| Oct 31 | 8 zips, 5 fix groups | 1 product sent back | A pasted error left a status reading "practicing", and a name made only of spaces still counted as a learner. |
| Oct 31 to Nov 1 | 37 zips, 4 fix groups | 0 blockers | Nothing to send back. |
| Nov 1 | 115 zips | 3 blockers in round one, 0 in round two | In three products, a pasted error outside the guarded cells flipped a blocked verdict to clean. All three did the same before our fix: older bugs the review surfaced. |
| Nov 2 to 3 | 21 products | 1 finding, the largest | Guards sat only on rows the sample data filled. Widening them added about 8,500 guarded cells. |
Win: the reviewer read what the tests could not
The last row of that table is the reason we keep a reviewer. Our automated checks recalculated every product and passed. They passed because they pasted test values into the filled rows only, the same rows the fixes had been planned from. The tests and the fixes shared one blind spot. The reviewer opened the files and looked at the rows below the sample, where a buyer keeps typing, and found them unguarded.
The first row matters too. A fix is a change, and changes break things. Two of the reviewer's blockers were not in the original product at all. The builders created them while fixing something else.
Loss: the reviewer only reads its brief
Every reviewer in this table was asked one question: do the fixed files behave correctly? None was asked whether the download matches what the product page promises. So none noticed that all 16 of our role kits shipped without the lessons their pages described. A writer drafting blog posts about the kits found it on November 5, while the kits were already on sale.
A reviewer catches what is in its brief, and nothing outside it. That is not a flaw in the AI. It is a flaw in the brief, and we wrote the brief.
What to take from this for your own AI work
- Use a second reviewer that did not do the work, and tell it to find what is wrong, not to confirm it is right.
- Make it read the output, not the summary. An AI's report of what it did is a claim. The file is the evidence.
- Test outside the sample. If your test data and your fix were both built from the same five examples, a passing test says only that those five work.
- Review the fix, not just the bug. In one batch, both blockers were new.
- Write the question you are not asking. Once a month, ask your reviewer to compare what you deliver against what you promise.
What we changed
Every guard plan now pastes into every row of every table, filled or not, instead of a sample. And the gap the role kits exposed is now a written rule: when a download changes, we search every place that describes it. That lesson is this week's RedHub Weekly.
FAQ
What is the RedHub Teardown?
A monthly post that takes one RedHub process apart and reports where it worked, where it failed, and the numbers behind both. It runs alongside the weekly RedHub Weekly.
Should a different AI review work an AI agent produced?
In RedHub's experience, yes. A separate reviewer told to find faults, reading the files rather than the builder's report, sent back work in four of five fix batches. It catches only what its brief asks it to check.
Why did RedHub's automated tests miss the unguarded rows?
The tests pasted values only into rows the sample data filled, which were the same rows the fixes were planned from. A test built from the same sample as the fix cannot see what the fix missed.