What the checks caught
Building this surfaced 11 real defects. The interesting part is which check found each one.
Dots show the check that caught each defect. No single check caught them all.
What it shows
- The structural checks (schema validation and the strict build) caught none of these, because they test form, not fidelity. A clean build does not mean correct content.
- The most valuable check was measuring completeness against the source (the coverage check), which exposed a factory-reset procedure that a well-meant cleanup step had silently deleted.
- Several defects were found only by reading the output against the source. Automated checks narrow the search; they do not replace looking.
The defects
| # | Defect | Caught by | How it showed | Fix |
|---|---|---|---|---|
| 1 | Cleanup deleted real instructions (a factory-reset procedure) | Coverage check | Coverage check showed 97.5% recall; missing words named the culprit | Keep figure text; drop only pure noise |
| 2 | Assumed headings appear in outline order (multi-column pages) | Converter warning | Converter warnings: six headings 'not located' | Locate each heading in its own page window |
| 3 | Hidden control characters in outline titles | Converter warning | Same warnings | Strip control characters |
| 4 | Page-number footer leaked into a section; empty stub chapters | Spot check vs source | Spot check against the PDF | Strip page numbers; skip title-only stubs |
| 5 | A download was for a different product (Play mk3 vs MK3) | Reading the output | Reading the document's own title | Not ingested; correct file matched by content |
| 6 | Other-language text leaked into English sections | Reading the output | Reading the converted output | Crop to the English column; cut at first non-English heading |
| 7 | Running page headers used as section titles | Reading the output | Reading the converted output | Explicit heading patterns per manual |
| 8 | Spec parser misread numbers (1,000 mA as 0; wrong rail; a row size as a width) | Evidence lines / flags | Wrong-looking numbers, the disagree flag, raw evidence lines | Line-anchored, sign-aware patterns; override with reasoning |
| 9 | A non-UTF-8 page crashed the fetcher | Crash | The crash | Tolerant decoding |
| 10 | A fix silently did not apply; the table looked fresh | Reading the output | A value that should have changed still read 0 | Confirm changes took effect; regression diff |
| 11 | Tags that match everywhere carry no information | Reading the output | Reading tag frequencies | Heading-only tags; drop the bare word 'gate' |
More: how correctness is checked.