Retrieval results
If software searches this knowledge base, does it find the right page, and which structuring choices help? Retrieval is scored on its own, with no language model, using a keyword search (BM25 keyword baseline) and 22 test questions (17 answerable; the rest test whether a system can say "I don't know").
hit@k: an expected page is in the top k results. MRR: how high the first correct page ranks (1.0 is best).
What changed when device metadata was indexed
Indexing manufacturer, model and device alongside the page text moved these questions (first correct rank):
| Question | Baseline | With metadata |
|---|---|---|
d01 |
4 | 1 |
d03 |
5 | 2 |
a01 |
2 | 1 |
a03 |
3 | 4 |
r03 |
not in top 5 | 3 |
m02 |
2 | 1 |
What it does and does not say
- Metadata helped, but with 17 answerable questions this is a lead to re-test, not a settled result.
- Indexing the tags too added nothing.
- Vocabulary mismatch is a content problem. One question asked for the back of a device; the manual only says rear panel, so a keyword search cannot connect them. Adding plain-language aliases to pages is the obvious next experiment.
- One miss (
a02) is the test's fault: several pages legitimately answer it, but only one was accepted. - Keyword search cannot abstain: it returns confident-looking results even when the answer is not in the knowledge base.
- No language-model-judged evaluation has been run, so nothing here says anything about answer quality.
More: evaluation. Questions that never found a page: a02, r01.