From b5f12a500cedae33ff37c79b95698d919c53c419 Mon Sep 17 00:00:00 2001 From: Austin Schaefer Date: Tue, 18 Aug 2026 14:49:53 +0200 Subject: [PATCH] Document a real retrieval limitation found via manual large-document testing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Manually tested chunking against a 13.7KB/12-chunk document (previous verification only used a 424-char single-chunk file, which never exercised multi-chunk retrieval). Chunking itself held up. Retrieval didn't: built a document with 6 near-identical distractor sections and only one true answer, and the fixed top-5 slots filled entirely with distractors, excluding the chunk that actually answered the query. Documented as a known limitation rather than fixed now — it takes a document engineered to trigger it (several chunks that all read as similar to the query), not a typical upload. --- README.md | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 516748e..6bfbb9f 100644 --- a/README.md +++ b/README.md @@ -86,11 +86,25 @@ cargo run -- -l info "your research topic" cargo run -- --doc ./notes.txt --doc ./research-docs/ "your research topic" ``` -Uploaded documents are embedded with a dedicated embedding model (see -`EMBEDDING_MODEL` in [`src/models.rs`](./src/models.rs)) into an in-memory -vector index, then the excerpts most relevant to the topic are retrieved and -folded into the researcher's task alongside anything it finds on the web — -the same footnote-citation scheme applies to both. +Uploaded documents are chunked (see `documents.rs`), embedded with a +dedicated embedding model (see `EMBEDDING_MODEL` in +[`src/models.rs`](./src/models.rs)) into an in-memory vector index, then the +excerpts most relevant to the topic are retrieved and folded into the +researcher's task alongside anything it finds on the web — the same +footnote-citation scheme applies to both. + +Known limitation: retrieval returns a fixed top-N chunks +(`retrieval::TOP_N_EXCERPTS`). A document with several chunks that all read +as similar to the query — several incident reports, several revisions of +the same section — can crowd out the one chunk that actually answers it, +since only the top N by similarity are ever returned regardless of how many +plausible candidates exist. Reproduced deliberately (a 13.7 KB / 12-chunk +document with 6 near-identical "incident report" sections, only one of +which had the real answer, was built specifically to stress this — the top +5 slots filled entirely with distractors and the answer chunk was +excluded), so it's a real edge case, not a hypothetical. Not fixed for now +since it takes a document engineered to trigger it, but worth knowing if a +report seems to be missing something you know is in an uploaded document. ## Project layout