diff --git a/README.md b/README.md index 516748e..6bfbb9f 100644 --- a/README.md +++ b/README.md @@ -86,11 +86,25 @@ cargo run -- -l info "your research topic" cargo run -- --doc ./notes.txt --doc ./research-docs/ "your research topic" ``` -Uploaded documents are embedded with a dedicated embedding model (see -`EMBEDDING_MODEL` in [`src/models.rs`](./src/models.rs)) into an in-memory -vector index, then the excerpts most relevant to the topic are retrieved and -folded into the researcher's task alongside anything it finds on the web — -the same footnote-citation scheme applies to both. +Uploaded documents are chunked (see `documents.rs`), embedded with a +dedicated embedding model (see `EMBEDDING_MODEL` in +[`src/models.rs`](./src/models.rs)) into an in-memory vector index, then the +excerpts most relevant to the topic are retrieved and folded into the +researcher's task alongside anything it finds on the web — the same +footnote-citation scheme applies to both. + +Known limitation: retrieval returns a fixed top-N chunks +(`retrieval::TOP_N_EXCERPTS`). A document with several chunks that all read +as similar to the query — several incident reports, several revisions of +the same section — can crowd out the one chunk that actually answers it, +since only the top N by similarity are ever returned regardless of how many +plausible candidates exist. Reproduced deliberately (a 13.7 KB / 12-chunk +document with 6 near-identical "incident report" sections, only one of +which had the real answer, was built specifically to stress this — the top +5 slots filled entirely with distractors and the answer chunk was +excluded), so it's a real edge case, not a hypothetical. Not fixed for now +since it takes a document engineered to trigger it, but worth knowing if a +report seems to be missing something you know is in an uploaded document. ## Project layout