Document a real retrieval limitation found via manual large-document testing
All checks were successful
CI / test (pull_request) Successful in 11m11s
All checks were successful
CI / test (pull_request) Successful in 11m11s
Manually tested chunking against a 13.7KB/12-chunk document (previous verification only used a 424-char single-chunk file, which never exercised multi-chunk retrieval). Chunking itself held up. Retrieval didn't: built a document with 6 near-identical distractor sections and only one true answer, and the fixed top-5 slots filled entirely with distractors, excluding the chunk that actually answered the query. Documented as a known limitation rather than fixed now — it takes a document engineered to trigger it (several chunks that all read as similar to the query), not a typical upload.
This commit is contained in:
parent
ff9f9455a6
commit
b5f12a500c
1 changed files with 19 additions and 5 deletions
24
README.md
24
README.md
|
|
@ -86,11 +86,25 @@ cargo run -- -l info "your research topic"
|
||||||
cargo run -- --doc ./notes.txt --doc ./research-docs/ "your research topic"
|
cargo run -- --doc ./notes.txt --doc ./research-docs/ "your research topic"
|
||||||
```
|
```
|
||||||
|
|
||||||
Uploaded documents are embedded with a dedicated embedding model (see
|
Uploaded documents are chunked (see `documents.rs`), embedded with a
|
||||||
`EMBEDDING_MODEL` in [`src/models.rs`](./src/models.rs)) into an in-memory
|
dedicated embedding model (see `EMBEDDING_MODEL` in
|
||||||
vector index, then the excerpts most relevant to the topic are retrieved and
|
[`src/models.rs`](./src/models.rs)) into an in-memory vector index, then the
|
||||||
folded into the researcher's task alongside anything it finds on the web —
|
excerpts most relevant to the topic are retrieved and folded into the
|
||||||
the same footnote-citation scheme applies to both.
|
researcher's task alongside anything it finds on the web — the same
|
||||||
|
footnote-citation scheme applies to both.
|
||||||
|
|
||||||
|
Known limitation: retrieval returns a fixed top-N chunks
|
||||||
|
(`retrieval::TOP_N_EXCERPTS`). A document with several chunks that all read
|
||||||
|
as similar to the query — several incident reports, several revisions of
|
||||||
|
the same section — can crowd out the one chunk that actually answers it,
|
||||||
|
since only the top N by similarity are ever returned regardless of how many
|
||||||
|
plausible candidates exist. Reproduced deliberately (a 13.7 KB / 12-chunk
|
||||||
|
document with 6 near-identical "incident report" sections, only one of
|
||||||
|
which had the real answer, was built specifically to stress this — the top
|
||||||
|
5 slots filled entirely with distractors and the answer chunk was
|
||||||
|
excluded), so it's a real edge case, not a hypothetical. Not fixed for now
|
||||||
|
since it takes a document engineered to trigger it, but worth knowing if a
|
||||||
|
report seems to be missing something you know is in an uploaded document.
|
||||||
|
|
||||||
## Project layout
|
## Project layout
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue