Add document embedding and retrieval via a dedicated embedding model #10

Merged
schaefera merged 10 commits from worktree-deep-research-max-turns-report into master 2026-08-19 15:37:33 +00:00
Showing only changes of commit b5f12a500c - Show all commits

View file

@ -86,11 +86,25 @@ cargo run -- -l info "your research topic"
cargo run -- --doc ./notes.txt --doc ./research-docs/ "your research topic"
```
Uploaded documents are embedded with a dedicated embedding model (see
`EMBEDDING_MODEL` in [`src/models.rs`](./src/models.rs)) into an in-memory
vector index, then the excerpts most relevant to the topic are retrieved and
folded into the researcher's task alongside anything it finds on the web —
the same footnote-citation scheme applies to both.
Uploaded documents are chunked (see `documents.rs`), embedded with a
dedicated embedding model (see `EMBEDDING_MODEL` in
[`src/models.rs`](./src/models.rs)) into an in-memory vector index, then the
excerpts most relevant to the topic are retrieved and folded into the
researcher's task alongside anything it finds on the web — the same
footnote-citation scheme applies to both.
Known limitation: retrieval returns a fixed top-N chunks
(`retrieval::TOP_N_EXCERPTS`). A document with several chunks that all read
as similar to the query — several incident reports, several revisions of
the same section — can crowd out the one chunk that actually answers it,
since only the top N by similarity are ever returned regardless of how many
plausible candidates exist. Reproduced deliberately (a 13.7 KB / 12-chunk
document with 6 near-identical "incident report" sections, only one of
which had the real answer, was built specifically to stress this — the top
5 slots filled entirely with distractors and the answer chunk was
excluded), so it's a real edge case, not a hypothetical. Not fixed for now
since it takes a document engineered to trigger it, but worth knowing if a
report seems to be missing something you know is in an uploaded document.
## Project layout