Add document embedding and retrieval via a dedicated embedding model #10
1 changed files with 19 additions and 5 deletions
24
README.md
24
README.md
|
|
@ -86,11 +86,25 @@ cargo run -- -l info "your research topic"
|
|||
cargo run -- --doc ./notes.txt --doc ./research-docs/ "your research topic"
|
||||
```
|
||||
|
||||
Uploaded documents are embedded with a dedicated embedding model (see
|
||||
`EMBEDDING_MODEL` in [`src/models.rs`](./src/models.rs)) into an in-memory
|
||||
vector index, then the excerpts most relevant to the topic are retrieved and
|
||||
folded into the researcher's task alongside anything it finds on the web —
|
||||
the same footnote-citation scheme applies to both.
|
||||
Uploaded documents are chunked (see `documents.rs`), embedded with a
|
||||
dedicated embedding model (see `EMBEDDING_MODEL` in
|
||||
[`src/models.rs`](./src/models.rs)) into an in-memory vector index, then the
|
||||
excerpts most relevant to the topic are retrieved and folded into the
|
||||
researcher's task alongside anything it finds on the web — the same
|
||||
footnote-citation scheme applies to both.
|
||||
|
||||
Known limitation: retrieval returns a fixed top-N chunks
|
||||
(`retrieval::TOP_N_EXCERPTS`). A document with several chunks that all read
|
||||
as similar to the query — several incident reports, several revisions of
|
||||
the same section — can crowd out the one chunk that actually answers it,
|
||||
since only the top N by similarity are ever returned regardless of how many
|
||||
plausible candidates exist. Reproduced deliberately (a 13.7 KB / 12-chunk
|
||||
document with 6 near-identical "incident report" sections, only one of
|
||||
which had the real answer, was built specifically to stress this — the top
|
||||
5 slots filled entirely with distractors and the answer chunk was
|
||||
excluded), so it's a real edge case, not a hypothetical. Not fixed for now
|
||||
since it takes a document engineered to trigger it, but worth knowing if a
|
||||
report seems to be missing something you know is in an uploaded document.
|
||||
|
||||
## Project layout
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue