Manually tested chunking against a 13.7KB/12-chunk document (previous
verification only used a 424-char single-chunk file, which never
exercised multi-chunk retrieval). Chunking itself held up. Retrieval
didn't: built a document with 6 near-identical distractor sections and
only one true answer, and the fixed top-5 slots filled entirely with
distractors, excluding the chunk that actually answered the query.
Documented as a known limitation rather than fixed now — it takes a
document engineered to trigger it (several chunks that all read as
similar to the query), not a typical upload.
Adds --doc (repeatable, file or directory) so the researcher can draw on
user-supplied documents alongside the web: documents.rs resolves paths
into embeddable text, retrieval.rs embeds them with a dedicated
embedding model (nomic-embed-text, separate from the chat models used
elsewhere) into an in-memory vector index and retrieves the excerpts
most relevant to the topic once up front, and researcher.rs folds those
excerpts into the researcher's task under the same footnote-citation
scheme already used for web sources. The embedding and retrieval phases
show progress the same way every other phase does — a spinner while
embedding, a summary line once excerpts are retrieved, tracing spans
for -l mode.
Verified against a live Ollama nomic-embed-text pull and a real
research round: a planted fact sheet was correctly ranked as the most
relevant of several embedded documents and appeared in the researcher's
task before its first turn.
deep_research is the only project this repo is meant to showcase, so the
Cargo workspace wrapping it and an unrelated side project no longer earns
its keep:
- swear_cleanup moved to a new standalone local repo (~/dev/swear_cleanup,
not pushed anywhere) via `git subtree split`, with its pre-workspace-
split history (when it lived at src/swear_cleanup/ in a single shared
crate) spliced onto its post-split history rather than starting from a
single flattened snapshot. FINDINGS.md, which was sitting at this repo's
root but was actually swear_cleanup's own build log, went with it.
- deep_research/{src,Cargo.toml,README.md,docs} moved to the repo root;
the [workspace] table collapsed into a plain [package] manifest with
dependency versions inlined from the old [workspace.dependencies].
- Cargo.toml keeps an explicit empty [workspace] table (not just omitted)
so that checking this repo out as a nested git worktree — this
project's own normal workflow — can't accidentally inherit a stale
ancestor directory's workspace manifest, which is exactly what broke
the build while testing this change from a worktree.
- .forgejo/workflows/deep_research-ci.yml -> ci.yml, dropping the now-
meaningless -p deep_research scoping and path filters (redundant when
it's the only thing in the repo).
- README.md and docs/case-study.md updated for the flattened commands
(cargo run/test with no -p flag); their relative links to each other
and to src/ were already correct since both moved together.
Verified: cargo build/test/clippy/fmt all clean from the new repo root.
