- collect_documents now partitions --doc paths into files vs.
directories up front and handles each with its own loop, instead of
a single branching if/else per path; the directory branch resolves
entries via a fallible iterator chain (? inside a Result-returning
map, collected with Result::transpose) rather than a nested for loop
- extracted chunk_document (path, splitter) -> Vec<Document> as a pure
function, and Document::from_chunk(path, index, chunk, total) to own
the chunk's source-string formatting, both previously inlined in a
function that mutated a shared Vec in place
- renamed push_document -> chunk_document to match what it now does
(reads and chunks a file into owned Documents) instead of what it
used to do (push one Document into a caller-supplied Vec)
- added unit tests for the reworked logic: multiple files/directories
in one call, a mix of both in one call, an unreadable file inside a
scanned directory (skipped, siblings kept) vs. an unreadable
directory itself (fails the whole call, unlike a file), and direct
tests of chunk_document and Document::from_chunk in isolation
Verified with cargo build/test (34 passed)/clippy -D warnings/fmt --check.
It didn't actually guard what it claimed to: std::fs::read_to_string
loads the whole file into memory before truncation ever ran, so it
never prevented the OOM risk it was documented as protecting against
— it only capped how much of the already-fully-read text got chunked
afterward. Directory scanning is non-recursive and --doc is an
explicit opt-in, so an oversized file is on the caller; chunking
already handles arbitrarily long documents correctly. Can add a
pre-read fs::metadata size check back later if real usage shows a
need for it.
- retrieval::retrieve_relevant now shows a spinner around the query-
embedding call and gets its own tracing span, matching every other
model-calling phase in the pipeline (it previously ran invisibly and
untraced)
- add progress::report() to dedupe the show_progress-gated eprintln!
pattern shared by research.rs and retrieval.rs
- documents::push_document builds its TextSplitter once per collect_documents
call instead of once per file, skips the char-truncation walk entirely
when a file is already under the limit, and pulls the source-string
branch out of the map closure
- researcher::gather_findings drops a doc_context emptiness check that
build_doc_context already guarantees
- documents.rs tests use tempfile::tempdir() instead of a hand-rolled
TempDir type
Verified with cargo build/test/clippy -D warnings/fmt --check.
Resolves the Cargo.lock conflict by regenerating it, and picks up
master's rename (deep_research -> doubleo7 crate/binary name, Cli ->
Doubleo7 struct) cleanly through everything this branch added
(--doc/documents.rs/retrieval.rs). Also renamed a leftover
"deep_research-test-" temp-dir prefix in documents.rs's tests for
consistency with the rebrand.
A flat 20K-char cutoff (copied from the fetched-web-page limit) silently
dropped everything past the first ~20KB of a larger file, and even
within the cutoff, embedding a whole multi-page document as one vector
made retrieval coarse — the vector just averages out whatever topics
the document covers.
Split each file into ~1500-char chunks via text-splitter (recursive
semantic-boundary splitting: paragraph > sentence > word, never mid-word)
and embed each chunk as its own document, tagged with its source and
part number. This removes the practical size ceiling — a large file
chunks the same way a short one does — and sharpens retrieval by letting
it surface the specific passage relevant to a query. It also incidentally
caps the worst-case retrieval payload: 5 chunks now tops out around
7500 chars versus the old worst case of 5 full 20K-char documents.
Verified against a live Ollama nomic-embed-text pull: a short document
still embeds as a single chunk, unchanged from before.
Adds --doc (repeatable, file or directory) so the researcher can draw on
user-supplied documents alongside the web: documents.rs resolves paths
into embeddable text, retrieval.rs embeds them with a dedicated
embedding model (nomic-embed-text, separate from the chat models used
elsewhere) into an in-memory vector index and retrieves the excerpts
most relevant to the topic once up front, and researcher.rs folds those
excerpts into the researcher's task under the same footnote-citation
scheme already used for web sources. The embedding and retrieval phases
show progress the same way every other phase does — a spinner while
embedding, a summary line once excerpts are retrieved, tracing spans
for -l mode.
Verified against a live Ollama nomic-embed-text pull and a real
research round: a planted fact sheet was correctly ranked as the most
relevant of several embedded documents and appeared in the researcher's
task before its first turn.
