schaefera/doubleo7
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Add document embedding and retrieval via a dedicated embedding model #10

Merged
schaefera merged 10 commits from worktree-deep-research-max-turns-report into master 2026-08-19 15:37:33 +00:00
Conversation 0 Commits 10 Files changed 12 +653 -11
claude-bot commented 2026-08-18 12:28:11 +00:00
Collaborator
Copy link

Summary

  • --doc <PATH> (repeatable, file or directory — directories are scanned one level deep) lets the researcher draw on user-supplied documents alongside the web
  • src/documents.rs resolves paths into embeddable text; src/retrieval.rs embeds them with a dedicated embedding model (nomic-embed-text, kept separate from the chat models used elsewhere) into an in-memory vector index and retrieves the excerpts most relevant to the topic once up front
  • src/researcher.rs folds retrieved excerpts into the researcher's task under the same footnote-citation scheme already used for web sources
  • Embedding/retrieval phases show progress the same way every other phase does — spinner while embedding, summary line once excerpts are retrieved, tracing spans in -l mode

Test plan

  • cargo build / cargo test (24 passed, 1 ignored) / cargo clippy --all-targets -- -D warnings / cargo fmt -- --check
  • Live smoke test against a real Ollama nomic-embed-text pull and a real research round: a planted fact sheet was correctly ranked as the most relevant of several embedded documents and appeared in the researcher's task before its first turn
## Summary - `--doc <PATH>` (repeatable, file or directory — directories are scanned one level deep) lets the researcher draw on user-supplied documents alongside the web - `src/documents.rs` resolves paths into embeddable text; `src/retrieval.rs` embeds them with a dedicated embedding model (`nomic-embed-text`, kept separate from the chat models used elsewhere) into an in-memory vector index and retrieves the excerpts most relevant to the topic once up front - `src/researcher.rs` folds retrieved excerpts into the researcher's task under the same footnote-citation scheme already used for web sources - Embedding/retrieval phases show progress the same way every other phase does — spinner while embedding, summary line once excerpts are retrieved, tracing spans in `-l` mode ## Test plan - [x] `cargo build` / `cargo test` (24 passed, 1 ignored) / `cargo clippy --all-targets -- -D warnings` / `cargo fmt -- --check` - [x] Live smoke test against a real Ollama `nomic-embed-text` pull and a real research round: a planted fact sheet was correctly ranked as the most relevant of several embedded documents and appeared in the researcher's task before its first turn
claude-bot added 1 commit 2026-08-18 12:28:12 +00:00
Add document embedding and retrieval via a dedicated embedding model
All checks were successful
CI / test (pull_request) Successful in 1m52s
Details
ec15893c9c 
Adds --doc (repeatable, file or directory) so the researcher can draw on
user-supplied documents alongside the web: documents.rs resolves paths
into embeddable text, retrieval.rs embeds them with a dedicated
embedding model (nomic-embed-text, separate from the chat models used
elsewhere) into an in-memory vector index and retrieves the excerpts
most relevant to the topic once up front, and researcher.rs folds those
excerpts into the researcher's task under the same footnote-citation
scheme already used for web sources. The embedding and retrieval phases
show progress the same way every other phase does — a spinner while
embedding, a summary line once excerpts are retrieved, tracing spans
for -l mode.

Verified against a live Ollama nomic-embed-text pull and a real
research round: a planted fact sheet was correctly ranked as the most
relevant of several embedded documents and appeared in the researcher's
task before its first turn.
schaefera added 1 commit 2026-08-18 12:43:19 +00:00
Chunk uploaded documents instead of truncating them at 20K chars
All checks were successful
CI / test (pull_request) Successful in 11m5s
Details
ff9f9455a6 
A flat 20K-char cutoff (copied from the fetched-web-page limit) silently
dropped everything past the first ~20KB of a larger file, and even
within the cutoff, embedding a whole multi-page document as one vector
made retrieval coarse — the vector just averages out whatever topics
the document covers.

Split each file into ~1500-char chunks via text-splitter (recursive
semantic-boundary splitting: paragraph > sentence > word, never mid-word)
and embed each chunk as its own document, tagged with its source and
part number. This removes the practical size ceiling — a large file
chunks the same way a short one does — and sharpens retrieval by letting
it surface the specific passage relevant to a query. It also incidentally
caps the worst-case retrieval payload: 5 chunks now tops out around
7500 chars versus the old worst case of 5 full 20K-char documents.

Verified against a live Ollama nomic-embed-text pull: a short document
still embeds as a single chunk, unchanged from before.
schaefera added 1 commit 2026-08-18 12:49:56 +00:00
Document a real retrieval limitation found via manual large-document testing
All checks were successful
CI / test (pull_request) Successful in 11m11s
Details
b5f12a500c 
Manually tested chunking against a 13.7KB/12-chunk document (previous
verification only used a 424-char single-chunk file, which never
exercised multi-chunk retrieval). Chunking itself held up. Retrieval
didn't: built a document with 6 near-identical distractor sections and
only one true answer, and the fixed top-5 slots filled entirely with
distractors, excluding the chunk that actually answered the query.

Documented as a known limitation rather than fixed now — it takes a
document engineered to trigger it (several chunks that all read as
similar to the query), not a typical upload.
schaefera added 1 commit 2026-08-18 14:19:44 +00:00
Merge master (rebrand to doubleo7) into this branch
All checks were successful
CI / test (pull_request) Successful in 3m38s
Details
8f4af514a3 
Resolves the Cargo.lock conflict by regenerating it, and picks up
master's rename (deep_research -> doubleo7 crate/binary name, Cli ->
Doubleo7 struct) cleanly through everything this branch added
(--doc/documents.rs/retrieval.rs). Also renamed a leftover
"deep_research-test-" temp-dir prefix in documents.rs's tests for
consistency with the rebrand.
schaefera added 1 commit 2026-08-19 13:04:08 +00:00
Merge master into worktree-deep-research-max-turns-report
Some checks failed
CI / test (pull_request) Failing after 54s
Details
53b4f572df
schaefera added 1 commit 2026-08-19 13:13:15 +00:00
refactor: cleanup pass on document embedding/retrieval
All checks were successful
CI / test (pull_request) Successful in 6m56s
Details
19c92ccd52 
- retrieval::retrieve_relevant now shows a spinner around the query-
  embedding call and gets its own tracing span, matching every other
  model-calling phase in the pipeline (it previously ran invisibly and
  untraced)
- add progress::report() to dedupe the show_progress-gated eprintln!
  pattern shared by research.rs and retrieval.rs
- documents::push_document builds its TextSplitter once per collect_documents
  call instead of once per file, skips the char-truncation walk entirely
  when a file is already under the limit, and pulls the source-string
  branch out of the map closure
- researcher::gather_findings drops a doc_context emptiness check that
  build_doc_context already guarantees
- documents.rs tests use tempfile::tempdir() instead of a hand-rolled
  TempDir type

Verified with cargo build/test/clippy -D warnings/fmt --check.
schaefera added 1 commit 2026-08-19 13:23:56 +00:00
refactor: drop MAX_FILE_CHARS cap on --doc file size
All checks were successful
CI / test (pull_request) Successful in 1m23s
Details
5e55861a3f 
It didn't actually guard what it claimed to: std::fs::read_to_string
loads the whole file into memory before truncation ever ran, so it
never prevented the OOM risk it was documented as protecting against
— it only capped how much of the already-fully-read text got chunked
afterward. Directory scanning is non-recursive and --doc is an
explicit opt-in, so an oversized file is on the caller; chunking
already handles arbitrarily long documents correctly. Can add a
pre-read fs::metadata size check back later if real usage shows a
need for it.
schaefera added 2 commits 2026-08-19 15:04:14 +00:00 
- collect_documents now partitions --doc paths into files vs.
  directories up front and handles each with its own loop, instead of
  a single branching if/else per path; the directory branch resolves
  entries via a fallible iterator chain (?  inside a Result-returning
  map, collected with Result::transpose) rather than a nested for loop
- extracted chunk_document (path, splitter) -> Vec<Document> as a pure
  function, and Document::from_chunk(path, index, chunk, total) to own
  the chunk's source-string formatting, both previously inlined in a
  function that mutated a shared Vec in place
- renamed push_document -> chunk_document to match what it now does
  (reads and chunks a file into owned Documents) instead of what it
  used to do (push one Document into a caller-supplied Vec)
- added unit tests for the reworked logic: multiple files/directories
  in one call, a mix of both in one call, an unreadable file inside a
  scanned directory (skipped, siblings kept) vs. an unreadable
  directory itself (fails the whole call, unlike a file), and direct
  tests of chunk_document and Document::from_chunk in isolation

Verified with cargo build/test (34 passed)/clippy -D warnings/fmt --check.
chore: refactor some code.
Some checks failed
CI / test (pull_request) Failing after 1m50s
Details
f916787962
schaefera approved these changes 2026-08-19 15:31:05 +00:00
schaefera added 1 commit 2026-08-19 15:34:38 +00:00
fix: skip chmod-0o000 permission tests when running as root
All checks were successful
CI / test (pull_request) Successful in 1m22s
Details
ba9b61fcf3 
CI runs the test suite as root inside an unmodified Docker base image
(data.forgejo.org/oci/node:20-bookworm), where chmod 0o000 doesn't
actually block reads — root bypasses Unix permission bits entirely.
The two new permission-based tests from the previous commit passed
locally (non-root) but failed in CI for exactly that reason. Skip them
under root via a raw geteuid() FFI check instead of asserting behavior
the OS isn't enforcing.

Verified with cargo build/test (34 passed)/clippy -D warnings/fmt --check.
schaefera merged commit 90502a3eed into master 2026-08-19 15:37:33 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
schaefera
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: schaefera/doubleo7#10
No description provided.
Delete branch "worktree-deep-research-max-turns-report"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?