CI runs the test suite as root inside an unmodified Docker base image
(data.forgejo.org/oci/node:20-bookworm), where chmod 0o000 doesn't
actually block reads — root bypasses Unix permission bits entirely.
The two new permission-based tests from the previous commit passed
locally (non-root) but failed in CI for exactly that reason. Skip them
under root via a raw geteuid() FFI check instead of asserting behavior
the OS isn't enforcing.
Verified with cargo build/test (34 passed)/clippy -D warnings/fmt --check.
- collect_documents now partitions --doc paths into files vs.
directories up front and handles each with its own loop, instead of
a single branching if/else per path; the directory branch resolves
entries via a fallible iterator chain (? inside a Result-returning
map, collected with Result::transpose) rather than a nested for loop
- extracted chunk_document (path, splitter) -> Vec<Document> as a pure
function, and Document::from_chunk(path, index, chunk, total) to own
the chunk's source-string formatting, both previously inlined in a
function that mutated a shared Vec in place
- renamed push_document -> chunk_document to match what it now does
(reads and chunks a file into owned Documents) instead of what it
used to do (push one Document into a caller-supplied Vec)
- added unit tests for the reworked logic: multiple files/directories
in one call, a mix of both in one call, an unreadable file inside a
scanned directory (skipped, siblings kept) vs. an unreadable
directory itself (fails the whole call, unlike a file), and direct
tests of chunk_document and Document::from_chunk in isolation
Verified with cargo build/test (34 passed)/clippy -D warnings/fmt --check.
It didn't actually guard what it claimed to: std::fs::read_to_string
loads the whole file into memory before truncation ever ran, so it
never prevented the OOM risk it was documented as protecting against
— it only capped how much of the already-fully-read text got chunked
afterward. Directory scanning is non-recursive and --doc is an
explicit opt-in, so an oversized file is on the caller; chunking
already handles arbitrarily long documents correctly. Can add a
pre-read fs::metadata size check back later if real usage shows a
need for it.
- retrieval::retrieve_relevant now shows a spinner around the query-
embedding call and gets its own tracing span, matching every other
model-calling phase in the pipeline (it previously ran invisibly and
untraced)
- add progress::report() to dedupe the show_progress-gated eprintln!
pattern shared by research.rs and retrieval.rs
- documents::push_document builds its TextSplitter once per collect_documents
call instead of once per file, skips the char-truncation walk entirely
when a file is already under the limit, and pulls the source-string
branch out of the map closure
- researcher::gather_findings drops a doc_context emptiness check that
build_doc_context already guarantees
- documents.rs tests use tempfile::tempdir() instead of a hand-rolled
TempDir type
Verified with cargo build/test/clippy -D warnings/fmt --check.
Ollama doesn't support tool_choice, so rig's tool-forced extractor
(client.extractor::<Review>()) never actually compelled the reviewer's
small local model to call the submit tool -- it just answered in prose,
and extraction exhausted its retries with "No data extracted" every time.
Switch to rig's typed-prompt API (agent.prompt_typed::<Review>()), which
uses Native output mode: Ollama's own `format` JSON-schema constraint on
the completion request, honored regardless of tool-calling ability.
Verified against a live local Ollama instance.
Resolves the Cargo.lock conflict by regenerating it, and picks up
master's rename (deep_research -> doubleo7 crate/binary name, Cli ->
Doubleo7 struct) cleanly through everything this branch added
(--doc/documents.rs/retrieval.rs). Also renamed a leftover
"deep_research-test-" temp-dir prefix in documents.rs's tests for
consistency with the rebrand.
A flat 20K-char cutoff (copied from the fetched-web-page limit) silently
dropped everything past the first ~20KB of a larger file, and even
within the cutoff, embedding a whole multi-page document as one vector
made retrieval coarse — the vector just averages out whatever topics
the document covers.
Split each file into ~1500-char chunks via text-splitter (recursive
semantic-boundary splitting: paragraph > sentence > word, never mid-word)
and embed each chunk as its own document, tagged with its source and
part number. This removes the practical size ceiling — a large file
chunks the same way a short one does — and sharpens retrieval by letting
it surface the specific passage relevant to a query. It also incidentally
caps the worst-case retrieval payload: 5 chunks now tops out around
7500 chars versus the old worst case of 5 full 20K-char documents.
Verified against a live Ollama nomic-embed-text pull: a short document
still embeds as a single chunk, unchanged from before.
Adds --doc (repeatable, file or directory) so the researcher can draw on
user-supplied documents alongside the web: documents.rs resolves paths
into embeddable text, retrieval.rs embeds them with a dedicated
embedding model (nomic-embed-text, separate from the chat models used
elsewhere) into an in-memory vector index and retrieves the excerpts
most relevant to the topic once up front, and researcher.rs folds those
excerpts into the researcher's task under the same footnote-citation
scheme already used for web sources. The embedding and retrieval phases
show progress the same way every other phase does — a spinner while
embedding, a summary line once excerpts are retrieved, tracing spans
for -l mode.
Verified against a live Ollama nomic-embed-text pull and a real
research round: a planted fact sheet was correctly ranked as the most
relevant of several embedded documents and appeared in the researcher's
task before its first turn.
deep_research is the only project this repo is meant to showcase, so the
Cargo workspace wrapping it and an unrelated side project no longer earns
its keep:
- swear_cleanup moved to a new standalone local repo (~/dev/swear_cleanup,
not pushed anywhere) via `git subtree split`, with its pre-workspace-
split history (when it lived at src/swear_cleanup/ in a single shared
crate) spliced onto its post-split history rather than starting from a
single flattened snapshot. FINDINGS.md, which was sitting at this repo's
root but was actually swear_cleanup's own build log, went with it.
- deep_research/{src,Cargo.toml,README.md,docs} moved to the repo root;
the [workspace] table collapsed into a plain [package] manifest with
dependency versions inlined from the old [workspace.dependencies].
- Cargo.toml keeps an explicit empty [workspace] table (not just omitted)
so that checking this repo out as a nested git worktree — this
project's own normal workflow — can't accidentally inherit a stale
ancestor directory's workspace manifest, which is exactly what broke
the build while testing this change from a worktree.
- .forgejo/workflows/deep_research-ci.yml -> ci.yml, dropping the now-
meaningless -p deep_research scoping and path filters (redundant when
it's the only thing in the repo).
- README.md and docs/case-study.md updated for the flattened commands
(cargo run/test with no -p flag); their relative links to each other
and to src/ were already correct since both moved together.
Verified: cargo build/test/clippy/fmt all clean from the new repo root.
deep_research and swear_cleanup were sharing one Cargo.toml, so every
build compiled clap/indicatif/scraper/chrono (only needed by
deep_research) even when just building swear_cleanup for its own
course work, and vice versa. Moves each into its own workspace member
crate (deep_research/, swear_cleanup/) with an independent Cargo.toml
declaring only the deps it actually uses; common deps/versions are
pinned once via [workspace.dependencies] so the two don't drift.
Verified `cargo build -p swear_cleanup` alone no longer pulls in
clap/indicatif/scraper/chrono (schemars still compiles for it, but
that's a direct transitive dependency of rig itself, not something
this split can avoid). Also verified the relocated deep_research
binary still runs end-to-end against live Ollama with correct
footnote citations and sources.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a clap-derive Cli (topic argument, --log-level flag) so the
research loop is a proper command-line tool instead of a raw
env::args().nth(1) read. Logging is now opt-in: tracing only
initializes a subscriber when --log-level is passed, so the terminal
stays clean by default. When logging is off, each research phase
(researcher/reviewer/writer) shows an indicatif spinner instead, so
the user isn't staring at a blank terminal during the 1-3 minute
Gemma tool-calling turns.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Interpolates today's date into the researcher's preamble so it can judge
source freshness instead of relying on training-cutoff knowledge, and
asks it to cite facts with bracketed footnote numbers backed by a
Sources list, which the writer agent is now instructed to preserve
through to the final report.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a reviewer step (gemma4-e4b, fresh context) between gathering and
writing: it uses rig's typed Extractor to judge whether the findings'
conclusions actually follow from their cited sources, rather than
relying on free-text parsing. research() is now a plain bounded loop —
"the least agentic design that solves the problem", per rig's own
workflow guidance — that reruns the researcher with the reviewer's
solid_findings/gaps feedback folded into the next round's task until
it approves or MAX_RESEARCH_ROUNDS runs out.
Agents were showing up as "Unnamed Agent" in rig's built-in gen_ai.*
spans. Naming them via .name(...) fixes that, and splitting the two
phases into #[tracing::instrument]-annotated functions wraps rig's
per-turn chat/execute_tool spans in a parent span per phase, making
the trace tree legible instead of a flat stream of chat calls.
Pins rig to the newest published crates.io release (0.41.0) instead of
the git main branch, and adapts swear_cleanup's revise.rs to that
release's API (OneOrMany::first() returns T directly, raw_completion
folded into CompletionResponse::raw_response).
The research agent (gemma4:26b) drives rig's AgentRunner tool-calling
loop with two lean #[rig::tool_macro] tools — a DuckDuckGo HTML search
and a page-text fetcher — to gather and cross-check findings. A second
agent (gemma4-e4b) turns those findings into a structured report; the
smaller/faster model suffices there since it's reformatting already-
digested notes rather than doing multi-step research reasoning.
Gemma now checks its own output (a fresh, stateless completion call,
not conversation history) before it's accepted as a seed or revision,
retrying up to 5 times if it's a refusal, meta-commentary describing
what it's about to write, or a list of multiple options instead of
one direct answer. Originally tried a separate CPU-only judge model
(critic-cpu) to avoid VRAM contention, but it was both far slower
(13-16s per judgment vs Gemma's own sub-second calls) and unreliable
on the exact failure patterns it was meant to catch — Gemma reviewing
itself turned out faster and more accurate, so critic-cpu is dropped
entirely.
Also required two rounds of prompt tuning, driven by live failures:
first adding concrete negative examples after the judge approved
outputs it should have rejected, then explicitly scoping the check to
format only after Gemma started rejecting its own genuinely hostile
(but well-formed) output — conflating "should I have generated this"
with the format question actually asked. Added integration tests
covering both directions (rejecting bad formats, accepting hostile-
but-well-formed content) as a fast regression check against an
expensive full generation loop.
Enable Rig's built-in tracing spans (model, token usage, cache hits,
latency) via tracing-subscriber, filterable through RUST_LOG and
defaulting to info level. Logs write to stderr so stdout stays
reserved for program output. Standardizes the remaining ad-hoc
println! diagnostics (server startup, per-iteration revision progress,
non-convergence) into structured tracing events at appropriate levels.
Gemma now seeds deliberately hostile text, Shieldstral scores it, and
Gemma revises its own output based on the score until it drops below a
safety threshold (or a max-iteration cap is hit, returning the best
attempt seen). Extracted into a new revise module: score() now borrows
instead of consuming its args so it can run repeatedly, and the
gemma-call/extract-text logic is shared between seed generation and
every revision instead of being duplicated.
Also adds FINDINGS.md logging what actually turned out to be real
obstacles vs. overblown vs. irrelevant while building this out.
Checks whether llama-server is already healthy on startup and spawns it
from configured binary/model paths if not, polling until ready. Server
infra config (binary, model path, host, port, context size) split out
of prompts.toml into its own server.toml, and all of it lives in a new
server module rather than inline in main.rs, alongside a single reused
HTTP client and a shared health-check helper. Gemma client setup now
runs concurrently with the server health-check/spawn since they're
independent.
