Write a partial report instead of erroring out when research hits max turns #8

Merged

schaefera merged 6 commits from worktree-deep-research-max-turns-report into master

2026-08-18 11:03:03 +00:00

claude-bot commented

2026-08-18 09:27:15 +00:00

Collaborator

When the researcher agent exhausts its MAX_RESEARCH_TURNS budget mid-investigation, the run previously just propagated rig's error and the whole research() call failed — losing all the searching/fetching work done up to that point.

rig-agent 0.41.0's PromptError::MaxTurnsError { chat_history, .. } carries the full chat history at the point the budget ran out, so this catches that specific error, reconstructs a findings dump from whatever assistant text and tool results (search/fetch output) are in that history, and still runs the writer pass on it — with a hard-coded disclaimer (guaranteed regardless of what the writer model does) plus writer instructions to flag the report as inconclusive and call out thin/missing evidence.

Other errors still propagate normally via ?.

Test plan

cargo check -p deep_research
cargo test -p deep_research
cargo clippy -p deep_research (clean)
Manual run against a topic that reliably exhausts the turn budget, to eyeball the disclaimer + partial report output

When the researcher agent exhausts its `MAX_RESEARCH_TURNS` budget mid-investigation, the run previously just propagated `rig`'s error and the whole `research()` call failed — losing all the searching/fetching work done up to that point. `rig-agent` 0.41.0's `PromptError::MaxTurnsError { chat_history, .. }` carries the full chat history at the point the budget ran out, so this catches that specific error, reconstructs a findings dump from whatever assistant text and tool results (search/fetch output) are in that history, and still runs the writer pass on it — with a hard-coded disclaimer (guaranteed regardless of what the writer model does) plus writer instructions to flag the report as inconclusive and call out thin/missing evidence. Other errors still propagate normally via `?`. ## Test plan - [x] `cargo check -p deep_research` - [x] `cargo test -p deep_research` - [x] `cargo clippy -p deep_research` (clean) - [ ] Manual run against a topic that reliably exhausts the turn budget, to eyeball the disclaimer + partial report output

claude-bot added 1 commit 2026-08-18 09:27:17 +00:00

Write a partial report instead of erroring out when research hits max turns 187d327144

When the researcher agent exhausts its turn budget mid-investigation, rig
now surfaces PromptError::MaxTurnsError with the chat history intact rather
than nothing at all. Catch it, reconstruct a findings dump from whatever
assistant text and tool results the run produced, and still write a report
from that — clearly flagged as incomplete — instead of propagating the
error and losing all the work.

schaefera added 1 commit 2026-08-18 09:30:50 +00:00

Flatten partial_findings_from_history's nested for/match/if-let pyramid 3ddc48fe18

Split into two filter_map-based helpers (assistant_text, tool_result_text)
and a shared truncate() so the extraction reads as a flat iterator chain
instead of four levels of nesting.

schaefera added 1 commit 2026-08-18 09:43:19 +00:00

Add unit tests for the decomposed max-turns recovery helpers e446046744

Covers truncate, assistant_text, tool_result_text, and
partial_findings_from_history in isolation (fallback text, ordering,
truncation, and filtering out non-text content). gather_findings/
write_report/research still need a live ollama client and aren't covered
here.

schaefera added 1 commit 2026-08-18 09:54:26 +00:00

Prototype an agentic summarizer for max-turns recovery 21030462b1

Add summarize_partial_history: a one-shot writer-model pass that turns an
annotated transcript (tool calls with their args, so a fetch's URL or a
search's query stays attached to its result, plus results and interim
notes) into a proper footnote-style findings dump, instead of the flat
concatenation partial_findings_from_history produces on its own.

It's wired in as the MaxTurnsError recovery path in gather_findings, but
partial_findings_from_history stays as the fallback for an empty transcript
or if the summarizer call itself fails — the one guaranteed recovery path
shouldn't have a second turn-budget/model failure as a single point of
failure.

Observability: instrumented with the same #[tracing::instrument(fields(
gen_ai.agent.name = ...))] + spinner pattern as the researcher/reviewer/
writer phases, with info!/warn! events on success, empty-transcript
skip, and summarizer failure.

Also: cargo fmt across the crate (unrelated formatting drift had
accumulated), and adds unit tests for the new transcript_lines /
annotated_transcript_from_history helpers.

schaefera added 1 commit 2026-08-18 10:25:35 +00:00

Switch search_web from DuckDuckGo HTML scraping to local SearXNG c150c67f1e

DuckDuckGo's HTML endpoint rate-limits after enough requests, and a
rate-limited response is indistinguishable from a genuine empty result —
which is exactly what burned a full 12-turn research run on 13 consecutive
"No results found" responses. Swapping to a local SearXNG instance's JSON
API (no HTML scraping needed) fixes both problems: SearXNG spreads queries
across multiple upstream engines instead of hammering one, and this
machine already runs an instance.

This tool is explicitly local-only and never released, so the base URL is
a plain default (localhost:8080) overridable via SEARXNG_URL, not a
general-purpose config surface. Evaluated the two third-party SearXNG
crates on crates.io first (searxng, searxng-client) — both are
single-maintainer v0.1.0 packages with no adoption signal and no official
alternative exists, so a hand-rolled reqwest + serde call was the better
bet for something this small.

Drops the DuckDuckGo-specific HTML parsing (parse_search_results,
resolve_ddg_redirect, the .result/.result__a/.result__snippet scraper
selectors) entirely — fetch_page's extract_readable_text still needs
scraper for arbitrary fetched pages, so that dependency stays.

Adds an #[ignore]'d live smoke test (search_web_returns_real_results_from_local_searxng)
for manually verifying against a running instance; not run by default
since there's no CI environment with SearXNG available.

schaefera added 1 commit 2026-08-18 10:59:07 +00:00

Split core.rs into one file per concern 6cd2d65158

core.rs had grown into a 612-line grab-bag mixing six unrelated concerns:
CLI arg parsing, logging setup, top-level orchestration, the researcher
agent phase, chat-history reconstruction utilities, the summarizer agent
phase, and the writer agent phase — while review.rs, tools.rs, stream.rs,
and progress.rs already correctly isolated their own concerns. This
splits core.rs to match that existing pattern instead of being the one
file that doesn't follow it:

- cli.rs — Cli struct + DEFAULT_TOPIC
- observability.rs — initialize_observability
- models.rs — RESEARCHER_MODEL / WRITER_MODEL (previously duplicated
  across call sites, now a single source of truth)
- history.rs — pure chat-history parsing/reconstruction helpers
  (partial_findings_from_history, annotated_transcript_from_history, and
  their private helpers), plus their unit tests. Also dedupes
  MAX_TOOL_RESULT_CHARS, which was previously defined twice.
- researcher.rs — gather_findings + GatheredFindings (the tool-calling
  research phase)
- summarizer.rs — summarize_partial_history (the max-turns recovery
  agent)
- writer.rs — write_report
- research.rs — the top-level research() orchestration loop

main.rs now only does argument parsing, logging setup, and the top-level
call — no orchestration logic of its own. Unit tests stay co-located
with the code they test per Rust convention (not pulled into separate
files) rather than under "prefer new files" — that applies to
production code organization here.

No behavior changes; cargo test/clippy/fmt all clean.