core.rs had grown into a 612-line grab-bag mixing six unrelated concerns:
CLI arg parsing, logging setup, top-level orchestration, the researcher
agent phase, chat-history reconstruction utilities, the summarizer agent
phase, and the writer agent phase — while review.rs, tools.rs, stream.rs,
and progress.rs already correctly isolated their own concerns. This
splits core.rs to match that existing pattern instead of being the one
file that doesn't follow it:
- cli.rs — Cli struct + DEFAULT_TOPIC
- observability.rs — initialize_observability
- models.rs — RESEARCHER_MODEL / WRITER_MODEL (previously duplicated
across call sites, now a single source of truth)
- history.rs — pure chat-history parsing/reconstruction helpers
(partial_findings_from_history, annotated_transcript_from_history, and
their private helpers), plus their unit tests. Also dedupes
MAX_TOOL_RESULT_CHARS, which was previously defined twice.
- researcher.rs — gather_findings + GatheredFindings (the tool-calling
research phase)
- summarizer.rs — summarize_partial_history (the max-turns recovery
agent)
- writer.rs — write_report
- research.rs — the top-level research() orchestration loop
main.rs now only does argument parsing, logging setup, and the top-level
call — no orchestration logic of its own. Unit tests stay co-located
with the code they test per Rust convention (not pulled into separate
files) rather than under "prefer new files" — that applies to
production code organization here.
No behavior changes; cargo test/clippy/fmt all clean.
DuckDuckGo's HTML endpoint rate-limits after enough requests, and a
rate-limited response is indistinguishable from a genuine empty result —
which is exactly what burned a full 12-turn research run on 13 consecutive
"No results found" responses. Swapping to a local SearXNG instance's JSON
API (no HTML scraping needed) fixes both problems: SearXNG spreads queries
across multiple upstream engines instead of hammering one, and this
machine already runs an instance.
This tool is explicitly local-only and never released, so the base URL is
a plain default (localhost:8080) overridable via SEARXNG_URL, not a
general-purpose config surface. Evaluated the two third-party SearXNG
crates on crates.io first (searxng, searxng-client) — both are
single-maintainer v0.1.0 packages with no adoption signal and no official
alternative exists, so a hand-rolled reqwest + serde call was the better
bet for something this small.
Drops the DuckDuckGo-specific HTML parsing (parse_search_results,
resolve_ddg_redirect, the .result/.result__a/.result__snippet scraper
selectors) entirely — fetch_page's extract_readable_text still needs
scraper for arbitrary fetched pages, so that dependency stays.
Adds an #[ignore]'d live smoke test (search_web_returns_real_results_from_local_searxng)
for manually verifying against a running instance; not run by default
since there's no CI environment with SearXNG available.
Add summarize_partial_history: a one-shot writer-model pass that turns an
annotated transcript (tool calls with their args, so a fetch's URL or a
search's query stays attached to its result, plus results and interim
notes) into a proper footnote-style findings dump, instead of the flat
concatenation partial_findings_from_history produces on its own.
It's wired in as the MaxTurnsError recovery path in gather_findings, but
partial_findings_from_history stays as the fallback for an empty transcript
or if the summarizer call itself fails — the one guaranteed recovery path
shouldn't have a second turn-budget/model failure as a single point of
failure.
Observability: instrumented with the same #[tracing::instrument(fields(
gen_ai.agent.name = ...))] + spinner pattern as the researcher/reviewer/
writer phases, with info!/warn! events on success, empty-transcript
skip, and summarizer failure.
Also: cargo fmt across the crate (unrelated formatting drift had
accumulated), and adds unit tests for the new transcript_lines /
annotated_transcript_from_history helpers.
Covers truncate, assistant_text, tool_result_text, and
partial_findings_from_history in isolation (fallback text, ordering,
truncation, and filtering out non-text content). gather_findings/
write_report/research still need a live ollama client and aren't covered
here.
Split into two filter_map-based helpers (assistant_text, tool_result_text)
and a shared truncate() so the extraction reads as a flat iterator chain
instead of four levels of nesting.
When the researcher agent exhausts its turn budget mid-investigation, rig
now surfaces PromptError::MaxTurnsError with the chat history intact rather
than nothing at all. Catch it, reconstruct a findings dump from whatever
assistant text and tool results the run produced, and still write a report
from that — clearly flagged as incomplete — instead of propagating the
error and losing all the work.
