DuckDuckGo's HTML endpoint rate-limits after enough requests, and a
rate-limited response is indistinguishable from a genuine empty result —
which is exactly what burned a full 12-turn research run on 13 consecutive
"No results found" responses. Swapping to a local SearXNG instance's JSON
API (no HTML scraping needed) fixes both problems: SearXNG spreads queries
across multiple upstream engines instead of hammering one, and this
machine already runs an instance.
This tool is explicitly local-only and never released, so the base URL is
a plain default (localhost:8080) overridable via SEARXNG_URL, not a
general-purpose config surface. Evaluated the two third-party SearXNG
crates on crates.io first (searxng, searxng-client) — both are
single-maintainer v0.1.0 packages with no adoption signal and no official
alternative exists, so a hand-rolled reqwest + serde call was the better
bet for something this small.
Drops the DuckDuckGo-specific HTML parsing (parse_search_results,
resolve_ddg_redirect, the .result/.result__a/.result__snippet scraper
selectors) entirely — fetch_page's extract_readable_text still needs
scraper for arbitrary fetched pages, so that dependency stays.
Adds an #[ignore]'d live smoke test (search_web_returns_real_results_from_local_searxng)
for manually verifying against a running instance; not run by default
since there's no CI environment with SearXNG available.
Covers truncate, assistant_text, tool_result_text, and
partial_findings_from_history in isolation (fallback text, ordering,
truncation, and filtering out non-text content). gather_findings/
write_report/research still need a live ollama client and aren't covered
here.
Wires up starter.rs -> core.rs (CLI parsing and observability init moved
into core, main.rs left as a thin entry point) and switches the report
phase from Agent::prompt to rig's stream_prompt, printing each text
delta to stdout as it arrives instead of waiting for the full response.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
deep_research and swear_cleanup were sharing one Cargo.toml, so every
build compiled clap/indicatif/scraper/chrono (only needed by
deep_research) even when just building swear_cleanup for its own
course work, and vice versa. Moves each into its own workspace member
crate (deep_research/, swear_cleanup/) with an independent Cargo.toml
declaring only the deps it actually uses; common deps/versions are
pinned once via [workspace.dependencies] so the two don't drift.
Verified `cargo build -p swear_cleanup` alone no longer pulls in
clap/indicatif/scraper/chrono (schemars still compiles for it, but
that's a direct transitive dependency of rig itself, not something
this split can avoid). Also verified the relocated deep_research
binary still runs end-to-end against live Ollama with correct
footnote citations and sources.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
