f2c10783db
deep_research is the only project this repo is meant to showcase, so the
Cargo workspace wrapping it and an unrelated side project no longer earns
its keep:
- swear_cleanup moved to a new standalone local repo (~/dev/swear_cleanup,
not pushed anywhere) via `git subtree split`, with its pre-workspace-
split history (when it lived at src/swear_cleanup/ in a single shared
crate) spliced onto its post-split history rather than starting from a
single flattened snapshot. FINDINGS.md, which was sitting at this repo's
root but was actually swear_cleanup's own build log, went with it.
- deep_research/{src,Cargo.toml,README.md,docs} moved to the repo root;
the [workspace] table collapsed into a plain [package] manifest with
dependency versions inlined from the old [workspace.dependencies].
- Cargo.toml keeps an explicit empty [workspace] table (not just omitted)
so that checking this repo out as a nested git worktree — this
project's own normal workflow — can't accidentally inherit a stale
ancestor directory's workspace manifest, which is exactly what broke
the build while testing this change from a worktree.
- .forgejo/workflows/deep_research-ci.yml -> ci.yml, dropping the now-
meaningless -p deep_research scoping and path filters (redundant when
it's the only thing in the repo).
- README.md and docs/case-study.md updated for the flattened commands
(cargo run/test with no -p flag); their relative links to each other
and to src/ were already correct since both moved together.
Verified: cargo build/test/clippy/fmt all clean from the new repo root.
111 lines
4.9 KiB
Markdown
111 lines
4.9 KiB
Markdown
# deep_research
|
||
|
||
A local-first, multi-agent deep-research CLI: give it a topic, it searches
|
||
the web, cross-checks what it finds, and writes up a cited report — entirely
|
||
on infrastructure you control, with no cloud LLM API key and no query ever
|
||
leaving your machine.
|
||
|
||
```
|
||
$ deep_research "trends in AI customer-support chatbots"
|
||
🔎 Researching...
|
||
🧐 Reviewing findings...
|
||
✍️ Writing report...
|
||
|
||
# AI Customer-Support Chatbots: 2025–2026 Trends
|
||
...
|
||
```
|
||
|
||
## Why this exists
|
||
|
||
This started as a "does deep research actually work end-to-end" exercise and
|
||
turned into a small case study in building an *agentic* system that survives
|
||
contact with reality: models that hit their turn budget mid-task, search
|
||
providers that rate-limit, and reviewers that reject good-faith work. The
|
||
[case study](./docs/case-study.md) walks through what broke and how each
|
||
failure was fixed, not just papered over.
|
||
|
||
## Architecture
|
||
|
||
Four small agents, each with one job, coordinated by plain Rust control
|
||
flow — not a framework's agent graph, not an LLM deciding when to stop:
|
||
|
||
```
|
||
┌─────────────┐ approve/reject ┌──────────┐
|
||
│ researcher │ ───────────────► │ reviewer │
|
||
│ (tool-using)│ ◄─────────────── │ │
|
||
└──────┬──────┘ gaps/feedback └────┬─────┘
|
||
│ turn budget exhausted │ approved,
|
||
│ mid-investigation │ or out of rounds
|
||
▼ ▼
|
||
┌──────────────┐ ┌──────────────┐
|
||
│ summarizer │──findings───►│ writer │──► report
|
||
│ (recovery) │ │ │
|
||
└──────────────┘ └──────────────┘
|
||
```
|
||
|
||
- **researcher** — a tool-calling agent (`search_web`, `fetch_page`) that
|
||
gathers and cross-checks evidence, capped at a fixed model-call budget so
|
||
a confused model can't loop forever.
|
||
- **reviewer** — a separate, fresh-context agent that checks the researcher's
|
||
conclusions actually follow from its cited sources, and either approves
|
||
the findings or hands back concrete gaps for another pass.
|
||
- **writer** — turns approved (or partial) findings into a structured,
|
||
footnoted report, streamed to the terminal as it's generated.
|
||
- **summarizer** (recovery path) — only runs when the researcher exhausts
|
||
its turn budget before concluding on its own. It reconstructs a proper
|
||
findings dump from the raw tool-call transcript rather than the run
|
||
simply failing; see the case study for why this exists and how it
|
||
degrades gracefully if the summarizer call itself fails.
|
||
|
||
Everything runs against local models via [Ollama](https://ollama.com) and a
|
||
self-hosted [SearXNG](https://searx.space) instance for search — no OpenAI/
|
||
Anthropic/Google API key, no third-party search API, nothing about the
|
||
research topic leaves the host it runs on. That's a deliberate constraint,
|
||
not a limitation: it's the same shape a privacy-sensitive customer
|
||
deployment would need.
|
||
|
||
## Running it
|
||
|
||
Prerequisites:
|
||
- [Ollama](https://ollama.com) running locally with a tool-calling-capable
|
||
model pulled (the researcher and reviewer/writer models are configured in
|
||
[`src/models.rs`](./src/models.rs))
|
||
- A local [SearXNG](https://docs.searxng.org/) instance with its JSON API
|
||
enabled (defaults to `http://localhost:8080`, overridable via
|
||
`SEARXNG_URL`)
|
||
|
||
```
|
||
cargo run -- "your research topic"
|
||
|
||
# or, with tracing spans on stderr instead of the progress spinner:
|
||
cargo run -- -l info "your research topic"
|
||
```
|
||
|
||
## Project layout
|
||
|
||
Split one concern per file rather than one large module:
|
||
|
||
| File | Responsibility |
|
||
|---|---|
|
||
| `main.rs` | Argument parsing, logging setup, and the single top-level call — no orchestration logic |
|
||
| `research.rs` | The research/review round loop |
|
||
| `researcher.rs` | The tool-calling research phase |
|
||
| `review.rs` | The reviewer agent |
|
||
| `summarizer.rs` | Max-turns recovery: reconstructs findings via a model call |
|
||
| `writer.rs` | Turns findings into the final streamed report |
|
||
| `history.rs` | Pure, unit-tested helpers for parsing a rig chat history into usable text |
|
||
| `tools.rs` | `search_web` (SearXNG) and `fetch_page` tool implementations |
|
||
| `stream.rs` | Drains a streaming prompt response to the terminal |
|
||
| `progress.rs` | The terminal spinner and per-phase emoji |
|
||
| `models.rs`, `cli.rs`, `observability.rs` | Small shared config: model names, CLI args, tracing setup |
|
||
|
||
## Testing
|
||
|
||
```
|
||
cargo test # unit tests — pure functions, no network
|
||
cargo test -- --ignored # + a live smoke test against SearXNG
|
||
cargo clippy --all-targets
|
||
```
|
||
|
||
CI (`.forgejo/workflows/ci.yml`) runs formatting, lint, build, and the unit
|
||
test suite on every push and PR.
|