c5c25c8b49
Adds a standalone binary (src/bin/ai_agents_spike.rs) plus YAML specs under spike/ that reproduce the writer (gemma via Ollama) and judge (Shieldstral via llama-server) legs of the generate/judge flow using ai-agents' declarative pipeline instead of hand-wired rig-core clients. Both legs verified working individually against real local models. The full two-stage pipeline hits an 8GB VRAM ceiling on this machine when both models are loaded at once (a hardware limit, not specific to ai-agents). ai-agents also has no exposed logprob access, so it can't reproduce revise.rs's actual scoring mechanism as-is. See spike/NOTES.md for the full writeup and verdict. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
64 lines
3.3 KiB
Markdown
64 lines
3.3 KiB
Markdown
# Spike: `ai-agents` (declarative YAML) vs. hand-wired `rig-core`
|
|
|
|
## What this proves
|
|
|
|
- `ai-agents` (crates.io `ai-agents` v1.0.0, Rust-native, no Python) can load an
|
|
agent purely from YAML and talk to both of this project's real backends:
|
|
- `spike/agents/writer.yaml` -> Ollama (`gemma4-e4b:latest`), matches `wire_gemma_client`
|
|
in `src/main.rs`.
|
|
- `spike/agents/judge.yaml` -> `provider: openai-compatible` against the local
|
|
`llama-server` (Shieldstral), matches `wire_shieldstral`.
|
|
- Both worked verbatim against this machine's real models/config, no mocking.
|
|
- `spike/pipeline.yaml` expresses the two-stage generate -> judge flow (this
|
|
project's `revise::generate_below_threshold` shape) as one declarative
|
|
`pipeline:` block with `spawner.auto_spawn` and `{{ stages.<id> }}`
|
|
templating, no manual Rust orchestration code.
|
|
|
|
## What it doesn't prove (real limitations found)
|
|
|
|
1. **VRAM ceiling, not a framework bug.** Running the full pipeline in one
|
|
process needs Ollama's gemma model and Shieldstral's llama-server (32k
|
|
ctx, ~5.6GB) resident at once. On this 8GB card that overflows CUDA
|
|
("out of memory" from Ollama's own `/api/chat`, not from `ai-agents`).
|
|
The writer and judge legs each work fine in isolation. This constraint is
|
|
identical for the existing rig-core code — nothing here is
|
|
`ai-agents`-specific — but it means an actual migration would need to
|
|
confirm the current app doesn't already skirt this same ceiling.
|
|
2. **No logprob-based scoring.** `revise.rs`'s real `score()` function reads
|
|
token logprobs off Shieldstral's response (see `models::ChatLogprobs`) to
|
|
get a continuous 0.0-1.0 score, not a yes/no string. `ai-agents`'
|
|
`Agent::chat()` returns plain text content; there's no exposed hook for
|
|
raw logprobs in the YAML/builder API surface I found. Reproducing the
|
|
current scoring behavior would mean dropping to `ai-agents`' lower-level
|
|
provider access (if any) or keeping rig-core for the judge call and only
|
|
using `ai-agents` for orchestration/prompt config — a hybrid, not a
|
|
clean swap.
|
|
3. Multi-turn revision loop (`MAX_REVISION_ITERATIONS`, feeding the previous
|
|
score back into the next prompt) isn't attempted here — the pipeline
|
|
stage in this spike is a single writer -> judge pass, not the full
|
|
generate/score/revise loop with a threshold-driven exit condition. The
|
|
`pipeline:` construct is one-shot; the retry/threshold loop would likely
|
|
need `states:`/`transitions:` (state machine) rather than `pipeline:`.
|
|
|
|
## Verdict
|
|
|
|
The declarative-YAML story checks out for *provider wiring and prompt
|
|
config* — that part is genuinely config, not code, and matches the
|
|
CrewAI-style ergonomics from the earlier conversation. It does **not**
|
|
cleanly cover this project's actual judge mechanism (logprob scoring), so
|
|
adopting it wholesale would be a partial rewrite of `revise.rs`'s scoring
|
|
logic, not a drop-in replacement. Worth revisiting if a future judge model
|
|
switches to yes/no-only verdicts, or if `ai-agents` grows raw-logprob
|
|
access.
|
|
|
|
## Reproducing
|
|
|
|
```
|
|
ollama serve # writer leg
|
|
# and/or
|
|
/home/austin/.local/share/llama.cpp/build/bin/llama-server \
|
|
-m /home/austin/ai/Shieldstral-1.0-3B-BF16.gguf --jinja -c 32768 \
|
|
--host 127.0.0.1 --port 8000 # judge leg
|
|
|
|
cargo run --bin ai_agents_spike
|
|
```
|