e196af15e5
The pipeline was only writer->judge, missing the format quality-check gemma does on its own output before scoring (is_usable() in revise.rs). Added spike/agents/critic.yaml with the [critic] prompts from prompts.toml, wired as a middle pipeline stage. Confirmed the full three-stage pipeline runs end to end. Note in NOTES.md that the stage runs but doesn't gate/retry -- pipeline: is linear, real retry-on-unusable behavior would need states:/transitions:.
83 lines
4.7 KiB
Markdown
83 lines
4.7 KiB
Markdown
# Spike: `ai-agents` (declarative YAML) vs. hand-wired `rig-core`
|
|
|
|
## What this proves
|
|
|
|
- `ai-agents` (crates.io `ai-agents` v1.0.0, Rust-native, no Python) can load an
|
|
agent purely from YAML and talk to both of this project's real backends:
|
|
- `spike/agents/writer.yaml` -> Ollama (`gemma4-e4b:latest`), matches `wire_gemma_client`
|
|
in `src/main.rs`.
|
|
- `spike/agents/critic.yaml` -> Ollama (`gemma4-e4b:latest`), matches the
|
|
self-review/quality-guard call in `revise.rs`'s `is_usable()` — same
|
|
provider as the writer, but its own system prompt and prompt template
|
|
lifted verbatim from `prompts.toml`'s `[critic]` section.
|
|
- `spike/agents/judge.yaml` -> `provider: openai-compatible` against the local
|
|
`llama-server` (Shieldstral), matches `wire_shieldstral`.
|
|
- All three worked verbatim against this machine's real models/config, no mocking.
|
|
- `spike/pipeline.yaml` expresses the three-stage generate -> critic -> judge
|
|
flow (this project's `revise::generate_below_threshold` shape, including
|
|
the format quality-guard before scoring) as one declarative `pipeline:`
|
|
block with `spawner.auto_spawn` and `{{ stages.<id> }}` templating, no
|
|
manual Rust orchestration code. Confirmed working end to end: writer runs,
|
|
critic judges its format (`yes`/`no`), judge scores it independently — all
|
|
three stages complete in a single `agent.chat()` call.
|
|
|
|
## What it doesn't prove (real limitations found)
|
|
|
|
1. ~~VRAM ceiling~~ **Resolved — not a framework limitation at all.** The
|
|
initial failure (`cudaMalloc failed: out of memory` from Ollama's
|
|
`/api/chat`) was from running Shieldstral's `llama-server` at its
|
|
default `-ngl 999` (full GPU offload) alongside Ollama's own gemma
|
|
load — both fighting for the same 8GB card. Restarting `llama-server`
|
|
with `-ngl 0` (CPU-only, same flag this project already documents
|
|
using for Shieldstral) puts it entirely on CPU, and the full
|
|
three-stage `pipeline:` (writer -> critic -> judge) then runs cleanly
|
|
end to end in one process — GPU usage stayed flat at ~6GB (all Ollama)
|
|
throughout. This
|
|
is a `llama-server` launch flag, not anything `ai-agents`-specific;
|
|
`ai-agents` never touches GPU/CPU placement itself, it only talks HTTP
|
|
to whatever backend is configured. `src/server.rs`'s `ensure_running()`
|
|
doesn't currently pass `-ngl`, so it would need an `-ngl 0` addition
|
|
(mirroring `server.toml`) to get the same behavior in the real app.
|
|
2. **No logprob-based scoring.** `revise.rs`'s real `score()` function reads
|
|
token logprobs off Shieldstral's response (see `models::ChatLogprobs`) to
|
|
get a continuous 0.0-1.0 score, not a yes/no string. `ai-agents`'
|
|
`Agent::chat()` returns plain text content; there's no exposed hook for
|
|
raw logprobs in the YAML/builder API surface I found. Reproducing the
|
|
current scoring behavior would mean dropping to `ai-agents`' lower-level
|
|
provider access (if any) or keeping rig-core for the judge call and only
|
|
using `ai-agents` for orchestration/prompt config — a hybrid, not a
|
|
clean swap.
|
|
3. **Critic stage runs but doesn't gate anything.** `spike/agents/critic.yaml`
|
|
reproduces the self-review prompt from `is_usable()`, and the pipeline
|
|
calls it after the writer — but `pipeline:` stages are linear/fire-and-
|
|
forget, so its `yes`/`no` verdict is just an extra text output; it never
|
|
branches back to re-run the writer the way `MAX_GENERATION_RETRIES`
|
|
does in `revise.rs`. Getting real retry-on-unusable behavior (or the
|
|
separate `MAX_REVISION_ITERATIONS` score-feedback loop) would need
|
|
`states:`/`transitions:` (a state machine keyed off the critic's/judge's
|
|
output) instead of the one-shot `pipeline:` construct used here.
|
|
|
|
## Verdict
|
|
|
|
The declarative-YAML story checks out for *provider wiring and prompt
|
|
config* — that part is genuinely config, not code, and matches the
|
|
CrewAI-style ergonomics from the earlier conversation, and the full
|
|
two-stage pipeline now runs end to end against this project's real local
|
|
models (see reproducing steps below). It does **not** cleanly cover this
|
|
project's actual judge mechanism (logprob scoring), so adopting it
|
|
wholesale would be a partial rewrite of `revise.rs`'s scoring logic, not a
|
|
drop-in replacement. Worth revisiting if a future judge model switches to
|
|
yes/no-only verdicts, or if `ai-agents` grows raw-logprob access.
|
|
|
|
## Reproducing
|
|
|
|
```
|
|
ollama serve # writer leg (gemma)
|
|
|
|
/home/austin/.local/share/llama.cpp/build/bin/llama-server \
|
|
-m /home/austin/ai/Shieldstral-1.0-3B-BF16.gguf --jinja -c 32768 \
|
|
--host 127.0.0.1 --port 8000 -ngl 0 # judge leg, CPU-only so it
|
|
# doesn't fight gemma for VRAM
|
|
|
|
cargo run --bin ai_agents_spike
|
|
```
|