doubleo7/spike/NOTES.md

Spike: ai-agents (declarative YAML) vs. hand-wired rig-core

What this proves

• ai-agents (crates.io ai-agents v1.0.0, Rust-native, no Python) can load an agent purely from YAML and talk to both of this project's real backends:
  • spike/agents/writer.yaml -> Ollama (gemma4-e4b:latest), matches wire_gemma_client in src/main.rs.
  • spike/agents/judge.yaml -> provider: openai-compatible against the local llama-server (Shieldstral), matches wire_shieldstral.
  • Both worked verbatim against this machine's real models/config, no mocking.
• spike/pipeline.yaml expresses the two-stage generate -> judge flow (this project's revise::generate_below_threshold shape) as one declarative pipeline: block with spawner.auto_spawn and {{ stages.<id> }} templating, no manual Rust orchestration code.

What it doesn't prove (real limitations found)

1. VRAM ceiling Resolved — not a framework limitation at all. The initial failure (cudaMalloc failed: out of memory from Ollama's /api/chat) was from running Shieldstral's llama-server at its default -ngl 999 (full GPU offload) alongside Ollama's own gemma load — both fighting for the same 8GB card. Restarting llama-server with -ngl 0 (CPU-only, same flag this project already documents using for Shieldstral) puts it entirely on CPU, and the full two-stage pipeline: (writer -> judge) then runs cleanly end to end in one process — GPU usage stayed flat at ~6GB (all Ollama) throughout. This is a llama-server launch flag, not anything ai-agents-specific; ai-agents never touches GPU/CPU placement itself, it only talks HTTP to whatever backend is configured. src/server.rs's ensure_running() doesn't currently pass -ngl, so it would need an -ngl 0 addition (mirroring server.toml) to get the same behavior in the real app.
2. No logprob-based scoring. revise.rs's real score() function reads token logprobs off Shieldstral's response (see models::ChatLogprobs) to get a continuous 0.0-1.0 score, not a yes/no string. ai-agents' Agent::chat() returns plain text content; there's no exposed hook for raw logprobs in the YAML/builder API surface I found. Reproducing the current scoring behavior would mean dropping to ai-agents' lower-level provider access (if any) or keeping rig-core for the judge call and only using ai-agents for orchestration/prompt config — a hybrid, not a clean swap.
3. Multi-turn revision loop (MAX_REVISION_ITERATIONS, feeding the previous score back into the next prompt) isn't attempted here — the pipeline stage in this spike is a single writer -> judge pass, not the full generate/score/revise loop with a threshold-driven exit condition. The pipeline: construct is one-shot; the retry/threshold loop would likely need states:/transitions: (state machine) rather than pipeline:.

Verdict

The declarative-YAML story checks out for provider wiring and prompt config — that part is genuinely config, not code, and matches the CrewAI-style ergonomics from the earlier conversation, and the full two-stage pipeline now runs end to end against this project's real local models (see reproducing steps below). It does not cleanly cover this project's actual judge mechanism (logprob scoring), so adopting it wholesale would be a partial rewrite of revise.rs's scoring logic, not a drop-in replacement. Worth revisiting if a future judge model switches to yes/no-only verdicts, or if ai-agents grows raw-logprob access.

Reproducing

ollama serve                      # writer leg (gemma)

/home/austin/.local/share/llama.cpp/build/bin/llama-server \
  -m /home/austin/ai/Shieldstral-1.0-3B-BF16.gguf --jinja -c 32768 \
  --host 127.0.0.1 --port 8000 -ngl 0   # judge leg, CPU-only so it
                                         # doesn't fight gemma for VRAM

cargo run --bin ai_agents_spike