doubleo7/spike/NOTES.md

Spike: ai-agents (declarative YAML) vs. hand-wired rig-core

What this proves

• ai-agents (crates.io ai-agents v1.0.0, Rust-native, no Python) can load an agent purely from YAML and talk to both of this project's real backends:
  • spike/agents/writer.yaml -> Ollama (gemma4-e4b:latest), matches wire_gemma_client in src/main.rs.
  • spike/agents/judge.yaml -> provider: openai-compatible against the local llama-server (Shieldstral), matches wire_shieldstral.
  • Both worked verbatim against this machine's real models/config, no mocking.
• spike/pipeline.yaml expresses the two-stage generate -> judge flow (this project's revise::generate_below_threshold shape) as one declarative pipeline: block with spawner.auto_spawn and {{ stages.<id> }} templating, no manual Rust orchestration code.

What it doesn't prove (real limitations found)

1. VRAM ceiling, not a framework bug. Running the full pipeline in one process needs Ollama's gemma model and Shieldstral's llama-server (32k ctx, ~5.6GB) resident at once. On this 8GB card that overflows CUDA ("out of memory" from Ollama's own /api/chat, not from ai-agents). The writer and judge legs each work fine in isolation. This constraint is identical for the existing rig-core code — nothing here is ai-agents-specific — but it means an actual migration would need to confirm the current app doesn't already skirt this same ceiling.
2. No logprob-based scoring. revise.rs's real score() function reads token logprobs off Shieldstral's response (see models::ChatLogprobs) to get a continuous 0.0-1.0 score, not a yes/no string. ai-agents' Agent::chat() returns plain text content; there's no exposed hook for raw logprobs in the YAML/builder API surface I found. Reproducing the current scoring behavior would mean dropping to ai-agents' lower-level provider access (if any) or keeping rig-core for the judge call and only using ai-agents for orchestration/prompt config — a hybrid, not a clean swap.
3. Multi-turn revision loop (MAX_REVISION_ITERATIONS, feeding the previous score back into the next prompt) isn't attempted here — the pipeline stage in this spike is a single writer -> judge pass, not the full generate/score/revise loop with a threshold-driven exit condition. The pipeline: construct is one-shot; the retry/threshold loop would likely need states:/transitions: (state machine) rather than pipeline:.

Verdict

The declarative-YAML story checks out for provider wiring and prompt config — that part is genuinely config, not code, and matches the CrewAI-style ergonomics from the earlier conversation. It does not cleanly cover this project's actual judge mechanism (logprob scoring), so adopting it wholesale would be a partial rewrite of revise.rs's scoring logic, not a drop-in replacement. Worth revisiting if a future judge model switches to yes/no-only verdicts, or if ai-agents grows raw-logprob access.

Reproducing

ollama serve                      # writer leg
# and/or
/home/austin/.local/share/llama.cpp/build/bin/llama-server \
  -m /home/austin/ai/Shieldstral-1.0-3B-BF16.gguf --jinja -c 32768 \
  --host 127.0.0.1 --port 8000    # judge leg

cargo run --bin ai_agents_spike