doubleo7/spike/NOTES.md

Spike: ai-agents (declarative YAML) vs. hand-wired rig-core

What this proves

• ai-agents (crates.io ai-agents v1.0.0, Rust-native, no Python) can load an agent purely from YAML and talk to both of this project's real backends:
  • spike/agents/writer.yaml -> Ollama (gemma4-e4b:latest), matches wire_gemma_client in src/main.rs.
  • spike/agents/critic.yaml -> Ollama (gemma4-e4b:latest), matches the self-review/quality-guard call in revise.rs's is_usable() — same provider as the writer, but its own system prompt and prompt template lifted verbatim from prompts.toml's [critic] section.
  • spike/agents/judge.yaml -> provider: openai-compatible against the local llama-server (Shieldstral), matches wire_shieldstral.
  • All three worked verbatim against this machine's real models/config, no mocking.
• spike/pipeline.yaml expresses the three-stage generate -> critic -> judge flow (this project's revise::generate_below_threshold shape, including the format quality-guard before scoring) as one declarative pipeline: block with spawner.auto_spawn and {{ stages.<id> }} templating, no manual Rust orchestration code. Confirmed working end to end: writer runs, critic judges its format (yes/no), judge scores it independently — all three stages complete in a single agent.chat() call.

What it doesn't prove (real limitations found)

1. VRAM ceiling Resolved — not a framework limitation at all. The initial failure (cudaMalloc failed: out of memory from Ollama's /api/chat) was from running Shieldstral's llama-server at its default -ngl 999 (full GPU offload) alongside Ollama's own gemma load — both fighting for the same 8GB card. Restarting llama-server with -ngl 0 (CPU-only, same flag this project already documents using for Shieldstral) puts it entirely on CPU, and the full three-stage pipeline: (writer -> critic -> judge) then runs cleanly end to end in one process — GPU usage stayed flat at ~6GB (all Ollama) throughout. This is a llama-server launch flag, not anything ai-agents-specific; ai-agents never touches GPU/CPU placement itself, it only talks HTTP to whatever backend is configured. src/server.rs's ensure_running() doesn't currently pass -ngl, so it would need an -ngl 0 addition (mirroring server.toml) to get the same behavior in the real app.

2. No logprob-based scoring. revise.rs's real score() function reads token logprobs off Shieldstral's response (see models::ChatLogprobs) to get a continuous 0.0-1.0 score, not a yes/no string. ai-agents' Agent::chat() returns plain text content; there's no exposed hook for raw logprobs in the YAML/builder API surface I found. Reproducing the current scoring behavior would mean dropping to ai-agents' lower-level provider access (if any) or keeping rig-core for the judge call and only using ai-agents for orchestration/prompt config — a hybrid, not a clean swap.

3. Critic stage runs but doesn't gate anything in pipeline:, and states:/transitions: doesn't cleanly fix that either. spike/agents/critic.yaml reproduces the self-review prompt from is_usable(), and pipeline: calls it after the writer — but its yes/no verdict is just an extra text output; pipeline: stages are linear/fire-and-forget, no branching back to retry the writer the way MAX_GENERATION_RETRIES does.

   Tried building the retry loop with states:/transitions: instead (spike/state_machine.yaml, src/bin/ai_agents_spike_states.rs): write -> critique -> (loop to write on "no", or advance to judge on "yes"), using a guard: expression on extracted context. Two real obstacles surfaced, both confirmed against the crate source (ai-agents-runtime-1.0.0, ai-agents-state-1.0.0) and by running it:

   • delegate: states have no per-turn input override. Only pipeline:/concurrent: stages get an input: Jinja template (ai-agents-state-1.0.0/src/config.rs PipelineStageEntry::Config). A bare delegate: critic state just forwards the parent conversation history, with no way to inject the "judge only the format" instruction — in practice the critic just echoed the writer's document back verbatim instead of answering yes/no. Switching the critique/judge states to single-stage pipeline: blocks (which do support input:) fixed this.
   • extract: context extractors read the state's incoming user_message, not its generated response. run_context_extractors_staged (ai-agents-runtime-1.0.0/src/runtime.rs:7320) builds its extraction prompt from user_message — the turn's input — never the assistant's (or a delegated/piped agent's) output. So an extract: block meant to capture "what the critic just answered" has nothing real to read; confirmed by RUST_LOG=debug showing no extraction activity at all around the state transition, and by the context.usable-gated transition to judge never firing even when the critic's actual answer was yes. The machine just stopped after write -> critique and returned the critic's raw response as chat()'s final output — i.e. the loop never proved out.
   • Separately, each agent.chat() call only appeared to advance one state transition (depth=1 in the logs) before returning, so even with working guards, driving the machine to a judge terminal state might require the caller to loop calling chat() per hop rather than getting one resolved answer per call the way pipeline: does.

   Bottom line: reproducing MAX_GENERATION_RETRIES declaratively isn't a matter of swapping pipeline: for states:/transitions: — the state-machine primitives here are built for turn-based conversational branching (routing user intent to sub-flows), not for gating on a sub-agent's structured verdict about text it just produced. That would need either a custom tool/hook that calls back into Rust to inspect stage output and decide the transition, or keeping the retry loop in hand-written Rust (as revise.rs already does) and using ai-agents only for the linear leg of the flow.

Verdict

The declarative-YAML story checks out for provider wiring and prompt config — that part is genuinely config, not code, and matches the CrewAI-style ergonomics from the earlier conversation, and the full two-stage pipeline now runs end to end against this project's real local models (see reproducing steps below). It does not cleanly cover this project's actual judge mechanism (logprob scoring), so adopting it wholesale would be a partial rewrite of revise.rs's scoring logic, not a drop-in replacement. Worth revisiting if a future judge model switches to yes/no-only verdicts, or if ai-agents grows raw-logprob access.

Reproducing

ollama serve                      # writer leg (gemma)

/home/austin/.local/share/llama.cpp/build/bin/llama-server \
  -m /home/austin/ai/Shieldstral-1.0-3B-BF16.gguf --jinja -c 32768 \
  --host 127.0.0.1 --port 8000 -ngl 0   # judge leg, CPU-only so it
                                         # doesn't fight gemma for VRAM

cargo run --bin ai_agents_spike           # linear pipeline: writer -> critic -> judge
cargo run --bin ai_agents_spike_states    # states:/transitions: retry-loop attempt (see limitation #3 above)