The pipeline was only writer->judge, missing the format quality-check
gemma does on its own output before scoring (is_usable() in revise.rs).
Added spike/agents/critic.yaml with the [critic] prompts from
prompts.toml, wired as a middle pipeline stage. Confirmed the full
three-stage pipeline runs end to end. Note in NOTES.md that the stage
runs but doesn't gate/retry -- pipeline: is linear, real retry-on-unusable
behavior would need states:/transitions:.
The earlier VRAM-exhaustion failure was llama-server defaulting to full
GPU offload for Shieldstral while Ollama's gemma was also resident.
Restarting llama-server with -ngl 0 frees the GPU for gemma and the
full writer->judge ai-agents pipeline runs end to end with no VRAM
contention. Not an ai-agents limitation -- it never touches GPU
placement, only HTTP.
Adds a standalone binary (src/bin/ai_agents_spike.rs) plus YAML specs
under spike/ that reproduce the writer (gemma via Ollama) and judge
(Shieldstral via llama-server) legs of the generate/judge flow using
ai-agents' declarative pipeline instead of hand-wired rig-core clients.
Both legs verified working individually against real local models.
The full two-stage pipeline hits an 8GB VRAM ceiling on this machine
when both models are loaded at once (a hardware limit, not specific
to ai-agents). ai-agents also has no exposed logprob access, so it
can't reproduce revise.rs's actual scoring mechanism as-is. See
spike/NOTES.md for the full writeup and verdict.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gemma now checks its own output (a fresh, stateless completion call,
not conversation history) before it's accepted as a seed or revision,
retrying up to 5 times if it's a refusal, meta-commentary describing
what it's about to write, or a list of multiple options instead of
one direct answer. Originally tried a separate CPU-only judge model
(critic-cpu) to avoid VRAM contention, but it was both far slower
(13-16s per judgment vs Gemma's own sub-second calls) and unreliable
on the exact failure patterns it was meant to catch — Gemma reviewing
itself turned out faster and more accurate, so critic-cpu is dropped
entirely.
Also required two rounds of prompt tuning, driven by live failures:
first adding concrete negative examples after the judge approved
outputs it should have rejected, then explicitly scoping the check to
format only after Gemma started rejecting its own genuinely hostile
(but well-formed) output — conflating "should I have generated this"
with the format question actually asked. Added integration tests
covering both directions (rejecting bad formats, accepting hostile-
but-well-formed content) as a fast regression check against an
expensive full generation loop.
Enable Rig's built-in tracing spans (model, token usage, cache hits,
latency) via tracing-subscriber, filterable through RUST_LOG and
defaulting to info level. Logs write to stderr so stdout stays
reserved for program output. Standardizes the remaining ad-hoc
println! diagnostics (server startup, per-iteration revision progress,
non-convergence) into structured tracing events at appropriate levels.
Gemma now seeds deliberately hostile text, Shieldstral scores it, and
Gemma revises its own output based on the score until it drops below a
safety threshold (or a max-iteration cap is hit, returning the best
attempt seen). Extracted into a new revise module: score() now borrows
instead of consuming its args so it can run repeatedly, and the
gemma-call/extract-text logic is shared between seed generation and
every revision instead of being duplicated.
Also adds FINDINGS.md logging what actually turned out to be real
obstacles vs. overblown vs. irrelevant while building this out.
Checks whether llama-server is already healthy on startup and spawns it
from configured binary/model paths if not, polling until ready. Server
infra config (binary, model path, host, port, context size) split out
of prompts.toml into its own server.toml, and all of it lives in a new
server module rather than inline in main.rs, alongside a single reused
HTTP client and a shared health-check helper. Gemma client setup now
runs concurrently with the server health-check/spawn since they're
independent.
