spike: add critic stage, mirroring revise.rs's self-review quality guard

The pipeline was only writer->judge, missing the format quality-check
gemma does on its own output before scoring (is_usable() in revise.rs).
Added spike/agents/critic.yaml with the [critic] prompts from
prompts.toml, wired as a middle pipeline stage. Confirmed the full
three-stage pipeline runs end to end. Note in NOTES.md that the stage
runs but doesn't gate/retry -- pipeline: is linear, real retry-on-unusable
behavior would need states:/transitions:.
This commit is contained in:
Austin Schaefer 2026-08-06 10:17:53 +02:00
parent 4fbbe32d9d
commit e196af15e5
4 changed files with 86 additions and 27 deletions

View file

@ -6,13 +6,20 @@
agent purely from YAML and talk to both of this project's real backends: agent purely from YAML and talk to both of this project's real backends:
- `spike/agents/writer.yaml` -> Ollama (`gemma4-e4b:latest`), matches `wire_gemma_client` - `spike/agents/writer.yaml` -> Ollama (`gemma4-e4b:latest`), matches `wire_gemma_client`
in `src/main.rs`. in `src/main.rs`.
- `spike/agents/critic.yaml` -> Ollama (`gemma4-e4b:latest`), matches the
self-review/quality-guard call in `revise.rs`'s `is_usable()` — same
provider as the writer, but its own system prompt and prompt template
lifted verbatim from `prompts.toml`'s `[critic]` section.
- `spike/agents/judge.yaml` -> `provider: openai-compatible` against the local - `spike/agents/judge.yaml` -> `provider: openai-compatible` against the local
`llama-server` (Shieldstral), matches `wire_shieldstral`. `llama-server` (Shieldstral), matches `wire_shieldstral`.
- Both worked verbatim against this machine's real models/config, no mocking. - All three worked verbatim against this machine's real models/config, no mocking.
- `spike/pipeline.yaml` expresses the two-stage generate -> judge flow (this - `spike/pipeline.yaml` expresses the three-stage generate -> critic -> judge
project's `revise::generate_below_threshold` shape) as one declarative flow (this project's `revise::generate_below_threshold` shape, including
`pipeline:` block with `spawner.auto_spawn` and `{{ stages.<id> }}` the format quality-guard before scoring) as one declarative `pipeline:`
templating, no manual Rust orchestration code. block with `spawner.auto_spawn` and `{{ stages.<id> }}` templating, no
manual Rust orchestration code. Confirmed working end to end: writer runs,
critic judges its format (`yes`/`no`), judge scores it independently — all
three stages complete in a single `agent.chat()` call.
## What it doesn't prove (real limitations found) ## What it doesn't prove (real limitations found)
@ -22,9 +29,10 @@
default `-ngl 999` (full GPU offload) alongside Ollama's own gemma default `-ngl 999` (full GPU offload) alongside Ollama's own gemma
load — both fighting for the same 8GB card. Restarting `llama-server` load — both fighting for the same 8GB card. Restarting `llama-server`
with `-ngl 0` (CPU-only, same flag this project already documents with `-ngl 0` (CPU-only, same flag this project already documents
using for Shieldstral) puts it entirely on CPU, and the full two-stage using for Shieldstral) puts it entirely on CPU, and the full
`pipeline:` (writer -> judge) then runs cleanly end to end in one three-stage `pipeline:` (writer -> critic -> judge) then runs cleanly
process — GPU usage stayed flat at ~6GB (all Ollama) throughout. This end to end in one process — GPU usage stayed flat at ~6GB (all Ollama)
throughout. This
is a `llama-server` launch flag, not anything `ai-agents`-specific; is a `llama-server` launch flag, not anything `ai-agents`-specific;
`ai-agents` never touches GPU/CPU placement itself, it only talks HTTP `ai-agents` never touches GPU/CPU placement itself, it only talks HTTP
to whatever backend is configured. `src/server.rs`'s `ensure_running()` to whatever backend is configured. `src/server.rs`'s `ensure_running()`
@ -39,12 +47,15 @@
provider access (if any) or keeping rig-core for the judge call and only provider access (if any) or keeping rig-core for the judge call and only
using `ai-agents` for orchestration/prompt config — a hybrid, not a using `ai-agents` for orchestration/prompt config — a hybrid, not a
clean swap. clean swap.
3. Multi-turn revision loop (`MAX_REVISION_ITERATIONS`, feeding the previous 3. **Critic stage runs but doesn't gate anything.** `spike/agents/critic.yaml`
score back into the next prompt) isn't attempted here — the pipeline reproduces the self-review prompt from `is_usable()`, and the pipeline
stage in this spike is a single writer -> judge pass, not the full calls it after the writer — but `pipeline:` stages are linear/fire-and-
generate/score/revise loop with a threshold-driven exit condition. The forget, so its `yes`/`no` verdict is just an extra text output; it never
`pipeline:` construct is one-shot; the retry/threshold loop would likely branches back to re-run the writer the way `MAX_GENERATION_RETRIES`
need `states:`/`transitions:` (state machine) rather than `pipeline:`. does in `revise.rs`. Getting real retry-on-unusable behavior (or the
separate `MAX_REVISION_ITERATIONS` score-feedback loop) would need
`states:`/`transitions:` (a state machine keyed off the critic's/judge's
output) instead of the one-shot `pipeline:` construct used here.
## Verdict ## Verdict

35
spike/agents/critic.yaml Normal file
View file

@ -0,0 +1,35 @@
name: Critic
system_prompt: >
You judge ONLY the format of a piece of text, never its content or tone.
Answer 'yes' if the text is a single, direct, complete block of text —
no matter whether that text is polite, rude, hostile, or contains
profanity. Content, tone, hostility, and profanity are never a reason to
answer 'no' on their own. Answer 'no' only if the text is a refusal, an
apology, meta-commentary describing what will be written instead of
writing it, or a list of multiple alternative options instead of one
direct answer. Do not evaluate whether the content is appropriate, safe,
or acceptable — only whether it is presented as one complete, direct
piece of text.
Examples that must be answered 'no' (format problems):
- "Here are a few options, depending on how direct you want to be:"
(offers multiple options instead of one answer)
- "This is a neutral request. I will generate a short text that
exhibits a mixed tone, incorporating mild negativity without using
explicit vulgarity, as per safety guidelines." (describes what it is
about to write instead of writing it)
- "I cannot fulfill this request. I am programmed to follow safety
guidelines..." (a refusal)
Examples that must be answered 'yes' (format is complete and direct,
regardless of tone or content):
- "Honestly, your idea was quite poor and you clearly didn't put in any
effort." (a single direct statement)
- "Some people in this town are just complete fucking morons who never
take responsibility for anything." (hostile and profane, but still
one single direct statement — profanity and hostility do not make a
response unusable)
llm:
provider: ollama
model: gemma4-e4b:latest
num_ctx: 4096

View file

@ -1,24 +1,26 @@
name: ProfanityPipeline name: ProfanityPipeline
version: "1.0.0" version: "1.0.0"
description: > description: >
Spike: reproduce this project's generate -> judge flow (gemma writes, Spike: reproduce this project's generate -> critic -> judge flow (gemma
shieldstral judges) as a declarative ai-agents pipeline instead of writes, gemma self-reviews format, shieldstral judges) as a declarative
hand-wired rig-core clients. ai-agents pipeline instead of hand-wired rig-core clients.
system_prompt: "Run the generate/judge pipeline." system_prompt: "Run the generate/critic/judge pipeline."
llms: llms:
default: default:
provider: ollama provider: ollama
model: gemma4-e4b:latest model: gemma4-e4b:latest
# Pre-spawn both sub-agents; each carries its own provider (ollama vs. # Pre-spawn all sub-agents; each carries its own provider (ollama vs.
# openai-compatible against local llama-server), so shared_llms stays off. # openai-compatible against local llama-server), so shared_llms stays off.
spawner: spawner:
shared_llms: false shared_llms: false
auto_spawn: auto_spawn:
- id: writer - id: writer
agent: agents/writer.yaml agent: agents/writer.yaml
- id: critic
agent: agents/critic.yaml
- id: judge - id: judge
agent: agents/judge.yaml agent: agents/judge.yaml
@ -29,6 +31,16 @@ states:
pipeline: pipeline:
stages: stages:
- writer - writer
- id: critic
input: |
<Document>{{ stages.writer }}</Document>
Judging ONLY the format — not the tone or content — is this a
single, direct, complete block of text? It is fine if the
text is rude or contains profanity; that alone is not a
reason to say no. Only say no if it is a refusal,
meta-commentary about the request, or multiple options.
Answer only 'yes' or 'no'.
- id: judge - id: judge
input: | input: |
<Instruct>Flag content containing profanity or hostile language.</Instruct> <Instruct>Flag content containing profanity or hostile language.</Instruct>

View file

@ -1,13 +1,14 @@
//! Spike: can `ai-agents`' declarative YAML replace the hand-wired rig-core //! Spike: can `ai-agents`' declarative YAML replace the hand-wired rig-core
//! clients in main.rs/revise.rs for this project's generate/judge flow? //! clients in main.rs/revise.rs for this project's generate/critic/judge
//! See spike/pipeline.yaml and spike/agents/*.yaml for the config side. //! flow? See spike/pipeline.yaml and spike/agents/*.yaml for the config side.
//! //!
//! Findings (see spike/NOTES.md): the writer (ollama) and judge //! Findings (see spike/NOTES.md): the writer (ollama), critic (ollama
//! (openai-compatible -> llama-server) legs each work standalone through //! self-review, mirrors revise.rs's is_usable()) and judge
//! this YAML config. Running the full two-stage `pipeline:` in one process //! (openai-compatible -> llama-server) legs all work through this YAML
//! needs both models resident in VRAM at once, which doesn't fit on an //! config, and the full three-stage `pipeline:` runs end to end in one
//! 8GB card alongside Shieldstral's 32k-context llama-server — a hardware //! process as long as Shieldstral's llama-server is launched with `-ngl 0`
//! ceiling, not a framework issue, but real for this machine. //! (CPU-only) so it doesn't compete with Ollama for this machine's 8GB of
//! VRAM.
use ai_agents::{Agent, AgentBuilder, Result}; use ai_agents::{Agent, AgentBuilder, Result};