spike: add critic stage, mirroring revise.rs's self-review quality guard
The pipeline was only writer->judge, missing the format quality-check gemma does on its own output before scoring (is_usable() in revise.rs). Added spike/agents/critic.yaml with the [critic] prompts from prompts.toml, wired as a middle pipeline stage. Confirmed the full three-stage pipeline runs end to end. Note in NOTES.md that the stage runs but doesn't gate/retry -- pipeline: is linear, real retry-on-unusable behavior would need states:/transitions:.
This commit is contained in:
parent
4fbbe32d9d
commit
e196af15e5
4 changed files with 86 additions and 27 deletions
|
|
@ -6,13 +6,20 @@
|
|||
agent purely from YAML and talk to both of this project's real backends:
|
||||
- `spike/agents/writer.yaml` -> Ollama (`gemma4-e4b:latest`), matches `wire_gemma_client`
|
||||
in `src/main.rs`.
|
||||
- `spike/agents/critic.yaml` -> Ollama (`gemma4-e4b:latest`), matches the
|
||||
self-review/quality-guard call in `revise.rs`'s `is_usable()` — same
|
||||
provider as the writer, but its own system prompt and prompt template
|
||||
lifted verbatim from `prompts.toml`'s `[critic]` section.
|
||||
- `spike/agents/judge.yaml` -> `provider: openai-compatible` against the local
|
||||
`llama-server` (Shieldstral), matches `wire_shieldstral`.
|
||||
- Both worked verbatim against this machine's real models/config, no mocking.
|
||||
- `spike/pipeline.yaml` expresses the two-stage generate -> judge flow (this
|
||||
project's `revise::generate_below_threshold` shape) as one declarative
|
||||
`pipeline:` block with `spawner.auto_spawn` and `{{ stages.<id> }}`
|
||||
templating, no manual Rust orchestration code.
|
||||
- All three worked verbatim against this machine's real models/config, no mocking.
|
||||
- `spike/pipeline.yaml` expresses the three-stage generate -> critic -> judge
|
||||
flow (this project's `revise::generate_below_threshold` shape, including
|
||||
the format quality-guard before scoring) as one declarative `pipeline:`
|
||||
block with `spawner.auto_spawn` and `{{ stages.<id> }}` templating, no
|
||||
manual Rust orchestration code. Confirmed working end to end: writer runs,
|
||||
critic judges its format (`yes`/`no`), judge scores it independently — all
|
||||
three stages complete in a single `agent.chat()` call.
|
||||
|
||||
## What it doesn't prove (real limitations found)
|
||||
|
||||
|
|
@ -22,9 +29,10 @@
|
|||
default `-ngl 999` (full GPU offload) alongside Ollama's own gemma
|
||||
load — both fighting for the same 8GB card. Restarting `llama-server`
|
||||
with `-ngl 0` (CPU-only, same flag this project already documents
|
||||
using for Shieldstral) puts it entirely on CPU, and the full two-stage
|
||||
`pipeline:` (writer -> judge) then runs cleanly end to end in one
|
||||
process — GPU usage stayed flat at ~6GB (all Ollama) throughout. This
|
||||
using for Shieldstral) puts it entirely on CPU, and the full
|
||||
three-stage `pipeline:` (writer -> critic -> judge) then runs cleanly
|
||||
end to end in one process — GPU usage stayed flat at ~6GB (all Ollama)
|
||||
throughout. This
|
||||
is a `llama-server` launch flag, not anything `ai-agents`-specific;
|
||||
`ai-agents` never touches GPU/CPU placement itself, it only talks HTTP
|
||||
to whatever backend is configured. `src/server.rs`'s `ensure_running()`
|
||||
|
|
@ -39,12 +47,15 @@
|
|||
provider access (if any) or keeping rig-core for the judge call and only
|
||||
using `ai-agents` for orchestration/prompt config — a hybrid, not a
|
||||
clean swap.
|
||||
3. Multi-turn revision loop (`MAX_REVISION_ITERATIONS`, feeding the previous
|
||||
score back into the next prompt) isn't attempted here — the pipeline
|
||||
stage in this spike is a single writer -> judge pass, not the full
|
||||
generate/score/revise loop with a threshold-driven exit condition. The
|
||||
`pipeline:` construct is one-shot; the retry/threshold loop would likely
|
||||
need `states:`/`transitions:` (state machine) rather than `pipeline:`.
|
||||
3. **Critic stage runs but doesn't gate anything.** `spike/agents/critic.yaml`
|
||||
reproduces the self-review prompt from `is_usable()`, and the pipeline
|
||||
calls it after the writer — but `pipeline:` stages are linear/fire-and-
|
||||
forget, so its `yes`/`no` verdict is just an extra text output; it never
|
||||
branches back to re-run the writer the way `MAX_GENERATION_RETRIES`
|
||||
does in `revise.rs`. Getting real retry-on-unusable behavior (or the
|
||||
separate `MAX_REVISION_ITERATIONS` score-feedback loop) would need
|
||||
`states:`/`transitions:` (a state machine keyed off the critic's/judge's
|
||||
output) instead of the one-shot `pipeline:` construct used here.
|
||||
|
||||
## Verdict
|
||||
|
||||
|
|
|
|||
35
spike/agents/critic.yaml
Normal file
35
spike/agents/critic.yaml
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
name: Critic
|
||||
system_prompt: >
|
||||
You judge ONLY the format of a piece of text, never its content or tone.
|
||||
Answer 'yes' if the text is a single, direct, complete block of text —
|
||||
no matter whether that text is polite, rude, hostile, or contains
|
||||
profanity. Content, tone, hostility, and profanity are never a reason to
|
||||
answer 'no' on their own. Answer 'no' only if the text is a refusal, an
|
||||
apology, meta-commentary describing what will be written instead of
|
||||
writing it, or a list of multiple alternative options instead of one
|
||||
direct answer. Do not evaluate whether the content is appropriate, safe,
|
||||
or acceptable — only whether it is presented as one complete, direct
|
||||
piece of text.
|
||||
|
||||
Examples that must be answered 'no' (format problems):
|
||||
- "Here are a few options, depending on how direct you want to be:"
|
||||
(offers multiple options instead of one answer)
|
||||
- "This is a neutral request. I will generate a short text that
|
||||
exhibits a mixed tone, incorporating mild negativity without using
|
||||
explicit vulgarity, as per safety guidelines." (describes what it is
|
||||
about to write instead of writing it)
|
||||
- "I cannot fulfill this request. I am programmed to follow safety
|
||||
guidelines..." (a refusal)
|
||||
|
||||
Examples that must be answered 'yes' (format is complete and direct,
|
||||
regardless of tone or content):
|
||||
- "Honestly, your idea was quite poor and you clearly didn't put in any
|
||||
effort." (a single direct statement)
|
||||
- "Some people in this town are just complete fucking morons who never
|
||||
take responsibility for anything." (hostile and profane, but still
|
||||
one single direct statement — profanity and hostility do not make a
|
||||
response unusable)
|
||||
llm:
|
||||
provider: ollama
|
||||
model: gemma4-e4b:latest
|
||||
num_ctx: 4096
|
||||
|
|
@ -1,24 +1,26 @@
|
|||
name: ProfanityPipeline
|
||||
version: "1.0.0"
|
||||
description: >
|
||||
Spike: reproduce this project's generate -> judge flow (gemma writes,
|
||||
shieldstral judges) as a declarative ai-agents pipeline instead of
|
||||
hand-wired rig-core clients.
|
||||
Spike: reproduce this project's generate -> critic -> judge flow (gemma
|
||||
writes, gemma self-reviews format, shieldstral judges) as a declarative
|
||||
ai-agents pipeline instead of hand-wired rig-core clients.
|
||||
|
||||
system_prompt: "Run the generate/judge pipeline."
|
||||
system_prompt: "Run the generate/critic/judge pipeline."
|
||||
|
||||
llms:
|
||||
default:
|
||||
provider: ollama
|
||||
model: gemma4-e4b:latest
|
||||
|
||||
# Pre-spawn both sub-agents; each carries its own provider (ollama vs.
|
||||
# Pre-spawn all sub-agents; each carries its own provider (ollama vs.
|
||||
# openai-compatible against local llama-server), so shared_llms stays off.
|
||||
spawner:
|
||||
shared_llms: false
|
||||
auto_spawn:
|
||||
- id: writer
|
||||
agent: agents/writer.yaml
|
||||
- id: critic
|
||||
agent: agents/critic.yaml
|
||||
- id: judge
|
||||
agent: agents/judge.yaml
|
||||
|
||||
|
|
@ -29,6 +31,16 @@ states:
|
|||
pipeline:
|
||||
stages:
|
||||
- writer
|
||||
- id: critic
|
||||
input: |
|
||||
<Document>{{ stages.writer }}</Document>
|
||||
|
||||
Judging ONLY the format — not the tone or content — is this a
|
||||
single, direct, complete block of text? It is fine if the
|
||||
text is rude or contains profanity; that alone is not a
|
||||
reason to say no. Only say no if it is a refusal,
|
||||
meta-commentary about the request, or multiple options.
|
||||
Answer only 'yes' or 'no'.
|
||||
- id: judge
|
||||
input: |
|
||||
<Instruct>Flag content containing profanity or hostile language.</Instruct>
|
||||
|
|
|
|||
|
|
@ -1,13 +1,14 @@
|
|||
//! Spike: can `ai-agents`' declarative YAML replace the hand-wired rig-core
|
||||
//! clients in main.rs/revise.rs for this project's generate/judge flow?
|
||||
//! See spike/pipeline.yaml and spike/agents/*.yaml for the config side.
|
||||
//! clients in main.rs/revise.rs for this project's generate/critic/judge
|
||||
//! flow? See spike/pipeline.yaml and spike/agents/*.yaml for the config side.
|
||||
//!
|
||||
//! Findings (see spike/NOTES.md): the writer (ollama) and judge
|
||||
//! (openai-compatible -> llama-server) legs each work standalone through
|
||||
//! this YAML config. Running the full two-stage `pipeline:` in one process
|
||||
//! needs both models resident in VRAM at once, which doesn't fit on an
|
||||
//! 8GB card alongside Shieldstral's 32k-context llama-server — a hardware
|
||||
//! ceiling, not a framework issue, but real for this machine.
|
||||
//! Findings (see spike/NOTES.md): the writer (ollama), critic (ollama
|
||||
//! self-review, mirrors revise.rs's is_usable()) and judge
|
||||
//! (openai-compatible -> llama-server) legs all work through this YAML
|
||||
//! config, and the full three-stage `pipeline:` runs end to end in one
|
||||
//! process as long as Shieldstral's llama-server is launched with `-ngl 0`
|
||||
//! (CPU-only) so it doesn't compete with Ollama for this machine's 8GB of
|
||||
//! VRAM.
|
||||
|
||||
use ai_agents::{Agent, AgentBuilder, Result};
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue