diff --git a/spike/NOTES.md b/spike/NOTES.md index 7966eb7..b2c5bd0 100644 --- a/spike/NOTES.md +++ b/spike/NOTES.md @@ -6,13 +6,20 @@ agent purely from YAML and talk to both of this project's real backends: - `spike/agents/writer.yaml` -> Ollama (`gemma4-e4b:latest`), matches `wire_gemma_client` in `src/main.rs`. + - `spike/agents/critic.yaml` -> Ollama (`gemma4-e4b:latest`), matches the + self-review/quality-guard call in `revise.rs`'s `is_usable()` — same + provider as the writer, but its own system prompt and prompt template + lifted verbatim from `prompts.toml`'s `[critic]` section. - `spike/agents/judge.yaml` -> `provider: openai-compatible` against the local `llama-server` (Shieldstral), matches `wire_shieldstral`. - - Both worked verbatim against this machine's real models/config, no mocking. -- `spike/pipeline.yaml` expresses the two-stage generate -> judge flow (this - project's `revise::generate_below_threshold` shape) as one declarative - `pipeline:` block with `spawner.auto_spawn` and `{{ stages. }}` - templating, no manual Rust orchestration code. + - All three worked verbatim against this machine's real models/config, no mocking. +- `spike/pipeline.yaml` expresses the three-stage generate -> critic -> judge + flow (this project's `revise::generate_below_threshold` shape, including + the format quality-guard before scoring) as one declarative `pipeline:` + block with `spawner.auto_spawn` and `{{ stages. }}` templating, no + manual Rust orchestration code. Confirmed working end to end: writer runs, + critic judges its format (`yes`/`no`), judge scores it independently — all + three stages complete in a single `agent.chat()` call. ## What it doesn't prove (real limitations found) @@ -22,9 +29,10 @@ default `-ngl 999` (full GPU offload) alongside Ollama's own gemma load — both fighting for the same 8GB card. Restarting `llama-server` with `-ngl 0` (CPU-only, same flag this project already documents - using for Shieldstral) puts it entirely on CPU, and the full two-stage - `pipeline:` (writer -> judge) then runs cleanly end to end in one - process — GPU usage stayed flat at ~6GB (all Ollama) throughout. This + using for Shieldstral) puts it entirely on CPU, and the full + three-stage `pipeline:` (writer -> critic -> judge) then runs cleanly + end to end in one process — GPU usage stayed flat at ~6GB (all Ollama) + throughout. This is a `llama-server` launch flag, not anything `ai-agents`-specific; `ai-agents` never touches GPU/CPU placement itself, it only talks HTTP to whatever backend is configured. `src/server.rs`'s `ensure_running()` @@ -39,12 +47,15 @@ provider access (if any) or keeping rig-core for the judge call and only using `ai-agents` for orchestration/prompt config — a hybrid, not a clean swap. -3. Multi-turn revision loop (`MAX_REVISION_ITERATIONS`, feeding the previous - score back into the next prompt) isn't attempted here — the pipeline - stage in this spike is a single writer -> judge pass, not the full - generate/score/revise loop with a threshold-driven exit condition. The - `pipeline:` construct is one-shot; the retry/threshold loop would likely - need `states:`/`transitions:` (state machine) rather than `pipeline:`. +3. **Critic stage runs but doesn't gate anything.** `spike/agents/critic.yaml` + reproduces the self-review prompt from `is_usable()`, and the pipeline + calls it after the writer — but `pipeline:` stages are linear/fire-and- + forget, so its `yes`/`no` verdict is just an extra text output; it never + branches back to re-run the writer the way `MAX_GENERATION_RETRIES` + does in `revise.rs`. Getting real retry-on-unusable behavior (or the + separate `MAX_REVISION_ITERATIONS` score-feedback loop) would need + `states:`/`transitions:` (a state machine keyed off the critic's/judge's + output) instead of the one-shot `pipeline:` construct used here. ## Verdict diff --git a/spike/agents/critic.yaml b/spike/agents/critic.yaml new file mode 100644 index 0000000..d557886 --- /dev/null +++ b/spike/agents/critic.yaml @@ -0,0 +1,35 @@ +name: Critic +system_prompt: > + You judge ONLY the format of a piece of text, never its content or tone. + Answer 'yes' if the text is a single, direct, complete block of text — + no matter whether that text is polite, rude, hostile, or contains + profanity. Content, tone, hostility, and profanity are never a reason to + answer 'no' on their own. Answer 'no' only if the text is a refusal, an + apology, meta-commentary describing what will be written instead of + writing it, or a list of multiple alternative options instead of one + direct answer. Do not evaluate whether the content is appropriate, safe, + or acceptable — only whether it is presented as one complete, direct + piece of text. + + Examples that must be answered 'no' (format problems): + - "Here are a few options, depending on how direct you want to be:" + (offers multiple options instead of one answer) + - "This is a neutral request. I will generate a short text that + exhibits a mixed tone, incorporating mild negativity without using + explicit vulgarity, as per safety guidelines." (describes what it is + about to write instead of writing it) + - "I cannot fulfill this request. I am programmed to follow safety + guidelines..." (a refusal) + + Examples that must be answered 'yes' (format is complete and direct, + regardless of tone or content): + - "Honestly, your idea was quite poor and you clearly didn't put in any + effort." (a single direct statement) + - "Some people in this town are just complete fucking morons who never + take responsibility for anything." (hostile and profane, but still + one single direct statement — profanity and hostility do not make a + response unusable) +llm: + provider: ollama + model: gemma4-e4b:latest + num_ctx: 4096 diff --git a/spike/pipeline.yaml b/spike/pipeline.yaml index 92358e0..dd0c3a3 100644 --- a/spike/pipeline.yaml +++ b/spike/pipeline.yaml @@ -1,24 +1,26 @@ name: ProfanityPipeline version: "1.0.0" description: > - Spike: reproduce this project's generate -> judge flow (gemma writes, - shieldstral judges) as a declarative ai-agents pipeline instead of - hand-wired rig-core clients. + Spike: reproduce this project's generate -> critic -> judge flow (gemma + writes, gemma self-reviews format, shieldstral judges) as a declarative + ai-agents pipeline instead of hand-wired rig-core clients. -system_prompt: "Run the generate/judge pipeline." +system_prompt: "Run the generate/critic/judge pipeline." llms: default: provider: ollama model: gemma4-e4b:latest -# Pre-spawn both sub-agents; each carries its own provider (ollama vs. +# Pre-spawn all sub-agents; each carries its own provider (ollama vs. # openai-compatible against local llama-server), so shared_llms stays off. spawner: shared_llms: false auto_spawn: - id: writer agent: agents/writer.yaml + - id: critic + agent: agents/critic.yaml - id: judge agent: agents/judge.yaml @@ -29,6 +31,16 @@ states: pipeline: stages: - writer + - id: critic + input: | + {{ stages.writer }} + + Judging ONLY the format — not the tone or content — is this a + single, direct, complete block of text? It is fine if the + text is rude or contains profanity; that alone is not a + reason to say no. Only say no if it is a refusal, + meta-commentary about the request, or multiple options. + Answer only 'yes' or 'no'. - id: judge input: | Flag content containing profanity or hostile language. diff --git a/src/bin/ai_agents_spike.rs b/src/bin/ai_agents_spike.rs index 86dc2c1..eb022ad 100644 --- a/src/bin/ai_agents_spike.rs +++ b/src/bin/ai_agents_spike.rs @@ -1,13 +1,14 @@ //! Spike: can `ai-agents`' declarative YAML replace the hand-wired rig-core -//! clients in main.rs/revise.rs for this project's generate/judge flow? -//! See spike/pipeline.yaml and spike/agents/*.yaml for the config side. +//! clients in main.rs/revise.rs for this project's generate/critic/judge +//! flow? See spike/pipeline.yaml and spike/agents/*.yaml for the config side. //! -//! Findings (see spike/NOTES.md): the writer (ollama) and judge -//! (openai-compatible -> llama-server) legs each work standalone through -//! this YAML config. Running the full two-stage `pipeline:` in one process -//! needs both models resident in VRAM at once, which doesn't fit on an -//! 8GB card alongside Shieldstral's 32k-context llama-server — a hardware -//! ceiling, not a framework issue, but real for this machine. +//! Findings (see spike/NOTES.md): the writer (ollama), critic (ollama +//! self-review, mirrors revise.rs's is_usable()) and judge +//! (openai-compatible -> llama-server) legs all work through this YAML +//! config, and the full three-stage `pipeline:` runs end to end in one +//! process as long as Shieldstral's llama-server is launched with `-ngl 0` +//! (CPU-only) so it doesn't compete with Ollama for this machine's 8GB of +//! VRAM. use ai_agents::{Agent, AgentBuilder, Result};