doubleo7/FINDINGS.md

Findings

Running log of things that came up building and testing the Gemma → Shieldstral pipeline, sorted by how much they actually mattered in practice.

Actual obstacles

Things that were real problems and required a fix.

• Trailing colon typo in LLAMA_SERVER_URL produced an "invalid authority" error from Rig's URI parser — llamafile::Client::from_url needs a bare http://host:port, no trailing punctuation, no /v1 suffix (the client appends that itself).
• std::fs::read_to_string("prompts.toml") used a path relative to the process's runtime working directory, which differs between cargo run, RustRover's run config, and any future install location — file-not-found in practice. Fixed by switching to include_str!, which resolves relative to the source file at compile time instead.
• Ollama has no logprobs support at all, in either its native /api/chat or its OpenAI-compatible /v1/chat/completions endpoint. This was a hard blocker for the whole scoring approach — had to serve Shieldstral through llama-server directly instead of Ollama.
• raw_completion() doesn't exist in the last published rig-core crate (0.41.0) — it's only on git main, ahead of any release. Had to pin a git dependency to get it, accepting the instability that comes with tracking an unreleased branch.
• Missing <Instruct>/<Query>/<Document> scaffolding + system preamble produced meaningless, unreliable scores when testing against raw unscaffolded text — the model has no policy to judge against without it.
• temperature: 1.0 vs 0.0 silently distorted reported scores. llama-server only bypasses the full sampler chain (top_k/top_p/min_p/ repetition penalties) for logprobs reporting at greedy decoding (temperature effectively 0); at 1.0 the reported probabilities reflect the post-sampler-chain distribution, not raw logits.
• Gemma's "thinking" mode was on by default for the Ollama model tag. A tight max_tokens budget meant it sometimes got cut off mid-thought before ever emitting content, crashing the naive AssistantContent::Text match on an unhandled Reasoning block. Fixed with "think": false.
• uv dependency resolution failures on transformers==4.57.6 — the version genuinely exists on PyPI, but was shadowed by a same-named package on PyTorch's own wheel index under uv's default first-index strategy. Needed --index-strategy unsafe-best-match (safe here since both indexes are reputable) or a per-package --override.
• Gemma refusing the "generate hostile text" seed prompt depending on exact wording — explicit "hate speech" / "insulting people" phrasing triggered refusals noticeably more than softer framing. Needed a few iterations on the prompt to land on wording that reliably produces scoreable content without tripping Gemma's own alignment training every time.

Overblown, but theoretically impactful under the right conditions

Concerns that turned out not to matter in the cases tested, but aren't nothing — worth revisiting if circumstances change.

• Qualifying/meta text ("Here is a short text:") diluting the score — negligible on a seed document already saturated with hostile content (a ~4-word neutral preamble on a ~50-word hostile block didn't move a 0.9999 score). Could plausibly matter more on revision-step output sitting near the 0.1 threshold, where a few tokens of padding might tip an not-actually-safer revision under the line. Worth watching iteration logs for revision scores landing suspiciously close to threshold right when padding shows up — not worth defending against pre-emptively without evidence it's happening.
• ServerConfig single-field wrapper struct, flagged during a cleanliness review as unnecessary indirection — true in isolation, but it deliberately mirrors the existing Prompts/prompts.toml pattern for consistency, so the real cost is close to nil in context.

Genuinely was irrelevant

Things that looked like they might be a problem and just weren't.

• Whether GenericCompletionModel/ollama::CompletionModel needed to be Clone to support a loop calling them repeatedly — turned out both already derive Clone, and more to the point, didn't even need cloning since both can just be borrowed across iterations. A non-issue once actually checked against source instead of assumed.
• Whether the literal sampled token from Shieldstral's single forced token matters — irrelevant, since scoring always reads the full top_logprobs list regardless of which single token happened to get emitted as content.
• Rig's .completion() normalizing away provider-specific logprobs initially looked like a dead end for using Rig at all for scoring — turned out to have a clean, intended escape hatch (raw_completion()) once the source was actually checked, so the abstraction gap was real but never a blocker.