doubleo7/FINDINGS.md

# Findings

Running log of things that came up building and testing the Gemma → Shieldstral
pipeline, sorted by how much they actually mattered in practice.

## Actual obstacles

Things that were real problems and required a fix.

- **Trailing colon typo in `LLAMA_SERVER_URL`** produced an "invalid authority"
  error from Rig's URI parser — `llamafile::Client::from_url` needs a bare
  `http://host:port`, no trailing punctuation, no `/v1` suffix (the client
  appends that itself).
- **`std::fs::read_to_string("prompts.toml")` used a path relative to the
  process's runtime working directory**, which differs between `cargo run`,
  RustRover's run config, and any future install location — file-not-found
  in practice. Fixed by switching to `include_str!`, which resolves relative
  to the source file at compile time instead.
- **Ollama has no logprobs support at all**, in either its native `/api/chat`
  or its OpenAI-compatible `/v1/chat/completions` endpoint. This was a hard
  blocker for the whole scoring approach — had to serve Shieldstral through
  `llama-server` directly instead of Ollama.
- **`raw_completion()` doesn't exist in the last published `rig-core` crate
  (0.41.0)** — it's only on git `main`, ahead of any release. Had to pin a
  git dependency to get it, accepting the instability that comes with
  tracking an unreleased branch.
- **Missing `<Instruct>/<Query>/<Document>` scaffolding + system preamble**
  produced meaningless, unreliable scores when testing against raw
  unscaffolded text — the model has no policy to judge against without it.
- **`temperature: 1.0` vs `0.0`** silently distorted reported scores.
  llama-server only bypasses the full sampler chain (top_k/top_p/min_p/
  repetition penalties) for logprobs reporting at greedy decoding
  (`temperature` effectively 0); at `1.0` the reported probabilities reflect
  the post-sampler-chain distribution, not raw logits.
- **Gemma's "thinking" mode was on by default** for the Ollama model tag. A
  tight `max_tokens` budget meant it sometimes got cut off mid-thought before
  ever emitting `content`, crashing the naive `AssistantContent::Text` match
  on an unhandled `Reasoning` block. Fixed with `"think": false`.
- **`uv` dependency resolution failures on `transformers==4.57.6`** — the
  version genuinely exists on PyPI, but was shadowed by a same-named package
  on PyTorch's own wheel index under uv's default `first-index` strategy.
  Needed `--index-strategy unsafe-best-match` (safe here since both indexes
  are reputable) or a per-package `--override`.
- **Gemma refusing the "generate hostile text" seed prompt** depending on
  exact wording — explicit "hate speech" / "insulting people" phrasing
  triggered refusals noticeably more than softer framing. Needed a few
  iterations on the prompt to land on wording that reliably produces
  scoreable content without tripping Gemma's own alignment training every
  time.

## Overblown, but theoretically impactful under the right conditions

Concerns that turned out not to matter in the cases tested, but aren't
nothing — worth revisiting if circumstances change.

- **Qualifying/meta text ("Here is a short text:") diluting the score** —
  negligible on a seed document already saturated with hostile content (a
  ~4-word neutral preamble on a ~50-word hostile block didn't move a 0.9999
  score). Could plausibly matter more on *revision*-step output sitting near
  the 0.1 threshold, where a few tokens of padding might tip an
  not-actually-safer revision under the line. Worth watching iteration logs
  for revision scores landing suspiciously close to threshold right when
  padding shows up — not worth defending against pre-emptively without
  evidence it's happening.
- **`ServerConfig` single-field wrapper struct**, flagged during a
  cleanliness review as unnecessary indirection — true in isolation, but it
  deliberately mirrors the existing `Prompts`/`prompts.toml` pattern for
  consistency, so the real cost is close to nil in context.

## Genuinely was irrelevant

Things that looked like they might be a problem and just weren't.

- **Whether `GenericCompletionModel`/`ollama::CompletionModel` needed to be
  `Clone`** to support a loop calling them repeatedly — turned out both
  already derive `Clone`, and more to the point, didn't even need cloning
  since both can just be borrowed across iterations. A non-issue once
  actually checked against source instead of assumed.
- **Whether the literal sampled token from Shieldstral's single forced token
  matters** — irrelevant, since scoring always reads the full
  `top_logprobs` list regardless of which single token happened to get
  emitted as `content`.
- **Rig's `.completion()` normalizing away provider-specific `logprobs`**
  initially looked like a dead end for using Rig at all for scoring — turned
  out to have a clean, intended escape hatch (`raw_completion()`) once the
  source was actually checked, so the abstraction gap was real but never a
  blocker.