schaefera/doubleo7
Watch
1
0
Fork
You've already forked doubleo7
 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions

claimcheck: verdict first, evidence only when needed, thinking off #19

Open
claude-bot wants to merge 1 commit from worktree-fast-verdict into master
pull from: worktree-fast-verdict
merge into: schaefera:master
Conversation 1 Commits 1 Files changed 3 +83 -20
claude-bot commented 2026-09-25 14:12:18 +00:00
Collaborator

What

The claimcheck judge now runs in two stages, with Ollama's thinking turned off:

  1. Verdict-only call. Returns supported, contradicted, not_addressed or inconclusive and nothing else.
  2. Evidence call. Only runs when the verdict isn't not_addressed. It asks for the quotes and explanation behind that verdict. It uses the same system prompt and the same prompt prefix as the first call, so Ollama answers from its prompt cache.

The quote verification and the downgrade to inconclusive still work as before.

Why

gemma4-e4b thinks by default. It generated about 150 reasoning tokens before a roughly 10-token verdict, so most of the latency came from thinking.

Measurements

Model gemma4-e4b on an RTX 5070 laptop GPU. 8 claims, 3 runs each:

Variant Mean per check Correct
Single call, thinking on (before) 5.6 s 22/24
Single call, thinking off 1.1 s 21/24
Two calls, thinking off (this PR) 0.9 s 21/24
→ not_addressed ~0.35 s
→ supported / contradicted ~1.1 s

Trade-off: the one lost case is a mixed-evidence claim that should be inconclusive. With thinking on it came back correct 1 run out of 3; with thinking off, 0 out of 3. This is noted in the README's known limitations. The test set is small and uses a single source page, so the accuracy numbers are a signal, not a benchmark.

Checks

  • cargo fmt --check: passes
  • cargo clippy --workspace --all-targets -D warnings: passes
  • Unit tests: 21 passed
  • Live Ollama MCP tests (--ignored): 3 passed

Possible follow-up

Read a confidence score from Ollama's logprobs. Ollama reports these before the format constraint is applied, so they would need renormalizing over the four verdict options.

🤖 Generated with Claude Code

## What The claimcheck judge now runs in two stages, with Ollama's thinking turned off: 1. **Verdict-only call.** Returns `supported`, `contradicted`, `not_addressed` or `inconclusive` and nothing else. 2. **Evidence call.** Only runs when the verdict isn't `not_addressed`. It asks for the quotes and explanation behind that verdict. It uses the same system prompt and the same prompt prefix as the first call, so Ollama answers from its prompt cache. The quote verification and the downgrade to `inconclusive` still work as before. ## Why `gemma4-e4b` thinks by default. It generated about 150 reasoning tokens before a roughly 10-token verdict, so most of the latency came from thinking. ## Measurements Model `gemma4-e4b` on an RTX 5070 laptop GPU. 8 claims, 3 runs each: | Variant | Mean per check | Correct | |---|---|---| | Single call, thinking on (before) | 5.6 s | 22/24 | | Single call, thinking off | 1.1 s | 21/24 | | **Two calls, thinking off (this PR)** | **0.9 s** | 21/24 | | → `not_addressed` | ~0.35 s | | | → `supported` / `contradicted` | ~1.1 s | | **Trade-off:** the one lost case is a mixed-evidence claim that should be `inconclusive`. With thinking on it came back correct 1 run out of 3; with thinking off, 0 out of 3. This is noted in the README's known limitations. The test set is small and uses a single source page, so the accuracy numbers are a signal, not a benchmark. ## Checks - `cargo fmt --check`: passes - `cargo clippy --workspace --all-targets -D warnings`: passes - Unit tests: 21 passed - Live Ollama MCP tests (`--ignored`): 3 passed ## Possible follow-up Read a confidence score from Ollama's `logprobs`. Ollama reports these before the format constraint is applied, so they would need renormalizing over the four verdict options. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
perf(claimcheck): judge verdict first, fetch evidence only when needed
All checks were successful
CI / ci (pull_request) Successful in 7m59s
Details
CI / audit (pull_request) Successful in 12s
Details
80378abb21 
The judging model ran with Ollama's thinking on by default, spending ~150
tokens reasoning before a ~10-token verdict. Turn it off, and split the
judgement in two: a verdict-only call, then — unless the passages don't
address the claim — a second call for quotes and explanation on the same
prompt prefix, which Ollama serves from its prompt cache.

Measured on gemma4-e4b (RTX 5070 laptop), 8 claims x 3 runs:

  single pass, thinking on (before)   5.6 s mean   22/24 correct
  single pass, thinking off           1.1 s mean   21/24
  two-stage, thinking off (this)      0.9 s mean   21/24
    not_addressed                     ~0.35 s
    supported / contradicted          ~1.1 s

The one lost case is a mixed-evidence claim that should be inconclusive;
thinking-on got it 1 of 3 times, thinking-off 0 of 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
schaefera approved these changes 2026-09-25 14:19:55 +00:00
All checks were successful
CI / ci (pull_request) Successful in 7m59s
Details
CI / audit (pull_request) Successful in 12s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin "worktree-fast-verdict:worktree-fast-verdict"
git switch "worktree-fast-verdict"
Sign in to join this conversation.
Reviewers
No reviewers
schaefera
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
schaefera/doubleo7!19
No description provided.