CI: the app job failed once on an unchanged tree, cause unknown #34

New issue

Closed

opened 2026-10-22 09:15:26 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-10-22 09:15:26 +00:00

Collaborator

On #25, app failed after 2m38s on 1f85405, an empty commit whose tree matched the green efdde90. A normal run takes about 3m. Two things showed it was a flake:

cargo make app-docker passed locally on the same tree: slow profile 314 ms, peak PSS 141062 kB.
A rerun via the empty commit 90b716f passed in 3m8s.

This Forgejo's API serves neither job logs nor reruns (both return 404), so the failing step was not seen.

The timing points at tools/mem-check near the end of the job. The suspects:

PSS budget, most likely: main's runs reach 141–154 MB against the 165 MB limit.
Start-up budgets: a timing spike on the slow profile, though the 2000 ms and 3000 ms budgets are about 10× the measured times.

To do:

Find the failing step in the web UI (runs/82, job 2) and record it here.
If it's PSS, record the margin across a few runs, then widen the budget or reduce noise. Don't just bump the number.
Make the failure easy to read from the outside, for example by making mem-check's last message the job's final line.

On #25, `app` failed after 2m38s on 1f85405, an empty commit whose tree matched the green efdde90. A normal run takes about 3m. Two things showed it was a flake: - `cargo make app-docker` passed locally on the same tree: slow profile 314 ms, peak PSS 141062 kB. - A rerun via the empty commit 90b716f passed in 3m8s. This Forgejo's API serves neither job logs nor reruns (both return 404), so the failing step was not seen. The timing points at `tools/mem-check` near the end of the job. The suspects: - **PSS budget, most likely:** main's runs reach 141–154 MB against the 165 MB limit. - **Start-up budgets:** a timing spike on the slow profile, though the 2000 ms and 3000 ms budgets are about 10× the measured times. To do: - [ ] Find the failing step in the web UI (runs/82, job 2) and record it here. - [ ] If it's PSS, record the margin across a few runs, then widen the budget or reduce noise. Don't just bump the number. - [ ] Make the failure easy to read from the outside, for example by making mem-check's last message the job's final line.

claude-bot added this to the v0.1 — reader milestone

2026-10-22 09:15:26 +00:00

claude-bot referenced this issue

2026-10-22 18:46:46 +00:00

E2E: isolated test cases in their own CI job; mem-check measures only (#38) #40

claude-bot commented

2026-10-22 20:06:13 +00:00

Author

Collaborator

Closed by the owner. It hasn't recurred since the runner moved off the CPU-steal-heavy host, and the app and e2e jobs now keep their logs as artifacts on failure. A new CI failure gets its own issue.

claude-bot closed this issue

2026-10-22 20:06:13 +00:00