CI: cut the e2e job's 17 minutes on one runner (after #148) #158

New issue

Open

opened 2026-11-03 14:31:03 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-11-03 14:31:03 +00:00

Collaborator

The e2e job takes about 17 minutes, which is too long while we stay on one runner. Do this after the redesign (#148).

Numbers (run 320, `033b7b5`)

Main run: 42 cases one after another, about 15.5 min (sum 931 s, median 16 s).
Slow-profile run: 4 cases, about 1 min.
Five slowest cases: 305 s, a third of the main run (light-ground 93 s, red-letters 72 s, module-manager 59 s, running-text 43 s, pinned-heading 38 s).
Whole pipeline if everything runs: about 38 min (rust 13.5, app 6, e2e 17, then click).

The time looks like waiting rather than CPU work: scroll-chapters took 14 s on a 16-core machine and 16 s on the 3-vCPU runner (one case only), and e2e_tap always spends 1 s and e2e_settle several seconds of quiet. This is not yet measured, hence step 1.

Plan, in order

Measure. Per-step timing in tools/e2e/run.sh and lib.sh, to show how much of the job is fixed sleep. The later steps' value depends on this.
The slowest cases. Find out why they take so long; red-letters is 39 lines but takes 72 s. Estimate: 2-3 min off.
Retry a failed case once, and report it as flaky. A flake currently costs a full pipeline re-run (#157 on PR #154). The flake count has to be visible in the job's output, so that real intermittent bugs aren't buried.
Signals in place of fixed sleeps. In --e2e mode the app logs when it is idle; e2e_tap and e2e_settle wait for that line. Estimate: 30-50% off, and it removes a source of flakes (#155, #157), but it touches many cases.
Only if still too slow: run 2-3 cases at once in run.sh. Each case already has its own Xvfb and log; shared resources such as the repo-server.py port still need checking. ci.yml keeps e2e and click apart because these cases get flaky under load, so timing-sensitive cases like scroll-chapters would run alone at the end.

Target: e2e at about 6-8 min (an estimate) without sharing the CPU between cases.

Kept for later, or not doing

Affected cases only on PRs, full suite on main: a big cut for small PRs, to add if PRs still feel slow. The path-to-case mapping goes stale, and a miss only shows up after merging.
One app reset between cases: small gain, as startup is probably a small part of a 16 s case (not measured). Leftover state could hide real bugs, and the kill-and-restore cases need a fresh process anyway.
Several CI jobs (shards): no help on one runner unless it runs jobs at the same time, and then they compete for the same 3 vCPUs.

🤖 Generated with Claude Code

The `e2e` job takes about 17 minutes, which is too long while we stay on one runner. Do this after the redesign (#148). ## Numbers (run 320, 033b7b5) - Main run: 42 cases one after another, about 15.5 min (sum 931 s, median 16 s). - Slow-profile run: 4 cases, about 1 min. - Five slowest cases: 305 s, a third of the main run (`light-ground` 93 s, `red-letters` 72 s, `module-manager` 59 s, `running-text` 43 s, `pinned-heading` 38 s). - Whole pipeline if everything runs: about 38 min (`rust` 13.5, `app` 6, `e2e` 17, then `click`). The time looks like waiting rather than CPU work: `scroll-chapters` took 14 s on a 16-core machine and 16 s on the 3-vCPU runner (one case only), and `e2e_tap` always spends 1 s and `e2e_settle` several seconds of quiet. This is not yet measured, hence step 1. ## Plan, in order 1. **Measure.** Per-step timing in `tools/e2e/run.sh` and `lib.sh`, to show how much of the job is fixed sleep. The later steps' value depends on this. 2. **The slowest cases.** Find out why they take so long; `red-letters` is 39 lines but takes 72 s. Estimate: 2-3 min off. 3. **Retry a failed case once, and report it as flaky.** A flake currently costs a full pipeline re-run (#157 on PR #154). The flake count has to be visible in the job's output, so that real intermittent bugs aren't buried. 4. **Signals in place of fixed sleeps.** In `--e2e` mode the app logs when it is idle; `e2e_tap` and `e2e_settle` wait for that line. Estimate: 30-50% off, and it removes a source of flakes (#155, #157), but it touches many cases. 5. **Only if still too slow: run 2-3 cases at once in `run.sh`.** Each case already has its own Xvfb and log; shared resources such as the `repo-server.py` port still need checking. `ci.yml` keeps `e2e` and `click` apart because these cases get flaky under load, so timing-sensitive cases like `scroll-chapters` would run alone at the end. Target: `e2e` at about 6-8 min (an estimate) without sharing the CPU between cases. ## Kept for later, or not doing - **Affected cases only on PRs, full suite on `main`:** a big cut for small PRs, to add if PRs still feel slow. The path-to-case mapping goes stale, and a miss only shows up after merging. - **One app reset between cases:** small gain, as startup is probably a small part of a 16 s case (not measured). Leftover state could hide real bugs, and the kill-and-restore cases need a fresh process anyway. - **Several CI jobs (shards):** no help on one runner unless it runs jobs at the same time, and then they compete for the same 3 vCPUs. 🤖 Generated with [Claude Code](https://claude.com/claude-code)