pkgwatch/src/config.rs

216 lines
7.2 KiB
Rust
Raw Normal View History

Add ARCHITECTURE.md and apply it to this PR's code Researched current industry practice on code organization/maintainability (Ousterhout's deep modules and information hiding, package-by-feature vs. package-by-layer, functional-core/imperative-shell testability, tech-debt prevention via ADR-equivalent inline rationale) and wrote it into ARCHITECTURE.md as a set of concrete, project-specific rules rather than a generic essay — each principle cites a real example already in this codebase or fixed by this commit. Cross-linked from SPEC.md, which stays about product design, not code organization. Applied it to this PR's own code: - Pulled process_package/fetch_and_verify/build_and_publish/run_review/ approve out of main.rs into a new pipeline.rs. main.rs's own main() had grown to 278 lines and zero tests by treating "it's just the entry point" as an excuse to skip separating logic from wiring; now main.rs is argv dispatch only. - Extracted decide_tier_action as a pure function (verification outcome + pending-state -> what to do), replacing dispatch logic that was previously inlined into a function that also made the real network/ build calls. Four unit tests, no I/O, covering all four outcomes. - Added a `//!` module doc comment to every file touched in this branch, each stating that module's one job in a sentence, per the "deep modules" principle the spec argues for. Coverage's reported total drops (94% -> 78%) because pipeline.rs is deliberately NOT excluded from it the way main.rs is, even though it's mostly the same kind of untestable I/O orchestration — excluding it would hide decide_tier_action's real unit-test coverage along with the untested parts. Noted inline in Makefile.toml/ci.yml so the number doesn't look like a quality regression at a glance. Also added a project reference memory pointing at ARCHITECTURE.md rather than duplicating its content there, per this session's own memory-hygiene rules (architecture/conventions are derivable from the repo and shouldn't be duplicated somewhere that can go stale). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-17 09:42:18 +00:00
//! Parses `packages.d/*.toml` into typed, in-memory `Package` records.
//! The only module that knows the TOML shape — everything downstream
//! works with `Package`/`Verification`/`SanityCheck`, never raw TOML.
use anyhow::{Context, Result};
use serde::Deserialize;
use std::collections::HashMap;
use std::path::Path;
#[derive(Debug, Deserialize)]
struct PackageFile {
package: HashMap<String, Package>,
}
#[derive(Debug, Deserialize, Clone)]
pub struct Package {
pub repo: String,
/// Exact GitHub release asset name (still not a glob — see
/// docs/SPEC.md > Architecture > Fetcher), optionally containing a
/// `{version}` placeholder for projects whose asset names embed the
/// version (e.g. `scaleway-cli_{version}_linux_amd64`). Substituted via
/// `checker::version_from_tag` before matching.
pub asset_pattern: String,
pub verification: Verification,
Close the loop: build, sanity-check, and publish for the first time Implements the last unimplemented pipeline stage from SPEC.md: PKGBUILD generation + makepkg (builder.rs), a post-build version sanity check (sanity.rs), and repo-add publishing (publisher.rs), wired into main.rs for both the tier 1-3 auto-publish path and a new tier 4-6 review queue (`pkgwatch review` / `pkgwatch review <name> --approve`, persisted via state::{load,save,clear}_pending_version, tracked separately from last-published-version since approving one release isn't a standing auto-publish grant for future ones). Publishing targets an existing, already-registered local pacman repo (~/.local/share/pacman/custom, `[custom]` in /etc/pacman.conf) rather than one pkgwatch invents — found already in real use for a hand-packaged AppImage, which resolves SPEC's open question on where the repo lives without pkgwatch ever touching pacman.conf. Publishing stops at `repo-add`; actually installing/upgrading (`pacman -Syu`/`pacman -S`) is left to the operator, not run automatically. Getting a real second package (scaleway-cli, tier 4) through the new pipeline immediately surfaced a real gap: its pacman package is named `scaleway-cli` but the actual binary is `scw` (confirmed via `pacman -Ql` against the currently-installed extra package) — without a way to declare that, the build would install alongside extra's package under the wrong name instead of shadowing it. Added `Package::binary_name` (config.rs) to cover it. Every upstream-controlled string (version, asset name, download URL) is validated before it touches generated shell content in the PKGBUILD template — rejects anything containing a single quote or newline, since values are embedded in single-quoted bash strings. Verified for real, end to end: uv (tier 2) auto-built and published against the real astral-sh/uv release with no human step; scaleway-cli (tier 4) queued for review, then approved via `pkgwatch review scaleway-cli --approve`, which re-verified, built, and published it — confirmed the built package contains exactly usr/bin/scw. Both landed in the real custom repo's database. Left scaleway-cli's real-repo review pending rather than approving it myself: the tier 4-6 gate exists for a human judgment call, not the agent's. 69 tests, cargo make ci clean (fmt, clippy, complexity, coverage, audit). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-17 09:13:13 +00:00
/// Name of the executable inside the built package, if it differs from
/// the package name itself — e.g. scaleway-cli's pacman package is
/// named `scaleway-cli` but its real binary is `scw` (discovered by
/// checking the currently-installed extra package, not guessable from
/// the repo name). Defaults to the package name when omitted.
pub binary_name: Option<String>,
/// Explicit path to the binary inside the extracted archive, relative
/// to `srcdir`, for archive layouts that don't match the "extracts into
/// a directory named after the archive stem" convention `builder.rs`
/// otherwise assumes (e.g. claude-code's tarball extracts a bare
/// `claude` file with no wrapping directory, and that inner filename
/// doesn't match the package name either). Only meaningful when
/// `asset_pattern` names an archive; ignored for bare-binary downloads,
/// where the downloaded file *is* the source path already. Defaults to
/// the stem/`binary_name` convention when omitted.
pub archive_binary_path: Option<String>,
Close the loop: build, sanity-check, and publish for the first time Implements the last unimplemented pipeline stage from SPEC.md: PKGBUILD generation + makepkg (builder.rs), a post-build version sanity check (sanity.rs), and repo-add publishing (publisher.rs), wired into main.rs for both the tier 1-3 auto-publish path and a new tier 4-6 review queue (`pkgwatch review` / `pkgwatch review <name> --approve`, persisted via state::{load,save,clear}_pending_version, tracked separately from last-published-version since approving one release isn't a standing auto-publish grant for future ones). Publishing targets an existing, already-registered local pacman repo (~/.local/share/pacman/custom, `[custom]` in /etc/pacman.conf) rather than one pkgwatch invents — found already in real use for a hand-packaged AppImage, which resolves SPEC's open question on where the repo lives without pkgwatch ever touching pacman.conf. Publishing stops at `repo-add`; actually installing/upgrading (`pacman -Syu`/`pacman -S`) is left to the operator, not run automatically. Getting a real second package (scaleway-cli, tier 4) through the new pipeline immediately surfaced a real gap: its pacman package is named `scaleway-cli` but the actual binary is `scw` (confirmed via `pacman -Ql` against the currently-installed extra package) — without a way to declare that, the build would install alongside extra's package under the wrong name instead of shadowing it. Added `Package::binary_name` (config.rs) to cover it. Every upstream-controlled string (version, asset name, download URL) is validated before it touches generated shell content in the PKGBUILD template — rejects anything containing a single quote or newline, since values are embedded in single-quoted bash strings. Verified for real, end to end: uv (tier 2) auto-built and published against the real astral-sh/uv release with no human step; scaleway-cli (tier 4) queued for review, then approved via `pkgwatch review scaleway-cli --approve`, which re-verified, built, and published it — confirmed the built package contains exactly usr/bin/scw. Both landed in the real custom repo's database. Left scaleway-cli's real-repo review pending rather than approving it myself: the tier 4-6 gate exists for a human judgment call, not the agent's. 69 tests, cargo make ci clean (fmt, clippy, complexity, coverage, audit). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-17 09:13:13 +00:00
/// Post-build correctness check (not a security control — see
/// docs/SPEC.md > Verification trust tiers). Runs `command` against the
Close the loop: build, sanity-check, and publish for the first time Implements the last unimplemented pipeline stage from SPEC.md: PKGBUILD generation + makepkg (builder.rs), a post-build version sanity check (sanity.rs), and repo-add publishing (publisher.rs), wired into main.rs for both the tier 1-3 auto-publish path and a new tier 4-6 review queue (`pkgwatch review` / `pkgwatch review <name> --approve`, persisted via state::{load,save,clear}_pending_version, tracked separately from last-published-version since approving one release isn't a standing auto-publish grant for future ones). Publishing targets an existing, already-registered local pacman repo (~/.local/share/pacman/custom, `[custom]` in /etc/pacman.conf) rather than one pkgwatch invents — found already in real use for a hand-packaged AppImage, which resolves SPEC's open question on where the repo lives without pkgwatch ever touching pacman.conf. Publishing stops at `repo-add`; actually installing/upgrading (`pacman -Syu`/`pacman -S`) is left to the operator, not run automatically. Getting a real second package (scaleway-cli, tier 4) through the new pipeline immediately surfaced a real gap: its pacman package is named `scaleway-cli` but the actual binary is `scw` (confirmed via `pacman -Ql` against the currently-installed extra package) — without a way to declare that, the build would install alongside extra's package under the wrong name instead of shadowing it. Added `Package::binary_name` (config.rs) to cover it. Every upstream-controlled string (version, asset name, download URL) is validated before it touches generated shell content in the PKGBUILD template — rejects anything containing a single quote or newline, since values are embedded in single-quoted bash strings. Verified for real, end to end: uv (tier 2) auto-built and published against the real astral-sh/uv release with no human step; scaleway-cli (tier 4) queued for review, then approved via `pkgwatch review scaleway-cli --approve`, which re-verified, built, and published it — confirmed the built package contains exactly usr/bin/scw. Both landed in the real custom repo's database. Left scaleway-cli's real-repo review pending rather than approving it myself: the tier 4-6 gate exists for a human judgment call, not the agent's. 69 tests, cargo make ci clean (fmt, clippy, complexity, coverage, audit). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-17 09:13:13 +00:00
/// freshly built binary and confirms `version_regex`'s capture group
/// matches the version pkgwatch believes it just built.
pub sanity_check: Option<SanityCheck>,
}
impl Package {
/// The name of the executable inside the built package: `binary_name`
/// if the package declares one, else `pkg_name` itself.
pub fn binary_name<'a>(&'a self, pkg_name: &'a str) -> &'a str {
self.binary_name.as_deref().unwrap_or(pkg_name)
}
}
#[derive(Debug, Deserialize, Clone)]
pub struct SanityCheck {
pub command: String,
pub version_regex: String,
}
#[derive(Debug, Deserialize, Clone)]
#[serde(tag = "method", rename_all = "kebab-case")]
pub enum Verification {
/// Tier 4: proves transport integrity only, not authorship. See
/// docs/SPEC.md > Verification trust tiers. `checksum_asset_pattern` may
/// also contain a `{version}` placeholder, same as `asset_pattern`.
SameOriginSha256 { checksum_asset_pattern: String },
/// Tier 2: GitHub build-provenance attestation, verified via `gh
/// attestation verify` rather than reimplementing sigstore in Rust.
GithubAttestation,
}
impl Verification {
pub fn tier(&self) -> u8 {
match self {
Verification::SameOriginSha256 { .. } => 4,
Verification::GithubAttestation => 2,
}
}
}
/// Loads every `*.toml` file in `dir` (the `packages.d/` layout from
/// docs/SPEC.md > Scaling to many packages), keyed by package name.
pub fn load_packages_dir(dir: &Path) -> Result<Vec<(String, Package)>> {
let mut out = Vec::new();
for entry in std::fs::read_dir(dir).with_context(|| format!("reading {}", dir.display()))? {
let path = entry?.path();
if path.extension().and_then(|e| e.to_str()) != Some("toml") {
continue;
}
let text = std::fs::read_to_string(&path)
.with_context(|| format!("reading {}", path.display()))?;
Set up project tooling to match the rest of ~/dev's Rust fleet Surveyed sporah/doubleo7/feedsignal/uy-immigration-watcher/notif-picker for conventions and replicated the current dominant pattern rather than inventing a new one: - Forgejo CI (.forgejo/workflows/ci.yml): build/test/audit jobs on the rust-ci runner label, cargo+sccache caching, cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo audit. Matches sporah/doubleo7/feedsignal/uy-immigration-watcher; notif-picker's docker-label/manual-toolchain-install variant looks like an earlier iteration superseded by this one. - Makefile.toml with format/format-check/lint/test/audit/build tasks and a `ci` task chaining them — copied from notif-picker's clean version, the only project that had this pattern. `cargo make ci` now runs the same checks locally that CI runs. - Explicit empty [workspace] in Cargo.toml (doubleo7's pattern) so a nested git-worktree checkout can't accidentally inherit an ancestor directory's workspace manifest. - rustfmt: no rustfmt.toml, matching every sibling project — default style is the established convention here, not an oversight. New for this fleet, since nothing else in ~/dev has it: a git-native pre-commit hook (.githooks/pre-commit, activated via `cargo make install-hooks` / `git config core.hooksPath .githooks`) that runs `cargo fmt` and re-stages whatever it reformats. Chose git's native hooksPath over the pre-commit(.com) framework or cargo-husky — no extra runtime dependency, hook is tracked and shareable, and nothing else here needs Python. Kept to formatting only; clippy/audit stay in CI, which already covers them and can run heavier checks than a commit hook should. Fixed one clippy finding (useless format! in checker.rs) and reformatted the existing code to match the now-enforced default rustfmt style. `cargo make ci` passes clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A2FEut5tVMNjeVjqhgVZbr
2026-09-11 08:13:29 +00:00
let file: PackageFile =
toml::from_str(&text).with_context(|| format!("parsing {}", path.display()))?;
out.extend(file.package);
}
Ok(out)
}
Add shift-left quality gates: cognitive complexity, coverage, dependency fix Adds two new gates to the existing format/lint/test/audit pipeline (Makefile.toml `cargo make ci`, .forgejo/workflows/ci.yml): - Cognitive complexity via clippy's nursery cognitive_complexity lint (clippy.toml, threshold 15), scoped to --bins so test code's naturally higher branch count doesn't get gated. Went with this over the closest real cyclomatic-complexity tool (rust-code-analysis-cli) because that crate hasn't shipped a release since Jan 2023. - Test coverage via cargo-llvm-cov, chosen over cargo-tarpaulin for friendlier behavior in containerized/dind CI (no ptrace). Report-only for now (no --fail-under-lines) since a real threshold needs real usage data first — see below. main.rs is excluded: it's orchestration glue exercised by the real end-to-end `cargo run`, not unit tests. Getting both gates running required writing pkgwatch's first tests (previously zero). To make the GitHub-facing modules unit-testable without hitting real github.com/api.github.com, added `GithubEndpoints` (src/github.rs) so checker/fetcher/verifier take injectable base URLs, and added mockito + tempfile as dev-dependencies. Result: 27 tests, 94% region / 96% line coverage excluding main.rs. Also: cargo audit (now wired into `cargo make ci`) immediately caught a real, currently-open advisory (RUSTSEC-2026-0285, published days ago) in the transitive rustls dependency — bumped 0.23.44 -> 0.23.45 to clear it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-17 07:16:14 +00:00
#[cfg(test)]
mod tests {
use super::*;
fn write(dir: &Path, name: &str, contents: &str) {
std::fs::write(dir.join(name), contents).unwrap();
}
#[test]
fn tier_matches_verification_method() {
assert_eq!(
Verification::SameOriginSha256 {
checksum_asset_pattern: "x".to_string()
}
.tier(),
4
);
assert_eq!(Verification::GithubAttestation.tier(), 2);
}
#[test]
fn loads_same_origin_sha256_package() {
let dir = tempfile::tempdir().unwrap();
write(
dir.path(),
"pkg.toml",
r#"
[package.pkg]
repo = "o/r"
asset_pattern = "pkg_{version}_linux_amd64"
[package.pkg.verification]
method = "same-origin-sha256"
checksum_asset_pattern = "SHA256SUMS"
"#,
);
let packages = load_packages_dir(dir.path()).unwrap();
assert_eq!(packages.len(), 1);
let (name, pkg) = &packages[0];
assert_eq!(name, "pkg");
assert_eq!(pkg.repo, "o/r");
assert_eq!(pkg.asset_pattern, "pkg_{version}_linux_amd64");
assert_eq!(pkg.verification.tier(), 4);
}
#[test]
fn loads_github_attestation_package() {
let dir = tempfile::tempdir().unwrap();
write(
dir.path(),
"pkg.toml",
r#"
[package.pkg]
repo = "o/r"
asset_pattern = "pkg-x86_64.tar.gz"
[package.pkg.verification]
method = "github-attestation"
"#,
);
let packages = load_packages_dir(dir.path()).unwrap();
assert_eq!(packages[0].1.verification.tier(), 2);
}
#[test]
fn loads_multiple_files_and_ignores_non_toml() {
let dir = tempfile::tempdir().unwrap();
write(
dir.path(),
"a.toml",
r#"
[package.a]
repo = "o/a"
asset_pattern = "a.tar.gz"
[package.a.verification]
method = "github-attestation"
"#,
);
write(
dir.path(),
"b.toml",
r#"
[package.b]
repo = "o/b"
asset_pattern = "b.tar.gz"
[package.b.verification]
method = "github-attestation"
"#,
);
write(dir.path(), "README.md", "not a package file");
let mut names: Vec<String> = load_packages_dir(dir.path())
.unwrap()
.into_iter()
.map(|(name, _)| name)
.collect();
names.sort();
assert_eq!(names, vec!["a", "b"]);
}
#[test]
fn errors_on_invalid_toml() {
let dir = tempfile::tempdir().unwrap();
write(dir.path(), "bad.toml", "not valid toml [[[");
assert!(load_packages_dir(dir.path()).is_err());
}
#[test]
fn empty_dir_yields_empty_list() {
let dir = tempfile::tempdir().unwrap();
assert!(load_packages_dir(dir.path()).unwrap().is_empty());
}
}