Compare commits

..

No commits in common. "master" and "worktree-add-systemd-timer" have entirely different histories.

17 changed files with 310 additions and 932 deletions

View file

@ -35,10 +35,8 @@ practice rather than asserted from habit — see Further reading.
`helpers/`) tends toward the opposite, and a single feature change ends `helpers/`) tends toward the opposite, and a single feature change ends
up touching files scattered across every layer. up touching files scattered across every layer.
**Rule**: modules are named after what they do in the pipeline **Rule**: modules are named after what they do in the pipeline
(`release_source`, `fetcher`, `verifier`, `builder`, `sanity`, (`checker`, `fetcher`, `verifier`, `builder`, `sanity`, `publisher`,
`publisher`, `state`), not generic buckets. (`release_source` is a directory module: the `state`), not generic buckets. A new pipeline stage gets a new module
`ReleaseSource` trait and one file per host, re-exported from its
`mod.rs` so the rest of the crate never names a host's file.) A new pipeline stage gets a new module
named after the stage, not a method bolted onto an existing one. named after the stage, not a method bolted onto an existing one.
**Anti-example to keep watching for**: a `utils.rs` grab-bag. `hash.rs` **Anti-example to keep watching for**: a `utils.rs` grab-bag. `hash.rs`
could look like one but isn't — it exists for exactly one piece of could look like one but isn't — it exists for exactly one piece of
@ -63,9 +61,8 @@ practice rather than asserted from habit — see Further reading.
Same testability goal as #3, applied to the specific ways this program Same testability goal as #3, applied to the specific ways this program
reaches outside itself. A consistent shape beats ad hoc mocking invented reaches outside itself. A consistent shape beats ad hoc mocking invented
per call site. per call site.
**Already in force**: `GithubEndpoints`/`ForgejoEndpoints` (the **Already in force**: `GithubEndpoints` (checker/fetcher/verifier),
`ReleaseSource` implementations, whose API root is what fetcher/verifier `repo_add_bin` and the pacman.conf path (publisher), `PKGWATCH_REPO_DIR`
take), `repo_add_bin` and the pacman.conf path (publisher), `PKGWATCH_REPO_DIR`
(main, for manual dry runs against a scratch repo instead of the real (main, for manual dry runs against a scratch repo instead of the real
one). A new external call follows the same shape: production code calls one). A new external call follows the same shape: production code calls
a thin wrapper with the real default; tests call the parameterized a thin wrapper with the real default; tests call the parameterized
@ -109,8 +106,8 @@ practice rather than asserted from habit — see Further reading.
directory, but the same information — why this way and not the obvious directory, but the same information — why this way and not the obvious
alternative — needs to live somewhere a future reader will actually see alternative — needs to live somewhere a future reader will actually see
it: the doc comment on the thing itself. it: the doc comment on the thing itself.
**Example already here**: `release_source/github.rs`'s doc comment on **Example already here**: `checker.rs`'s doc comment on
`GithubEndpoints::latest_release` explains why the newest Atom-feed entry isn't `latest_github_release` explains why the newest Atom-feed entry isn't
trusted outright (scaleway-cli's `-dbg1` tag has no real Release behind trusted outright (scaleway-cli's `-dbg1` tag has no real Release behind
it) — the reasoning lives right next to the code it justifies, not in a it) — the reasoning lives right next to the code it justifies, not in a
commit message or a separate design doc no one will find later. commit message or a separate design doc no one will find later.

View file

@ -287,33 +287,17 @@ implements `repo`, `asset_pattern`, and `verification.method`
matches the line by filename instead of assuming a single-hash file. matches the line by filename instead of assuming a single-hash file.
- Separately, scaleway-cli's Atom feed lists a `vX.Y.Z-dbg1` tag newest, - Separately, scaleway-cli's Atom feed lists a `vX.Y.Z-dbg1` tag newest,
with no real Release object behind it (`releases/tags/<tag>` 404s) — with no real Release object behind it (`releases/tags/<tag>` 404s) —
the GitHub source's `latest_release` now confirms each feed candidate `checker::latest_github_release` now confirms each feed candidate
against the releases API in feed order rather than trusting the first against the releases API in feed order rather than trusting the first
entry outright. entry outright.
`sanity_check` and `binary_name` are now real, implemented fields (see `sanity_check` and `binary_name` are now real, implemented fields (see
Builder/Sanity checker above) — added `packages.d/uv.toml`'s and Builder/Sanity checker above) — added `packages.d/uv.toml`'s and
`packages.d/scaleway-cli.toml`'s own `sanity_check` blocks, and `packages.d/scaleway-cli.toml`'s own `sanity_check` blocks, and
scaleway-cli's `binary_name = "scw"`. `source` is implemented too, with scaleway-cli's `binary_name = "scw"`. `source`, `check_method`, and
two values: `github-release` (the default when omitted, so existing `check_interval` are still schema sketch, not yet read by the code — the
configs are unchanged) and `forgejo-release`, which also requires a PoC only knows how to check GitHub-release sources, on a single one-shot
`base_url` (see Checker below). `check_method` and `check_interval` are run rather than a scheduled loop.
still schema sketch, not yet read by the code — checks are a fixed hourly
tick, not per-package.
```toml
# A package released from a Forgejo instance instead of GitHub. Only
# `same-origin-sha256` is valid here: `github-attestation` needs GitHub.
[package.mytool]
source = "forgejo-release"
base_url = "https://code.austinschaefer.com"
repo = "schaefera/mytool"
asset_pattern = "mytool-linux-x86_64.tar.gz"
[package.mytool.verification]
method = "same-origin-sha256"
checksum_asset_pattern = "SHA256SUMS"
```
Build/publish/review-queue (`makepkg`, `repo-add`, tier 46 human review) Build/publish/review-queue (`makepkg`, `repo-add`, tier 46 human review)
are now implemented too — see Builder/Sanity checker/Publisher/Reviewer are now implemented too — see Builder/Sanity checker/Publisher/Reviewer
@ -346,62 +330,19 @@ Open questions on the schema:
- **Config loader**: parses `packages.d/*.toml` into an in-memory package - **Config loader**: parses `packages.d/*.toml` into an in-memory package
list. *(Implemented — `src/config.rs`.)* list. *(Implemented — `src/config.rs`.)*
- **Paths**: where config, state and work files live, resolved by
`src/paths.rs` per the XDG base-directory spec rather than the current
working directory, so an installed binary behaves the same wherever it's
launched from. *(Implemented.)*
| What | Default | XDG variable | Override |
|---|---|---|---|
| Package declarations (`packages.d/*.toml`) | `~/.config/pkgwatch/packages.d` | `XDG_CONFIG_HOME` | `PKGWATCH_CONFIG_DIR` (the dir *containing* `packages.d`) |
| Last-published / pending versions | `~/.local/state/pkgwatch` | `XDG_STATE_HOME` | `PKGWATCH_STATE_DIR` |
| Downloads and build trees (safe to delete) | `~/.cache/pkgwatch` | `XDG_CACHE_HOME` | `PKGWATCH_WORK_DIR` |
Precedence per directory: override, then the XDG variable, then the
default under `$HOME`; an empty variable counts as unset. The overrides
are used verbatim (no `pkgwatch/` suffix) and exist for dry runs against
scratch directories, like `PKGWATCH_REPO_DIR` does for the pacman repo.
The XDG variables and `$HOME` must be absolute paths: a relative XDG
value is ignored, as the XDG spec requires, and a relative `$HOME` is an
error.
The checkout's `packages.d/` is no longer read on its own; it's just the
source to link from. Migrating from the old cwd-relative layout: move
`state/` to the state dir and copy or symlink `packages.d/` into the
config dir; `work/` is cache and can simply be dropped.
- **Checker**: per source type, resolves "what's the latest version" — - **Checker**: per source type, resolves "what's the latest version" —
likely reuses `nvchecker`'s logic/sources conceptually for non-GitHub likely reuses `nvchecker`'s logic/sources conceptually for non-GitHub
sources eventually. For GitHub sources, prefers the `github-atom` feed sources eventually. For GitHub sources, prefers the `github-atom` feed
(see Scaling > Check method) over unconditional REST polling. (see Scaling > Check method) over unconditional REST polling.
*(Implemented for GitHub and Forgejo. `src/release_source/` defines the *(Implemented for GitHub only — `src/checker.rs` regex-matches the first
`ReleaseSource` trait (latest release + API root); each host implements `releases/tag/<tag>` link in the feed rather than doing a full XML parse;
it in its own file (`github.rs`, `forgejo.rs`), and the module's fine while the feed's newest-entry-first shape holds, revisit if that
`for_package` picks one per package, so adding a host doesn't touch ever changes. `check_interval`/per-package cadence not wired up yet —
existing ones. The rest of the crate imports the trait and hosts from the PoC is a single one-shot run, not a scheduled loop.)*
`crate::release_source`, which re-exports them. A trait
rather than an enum match because there are now two real hosts with
genuinely different logic. GitHub regex-matches the first
`releases/tag/<tag>` link in the feed rather than doing a full XML
parse; fine while the feed's newest-entry-first shape holds, revisit if
that ever changes. Forgejo is one call to
`<base_url>/api/v1/repos/<repo>/releases/latest`, which already returns
only the newest non-draft, non-prerelease release, so it needs none of
GitHub's confirm-each-tag step. `check_interval`/per-package cadence not
wired up yet — checks are a fixed hourly tick.)*
The HTTP is hand-rolled on the `reqwest` already in the tree, not an
API-client crate: pkgwatch needs two `GET`s, GitHub's check deliberately
uses an Atom feed no API crate covers (to stay off the rate-limited REST
API), and the release-by-tag call is shared verbatim by both hosts.
`octocrab` is async against our blocking `reqwest` with a default tree
larger than pkgwatch's whole current one; `forgejo-api` is a generated
binding of the entire API for one endpoint. Revisit if pkgwatch needs
authenticated or write API calls.
- **Fetcher**: downloads the artifact (and any checksum/signature/ - **Fetcher**: downloads the artifact (and any checksum/signature/
attestation companion) for a resolved version. *(Implemented — attestation companion) for a resolved version. *(Implemented —
`src/fetcher.rs`, via the GitHub or Forgejo releases API — same `src/fetcher.rs`, via the GitHub releases API; exact asset-name match,
`releases/tags/<tag>` endpoint and JSON shape on both; exact asset-name not a glob.)*
match, not a glob.)*
- **Verifier**: tier-specific verification implementations, dispatched via - **Verifier**: tier-specific verification implementations, dispatched via
a `Verification` enum matched on `method` (an internally-tagged serde a `Verification` enum matched on `method` (an internally-tagged serde
enum) rather than a trait — simpler while there are only two methods; enum) rather than a trait — simpler while there are only two methods;
@ -478,10 +419,8 @@ Open questions on the schema:
systemctl --user enable --now pkgwatch.timer systemctl --user enable --now pkgwatch.timer
``` ```
*`pkgwatch.service` runs the release binary from the checkout and sets *`pkgwatch.service` sets `WorkingDirectory` to `~/dev/pkgwatch` because
no `WorkingDirectory`: config, state and work dirs come from the XDG `packages.d/`, `state/` and `work/` resolve relative to cwd.)*
paths above, so the service needs `~/.config/pkgwatch/packages.d` set
up first — see the migration note under Paths.)*
- **Notifications**: `notifier.rs` sends a desktop notification - **Notifications**: `notifier.rs` sends a desktop notification
(`notify-send`) when a tier 4-6 release is newly queued for review or a (`notify-send`) when a tier 4-6 release is newly queued for review or a
tier 1-3 release is published; both are best-effort and never fail a tier 1-3 release is published; both are best-effort and never fail a
@ -526,8 +465,8 @@ Open questions on the schema:
confirmed correct — no build-provenance attestations upstream. confirmed correct — no build-provenance attestations upstream.
Required adding `{version}`-placeholder support to `asset_pattern`/ Required adding `{version}`-placeholder support to `asset_pattern`/
`checksum_asset_pattern`, filename-matched parsing of combined `checksum_asset_pattern`, filename-matched parsing of combined
multi-asset checksum files, and having the GitHub source's multi-asset checksum files, and having `latest_github_release`
`latest_release` confirm each Atom-feed candidate against the releases API (this confirm each Atom-feed candidate against the releases API (this
repo's newest feed entry, a `-dbg1` tag, has no real Release behind repo's newest feed entry, a `-dbg1` tag, has no real Release behind
it). Still just flags for human review, same as any tier 4-6 pass — it). Still just flags for human review, same as any tier 4-6 pass —
not auto-installed; see the unchecked build/publish item below. not auto-installed; see the unchecked build/publish item below.
@ -553,7 +492,7 @@ Open questions on the schema:
- [ ] Not yet implemented: `pkgwatch review <name> --reject` (a pending - [ ] Not yet implemented: `pkgwatch review <name> --reject` (a pending
review can only be approved or left pending, not dismissed), review can only be approved or left pending, not dismissed),
per-package `check_interval` (the timer is a fixed hourly tick), per-package `check_interval` (the timer is a fixed hourly tick),
sources other than GitHub and Forgejo releases, `minisign`/tier-1 non-GitHub sources, `minisign`/tier-1
method, retention/pruning of old versions in the local repo (see method, retention/pruning of old versions in the local repo (see
Scaling > Local repo retention), staggering/auth for GitHub API Scaling > Local repo retention), staggering/auth for GitHub API
rate limits at higher package counts. rate limits at higher package counts.

View file

@ -1,6 +1,45 @@
//! Turns a release tag into a version string. What the latest tag *is* use crate::github::GithubEndpoints;
//! comes from a `ReleaseSource` (see `release_source`); this is the one piece of use anyhow::{Result, bail};
//! the check stage that isn't host-specific. use regex::Regex;
/// Resolves the latest release tag for `repo` via its public Atom feed.
///
/// Deliberately not a full XML parse: the feed lists entries newest-first,
/// and each `<link rel="alternate" .../releases/tag/<tag>"/>` is matched
/// in document order. Revisit with a real XML parser if GitHub's feed
/// shape ever changes.
///
/// The feed can list a tag newer than any tag with a real Release object
/// behind it — observed on scaleway/scaleway-cli, which pushes a
/// `vX.Y.Z-dbg1` tag (no corresponding Release; `releases/tags/<tag>`
/// 404s) right after each real release, and that tag sorts newest in the
/// feed. So each candidate is confirmed against the releases API in feed
/// order, returning the first that actually resolves.
pub fn latest_github_release(
client: &reqwest::blocking::Client,
endpoints: &GithubEndpoints,
repo: &str,
) -> Result<String> {
let url = format!("{}/{repo}/releases.atom", endpoints.web);
let body = client.get(&url).send()?.error_for_status()?.text()?;
let re = Regex::new(r#"releases/tag/([^"]+)""#)?;
let mut candidates = re
.captures_iter(&body)
.map(|caps| caps[1].to_string())
.peekable();
if candidates.peek().is_none() {
bail!("no release tag found in {url}");
}
for tag in candidates {
let release_url = format!("{}/repos/{repo}/releases/tags/{tag}", endpoints.api);
if client.get(&release_url).send()?.status().is_success() {
return Ok(tag);
}
}
bail!("no release tag in {url} resolved to a real release via the API")
}
/// Strips a leading `v` from a release tag, e.g. `v2.62.0` -> `2.62.0`. /// Strips a leading `v` from a release tag, e.g. `v2.62.0` -> `2.62.0`.
/// ///
@ -26,4 +65,84 @@ mod tests {
fn version_from_tag_leaves_bare_version_unchanged() { fn version_from_tag_leaves_bare_version_unchanged() {
assert_eq!(version_from_tag("0.12.15"), "0.12.15"); assert_eq!(version_from_tag("0.12.15"), "0.12.15");
} }
fn atom_feed(tags: &[&str]) -> String {
let entries: String = tags
.iter()
.map(|t| {
format!(r#"<link rel="alternate" href="https://github.com/o/r/releases/tag/{t}"/>"#)
})
.collect();
format!("<feed>{entries}</feed>")
}
#[test]
fn latest_github_release_skips_tags_with_no_real_release() {
let mut server = mockito::Server::new();
let endpoints = GithubEndpoints {
web: server.url(),
api: server.url(),
};
// Mirrors the real scaleway-cli case: newest feed entry (a -dbg1
// tag) has no Release object behind it and 404s.
let _feed = server
.mock("GET", "/o/r/releases.atom")
.with_status(200)
.with_body(atom_feed(&["v2.62.0-dbg1", "v2.62.0"]))
.create();
let _missing = server
.mock("GET", "/repos/o/r/releases/tags/v2.62.0-dbg1")
.with_status(404)
.create();
let _real = server
.mock("GET", "/repos/o/r/releases/tags/v2.62.0")
.with_status(200)
.with_body("{}")
.create();
let client = reqwest::blocking::Client::new();
let tag = latest_github_release(&client, &endpoints, "o/r").unwrap();
assert_eq!(tag, "v2.62.0");
}
#[test]
fn latest_github_release_errors_when_feed_has_no_tags() {
let mut server = mockito::Server::new();
let endpoints = GithubEndpoints {
web: server.url(),
api: server.url(),
};
let _feed = server
.mock("GET", "/o/r/releases.atom")
.with_status(200)
.with_body("<feed></feed>")
.create();
let client = reqwest::blocking::Client::new();
let err = latest_github_release(&client, &endpoints, "o/r").unwrap_err();
assert!(err.to_string().contains("no release tag found"));
}
#[test]
fn latest_github_release_errors_when_no_candidate_resolves() {
let mut server = mockito::Server::new();
let endpoints = GithubEndpoints {
web: server.url(),
api: server.url(),
};
let _feed = server
.mock("GET", "/o/r/releases.atom")
.with_status(200)
.with_body(atom_feed(&["v1.0.0-dbg1"]))
.create();
let _missing = server
.mock("GET", "/repos/o/r/releases/tags/v1.0.0-dbg1")
.with_status(404)
.create();
let client = reqwest::blocking::Client::new();
let err = latest_github_release(&client, &endpoints, "o/r").unwrap_err();
assert!(err.to_string().contains("resolved to a real release"));
}
} }

View file

@ -2,7 +2,7 @@
//! The only module that knows the TOML shape — everything downstream //! The only module that knows the TOML shape — everything downstream
//! works with `Package`/`Verification`/`SanityCheck`, never raw TOML. //! works with `Package`/`Verification`/`SanityCheck`, never raw TOML.
use anyhow::{Context, Result, bail}; use anyhow::{Context, Result};
use serde::Deserialize; use serde::Deserialize;
use std::collections::{BTreeMap, HashMap}; use std::collections::{BTreeMap, HashMap};
use std::path::Path; use std::path::Path;
@ -12,29 +12,10 @@ struct PackageFile {
package: HashMap<String, Package>, package: HashMap<String, Package>,
} }
/// Where a package's releases are published: the `source` key from
/// docs/SPEC.md > Config schema. Omitted means GitHub, so every existing
/// `packages.d/*.toml` keeps working.
#[derive(Debug, Deserialize, Clone, Copy, Default, PartialEq, Eq)]
#[serde(rename_all = "kebab-case")]
pub enum Source {
#[default]
GithubRelease,
/// A Forgejo (or Gitea) instance; needs `base_url` too.
ForgejoRelease,
}
#[derive(Debug, Deserialize, Clone)] #[derive(Debug, Deserialize, Clone)]
pub struct Package { pub struct Package {
/// `owner/name` on whichever `source` hosts it.
pub repo: String, pub repo: String,
#[serde(default)] /// Exact GitHub release asset name (still not a glob — see
pub source: Source,
/// Web root of the Forgejo instance, e.g. `https://code.austinschaefer.com`
/// (the API lives under `/api/v1`). Required for, and only meaningful
/// with, `source = "forgejo-release"`.
pub base_url: Option<String>,
/// Exact release asset name (still not a glob — see
/// docs/SPEC.md > Architecture > Fetcher), optionally containing a /// docs/SPEC.md > Architecture > Fetcher), optionally containing a
/// `{version}` placeholder for projects whose asset names embed the /// `{version}` placeholder for projects whose asset names embed the
/// version (e.g. `scaleway-cli_{version}_linux_amd64`). Substituted via /// version (e.g. `scaleway-cli_{version}_linux_amd64`). Substituted via
@ -79,32 +60,6 @@ pub struct Package {
} }
impl Package { impl Package {
/// Rejects combinations that can't work, once at load time rather than
/// as a confusing failure deep in a run (see docs/ARCHITECTURE.md >
/// "validate at the boundary, once").
fn validate(&self, name: &str) -> Result<()> {
match (self.source, &self.base_url) {
(Source::GithubRelease, None) => {}
(Source::GithubRelease, Some(_)) => {
// Silently ignoring it would hide a mistyped `source`.
bail!("{name}: base_url only applies to source = \"forgejo-release\"");
}
(Source::ForgejoRelease, None) => {
bail!("{name}: source = \"forgejo-release\" needs a base_url");
}
(Source::ForgejoRelease, Some(url)) => {
if !url.starts_with("https://") && !url.starts_with("http://") {
bail!("{name}: base_url '{url}' must start with http:// or https://");
}
// `gh attestation verify` only speaks GitHub's attestation API.
if matches!(self.verification, Verification::GithubAttestation) {
bail!("{name}: github-attestation verification needs a GitHub source");
}
}
}
Ok(())
}
/// The name of the executable inside the built package: `binary_name` /// The name of the executable inside the built package: `binary_name`
/// if the package declares one, else `pkg_name` itself. /// if the package declares one, else `pkg_name` itself.
pub fn binary_name<'a>(&'a self, pkg_name: &'a str) -> &'a str { pub fn binary_name<'a>(&'a self, pkg_name: &'a str) -> &'a str {
@ -152,10 +107,6 @@ pub fn load_packages_dir(dir: &Path) -> Result<Vec<(String, Package)>> {
.with_context(|| format!("reading {}", path.display()))?; .with_context(|| format!("reading {}", path.display()))?;
let file: PackageFile = let file: PackageFile =
toml::from_str(&text).with_context(|| format!("parsing {}", path.display()))?; toml::from_str(&text).with_context(|| format!("parsing {}", path.display()))?;
for (name, pkg) in &file.package {
pkg.validate(name)
.with_context(|| format!("in {}", path.display()))?;
}
out.extend(file.package); out.extend(file.package);
} }
Ok(out) Ok(out)
@ -307,79 +258,4 @@ mod tests {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
assert!(load_packages_dir(dir.path()).unwrap().is_empty()); assert!(load_packages_dir(dir.path()).unwrap().is_empty());
} }
/// One `[package.p]` with the given extra top-level lines and
/// verification table, loaded through the real loader so validation
/// runs too.
fn load_one(extra: &str, verification: &str) -> Result<Package> {
let dir = tempfile::tempdir().unwrap();
write(
dir.path(),
"p.toml",
&format!(
"[package.p]\nrepo = \"o/r\"\nasset_pattern = \"x\"\n{extra}\n\
[package.p.verification]\n{verification}\n"
),
);
let mut loaded = load_packages_dir(dir.path())?;
Ok(loaded.remove(0).1)
}
const SAME_ORIGIN: &str = "method = \"same-origin-sha256\"\nchecksum_asset_pattern = \"SUMS\"";
const ATTESTATION: &str = "method = \"github-attestation\"";
const FORGEJO: &str = "source = \"forgejo-release\"\nbase_url = \"https://forge.example.com\"";
#[test]
fn source_defaults_to_github_release() {
let pkg = load_one("", ATTESTATION).unwrap();
assert_eq!(pkg.source, Source::GithubRelease);
assert_eq!(pkg.base_url, None);
}
#[test]
fn loads_explicit_github_release_source() {
let pkg = load_one("source = \"github-release\"", ATTESTATION).unwrap();
assert_eq!(pkg.source, Source::GithubRelease);
}
#[test]
fn loads_forgejo_release_source() {
let pkg = load_one(FORGEJO, SAME_ORIGIN).unwrap();
assert_eq!(pkg.source, Source::ForgejoRelease);
assert_eq!(pkg.base_url.as_deref(), Some("https://forge.example.com"));
}
#[test]
fn rejects_forgejo_release_without_base_url() {
let err = load_one("source = \"forgejo-release\"", SAME_ORIGIN).unwrap_err();
assert!(format!("{err:#}").contains("needs a base_url"));
}
#[test]
fn rejects_base_url_on_a_github_source() {
let err = load_one("base_url = \"https://forge.example.com\"", SAME_ORIGIN).unwrap_err();
assert!(format!("{err:#}").contains("only applies to source"));
}
#[test]
fn rejects_forgejo_base_url_without_scheme() {
let err = load_one(
"source = \"forgejo-release\"\nbase_url = \"forge.example.com\"",
SAME_ORIGIN,
)
.unwrap_err();
assert!(format!("{err:#}").contains("must start with http"));
}
#[test]
fn rejects_github_attestation_on_a_forgejo_source() {
let err = load_one(FORGEJO, ATTESTATION).unwrap_err();
assert!(format!("{err:#}").contains("needs a GitHub source"));
}
#[test]
fn rejects_unknown_source() {
assert!(load_one("source = \"gitlab-release\"", SAME_ORIGIN).is_err());
}
} }

View file

@ -1,7 +1,8 @@
//! Downloads a named release asset (from GitHub or Forgejo) to a local //! Downloads a named GitHub release asset to a local path. The only
//! path. The only module that talks to the releases API for asset bytes — //! module that talks to the releases API for asset bytes — `checker` only
//! `release_source` only resolves version tags, never downloads. //! resolves version tags, never downloads.
use crate::github::GithubEndpoints;
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use serde::Deserialize; use serde::Deserialize;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
@ -28,18 +29,16 @@ pub struct DownloadedAsset {
} }
/// Downloads the release asset named exactly `asset_name` for `repo`@`tag` /// Downloads the release asset named exactly `asset_name` for `repo`@`tag`
/// into `dest_dir`, returning the local path and its origin URL. `api` is /// into `dest_dir`, returning the local path and its origin URL.
/// the releases API root (see `ReleaseSource::api`); GitHub and Forgejo
/// serve the same endpoint and JSON shape under it.
pub fn download_asset( pub fn download_asset(
client: &reqwest::blocking::Client, client: &reqwest::blocking::Client,
api: &str, endpoints: &GithubEndpoints,
repo: &str, repo: &str,
tag: &str, tag: &str,
asset_name: &str, asset_name: &str,
dest_dir: &Path, dest_dir: &Path,
) -> Result<DownloadedAsset> { ) -> Result<DownloadedAsset> {
let api_url = format!("{api}/repos/{repo}/releases/tags/{tag}"); let api_url = format!("{}/repos/{repo}/releases/tags/{tag}", endpoints.api);
let release: Release = client let release: Release = client
.get(&api_url) .get(&api_url)
.send()? .send()?
@ -74,7 +73,10 @@ mod tests {
#[test] #[test]
fn download_asset_writes_matching_asset_to_dest_dir() { fn download_asset_writes_matching_asset_to_dest_dir() {
let mut server = mockito::Server::new(); let mut server = mockito::Server::new();
let api = server.url(); let endpoints = GithubEndpoints {
web: server.url(),
api: server.url(),
};
let asset_url = format!("{}/download/thing.tar.gz", server.url()); let asset_url = format!("{}/download/thing.tar.gz", server.url());
let release_body = format!( let release_body = format!(
r#"{{"assets": [{{"name": "thing.tar.gz", "browser_download_url": "{asset_url}"}}]}}"# r#"{{"assets": [{{"name": "thing.tar.gz", "browser_download_url": "{asset_url}"}}]}}"#
@ -94,7 +96,7 @@ mod tests {
let dest_dir = tempfile::tempdir().unwrap(); let dest_dir = tempfile::tempdir().unwrap();
let asset = download_asset( let asset = download_asset(
&client, &client,
&api, &endpoints,
"o/r", "o/r",
"v1.0.0", "v1.0.0",
"thing.tar.gz", "thing.tar.gz",
@ -110,7 +112,10 @@ mod tests {
#[test] #[test]
fn download_asset_errors_when_no_asset_matches() { fn download_asset_errors_when_no_asset_matches() {
let mut server = mockito::Server::new(); let mut server = mockito::Server::new();
let api = server.url(); let endpoints = GithubEndpoints {
web: server.url(),
api: server.url(),
};
let _release = server let _release = server
.mock("GET", "/repos/o/r/releases/tags/v1.0.0") .mock("GET", "/repos/o/r/releases/tags/v1.0.0")
.with_status(200) .with_status(200)
@ -121,7 +126,7 @@ mod tests {
let dest_dir = tempfile::tempdir().unwrap(); let dest_dir = tempfile::tempdir().unwrap();
let err = download_asset( let err = download_asset(
&client, &client,
&api, &endpoints,
"o/r", "o/r",
"v1.0.0", "v1.0.0",
"thing.tar.gz", "thing.tar.gz",
@ -134,7 +139,10 @@ mod tests {
#[test] #[test]
fn download_asset_errors_when_release_not_found() { fn download_asset_errors_when_release_not_found() {
let mut server = mockito::Server::new(); let mut server = mockito::Server::new();
let api = server.url(); let endpoints = GithubEndpoints {
web: server.url(),
api: server.url(),
};
let _release = server let _release = server
.mock("GET", "/repos/o/r/releases/tags/v1.0.0") .mock("GET", "/repos/o/r/releases/tags/v1.0.0")
.with_status(404) .with_status(404)
@ -144,7 +152,7 @@ mod tests {
let dest_dir = tempfile::tempdir().unwrap(); let dest_dir = tempfile::tempdir().unwrap();
let err = download_asset( let err = download_asset(
&client, &client,
&api, &endpoints,
"o/r", "o/r",
"v1.0.0", "v1.0.0",
"thing.tar.gz", "thing.tar.gz",

29
src/github.rs Normal file
View file

@ -0,0 +1,29 @@
/// Base URLs for GitHub's public web host (Atom feeds, release pages) and
/// its REST API, factored out so tests can point both at a local mock
/// server instead of the real github.com/api.github.com.
#[derive(Debug, Clone)]
pub struct GithubEndpoints {
pub web: String,
pub api: String,
}
impl Default for GithubEndpoints {
fn default() -> Self {
Self {
web: "https://github.com".to_string(),
api: "https://api.github.com".to_string(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn default_points_at_real_github() {
let endpoints = GithubEndpoints::default();
assert_eq!(endpoints.web, "https://github.com");
assert_eq!(endpoints.api, "https://api.github.com");
}
}

View file

@ -6,12 +6,11 @@ mod builder;
mod checker; mod checker;
mod config; mod config;
mod fetcher; mod fetcher;
mod github;
mod hash; mod hash;
mod notifier; mod notifier;
mod paths;
mod pipeline; mod pipeline;
mod publisher; mod publisher;
mod release_source;
mod sanity; mod sanity;
mod state; mod state;
#[cfg(test)] #[cfg(test)]

View file

@ -1,184 +0,0 @@
//! Decides where pkgwatch's config, state, and work directories live on
//! disk — the only module that reads the environment to answer that; every
//! other module takes the directories it needs as parameters. (The pacman
//! repo dir, `pipeline::custom_repo_dir`, is resolved separately.)
//!
//! These follow the XDG base-directory spec instead of the current working
//! directory, so an installed `/usr/bin/pkgwatch` behaves the same
//! wherever it's launched from (a systemd unit, a shell, another checkout)
//! instead of only working from inside the repo.
use anyhow::{Context, Result};
use std::path::{Path, PathBuf};
const APP_DIR: &str = "pkgwatch";
#[derive(Debug, PartialEq, Eq)]
pub struct Paths {
/// `*.toml` package declarations. Config: hand-edited, worth backing up.
pub packages_dir: PathBuf,
/// Last-published/pending versions. State: small, but losing it makes
/// every package look new, so it isn't cache.
pub state_dir: PathBuf,
/// Downloaded artifacts and build trees. Cache: safe to delete.
pub work_dir: PathBuf,
}
impl Paths {
pub fn from_env() -> Result<Self> {
Self::resolve(|key| std::env::var(key).ok())
}
/// `getenv` is injectable so tests don't mutate the process-global
/// environment, which would race with `cargo test`'s parallel threads.
///
/// Each directory has a pkgwatch-specific override, then the matching
/// XDG variable, then the XDG default under `$HOME`; empty counts as
/// unset. The overrides exist for dry runs against scratch
/// directories, like `PKGWATCH_REPO_DIR` does for the pacman repo, so
/// they're used verbatim. The XDG variables and `$HOME` must be
/// absolute: the spec says to ignore a relative XDG value, and honoring
/// one would bring back the cwd dependence this module exists to remove.
fn resolve(getenv: impl Fn(&str) -> Option<String>) -> Result<Self> {
let base = |override_var: &str, xdg_var: &str, home_subpath: &str| -> Result<PathBuf> {
if let Some(dir) = non_empty(&getenv, override_var) {
return Ok(PathBuf::from(dir));
}
if let Some(dir) = absolute(&getenv, xdg_var) {
return Ok(Path::new(&dir).join(APP_DIR));
}
let home = absolute(&getenv, "HOME").context("HOME is not set to an absolute path")?;
Ok(Path::new(&home).join(home_subpath).join(APP_DIR))
};
Ok(Self {
packages_dir: base("PKGWATCH_CONFIG_DIR", "XDG_CONFIG_HOME", ".config")?
.join("packages.d"),
state_dir: base("PKGWATCH_STATE_DIR", "XDG_STATE_HOME", ".local/state")?,
work_dir: base("PKGWATCH_WORK_DIR", "XDG_CACHE_HOME", ".cache")?,
})
}
}
/// The XDG spec says an empty variable must be treated as unset.
fn non_empty(getenv: &impl Fn(&str) -> Option<String>, key: &str) -> Option<String> {
getenv(key).filter(|v| !v.is_empty())
}
/// Like `non_empty`, but also drops relative values (an empty string isn't
/// absolute either, so this subsumes the empty check).
fn absolute(getenv: &impl Fn(&str) -> Option<String>, key: &str) -> Option<String> {
getenv(key).filter(|v| Path::new(v).is_absolute())
}
#[cfg(test)]
mod tests {
use super::*;
use std::collections::HashMap;
fn env(pairs: &[(&str, &str)]) -> impl Fn(&str) -> Option<String> {
let map: HashMap<String, String> = pairs
.iter()
.map(|(k, v)| (k.to_string(), v.to_string()))
.collect();
move |key| map.get(key).cloned()
}
#[test]
fn defaults_live_under_home() {
let paths = Paths::resolve(env(&[("HOME", "/home/u")])).unwrap();
assert_eq!(
paths,
Paths {
packages_dir: "/home/u/.config/pkgwatch/packages.d".into(),
state_dir: "/home/u/.local/state/pkgwatch".into(),
work_dir: "/home/u/.cache/pkgwatch".into(),
}
);
}
#[test]
fn xdg_variables_override_home_defaults() {
let paths = Paths::resolve(env(&[
("HOME", "/home/u"),
("XDG_CONFIG_HOME", "/xdg/config"),
("XDG_STATE_HOME", "/xdg/state"),
("XDG_CACHE_HOME", "/xdg/cache"),
]))
.unwrap();
assert_eq!(
paths.packages_dir,
Path::new("/xdg/config/pkgwatch/packages.d")
);
assert_eq!(paths.state_dir, Path::new("/xdg/state/pkgwatch"));
assert_eq!(paths.work_dir, Path::new("/xdg/cache/pkgwatch"));
}
#[test]
fn pkgwatch_overrides_win_and_are_used_verbatim() {
let paths = Paths::resolve(env(&[
("HOME", "/home/u"),
("XDG_STATE_HOME", "/xdg/state"),
("PKGWATCH_CONFIG_DIR", "/scratch/cfg"),
("PKGWATCH_STATE_DIR", "/scratch/state"),
("PKGWATCH_WORK_DIR", "/scratch/work"),
]))
.unwrap();
// No `pkgwatch/` suffix appended to an explicit override.
assert_eq!(paths.packages_dir, Path::new("/scratch/cfg/packages.d"));
assert_eq!(paths.state_dir, Path::new("/scratch/state"));
assert_eq!(paths.work_dir, Path::new("/scratch/work"));
}
#[test]
fn empty_variables_are_treated_as_unset() {
let paths = Paths::resolve(env(&[
("HOME", "/home/u"),
("XDG_STATE_HOME", ""),
("PKGWATCH_WORK_DIR", ""),
]))
.unwrap();
assert_eq!(paths.state_dir, Path::new("/home/u/.local/state/pkgwatch"));
assert_eq!(paths.work_dir, Path::new("/home/u/.cache/pkgwatch"));
}
#[test]
fn relative_xdg_variables_are_ignored() {
let paths = Paths::resolve(env(&[
("HOME", "/home/u"),
("XDG_CONFIG_HOME", "rel/config"),
("XDG_STATE_HOME", "./state"),
("XDG_CACHE_HOME", "cache"),
]))
.unwrap();
assert_eq!(
paths.packages_dir,
Path::new("/home/u/.config/pkgwatch/packages.d")
);
assert_eq!(paths.state_dir, Path::new("/home/u/.local/state/pkgwatch"));
assert_eq!(paths.work_dir, Path::new("/home/u/.cache/pkgwatch"));
}
#[test]
fn relative_home_is_an_error() {
let err = Paths::resolve(env(&[("HOME", "relative/home")])).unwrap_err();
assert!(err.to_string().contains("HOME"));
}
#[test]
fn errors_when_nothing_locates_home() {
let err = Paths::resolve(env(&[])).unwrap_err();
assert!(err.to_string().contains("HOME"));
}
#[test]
fn no_home_needed_when_every_dir_is_overridden() {
let paths = Paths::resolve(env(&[
("PKGWATCH_CONFIG_DIR", "/c"),
("PKGWATCH_STATE_DIR", "/s"),
("PKGWATCH_WORK_DIR", "/w"),
]))
.unwrap();
assert_eq!(paths.state_dir, Path::new("/s"));
}
}

View file

@ -9,16 +9,18 @@ use crate::builder;
use crate::checker; use crate::checker;
use crate::config::{self, Package}; use crate::config::{self, Package};
use crate::fetcher::{self, DownloadedAsset}; use crate::fetcher::{self, DownloadedAsset};
use crate::github::GithubEndpoints;
use crate::notifier::{self, Event}; use crate::notifier::{self, Event};
use crate::paths::Paths;
use crate::publisher; use crate::publisher;
use crate::release_source::{self, ReleaseSource};
use crate::sanity; use crate::sanity;
use crate::state; use crate::state;
use crate::verifier::{self, VerificationResult}; use crate::verifier::{self, VerificationResult};
use anyhow::{Context, Result, bail}; use anyhow::{Context, Result, bail};
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
const PACKAGES_DIR: &str = "packages.d";
const STATE_DIR: &str = "state";
const WORK_DIR: &str = "work";
/// Not a repo pkgwatch invents: this is the existing, already-registered /// Not a repo pkgwatch invents: this is the existing, already-registered
/// local pacman repo on this box (see `[custom]` in /etc/pacman.conf and /// local pacman repo on this box (see `[custom]` in /etc/pacman.conf and
/// its `Server = file://...` line). pkgwatch adds packages to it; it does /// its `Server = file://...` line). pkgwatch adds packages to it; it does
@ -42,28 +44,14 @@ fn custom_repo_dir() -> Result<PathBuf> {
Ok(Path::new(&home).join(CUSTOM_REPO_SUBPATH)) Ok(Path::new(&home).join(CUSTOM_REPO_SUBPATH))
} }
/// Adds a hint to the bare "No such file" a missing config dir would give:
/// the dir is no longer relative to the cwd, so a checkout's `packages.d/`
/// isn't picked up on its own (see docs/SPEC.md > Paths).
fn load_packages(packages_dir: &Path) -> Result<Vec<(String, Package)>> {
config::load_packages_dir(packages_dir).with_context(|| {
format!(
"no package config at {} (set PKGWATCH_CONFIG_DIR to the directory containing \
packages.d, or see docs/SPEC.md > Paths for moving a checkout's packages.d/ there)",
packages_dir.display()
)
})
}
pub fn run_check() -> Result<()> { pub fn run_check() -> Result<()> {
let client = build_client()?; let client = build_client()?;
let Paths { let endpoints = GithubEndpoints::default();
packages_dir, let packages_dir = Path::new(PACKAGES_DIR);
state_dir, let state_dir = Path::new(STATE_DIR);
work_dir, let work_dir = Path::new(WORK_DIR);
} = Paths::from_env()?;
let packages = load_packages(&packages_dir)?; let packages = config::load_packages_dir(packages_dir)?;
if packages.is_empty() { if packages.is_empty() {
println!("no packages configured under {}/", packages_dir.display()); println!("no packages configured under {}/", packages_dir.display());
return Ok(()); return Ok(());
@ -72,10 +60,7 @@ pub fn run_check() -> Result<()> {
let mut any_failed = false; let mut any_failed = false;
for (name, pkg) in &packages { for (name, pkg) in &packages {
println!("== {name} ({}) ==", pkg.repo); println!("== {name} ({}) ==", pkg.repo);
let result = release_source::for_package(pkg).and_then(|host| { if let Err(err) = process_package(&client, &endpoints, state_dir, work_dir, name, pkg) {
process_package(&client, host.as_ref(), &state_dir, &work_dir, name, pkg)
});
if let Err(err) = result {
eprintln!(" error: {err:#}"); eprintln!(" error: {err:#}");
any_failed = true; any_failed = true;
} }
@ -120,13 +105,13 @@ fn decide_tier_action(tier: u8, passed: bool, already_pending_this_version: bool
fn process_package( fn process_package(
client: &reqwest::blocking::Client, client: &reqwest::blocking::Client,
host: &dyn ReleaseSource, endpoints: &GithubEndpoints,
state_dir: &Path, state_dir: &Path,
work_dir: &Path, work_dir: &Path,
name: &str, name: &str,
pkg: &Package, pkg: &Package,
) -> Result<()> { ) -> Result<()> {
let latest = host.latest_release(client, &pkg.repo)?; let latest = checker::latest_github_release(client, endpoints, &pkg.repo)?;
let last_seen = state::load_last_version(state_dir, name); let last_seen = state::load_last_version(state_dir, name);
if last_seen.as_deref() == Some(latest.as_str()) { if last_seen.as_deref() == Some(latest.as_str()) {
println!(" up to date at {latest}"); println!(" up to date at {latest}");
@ -134,7 +119,7 @@ fn process_package(
} }
println!(" new version detected: {latest} (previously: {last_seen:?})"); println!(" new version detected: {latest} (previously: {last_seen:?})");
let fetched = fetch_and_verify(client, host, work_dir, name, pkg, &latest)?; let fetched = fetch_and_verify(client, endpoints, work_dir, name, pkg, &latest)?;
println!(" fetched {}", fetched.asset.path.display()); println!(" fetched {}", fetched.asset.path.display());
println!( println!(
" verification (tier {}): {} — {}", " verification (tier {}): {} — {}",
@ -207,7 +192,7 @@ struct FetchVerifyResult {
/// trusting a possibly-stale flag from an earlier run). /// trusting a possibly-stale flag from an earlier run).
fn fetch_and_verify( fn fetch_and_verify(
client: &reqwest::blocking::Client, client: &reqwest::blocking::Client,
host: &dyn ReleaseSource, endpoints: &GithubEndpoints,
work_dir: &Path, work_dir: &Path,
name: &str, name: &str,
pkg: &Package, pkg: &Package,
@ -217,11 +202,10 @@ fn fetch_and_verify(
let asset_name = pkg.asset_pattern.replace("{version}", &version); let asset_name = pkg.asset_pattern.replace("{version}", &version);
let dest_dir = work_dir.join(name).join(tag); let dest_dir = work_dir.join(name).join(tag);
let api = host.api(); let asset = fetcher::download_asset(client, endpoints, &pkg.repo, tag, &asset_name, &dest_dir)?;
let asset = fetcher::download_asset(client, api, &pkg.repo, tag, &asset_name, &dest_dir)?;
let verification = verifier::verify( let verification = verifier::verify(
client, client,
api, endpoints,
&pkg.verification, &pkg.verification,
&pkg.repo, &pkg.repo,
tag, tag,
@ -283,18 +267,16 @@ fn build_and_publish(
} }
pub fn run_review(args: &[String]) -> Result<()> { pub fn run_review(args: &[String]) -> Result<()> {
let Paths { let packages_dir = Path::new(PACKAGES_DIR);
packages_dir, let state_dir = Path::new(STATE_DIR);
state_dir, let work_dir = Path::new(WORK_DIR);
work_dir, let packages = config::load_packages_dir(packages_dir)?;
} = Paths::from_env()?;
let packages = load_packages(&packages_dir)?;
match args { match args {
[] => { [] => {
let mut any = false; let mut any = false;
for (name, _) in &packages { for (name, _) in &packages {
if let Some(pending) = state::load_pending_version(&state_dir, name) { if let Some(pending) = state::load_pending_version(state_dir, name) {
println!( println!(
"{name}: {pending} pending review (run `pkgwatch review {name} --approve`)" "{name}: {pending} pending review (run `pkgwatch review {name} --approve`)"
); );
@ -310,9 +292,9 @@ pub fn run_review(args: &[String]) -> Result<()> {
let (_, pkg) = packages.iter().find(|(n, _)| n == name).with_context(|| { let (_, pkg) = packages.iter().find(|(n, _)| n == name).with_context(|| {
format!("no package named '{name}' in {}/", packages_dir.display()) format!("no package named '{name}' in {}/", packages_dir.display())
})?; })?;
let tag = state::load_pending_version(&state_dir, name) let tag = state::load_pending_version(state_dir, name)
.with_context(|| format!("'{name}' has no pending review"))?; .with_context(|| format!("'{name}' has no pending review"))?;
approve(&state_dir, &work_dir, name, pkg, &tag) approve(state_dir, work_dir, name, pkg, &tag)
} }
_ => bail!("usage: pkgwatch review [<name> --approve]"), _ => bail!("usage: pkgwatch review [<name> --approve]"),
} }
@ -320,11 +302,11 @@ pub fn run_review(args: &[String]) -> Result<()> {
fn approve(state_dir: &Path, work_dir: &Path, name: &str, pkg: &Package, tag: &str) -> Result<()> { fn approve(state_dir: &Path, work_dir: &Path, name: &str, pkg: &Package, tag: &str) -> Result<()> {
let client = build_client()?; let client = build_client()?;
let host = release_source::for_package(pkg)?; let endpoints = GithubEndpoints::default();
// Re-verify rather than trusting the earlier flag: the artifact at // Re-verify rather than trusting the earlier flag: the artifact at
// this tag could in principle have changed since it was queued. // this tag could in principle have changed since it was queued.
let fetched = fetch_and_verify(&client, host.as_ref(), work_dir, name, pkg, tag)?; let fetched = fetch_and_verify(&client, &endpoints, work_dir, name, pkg, tag)?;
if !fetched.verification.passed { if !fetched.verification.passed {
bail!( bail!(
"re-verification failed on approve: {}", "re-verification failed on approve: {}",
@ -346,8 +328,6 @@ fn approve(state_dir: &Path, work_dir: &Path, name: &str, pkg: &Package, tag: &s
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use crate::release_source::{ForgejoEndpoints, GithubEndpoints};
use crate::test_support::same_origin_package;
#[test] #[test]
fn decide_tier_action_failed_verification_overrides_everything() { fn decide_tier_action_failed_verification_overrides_everything() {
@ -379,67 +359,6 @@ mod tests {
assert_eq!(decide_tier_action(4, true, true), TierAction::StillPending); assert_eq!(decide_tier_action(4, true, true), TierAction::StillPending);
} }
/// Mocks the release-tag, asset, and checksum endpoints for `o/r@v1.0.0`
/// with a checksum that doesn't match the asset, so verification fails.
/// These are identical on GitHub and Forgejo (see `ReleaseSource::api`); only
/// how the latest tag is found differs per test. Returned mocks must
/// stay alive for the test's duration.
fn mock_release_with_bad_checksum(server: &mut mockito::ServerGuard) -> Vec<mockito::Mock> {
let asset_url = format!("{}/download/thing.tar.gz", server.url());
let sums_url = format!("{}/download/SHA256SUMS", server.url());
let release_body = format!(
r#"{{"assets": [
{{"name": "thing.tar.gz", "browser_download_url": "{asset_url}"}},
{{"name": "SHA256SUMS", "browser_download_url": "{sums_url}"}}
]}}"#
);
vec![
server
.mock("GET", "/repos/o/r/releases/tags/v1.0.0")
.with_status(200)
.with_body(release_body)
.create(),
server
.mock("GET", "/download/thing.tar.gz")
.with_status(200)
.with_body(b"artifact-bytes".as_slice())
.create(),
// Wrong hash for "artifact-bytes" — forces a verification failure.
server
.mock("GET", "/download/SHA256SUMS")
.with_status(200)
.with_body(
"0000000000000000000000000000000000000000000000000000000000000000 thing.tar.gz\n",
)
.create(),
]
}
/// Runs `process_package` for a package whose checksum is wrong and
/// asserts it errors with "verification failed" while leaving no trace
/// in state.
fn assert_verification_failure_is_an_error(host: &dyn ReleaseSource, pkg: &Package) {
let client = reqwest::blocking::Client::new();
let state_dir = tempfile::tempdir().unwrap();
let work_dir = tempfile::tempdir().unwrap();
let err = process_package(
&client,
host,
state_dir.path(),
work_dir.path(),
"thing",
pkg,
)
.unwrap_err();
assert!(err.to_string().contains("verification failed"));
// Neither published nor queued for review — a failed verification
// shouldn't leave any trace in state.
assert_eq!(state::load_last_version(state_dir.path(), "thing"), None);
assert_eq!(state::load_pending_version(state_dir.path(), "thing"), None);
}
/// Regression test for the exit-code gap this PR fixes: a verification /// Regression test for the exit-code gap this PR fixes: a verification
/// failure previously returned `Ok(())` from `process_package`, so /// failure previously returned `Ok(())` from `process_package`, so
/// `run_check` never counted it as a failure and the process exited 0 /// `run_check` never counted it as a failure and the process exited 0
@ -450,10 +369,11 @@ mod tests {
#[test] #[test]
fn process_package_returns_err_on_verification_failure() { fn process_package_returns_err_on_verification_failure() {
let mut server = mockito::Server::new(); let mut server = mockito::Server::new();
let github = GithubEndpoints { let endpoints = GithubEndpoints {
web: server.url(), web: server.url(),
api: server.url(), api: server.url(),
}; };
let feed = format!( let feed = format!(
r#"<feed><link rel="alternate" href="{}/o/r/releases/tag/v1.0.0"/></feed>"#, r#"<feed><link rel="alternate" href="{}/o/r/releases/tag/v1.0.0"/></feed>"#,
server.url() server.url()
@ -463,27 +383,63 @@ mod tests {
.with_status(200) .with_status(200)
.with_body(feed) .with_body(feed)
.create(); .create();
let _release_mocks = mock_release_with_bad_checksum(&mut server);
assert_verification_failure_is_an_error(&github, &same_origin_package("")); let asset_url = format!("{}/download/thing.tar.gz", server.url());
} let sums_url = format!("{}/download/SHA256SUMS", server.url());
let release_body = format!(
/// Same as above but through a Forgejo source, proving the whole r#"{{"assets": [
/// check -> fetch -> verify path works there too: the error is the {{"name": "thing.tar.gz", "browser_download_url": "{asset_url}"}},
/// verification failure, not a fetch or check failure on the way to it. {{"name": "SHA256SUMS", "browser_download_url": "{sums_url}"}}
#[test] ]}}"#
fn process_package_returns_err_on_verification_failure_via_forgejo() { );
let mut server = mockito::Server::new(); let _release = server
let forgejo = ForgejoEndpoints { api: server.url() }; .mock("GET", "/repos/o/r/releases/tags/v1.0.0")
let _latest = server
.mock("GET", "/repos/o/r/releases/latest")
.with_status(200) .with_status(200)
.with_body(r#"{"tag_name": "v1.0.0"}"#) .with_body(release_body)
.create();
let _asset = server
.mock("GET", "/download/thing.tar.gz")
.with_status(200)
.with_body(b"artifact-bytes".as_slice())
.create();
// Wrong hash for "artifact-bytes" — forces a verification failure.
let _sums = server
.mock("GET", "/download/SHA256SUMS")
.with_status(200)
.with_body(
"0000000000000000000000000000000000000000000000000000000000000000 thing.tar.gz\n",
)
.create(); .create();
let _release_mocks = mock_release_with_bad_checksum(&mut server);
// The package's own `source` is irrelevant here: `forgejo` is let pkg: Package = toml::from_str(
// hand-built to point at the mock server, bypassing `for_package`. r#"
assert_verification_failure_is_an_error(&forgejo, &same_origin_package("")); repo = "o/r"
asset_pattern = "thing.tar.gz"
[verification]
method = "same-origin-sha256"
checksum_asset_pattern = "SHA256SUMS"
"#,
)
.unwrap();
let client = reqwest::blocking::Client::new();
let state_dir = tempfile::tempdir().unwrap();
let work_dir = tempfile::tempdir().unwrap();
let err = process_package(
&client,
&endpoints,
state_dir.path(),
work_dir.path(),
"thing",
&pkg,
)
.unwrap_err();
assert!(err.to_string().contains("verification failed"));
// Neither published nor queued for review — a failed verification
// shouldn't leave any trace in state.
assert_eq!(state::load_last_version(state_dir.path(), "thing"), None);
assert_eq!(state::load_pending_version(state_dir.path(), "thing"), None);
} }
} }

View file

@ -1,20 +0,0 @@
//! The `ReleaseSource` trait: what the pipeline needs from a hosting
//! service a package's releases are published on. Each host implements it
//! in its own file next to this one, so per-host logic never accumulates
//! here.
use anyhow::Result;
/// `Debug` so a `Box<dyn ReleaseSource>` can sit in a `Result` that tests
/// unwrap.
pub trait ReleaseSource: std::fmt::Debug {
/// The latest release tag for `repo`.
fn latest_release(&self, client: &reqwest::blocking::Client, repo: &str) -> Result<String>;
/// The releases API root, which `fetcher` and `verifier` build their
/// own paths under. GitHub and Forgejo both serve
/// `/repos/{owner}/{repo}/releases/tags/{tag}` there with the same
/// `assets[].{name, browser_download_url}` shape, which is why those
/// stages need only this and not the source itself.
fn api(&self) -> &str;
}

View file

@ -1,112 +0,0 @@
//! A Forgejo (or Gitea) instance as a release source: its API root, and
//! how to find a repo's latest release there.
use super::ReleaseSource;
use anyhow::{Context, Result, bail};
use serde::Deserialize;
#[derive(Debug, Clone)]
pub struct ForgejoEndpoints {
/// The instance's API root, i.e. `<base_url>/api/v1`.
pub api: String,
}
impl ForgejoEndpoints {
/// `base_url` is the instance's web root, e.g.
/// `https://code.austinschaefer.com`; a trailing slash is tolerated.
pub fn from_base_url(base_url: &str) -> Self {
Self {
api: format!("{}/api/v1", base_url.trim_end_matches('/')),
}
}
}
impl ReleaseSource for ForgejoEndpoints {
/// One call, unlike GitHub's feed-then-confirm dance: Forgejo's
/// `releases/latest` already returns only the newest non-draft,
/// non-prerelease *release object*, so a stray tag with no release
/// behind it (GitHub's scaleway-cli `-dbg1` problem) can't be returned.
fn latest_release(&self, client: &reqwest::blocking::Client, repo: &str) -> Result<String> {
#[derive(Deserialize)]
struct Latest {
tag_name: String,
}
let url = format!("{}/repos/{repo}/releases/latest", self.api);
let response = client.get(&url).send()?;
// Forgejo answers 404 both for an unknown repo and for one with no
// releases yet — the common state for a project's very first
// release.
if response.status() == reqwest::StatusCode::NOT_FOUND {
bail!("no published release found at {url} (repo missing, or nothing released yet)");
}
let latest: Latest = response
.error_for_status()
.with_context(|| format!("fetching latest release from {url}"))?
.json()?;
Ok(latest.tag_name)
}
fn api(&self) -> &str {
&self.api
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn from_base_url_appends_api_v1() {
let endpoints = ForgejoEndpoints::from_base_url("https://code.example.com");
assert_eq!(endpoints.api(), "https://code.example.com/api/v1");
}
#[test]
fn from_base_url_tolerates_trailing_slash() {
let endpoints = ForgejoEndpoints::from_base_url("https://code.example.com/");
assert_eq!(endpoints.api(), "https://code.example.com/api/v1");
}
#[test]
fn latest_release_returns_tag_name() {
let mut server = mockito::Server::new();
let _latest = server
.mock("GET", "/repos/o/r/releases/latest")
.with_status(200)
.with_body(r#"{"tag_name": "v0.1.0", "assets": []}"#)
.create();
let client = reqwest::blocking::Client::new();
let endpoints = ForgejoEndpoints { api: server.url() };
assert_eq!(endpoints.latest_release(&client, "o/r").unwrap(), "v0.1.0");
}
#[test]
fn latest_release_names_the_no_releases_case() {
let mut server = mockito::Server::new();
let _latest = server
.mock("GET", "/repos/o/r/releases/latest")
.with_status(404)
.create();
let client = reqwest::blocking::Client::new();
let endpoints = ForgejoEndpoints { api: server.url() };
let err = endpoints.latest_release(&client, "o/r").unwrap_err();
assert!(err.to_string().contains("no published release"));
}
#[test]
fn latest_release_surfaces_server_errors() {
let mut server = mockito::Server::new();
let _latest = server
.mock("GET", "/repos/o/r/releases/latest")
.with_status(500)
.create();
let client = reqwest::blocking::Client::new();
let endpoints = ForgejoEndpoints { api: server.url() };
let err = endpoints.latest_release(&client, "o/r").unwrap_err();
assert!(err.to_string().contains("fetching latest release"));
}
}

View file

@ -1,166 +0,0 @@
//! GitHub as a release source: its endpoints, and how to find a repo's
//! latest release there.
use super::ReleaseSource;
use anyhow::{Result, bail};
use regex::Regex;
/// Base URLs for GitHub's public web host (Atom feeds, release pages) and
/// its REST API, factored out so tests can point both at a local mock
/// server instead of the real github.com/api.github.com.
#[derive(Debug, Clone)]
pub struct GithubEndpoints {
pub web: String,
pub api: String,
}
impl Default for GithubEndpoints {
fn default() -> Self {
Self {
web: "https://github.com".to_string(),
api: "https://api.github.com".to_string(),
}
}
}
impl ReleaseSource for GithubEndpoints {
/// Resolves the latest release tag for `repo` via its public Atom feed.
///
/// Deliberately not a full XML parse: the feed lists entries
/// newest-first, and each `<link rel="alternate"
/// .../releases/tag/<tag>"/>` is matched in document order. Revisit
/// with a real XML parser if GitHub's feed shape ever changes.
///
/// The feed can list a tag newer than any tag with a real Release
/// object behind it — observed on scaleway/scaleway-cli, which pushes a
/// `vX.Y.Z-dbg1` tag (no corresponding Release; `releases/tags/<tag>`
/// 404s) right after each real release, and that tag sorts newest in
/// the feed. So each candidate is confirmed against the releases API in
/// feed order, returning the first that actually resolves.
fn latest_release(&self, client: &reqwest::blocking::Client, repo: &str) -> Result<String> {
let url = format!("{}/{repo}/releases.atom", self.web);
let body = client.get(&url).send()?.error_for_status()?.text()?;
let re = Regex::new(r#"releases/tag/([^"]+)""#)?;
let mut candidates = re
.captures_iter(&body)
.map(|caps| caps[1].to_string())
.peekable();
if candidates.peek().is_none() {
bail!("no release tag found in {url}");
}
for tag in candidates {
let release_url = format!("{}/repos/{repo}/releases/tags/{tag}", self.api);
if client.get(&release_url).send()?.status().is_success() {
return Ok(tag);
}
}
bail!("no release tag in {url} resolved to a real release via the API")
}
fn api(&self) -> &str {
&self.api
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn default_points_at_real_github() {
let endpoints = GithubEndpoints::default();
assert_eq!(endpoints.web, "https://github.com");
assert_eq!(endpoints.api, "https://api.github.com");
}
fn atom_feed(tags: &[&str]) -> String {
let entries: String = tags
.iter()
.map(|t| {
format!(r#"<link rel="alternate" href="https://github.com/o/r/releases/tag/{t}"/>"#)
})
.collect();
format!("<feed>{entries}</feed>")
}
#[test]
fn latest_release_skips_tags_with_no_real_release() {
let mut server = mockito::Server::new();
let endpoints = GithubEndpoints {
web: server.url(),
api: server.url(),
};
// Mirrors the real scaleway-cli case: newest feed entry (a -dbg1
// tag) has no Release object behind it and 404s.
let _feed = server
.mock("GET", "/o/r/releases.atom")
.with_status(200)
.with_body(atom_feed(&["v2.62.0-dbg1", "v2.62.0"]))
.create();
let _missing = server
.mock("GET", "/repos/o/r/releases/tags/v2.62.0-dbg1")
.with_status(404)
.create();
let _real = server
.mock("GET", "/repos/o/r/releases/tags/v2.62.0")
.with_status(200)
.with_body("{}")
.create();
let client = reqwest::blocking::Client::new();
let tag = endpoints.latest_release(&client, "o/r").unwrap();
assert_eq!(tag, "v2.62.0");
}
#[test]
fn latest_release_errors_when_feed_has_no_tags() {
let mut server = mockito::Server::new();
let endpoints = GithubEndpoints {
web: server.url(),
api: server.url(),
};
let _feed = server
.mock("GET", "/o/r/releases.atom")
.with_status(200)
.with_body("<feed></feed>")
.create();
let client = reqwest::blocking::Client::new();
let err = endpoints.latest_release(&client, "o/r").unwrap_err();
assert!(err.to_string().contains("no release tag found"));
}
#[test]
fn latest_release_errors_when_no_candidate_resolves() {
let mut server = mockito::Server::new();
let endpoints = GithubEndpoints {
web: server.url(),
api: server.url(),
};
let _feed = server
.mock("GET", "/o/r/releases.atom")
.with_status(200)
.with_body(atom_feed(&["v1.0.0-dbg1"]))
.create();
let _missing = server
.mock("GET", "/repos/o/r/releases/tags/v1.0.0-dbg1")
.with_status(404)
.create();
let client = reqwest::blocking::Client::new();
let err = endpoints.latest_release(&client, "o/r").unwrap_err();
assert!(err.to_string().contains("resolved to a real release"));
}
#[test]
fn api_is_the_configured_api_root() {
let endpoints = GithubEndpoints {
web: "http://w".into(),
api: "http://a".into(),
};
assert_eq!(endpoints.api(), "http://a");
}
}

View file

@ -1,57 +0,0 @@
//! Where a package's releases are published: the `ReleaseSource` trait,
//! one file per host implementing it, and `for_package`, which picks the
//! implementation for a package from its configured `source`. The
//! submodules are private and re-exported here, so the rest of the crate
//! imports everything from `crate::release_source` and never a host's
//! file.
mod contract;
mod forgejo;
mod github;
pub use contract::ReleaseSource;
pub use forgejo::ForgejoEndpoints;
pub use github::GithubEndpoints;
use crate::config::{Package, Source};
use anyhow::{Context, Result};
/// Defensive: errors only if a `forgejo-release` package has no
/// `base_url`, which `config::load_packages_dir` already guarantees.
pub fn for_package(pkg: &Package) -> Result<Box<dyn ReleaseSource>> {
match pkg.source {
Source::GithubRelease => Ok(Box::new(GithubEndpoints::default())),
Source::ForgejoRelease => {
let base_url = pkg
.base_url
.as_deref()
.context("source = \"forgejo-release\" needs a base_url")?;
Ok(Box::new(ForgejoEndpoints::from_base_url(base_url)))
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::test_support::same_origin_package as package;
#[test]
fn github_source_uses_real_github() {
let source = for_package(&package("")).unwrap();
assert_eq!(source.api(), "https://api.github.com");
}
#[test]
fn forgejo_source_uses_the_instances_api() {
let pkg = package("source = \"forgejo-release\"\nbase_url = \"https://code.example.com\"");
let source = for_package(&pkg).unwrap();
assert_eq!(source.api(), "https://code.example.com/api/v1");
}
#[test]
fn forgejo_source_without_base_url_errors() {
let err = for_package(&package("source = \"forgejo-release\"")).unwrap_err();
assert!(err.to_string().contains("needs a base_url"));
}
}

View file

@ -1,6 +1,6 @@
//! Persists two independent per-package facts as plain files: the last //! Persists two independent per-package facts as plain files: the last
//! published version, and any version currently pending human review. //! published version, and any version currently pending human review.
//! The only module that touches the state directory (see `paths.rs`) on disk. //! The only module that touches `state/` on disk.
use anyhow::Result; use anyhow::Result;
use std::path::Path; use std::path::Path;

View file

@ -1,11 +1,10 @@
//! Test-only fixture helpers shared across modules' `#[cfg(test)]` code //! Test-only fixture helpers shared across modules' `#[cfg(test)]` code
//! — not production code, and not built outside `cargo test`. See //! (`publisher`, `sanity`, `notifier`) — not production code, and not built outside
//! docs/ARCHITECTURE.md > "organize by pipeline stage, not by layer": this //! `cargo test`. See docs/ARCHITECTURE.md > "organize by pipeline stage, not
//! exists to remove specific pieces of duplication (near-identical copies //! by layer": this exists to remove one specific piece of duplication
//! of "write an executable shell script" and of "build a same-origin //! (two near-identical copies of "write an executable shell script"), not
//! `Package` from TOML"), not as a general test-utils dump. //! as a general test-utils dump.
use crate::config::Package;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::process::Command; use std::process::Command;
use std::time::{Duration, Instant}; use std::time::{Duration, Instant};
@ -53,22 +52,3 @@ fn wait_until_executable(path: &Path) {
} }
} }
} }
/// A `same-origin-sha256` `Package` for repo `o/r`, asset `thing.tar.gz`,
/// checksum asset `SHA256SUMS`. `extra` is spliced in as top-level keys
/// before the `[verification]` table (e.g. `source`/`base_url`), and is
/// parsed directly rather than through `config::load_packages_dir`, so it
/// skips load-time validation.
pub(crate) fn same_origin_package(extra: &str) -> Package {
toml::from_str(&format!(
r#"
repo = "o/r"
asset_pattern = "thing.tar.gz"
{extra}
[verification]
method = "same-origin-sha256"
checksum_asset_pattern = "SHA256SUMS"
"#
))
.unwrap()
}

View file

@ -6,6 +6,7 @@
use crate::checker::version_from_tag; use crate::checker::version_from_tag;
use crate::config::Verification; use crate::config::Verification;
use crate::fetcher; use crate::fetcher;
use crate::github::GithubEndpoints;
use crate::hash; use crate::hash;
use anyhow::{Context, Result, bail}; use anyhow::{Context, Result, bail};
use std::path::Path; use std::path::Path;
@ -22,7 +23,7 @@ pub struct VerificationResult {
/// each tier does and does not prove. /// each tier does and does not prove.
pub fn verify( pub fn verify(
client: &reqwest::blocking::Client, client: &reqwest::blocking::Client,
api: &str, endpoints: &GithubEndpoints,
verification: &Verification, verification: &Verification,
repo: &str, repo: &str,
tag: &str, tag: &str,
@ -35,8 +36,14 @@ pub fn verify(
} => { } => {
let checksum_asset_name = let checksum_asset_name =
checksum_asset_pattern.replace("{version}", version_from_tag(tag)); checksum_asset_pattern.replace("{version}", version_from_tag(tag));
let checksum_asset = let checksum_asset = fetcher::download_asset(
fetcher::download_asset(client, api, repo, tag, &checksum_asset_name, dest_dir)?; client,
endpoints,
repo,
tag,
&checksum_asset_name,
dest_dir,
)?;
let checksum_text = std::fs::read_to_string(&checksum_asset.path)?; let checksum_text = std::fs::read_to_string(&checksum_asset.path)?;
let artifact_name = artifact_path let artifact_name = artifact_path
.file_name() .file_name()
@ -177,7 +184,10 @@ mod tests {
#[test] #[test]
fn verify_same_origin_sha256_passes_on_matching_checksum() { fn verify_same_origin_sha256_passes_on_matching_checksum() {
let mut server = mockito::Server::new(); let mut server = mockito::Server::new();
let api = server.url(); let endpoints = GithubEndpoints {
web: server.url(),
api: server.url(),
};
let dest_dir = tempfile::tempdir().unwrap(); let dest_dir = tempfile::tempdir().unwrap();
let artifact_path = dest_dir.path().join("thing.tar.gz"); let artifact_path = dest_dir.path().join("thing.tar.gz");
std::fs::write(&artifact_path, b"hello world").unwrap(); std::fs::write(&artifact_path, b"hello world").unwrap();
@ -204,7 +214,7 @@ mod tests {
let client = reqwest::blocking::Client::new(); let client = reqwest::blocking::Client::new();
let result = verify( let result = verify(
&client, &client,
&api, &endpoints,
&verification, &verification,
"o/r", "o/r",
"v1.0.0", "v1.0.0",
@ -220,7 +230,10 @@ mod tests {
#[test] #[test]
fn verify_same_origin_sha256_fails_on_mismatched_checksum() { fn verify_same_origin_sha256_fails_on_mismatched_checksum() {
let mut server = mockito::Server::new(); let mut server = mockito::Server::new();
let api = server.url(); let endpoints = GithubEndpoints {
web: server.url(),
api: server.url(),
};
let dest_dir = tempfile::tempdir().unwrap(); let dest_dir = tempfile::tempdir().unwrap();
let artifact_path = dest_dir.path().join("thing.tar.gz"); let artifact_path = dest_dir.path().join("thing.tar.gz");
std::fs::write(&artifact_path, b"hello world").unwrap(); std::fs::write(&artifact_path, b"hello world").unwrap();
@ -248,7 +261,7 @@ mod tests {
let client = reqwest::blocking::Client::new(); let client = reqwest::blocking::Client::new();
let result = verify( let result = verify(
&client, &client,
&api, &endpoints,
&verification, &verification,
"o/r", "o/r",
"v1.0.0", "v1.0.0",

View file

@ -1,9 +1,9 @@
# User-level oneshot: one check -> fetch -> verify -> build -> publish pass. # User-level oneshot: one check -> fetch -> verify -> build -> publish pass.
# Install: see docs/SPEC.md > Scheduling. # Install: see docs/SPEC.md > Scheduling.
# #
# No WorkingDirectory: config, state and work dirs come from the XDG paths # WorkingDirectory matters: packages.d/, state/ and work/ are all resolved
# in src/paths.rs (see docs/SPEC.md > Paths), not the cwd. # relative to the cwd, so this must be the main checkout, not a worktree.
# The binary is the release build in the main checkout (`cargo build # The binary is the release build in that same checkout (`cargo build
# --release`), so a rebuild is what picks up code changes. # --release`), so a rebuild is what picks up code changes.
[Unit] [Unit]
Description=pkgwatch: check tracked packages for new upstream releases Description=pkgwatch: check tracked packages for new upstream releases
@ -11,6 +11,7 @@ OnFailure=pkgwatch-failure.service
[Service] [Service]
Type=oneshot Type=oneshot
WorkingDirectory=%h/dev/pkgwatch
ExecStart=%h/dev/pkgwatch/target/release/pkgwatch ExecStart=%h/dev/pkgwatch/target/release/pkgwatch
# Builds (makepkg, large Go/Rust binaries) can legitimately take a while. # Builds (makepkg, large Go/Rust binaries) can legitimately take a while.
TimeoutStartSec=30min TimeoutStartSec=30min