Architecture and design

The patcher–runner model

Hyrum separates the work of modifying a charm’s dependencies (patching) from the work of running a check (running):

• Patchers are synchronous context managers. They touch the filesystem, may shell out to poetry lock or uv lock, and must restore every file on exit, whether or not the run succeeded.

• Runners are async. They spawn a subprocess (tox or make) and wait for it to exit, returning a structured RunResult.

Because patching involves slow, blocking subprocesses (lockfile regeneration can take minutes), the pool runs each patcher’s apply in a thread (asyncio.to_thread) so that concurrent workers overlap their lock subprocesses rather than waiting in sequence.

The async worker pool

The pool is a simple queue-based design:

1. All charm paths are loaded into an asyncio.Queue.

2. N concurrent consumer coroutines (controlled by --workers) each pull from the queue, patch, run, and report.

3. Results are collected in a list and sorted by repo path before display.

The pool deliberately does not use asyncio.Semaphore or structured concurrency beyond asyncio.gather on the consumer tasks. The queue approach means each worker is idle for at most one charm at a time and work is distributed evenly as workers complete.

Sharing a charms directory

Patching a charm means rewriting its working tree, so two runs over one charms directory would otherwise interleave: the second takes the first’s patched tree for the original, and each then reports results for whichever patch won. check therefore takes an advisory lock per charm, held from the patch to the restore, and a second run waits for that charm rather than joining it. Locks live in <charms-dir>/.hyrum-locks/. --no-lock turns this off, at the cost of that guarantee.

clean takes the same lock, for the same reason from the other side: the artefacts it reclaims — .tox, .venv, tool caches — are the ones a running check is using, so removing them mid-run would break it. It locks each charm before removing that charm’s artefacts, and has its own --no-lock. --dry-run removes nothing and so waits for nothing.

Outcome statuses and attribution

pool.Outcome normalises across three paths through the pool:

• A pre-pool skip (filtered out before patching): status='skipped'.

• A patcher failure: status='patcher_error' with the error message in outcome.error.

• A runner result (pass, fail, no-target, timeout, or a runner that could not be launched): status from RunStatus.

The distinction between the infrastructure statuses and failed is important: patcher_error means hyrum could not apply the patch, and runner_error means tox or make itself could not be launched. A failed outcome means the charm’s own tests reported failure. Mixing these together would make the “N charms broke” count misleading.

Attribution only works if each layer reports its own faults, so the runners catch OSError on launch and return a runner_error result rather than letting the exception escape into the pool, where it would have been recorded against the patcher. The pool’s catch-all therefore blames neither layer: it says only that the error was unexpected.

The same reasoning motivates the preflight that runs before the pool starts. An unresolvable --patch ref patches cleanly and only breaks when the runner installs the dependency, so every charm would be recorded as failed — a plausible-looking pile of regressions caused by one typo. Checking each distinct git ref with git ls-remote, and each runner executable with a PATH lookup, turns that into one error message before any charm runs.

Each non-passing outcome also carries a one-line summary, extracted heuristically from the runner’s output: a pytest tally, an exception class, a missing build tool, a resolver error. It exists so that a comparison table is readable without opening the log files — the shape of a failure is usually enough to tell a genuine regression from host noise.

Skips versus errors

Patchers signal two different kinds of “this did not happen”: PatcherError, meaning the patch should have applied but could not, and PatcherSkip, meaning there was nothing to patch. The second is not a failure — a charm that never depended on the library you are testing tells you nothing about your change, and reporting it as an error would inflate the numbers exactly where the fleet is largest. Skips carry a machine-readable reason, so the tally can separate the ordinary cases (dep_not_declared, vendored_lib_absent) from the one that deserves attention (malformed_pyproject).

Signal vs noise

Hyrum produces a table of outcomes, not a verdict. Two factors prevent treating any single run’s numbers as ground truth:

• Pre-existing failures. Many charm repositories have failing tests that predate any change under test. A baseline run (--no-patch) establishes how many charms fail without any modification, providing a comparison point.

• Flaky tests. Some tests are non-deterministic. A charm that fails once may pass on a re-run.

The intended workflow is to run hyrum twice (once without patching, once with) and compare the delta. Charms that move from passed to failed in the patched run are the likely regressions introduced by the change.

That comparison is built in rather than left to the reader. Every run serialises its outcomes to JSON, keyed on each charm’s path relative to the charms directory so that runs from different hosts still line up, and hyrum compare reduces two such files to the three categories that carry information: new failures, resolved, and new errors. Saving is on by default — a rolling pair of files per target — on the grounds that the second run is where the value is, and needing to have known in advance to pass a flag would waste the first one.