Running tests

The provium binary is what you run. This page covers the day-to-day patterns. The full flag inventory is on the CLI reference.

The basic invocation #

provium tests/

Walks tests/ for *.test.lua files and runs each. Default output:

PASS tests/smoke.test.lua (3 passed, 0 failed, 0 skipped)
FAIL tests/networking/partition.test.lua (2 passed, 1 failed, 0 skipped)
    FAIL split-brain
        client A could not reach client B after partition

2 file(s); 5 passed, 1 failed, 0 skipped; 12.34s

A single file works too:

provium tests/networking/partition.test.lua

If you omit paths entirely, the current directory is scanned recursively.

Selecting what runs #

--filter <substring> #

Match against the test-root-relative path:

provium tests/ --filter networking
provium tests/ --filter partition.test

--rerun-failed #

After every run with at least one failure, Provium writes the canonical paths to ~/.cache/provium/rerun.json. --rerun-failed intersects discovered files against this list:

provium tests/ --rerun-failed

If no prior failure state exists, --rerun-failed exits cleanly with a notice (running the FULL suite is almost never what the user wanted).

A clean run leaves the failed-state file alone, so the failed set persists until you fix the failures.

--since <reference-file> #

Run only files whose mtime is newer than the reference file:

provium tests/ --since main.lua          # files edited after main.lua
provium tests/ --since .last-checkpoint  # custom marker

Useful for "what's changed since I last ran".

Per-test filters #

Tag and slow filtering happen inside each file (the test body's meta is what's matched):

provium tests/ --tag smoke
provium tests/ --tag smoke --tag fast    # OR: any tag matches
provium tests/ --no-tag flaky            # exclude tests tagged flaky
provium tests/ --include-slow            # include meta.slow tests

--no-tag wins over --tag (intersect semantics).

For filtering on arbitrary meta fields (not just tags), use --tag-meta KEY=VALUE:

provium tests/ --tag-meta subsystems=peinit
provium tests/ --tag-meta subsystems=peinit --tag-meta subsystems=loregd   # OR within KEY
provium tests/ --tag-meta subsystems=peinit --tag-meta area=boot            # AND across KEYs
provium tests/ --no-tag-meta flaky=true

This is the recommended pattern when test files are organised by user-visible scenario rather than by code subsystem — annotate tests with subsystems = {"peinit", "loregd"} (or any other axis you care about) and let CI compute "what to run" from the changeset:

# CI: PR touched the peinit subsystem
provium tests/ --tag-meta subsystems=peinit

See meta tags for the convention.

Skipped tests still appear in outcomes — you'll see SKIP <name> in -v mode, with the filter expression as the reason.

Output modes #

Default (compact) #

FAIL tests/x.test.lua (2 passed, 1 failed, 0 skipped)
    FAIL the failing case
        message lines

Verbose (-v) #

Every test status, not just failures:

PASS tests/x.test.lua (3 passed, 0 failed, 0 skipped)
    PASS first
    PASS second
    PASS third

Quiet (-q) #

Only failures show. Files with all passing / skipped are silent. The summary line still prints.

JSON (--json) #

Line-delimited JSON, one object per file:

{"path":"tests/x.test.lua","timeout":"in_time","passed":true,"chunk_error":null,"tests":[{"name":"…","status":"passed","message":null,"log":[]}]}

Useful for piping into another tool. Mutually exclusive with --events-stdout.

Msgpack events (--events-stdout) #

Length-prefixed msgpack frames over stdout. The human-readable / JSON renderer redirects to stderr automatically. Useful with provium-coverage:

provium tests/ --events-stdout | provium-coverage

See events and coverage for the full event-stream story.

Watch mode #

provium tests/ --watch

Polls every 500 ms for stat-changed files; re-runs when one changes. Use Ctrl-C to exit. The empty-set case is fine — the watcher rescans every tick, so dropping a *.test.lua into the watched root after launch triggers discovery.

--rerun-failed is automatically dropped under --watch so file-edit detection works as expected (otherwise each tick would re-run only the original failed set forever).

Per-file timeouts #

provium tests/ --timeout 300            # 5 min (default)
provium tests/ --timeout 30s
provium tests/ --timeout 10m
provium tests/ --timeout 500ms
provium tests/ --timeout 0              # disable

Bare integers are seconds. Suffixes accepted: ms, s, m, h. 0 disables the timeout entirely.

When a file's timeout fires, the harness records it as timed_out, marks the file as failed for exit-code purposes, and emits a file_completed event with status = "timed_out".

Pool and CPU controls #

By default Provium uses 80 % of host RAM and the full host CPU count for the resource pool. Override:

provium tests/ --mem 16G                 # explicit pool memory
provium tests/ --cpus 8                  # explicit pool vCPUs
provium tests/ --cpu-overcommit 1.5      # multiplier on --cpus

See pools and parallelism for how the pool, claims, and dispatcher interact — including when overcommit is safe. Flag defaults and the clamp range are in the CLI reference.

Dev-mode flags #

--no-preflight #

Skip the /dev/kvm / iproute2 / nft / qemu / CAP_NET_ADMIN checks. Use in containers or CI environments where you know the environment is fine but the checks would fail.

--no-ksm #

Skip Kernel Same-page Merging tuning at startup. Default tunes /sys/kernel/mm/ksm/* per the design's pool-density goals; pass this on shared dev hosts where you don't want global tuning.

--vmm local #

Use the in-process LocalAgent backend instead of QEMU. Useful when KVM isn't available (CI, dev-on-laptop). The local backend does not actually boot a kernel — it gives the host bindings something to dispatch against, but tests that rely on guest-side behaviour (running commands, file I/O) won't work.

provium tests/ --vmm local

Subcommands #

When a subcommand is given, the test-runner mode is suppressed.

provium repl <profile> #

Boot a VM and drop into an interactive Lua REPL:

provium repl peios                  # cold-boot the peios profile
provium repl peios --name dev       # custom VM name
provium repl --fixture base         # resume from a fixture

Useful for poking at a guest interactively. The full Provium API is available — vm:run, vm:read_file, vm:tail_file, etc.

provium fixture <op> #

Manage the fixture cache:

provium fixture list                       # show every cached entry
provium fixture build path/to/fixture      # force-build
provium fixture rebuild path/to/fixture    # evict and rebuild
provium fixture clean                      # wipe the cache
provium fixture stale                      # list fixtures whose source doesn't match any cache entry

See fixtures and dependencies for when each is useful.

provium prepare [profile] #

Run dynamic profiles' build commands without booting anything — one profile by name, or every profile that declares a build. Pre-warm images with provium prepare && provium tests/ --no-build; see dynamic profiles.

provium list #

List discovered tests / fixtures without running anything:

provium list                 # tests
provium list --fixtures      # fixtures

provium lsp-setup [dir] #

Write Lua Language Server stubs and a .luarc.json into a test directory so the harness globals resolve in your editor — see the CLI reference.

Exit codes #

For CI, three values: 0 means every file passed (or skipped), 1 means at least one file did not finish cleanly, 2 means an internal error (config, pre-flight, dispatcher). The exit code is deliberately clamped to those three — the precise triggers and the clamping rationale are in the CLI reference.

CI patterns #

"Run everything, fail on any failure" #

#!/bin/sh
provium tests/

Exit code is your test result. Capture stdout for the summary, stderr for the human-readable lines (when --json is set).

"Run with structured output" #

provium tests/ --json | tee results.jsonl

Each line is a complete {"path":…,"tests":[…]} object. Easy to pipe into a custom dashboard or per-test reporter.

"Save events for post-hoc analysis" #

provium tests/ --save-events events.msgpack
provium-coverage --from events.msgpack > coverage.html

The msgpack file is portable; you can analyse it on a different host.

"Fail fast" #

provium tests/ --fail-fast

Stops after the first failed file. Useful for tight inner loops where you want to see the first failure quickly.

"Slow CI vs fast CI" #

# Fast: skip slow tests by default.
provium tests/

# Slow / nightly: run everything.
provium tests/ --include-slow

meta.slow = true tests are skipped by default. Tag them in the test bodies, and your dev loop stays fast while nightly catches the slow stuff.

"Parallel CI sharding" #

There's no built-in shard splitter, but provium accepts any number of path arguments, so split the file list in your CI script and pass each shard its own files:

# $SHARDS = total shard count, $INDEX = this shard (0-based).
git ls-files 'tests/**/*.test.lua' | awk "NR % $SHARDS == $INDEX" | xargs provium

Each shard runs a disjoint subset of files. Don't try to build a shard with --filter — it's a single substring match against the relative path, not a pattern list.

See also #

• CLI reference — every flag, every env var.
• Events and coverage — --save-events, --events-stdout, --coverage.
• Pools and parallelism — how the resource pool works.

Provium emits a structured msgpack event stream covering every file, test, VM, fixture, and pool transition. This page covers the practical side of consuming it; the wire format is on events.

Why events #

The human-readable PASS / FAIL summary is fine for the dev loop, but not enough for:

• Coverage reports (which test exercised which spec section?).
• Dashboards (live view of pool state, in-flight files, fixture builds).
• Post-hoc analysis ("why did this CI run take 4× longer than yesterday?").
• Debugging fixture rebuilds (which file is waiting on which build lock?).

For all of these, the event stream is what you want.

Output channels #

Provium can route events to three places independently. Combine flags freely.

--save-events <path> — to a file #

provium tests/ --save-events events.msgpack

Writes length-prefixed msgpack frames to the file. Use for post-hoc analysis or replay.

Under --watch, the file is truncated at each iteration so it contains exactly the most recent run.

--events-stdout — to stdout #

provium tests/ --events-stdout | provium-coverage

Streams events on stdout. The human-readable / JSON renderer redirects to stderr so consumers piping provium --events-stdout | … don't see human text interleaved into their msgpack parser. (provium-coverage reads stdin when --from is omitted, so the bare pipe above is the whole pipeline.)

Mutually exclusive with --json.

--events-socket <path> — over a Unix socket #

provium tests/ --watch --events-socket /tmp/provium.sock
# in another terminal — any client that can read a Unix socket:
socat - UNIX-CONNECT:/tmp/provium.sock | provium-coverage

The binary listens, accepts connections, and fans out frames. Useful for live dashboards. The socket persists across --watch iterations so connected clients aren't dropped on every re-run.

Coverage post-run #

--coverage is sugar over --save-events + a post-run pipe to provium-coverage:

provium tests/ --coverage

Internally:

1. If --save-events <path> is also set, Provium uses that file.
2. Otherwise, Provium tees events to a scratch tempfile ($TMPDIR/provium-coverage-<pid>.msgpack) with a sibling marker file.
3. After the run, runs provium-coverage --from <path>.
4. Cleans up the tempfile if Provium owns it (the marker is the proof).

Failures from the post-run pipe propagate as the exit code — a coverage failure shows up as exit 1.

The cleanup is robust: SIGINT / SIGTERM handlers and an atexit hook clear the tempfile even on a killed run, so CI won't accumulate orphaned msgpack files.

What's in the stream #

The stream covers five areas: file lifecycle (file_discovered → file_dispatched → file_completed, with file_blocked when a file waits on the pool), test outcomes (test_started then one of test_passed / test_failed / test_skipped), VM lifecycle (vm_spawned / vm_shutdown), pool and claim activity, and fixture builds (fixture_build_started / _done / _waiting and fixture_cache_hit).

Every frame is {ts, kind, payload}. The full variant list and every payload field are in the events reference.

Replay and analysis #

provium tests/ --save-events events.msgpack
provium-coverage --from events.msgpack

The msgpack file is portable. You can analyse it on a different host than the one that produced it, or run multiple consumers against the same file:

provium-coverage --from events.msgpack > coverage.html
my-custom-tool --from events.msgpack > metrics.json

Building your own consumer #

The protocol is provium-protocol's EventFrame over length-prefixed msgpack frames (the same framing as the agent wire protocol). The framing helpers in provium-protocol::frame are exposed for reuse.

A minimal Rust consumer:

use provium_protocol::events::{Event, EventFrame};
use provium_protocol::frame::{read_frame, DEFAULT_MAX_FRAME_BYTES};
use std::fs::File;
use std::io::BufReader;

fn main() -> std::io::Result<()> {
    let f = File::open("events.msgpack")?;
    let mut r = BufReader::new(f);
    loop {
        let frame: EventFrame = match read_frame(&mut r, DEFAULT_MAX_FRAME_BYTES) {
            Ok(f) => f,
            Err(_) => break,  // EOF or malformed; production code should distinguish
        };
        match frame.event {
            Event::TestPassed(t) => println!("PASS {}::{}", t.path, t.name),
            Event::TestFailed(t) => println!("FAIL {}::{} — {}", t.path, t.name, t.reason),
            _ => {}
        }
    }
    Ok(())
}

For other languages, use any msgpack library that handles the length-prefix framing (e.g. read 4 bytes BE length, then read N bytes, then deserialise as msgpack).

Live multiplexing for dashboards #

--events-socket is the right shape when you have a long-running dashboard:

# Provium runs in watch mode; events flow over the socket.
provium tests/ --watch --events-socket /tmp/provium.sock

# Dashboard connects once and stays connected across reruns.
my-dashboard --from-socket /tmp/provium.sock

The socket persists across --watch iterations precisely so dashboards don't have to reconnect on every re-run.

For non-dashboard pipelines (CI feeding a coverage tool), prefer --events-stdout — simpler, no socket-file management.

Observability event guarantees #

The harness guarantees ordering and pairing invariants (exactly one file_completed per dispatched file, exactly one terminal event per started test, paired claim and fixture-build events) — the full list is in events reference — Event guarantees. Consumers can rely on these to track in-flight state, e.g. counting "in-flight files" as file_dispatched - file_completed.

Practical patterns #

"Why did this run take so long?" #

provium tests/ --save-events events.msgpack
# parse events.msgpack to find the longest file_completed.duration_ns

"Which fixtures rebuilt?" #

provium tests/ --save-events events.msgpack
# count fixture_build_started events

If many fixtures are rebuilding, something invalidated the cache — usually a kernel swap or a helper edit.

"Spec coverage report" #

provium tests/ --include-slow --coverage

provium-coverage reads meta.spec from each test_passed / test_failed event and produces a spec-section report.

"What was the guest doing when this test failed?" #

The test_failed event includes console_excerpt — the last 4 KiB of every booted VM's console log captured at the failure moment. Useful when failures are kernel-side and reproducing locally is expensive.

See also #

• Events reference — every event, every field.
• Protocol version — wire-shape pinning.
• The CLI — --save-events, --events-stdout, --events-socket, --coverage.

Fixtures are pre-built VM (or whole-lab) snapshots that test files restore from instead of re-doing setup work. This page covers the cache model end-to-end.

The exhaustive method reference is on Lab (vm_fixture, lab_fixture); the userdata is on Snapshot.

Discovery #

A fixture is any *.fixture.lua file under one of the roots directories listed in provium.toml. Test files reference fixtures by their test-root-relative path with the .fixture.lua suffix omitted:

tests/
  fixtures/
    base.fixture.lua            -- referenced as "fixtures/base"
    cluster.fixture.lua         -- referenced as "fixtures/cluster"
  uses-base.test.lua

-- in uses-base.test.lua:
local vm = provium:vm_fixture("fixtures/base")

If the same name resolves under multiple roots, the first-listed root wins. Missing fixtures error at restore time with fixture \X` not found in any test root`.

Two fixture shapes #

Single-VM fixture (vm_fixture) #

The chunk ends with return vm:snapshot():

-- tests/fixtures/base.fixture.lua
local vm = provium:vm("base", "peios"):boot()
vm:run("seq 1000000 > /srv/corpus"):assert_ok()
return vm:snapshot()

Cached as <key>.snap — a single sparse zstd-compressed file.

Lab fixture (lab_fixture) #

The chunk ends with return provium:snapshot():

-- tests/fixtures/cluster.fixture.lua
local lan = provium:bridge("lan")
local a = provium:vm("a", "peios"):boot()
local b = provium:vm("b", "peios"):boot()
lan:attach({a, b})
return provium:snapshot()

Cached as <key>.lab/ — a directory containing per-VM .snap files plus a lab.json index.

What ends up in the cache key #

The cache key is a SHA-256 of, in order:

1. The fixture file's source bytes.
2. Every transitively-referenced fixture's key (so vm_fixture("derived") calling vm_fixture("base") invalidates when base changes).
3. Every require()d helper's source bytes (recursively — helpers that require other helpers fold in too).
4. The kernel and initrd identifier of EVERY profile in provium.toml (sorted by name for determinism).
5. Every external host-file declared with vm:push_file("…", …) or lab:depends_on_file("…") (path + mtime + size). See External host-file deps.

Edit any of those, and the next run rebuilds the fixture. This is intentional:

• A fixture builder editing the file → rebuild.
• A helper module the fixture requires gets edited → rebuild.
• A different fixture the fixture references gets rebuilt → rebuild this one too.
• A new kernel image is dropped in → every fixture rebuilds.

Multi-profile cache-key folding means a kernel change on profile B invalidates fixtures even if they only ever boot under profile A. This is a deliberate over-invalidation: it's safer than serving a fixture built against a now-stale kernel.

External host-file deps #

Fixtures routinely push host-side files — a freshly-built binary, a config template, a test corpus — into the guest. The cache key folds those host files in automatically so editing the file on the host invalidates the snapshot.

vm:push_file(host_path, guest_path, opts?) #

Reads host_path on the host and writes its bytes to guest_path in the guest. The host file's identifier (path + mtime + size) is folded into the fixture's cache key by default — rebuild the binary, get a fresh fixture next run.

-- tests/fixtures/uapi-ready.fixture.lua
local vm = provium:vm("uapi", "peios"):boot()
vm:push_file("../peios-uapi/target/release/peios-uapi", "/usr/bin/peios-uapi")
vm:run("/usr/bin/peios-uapi --setup"):assert_ok()
return vm:snapshot()

Relative host_path is resolved against the directory containing the fixture file (or the directory of the require'd helper that calls push_file), not the cwd at invocation time. That keeps things stable regardless of where provium was launched from.

To opt out for a specific call — e.g. a large test corpus you don't want included in the key — pass auto_dep = false:

vm:push_file("../big-corpus.tar", "/data.tar", {auto_dep = false})

The opt-out is detected by static source scanning, so the auto_dep = false must be a literal in the call site. A variable like {auto_dep = x} does NOT opt out (the default is to track).

lab:depends_on_file(host_path) #

Declares an external host-file as a fixture-cache dependency without doing any I/O. Useful when the fixture reads the file on the host side — e.g. a config template parsed in Lua, or a binary the build script invokes locally before pushing post-processed output.

provium:depends_on_file("../templates/sshd_config.in")
local rendered = render_template("../templates/sshd_config.in", {port = 2222})
vm:write_file("/etc/ssh/sshd_config", rendered)
return vm:snapshot()

provium: is the root lab; sub-labs (provium:lab("dc1"):depends_on_file(...)) work too. All declarations on any lab in the fixture fold into the same fixture-level cache key.

Limits of static scanning #

The fold is driven by scanning the fixture and its helpers for literal-string arguments to push_file and depends_on_file. Two cases fall outside that:

• Variable host paths: vm:push_file(my_path, "/foo") doesn't fold (the scanner can't resolve my_path). Either inline the literal or follow up with an explicit lab:depends_on_file("…").
• Generated paths: vm:push_file("build/" .. arch, "/bin/foo") doesn't fold either.

The fix is the same in both cases: add a separate literal lab:depends_on_file("…") declaration alongside the dynamic call. If a fixture's deps genuinely can't be expressed as literals, fall back to provium fixture rebuild <name> after host-side changes.

Cache layout #

~/.cache/provium/fixtures/
  c5e6f8….snap            -- single-VM fixture snapshot (sparse, zstd)
  c5e6f8….lock            -- per-key build lock
  abc123….lab/            -- lab-fixture directory
    lab.json
    a.snap
    b.snap
  abc123….lab.lock

Override the cache directory in provium.toml:

[provium]
cache_dir = "/var/cache/provium/fixtures"

Default: ~/.cache/provium/fixtures/.

Build flow #

When a test calls provium:vm_fixture("name"):

flowchart TD
  Start[vm_fixture call] --> KeyComp[Compute key — source + deps + kernels]
  KeyComp --> Lookup{Key in cache?}
  Lookup -->|yes| Restore[Restore from cache]
  Lookup -->|no| Lock[Acquire build lock]
  Lock --> Recheck{Peer built it while we waited?}
  Recheck -->|yes| Restore
  Recheck -->|no| Build[Build under lock]
  Build --> Compress[Sparse + zstd]
  Compress --> Install[Atomic install]
  Install --> Restore
  Restore --> Return[Return VM]

Build under lock ensures only one process builds a given fixture even when 16 test files reference the same fixture. Other files queue on the lock and either restore from the freshly-installed cache (if the holder finished cleanly) or rebuild themselves (if the holder crashed).

fixture_build_started / fixture_build_done events frame each build. fixture_build_waiting fires when a file queues behind a peer; the payload includes held_by_file so dashboards can show "file X is waiting on file Y to build fixture Z."

Atomic install #

Lab fixtures use renameat2(RENAME_EXCHANGE) to swap the freshly-built <key>.lab/ with any existing one. If the kernel doesn't support the syscall, falls back to a sibling-rename + cleanup.

Single-VM fixtures use plain rename after sparse + zstd compression. Both paths leave the cache in a coherent state — readers either see the old version or the new, never half-installed.

Eviction #

LRU eviction runs at every provium startup, before tests dispatch:

1. Sum file sizes under cache_dir.
2. Sort entries by access time (atime).
3. Delete oldest until total is ≤ cache_max_size.

Default cache_max_size: 20 GiB. Override in provium.toml:

[provium]
cache_max_size = "100G"

Each successful restore bumps the entry's atime so frequently-used fixtures stay hot. Without the bump, relatime would let every entry's atime collapse together and eviction would degrade to filesystem order.

Corrupt entries #

If a restore fails (decompression error, version mismatch not caught by the key), the harness:

1. Prints a warning to stderr naming the fixture.
2. Evicts the entry.
3. Falls through to the rebuild path.

This is logged but not fatal — the next restore attempt builds fresh.

CLI management #

provium fixture list #

   23.45MiB  vm   c5e6f8…
   78.12MiB  lab  abc123…

2 entries, 101.57MiB

Each entry shows size, kind (vm or lab), and the cache key. Useful for "what's hot in my cache?"

provium fixture build <path> #

Force-build the named fixture. If the entry is already cached, prints already built: <path> and exits.

provium fixture build fixtures/base

Useful in CI to warm the cache before the test run.

provium fixture rebuild <path> #

Evict and rebuild. Useful when you know the fixture should change but the harness's cache key didn't catch it (rare):

provium fixture rebuild fixtures/base

provium fixture clean #

Wipe the entire cache directory:

provium fixture clean

The next run rebuilds everything from scratch. Slow but safe.

provium fixture stale #

List fixtures whose source / dep / kernel hash doesn't match any cached entry:

provium fixture stale
  tests/fixtures/base.fixture.lua

1 stale fixture(s)

Useful to check "what would the next provium run rebuild?" without running anything.

Performance notes #

Fixture restores are fast — typical single-VM snapshot restore is < 1 second on warm caches because:

1. The snapshot file is sparse-zstd compressed at build time, so reading and decompressing is dominated by kernel buffer cache hits.
2. The harness uses renameat2(RENAME_EXCHANGE) for atomic install (no observable in-between state for readers).
3. Each successful restore bumps atime, so popular fixtures stay LRU-hot.

Fixture builds are slow — they involve actually booting a VM, doing the setup, and snapshotting. Build them once per cache-key change and amortise across every test that references them.

Common patterns #

"Make a clean baseline available everywhere" #

-- tests/fixtures/clean.fixture.lua
local vm = provium:vm("clean", "peios"):boot()
return vm:snapshot()

Then every test starts from a fresh boot without paying the boot cost:

test("…", function(t)
    local vm = provium:vm_fixture("fixtures/clean")
    -- … fresh-boot semantics …
end)

"Stack fixtures to amortise expensive setup" #

-- tests/fixtures/with-corpus.fixture.lua
local vm = provium:vm_fixture("fixtures/clean")
vm:run("seq 1000000 > /srv/corpus"):assert_ok()
return vm:snapshot()

with-corpus is keyed off clean's key — when clean rebuilds, with-corpus rebuilds too. When clean is a cache hit, with-corpus only pays the corpus-build cost.

"Cache a multi-VM topology" #

-- tests/fixtures/two-node-cluster.fixture.lua
local lan = provium:bridge("lan")
local a, b = provium:vm("a", "peios"):boot(), provium:vm("b", "peios"):boot()
lan:attach({a, b})
a:run("ip addr add 10.0.0.1/24 dev eth0 && ip link set eth0 up")
b:run("ip addr add 10.0.0.2/24 dev eth0 && ip link set eth0 up")
return provium:snapshot()

test("…", function(t)
    local cluster = provium:lab_fixture("fixtures/two-node-cluster")
    cluster.a:run("ping -c 1 -W 1 10.0.0.2"):assert_ok()
end)

The whole topology — bridge, both VMs, partition state, attached disks, everything — is one cached unit.

"Pre-warm the cache in CI" #

Add to your CI script:

provium fixture build fixtures/clean
provium fixture build fixtures/with-corpus
provium fixture build fixtures/two-node-cluster
provium tests/  # now every test sees cache hits

Without pre-warming, the first test to reference each fixture pays the build cost serially.

What can invalidate the cache #

Adding a new profile invalidates the cache (the new profile's kernel/initrd are folded in even if no test uses the new profile). This is by design.

See also #

• Lab reference — vm_fixture, lab_fixture.
• Snapshot reference — what fixture builders return.
• provium.toml reference — cache_dir, cache_max_size.
• The CLI — provium fixture … subcommands.

Provium's dispatcher runs multiple test files in parallel against a shared resource pool. Each file declares what it needs (or accepts the per-file overhead default), and the dispatcher schedules as many as fit. This page covers how to think about that for your test corpus.

The pool #

When provium starts, it builds one pool with two budgets:

The pool tracks total and available; every dispatch takes from available, every release returns. A pool_state event fires every second with {used, available} so dashboards see live utilisation.

If you want to be conservative on a busy host:

provium tests/ --mem 8G --cpus 4

If you have a beefy CI box and want maximum throughput:

provium tests/ --mem 64G --cpus 32 --cpu-overcommit 1.5

--cpu-overcommit is clamped to [0.5, 8.0]. Default 1.0 (strict — no oversubscription). 1.5 or 2.0 allows oversubscription if the workload tolerates scheduling jitter.

Per-file claims #

Each test file may declare what it needs:

provium:claim({memory = "4G", cpus = 4})

test("…", function(t) end)
test("…", function(t) end)

The claim is taken at file dispatch and released at file completion. A second :claim errors — lab claim already held; one-shot per lab.

Without a claim, the dispatcher uses the per-file overhead default (50 MiB memory, 0 CPU). A file that boots 3 VMs at 1 GiB each but doesn't claim anything gets through the pool gate immediately and then might OOM the host because the overhead default is way under the actual demand.

Rule of thumb: claim memory equal to the sum of expected VM memory budgets plus a small buffer; claim CPUs equal to the sum of expected VM vCPU budgets.

-- 3 VMs × (2G memory, 2 CPUs each):
provium:claim({memory = "7G", cpus = 7})  -- 6G VMs + 1G buffer; 6 vCPUs + 1

File dispatch flow #

flowchart LR
  Disc[file_discovered] --> Wait{pool has room?}
  Wait -->|no| Block[file_blocked]
  Block --> Wait
  Wait -->|yes| Disp[file_dispatched]
  Disp --> Acq[claim_acquired]
  Acq --> Run[Run tests in file]
  Run --> Comp[file_completed]
  Comp --> Rel[claim_released]

Files queue at the pool until their reservation fits. The order is stable — the dispatcher takes files in discovery order, so a file that doesn't fit blocks subsequent files (FIFO). For now there's no priority or backfill; if you need a specific file to run first, list its path explicitly.

PSI throttling #

On Linux hosts with pressure stall information (PSI) available, Provium spawns a pressure monitor at startup. It polls CPU and memory pressure once a second; when either crosses the threshold (10 % some-pressure averaged over 10 seconds), the dispatcher pauses new file dispatches until pressure drops.

The threshold and poll interval are fixed — they are not configurable from the CLI.

Files that are already in flight continue. The throttling only delays new dispatch.

When PSI throttling is the cause of a file_blocked, the event's reason field is psi_pressure (versus pool_full for "the pool can't afford this file's claim").

KSM tuning #

Kernel Same-page Merging deduplicates identical pages across VMs. When you boot many VMs from the same kernel + initrd, KSM can reclaim significant memory. Provium tunes /sys/kernel/mm/ksm/* at startup unless --no-ksm is passed.

The KSM tuning runs once per provium invocation, setting run = 1, pages_to_scan = 1000, and sleep_millisecs = 20. Best-effort: a one-line summary goes to stderr.

provium: ksm: tuned (3 knobs)

If your host is shared with non-Provium workloads, pass --no-ksm so Provium doesn't change global tuning.

What blocks parallelism #

Several things prevent unlimited parallelism even when the pool has room:

Of these, fixture build lock is the most common surprise. If 16 test files all reference fixtures/base and the cache is cold, all 16 queue on the build lock; only one builds, the rest wait. Pre-warm with provium fixture build fixtures/base to amortise.

Inspecting parallelism in a run #

The event stream gives you the full picture:

provium tests/ --save-events events.msgpack

Then walk the stream:

• file_discovered events at the start tell you the universe.
• file_dispatched events tell you what actually ran in parallel (count of in-flight = file_dispatched - file_completed).
• file_blocked events with reason tell you why something queued.
• pool_state events at 1 Hz give you a usage timeline.

provium-coverage summarises this in its run report; for ad-hoc debugging, parse the msgpack with the snippet in events and coverage.

Tuning patterns #

"I want to maximise throughput" #

provium tests/ --cpus $(nproc) --mem $(awk '/MemTotal/ {printf "%dG", $2/1024/1024 - 4}' /proc/meminfo)

Use everything except 4 GiB of RAM and overcommit CPUs gently:

provium tests/ --cpu-overcommit 1.5

"I want to be conservative on a shared host" #

provium tests/ --cpus 4 --mem 8G --no-ksm

"I want to detect over-subscription" #

Watch the pool_state and file_blocked event streams. Frequent file_blocked with reason = pool_full means files are claiming more than the pool can serve in parallel — either the pool is too small or claims are too generous. Frequent file_blocked with reason = psi_pressure means the host is genuinely overloaded — reduce parallelism (--cpus, --mem) or move other workloads off the host.

"I want to debug a slow run" #

provium tests/ --save-events events.msgpack

Parse the msgpack for the longest file_completed.duration_ns. Then look at that file's test_started / test_passed events to see which test(s) are slow. Cross-reference with vm_spawned events to see how many VMs were involved.

"I want to test for resource leaks" #

The pool's available budget should return to its full value after each claim_released. Watch pool_state over time — if available is drifting down monotonically, something is leaking. The most common culprits:

• A test that exits via panic without releasing a claim — but the dispatcher releases on file_completed regardless of how it ended, so this shouldn't happen.
• A bug in the harness — file an issue with the event stream.

VMs vs files #

A common confusion: the pool tracks per-file resources, not per-VM. The dispatcher reserves the file's full claim at dispatch and holds it until file completion, regardless of how many VMs the file actually boots concurrently.

-- This file claims 4G, but only ever has one VM live at a time:
provium:claim({memory = "4G"})

test("a", function(t)
    local vm = provium:vm("v", "peios"):boot()
    vm:shutdown()  -- VM gone, but claim still held
end)

test("b", function(t)
    local vm = provium:vm("v", "peios"):boot()
end)

The claim doesn't release between tests. If you want fine-grained reservation, you'd need a smaller claim and rely on the dispatcher's per-file overhead — but the trade-off is potential OOM if the claim is too small for the actual peak.

For the typical case, claim for the file's worst-case peak.

See also #

• The CLI — --mem, --cpus, --cpu-overcommit, --no-ksm.
• Lab reference — lab:claim.
• Events — pool_state, file_blocked, claim_acquired, claim_released.
• Fixtures and dependencies — fixture build lock as a parallelism limiter.