# Running tests --- # Running tests with the CLI _Provium / Running tests_ > Patterns for the provium binary — discovery, filtering, the watch loop, --rerun-failed, --since, --tag, --include-slow, JSON output, and exit codes for CI. The `provium` binary is what you run. This page covers the day-to-day patterns. The full flag inventory is on [the CLI reference](/provium/reference/cli.md). ## The basic invocation ``` provium tests/ ``` Walks `tests/` for `*.test.lua` files and runs each. Default output: ``` PASS tests/smoke.test.lua (3 passed, 0 failed, 0 skipped) FAIL tests/networking/partition.test.lua (2 passed, 1 failed, 0 skipped) FAIL split-brain client A could not reach client B after partition 2 file(s); 5 passed, 1 failed, 0 skipped; 12.34s ``` A single file works too: ``` provium tests/networking/partition.test.lua ``` If you omit paths entirely, the current directory is scanned recursively. ## Selecting what runs ### `--filter ` Match against the test-root-relative path: ``` provium tests/ --filter networking provium tests/ --filter partition.test ``` ### `--rerun-failed` After every run with at least one failure, Provium writes the canonical paths to `~/.cache/provium/rerun.json`. `--rerun-failed` intersects discovered files against this list: ``` provium tests/ --rerun-failed ``` If no prior failure state exists, `--rerun-failed` exits cleanly with a notice (running the FULL suite is almost never what the user wanted). A clean run leaves the failed-state file alone, so the failed set persists until you fix the failures. ### `--since ` Run only files whose mtime is newer than the reference file: ``` provium tests/ --since main.lua # files edited after main.lua provium tests/ --since .last-checkpoint # custom marker ``` Useful for "what's changed since I last ran". ### Per-test filters Tag and slow filtering happen inside each file (the test body's `meta` is what's matched): ``` provium tests/ --tag smoke provium tests/ --tag smoke --tag fast # OR: any tag matches provium tests/ --no-tag flaky # exclude tests tagged flaky provium tests/ --include-slow # include meta.slow tests ``` `--no-tag` wins over `--tag` (intersect semantics). For filtering on arbitrary meta fields (not just `tags`), use `--tag-meta KEY=VALUE`: ``` provium tests/ --tag-meta subsystems=peinit provium tests/ --tag-meta subsystems=peinit --tag-meta subsystems=loregd # OR within KEY provium tests/ --tag-meta subsystems=peinit --tag-meta area=boot # AND across KEYs provium tests/ --no-tag-meta flaky=true ``` This is the recommended pattern when test files are organised by user-visible scenario rather than by code subsystem — annotate tests with `subsystems = {"peinit", "loregd"}` (or any other axis you care about) and let CI compute "what to run" from the changeset: ```bash # CI: PR touched the peinit subsystem provium tests/ --tag-meta subsystems=peinit ``` See [meta tags](/provium/reference/meta-tags.md) for the convention. Skipped tests still appear in outcomes — you'll see `SKIP ` in `-v` mode, with the filter expression as the reason. ## Output modes ### Default (compact) ``` FAIL tests/x.test.lua (2 passed, 1 failed, 0 skipped) FAIL the failing case message lines ``` ### Verbose (`-v`) Every test status, not just failures: ``` PASS tests/x.test.lua (3 passed, 0 failed, 0 skipped) PASS first PASS second PASS third ``` ### Quiet (`-q`) Only failures show. Files with all passing / skipped are silent. The summary line still prints. ### JSON (`--json`) Line-delimited JSON, one object per file: ```json {"path":"tests/x.test.lua","timeout":"in_time","passed":true,"chunk_error":null,"tests":[{"name":"…","status":"passed","message":null,"log":[]}]} ``` Useful for piping into another tool. Mutually exclusive with `--events-stdout`. ### Msgpack events (`--events-stdout`) Length-prefixed msgpack frames over stdout. The human-readable / JSON renderer redirects to stderr automatically. Useful with `provium-coverage`: ``` provium tests/ --events-stdout | provium-coverage ``` See [events and coverage](/provium/running-tests/events-and-coverage.md) for the full event-stream story. ## Watch mode ``` provium tests/ --watch ``` Polls every 500 ms for stat-changed files; re-runs when one changes. Use Ctrl-C to exit. The empty-set case is fine — the watcher rescans every tick, so dropping a `*.test.lua` into the watched root after launch triggers discovery. `--rerun-failed` is automatically dropped under `--watch` so file-edit detection works as expected (otherwise each tick would re-run only the original failed set forever). ## Per-file timeouts ``` provium tests/ --timeout 300 # 5 min (default) provium tests/ --timeout 30s provium tests/ --timeout 10m provium tests/ --timeout 500ms provium tests/ --timeout 0 # disable ``` Bare integers are seconds. Suffixes accepted: `ms`, `s`, `m`, `h`. `0` disables the timeout entirely. When a file's timeout fires, the harness records it as `timed_out`, marks the file as failed for exit-code purposes, and emits a `file_completed` event with `status = "timed_out"`. ## Pool and CPU controls By default Provium uses 80 % of host RAM and the full host CPU count for the resource pool. Override: ``` provium tests/ --mem 16G # explicit pool memory provium tests/ --cpus 8 # explicit pool vCPUs provium tests/ --cpu-overcommit 1.5 # multiplier on --cpus ``` See [pools and parallelism](/provium/running-tests/pools-and-parallelism.md) for how the pool, claims, and dispatcher interact — including when overcommit is safe. Flag defaults and the clamp range are in the [CLI reference](/provium/reference/cli.md#vmm-and-resources). ## Dev-mode flags ### `--no-preflight` Skip the `/dev/kvm` / iproute2 / nft / qemu / `CAP_NET_ADMIN` checks. Use in containers or CI environments where you know the environment is fine but the checks would fail. ### `--no-ksm` Skip Kernel Same-page Merging tuning at startup. Default tunes `/sys/kernel/mm/ksm/*` per the design's pool-density goals; pass this on shared dev hosts where you don't want global tuning. ### `--vmm local` Use the in-process LocalAgent backend instead of QEMU. Useful when KVM isn't available (CI, dev-on-laptop). The local backend does not actually boot a kernel — it gives the host bindings something to dispatch against, but tests that rely on guest-side behaviour (running commands, file I/O) won't work. ``` provium tests/ --vmm local ``` ## Subcommands When a subcommand is given, the test-runner mode is suppressed. ### `provium repl ` Boot a VM and drop into an interactive Lua REPL: ``` provium repl peios # cold-boot the peios profile provium repl peios --name dev # custom VM name provium repl --fixture base # resume from a fixture ``` Useful for poking at a guest interactively. The full Provium API is available — `vm:run`, `vm:read_file`, `vm:tail_file`, etc. ### `provium fixture ` Manage the fixture cache: ``` provium fixture list # show every cached entry provium fixture build path/to/fixture # force-build provium fixture rebuild path/to/fixture # evict and rebuild provium fixture clean # wipe the cache provium fixture stale # list fixtures whose source doesn't match any cache entry ``` See [fixtures and dependencies](/provium/running-tests/fixtures-and-dependencies.md) for when each is useful. ### `provium prepare [profile]` Run dynamic profiles' `build` commands without booting anything — one profile by name, or every profile that declares a `build`. Pre-warm images with `provium prepare && provium tests/ --no-build`; see [dynamic profiles](/provium/configuration/dynamic-profiles.md). ### `provium list` List discovered tests / fixtures without running anything: ``` provium list # tests provium list --fixtures # fixtures ``` ### `provium lsp-setup [dir]` Write Lua Language Server stubs and a `.luarc.json` into a test directory so the harness globals resolve in your editor — see [the CLI reference](/provium/reference/cli.md#provium-lsp-setup-dir---force). ## Exit codes For CI, three values: `0` means every file passed (or skipped), `1` means at least one file did not finish cleanly, `2` means an internal error (config, pre-flight, dispatcher). The exit code is deliberately clamped to those three — the precise triggers and the clamping rationale are in the [CLI reference](/provium/reference/cli.md#exit-codes). ## CI patterns ### "Run everything, fail on any failure" ``` #!/bin/sh provium tests/ ``` Exit code is your test result. Capture stdout for the summary, stderr for the human-readable lines (when `--json` is set). ### "Run with structured output" ``` provium tests/ --json | tee results.jsonl ``` Each line is a complete `{"path":…,"tests":[…]}` object. Easy to pipe into a custom dashboard or per-test reporter. ### "Save events for post-hoc analysis" ``` provium tests/ --save-events events.msgpack provium-coverage --from events.msgpack > coverage.html ``` The msgpack file is portable; you can analyse it on a different host. ### "Fail fast" ``` provium tests/ --fail-fast ``` Stops after the first failed file. Useful for tight inner loops where you want to see the first failure quickly. ### "Slow CI vs fast CI" ``` # Fast: skip slow tests by default. provium tests/ # Slow / nightly: run everything. provium tests/ --include-slow ``` `meta.slow = true` tests are skipped by default. Tag them in the test bodies, and your dev loop stays fast while nightly catches the slow stuff. ### "Parallel CI sharding" There's no built-in shard splitter, but `provium` accepts any number of path arguments, so split the file list in your CI script and pass each shard its own files: ``` # $SHARDS = total shard count, $INDEX = this shard (0-based). git ls-files 'tests/**/*.test.lua' | awk "NR % $SHARDS == $INDEX" | xargs provium ``` Each shard runs a disjoint subset of files. Don't try to build a shard with `--filter` — it's a single substring match against the relative path, not a pattern list. ## See also - [CLI reference](/provium/reference/cli.md) — every flag, every env var. - [Events and coverage](/provium/running-tests/events-and-coverage.md) — `--save-events`, `--events-stdout`, `--coverage`. - [Pools and parallelism](/provium/running-tests/pools-and-parallelism.md) — how the resource pool works. --- # Events and coverage _Provium / Running tests_ > How to capture Provium's structured event stream — to a file, to stdout, or over a Unix socket — and feed it to provium-coverage or your own consumer. Provium emits a structured msgpack event stream covering every file, test, VM, fixture, and pool transition. This page covers the practical side of consuming it; the wire format is on [events](/provium/reference/events.md). ## Why events The human-readable `PASS / FAIL` summary is fine for the dev loop, but not enough for: - Coverage reports (which test exercised which spec section?). - Dashboards (live view of pool state, in-flight files, fixture builds). - Post-hoc analysis ("why did this CI run take 4× longer than yesterday?"). - Debugging fixture rebuilds (which file is waiting on which build lock?). For all of these, the event stream is what you want. ## Output channels Provium can route events to three places independently. Combine flags freely. ### `--save-events ` — to a file ``` provium tests/ --save-events events.msgpack ``` Writes length-prefixed msgpack frames to the file. Use for post-hoc analysis or replay. Under `--watch`, the file is truncated at each iteration so it contains exactly the most recent run. ### `--events-stdout` — to stdout ``` provium tests/ --events-stdout | provium-coverage ``` Streams events on stdout. The human-readable / JSON renderer redirects to stderr so consumers piping `provium --events-stdout | …` don't see human text interleaved into their msgpack parser. (`provium-coverage` reads stdin when `--from` is omitted, so the bare pipe above is the whole pipeline.) Mutually exclusive with `--json`. ### `--events-socket ` — over a Unix socket ``` provium tests/ --watch --events-socket /tmp/provium.sock # in another terminal — any client that can read a Unix socket: socat - UNIX-CONNECT:/tmp/provium.sock | provium-coverage ``` The binary listens, accepts connections, and fans out frames. Useful for live dashboards. The socket persists across `--watch` iterations so connected clients aren't dropped on every re-run. ## Coverage post-run `--coverage` is sugar over `--save-events` + a post-run pipe to `provium-coverage`: ``` provium tests/ --coverage ``` Internally: 1. If `--save-events ` is also set, Provium uses that file. 2. Otherwise, Provium tees events to a scratch tempfile (`$TMPDIR/provium-coverage-.msgpack`) with a sibling marker file. 3. After the run, runs `provium-coverage --from `. 4. Cleans up the tempfile if Provium owns it (the marker is the proof). Failures from the post-run pipe propagate as the exit code — a coverage failure shows up as exit `1`. The cleanup is robust: SIGINT / SIGTERM handlers and an `atexit` hook clear the tempfile even on a killed run, so CI won't accumulate orphaned msgpack files. ## What's in the stream The stream covers five areas: file lifecycle (`file_discovered` → `file_dispatched` → `file_completed`, with `file_blocked` when a file waits on the pool), test outcomes (`test_started` then one of `test_passed` / `test_failed` / `test_skipped`), VM lifecycle (`vm_spawned` / `vm_shutdown`), pool and claim activity, and fixture builds (`fixture_build_started` / `_done` / `_waiting` and `fixture_cache_hit`). Every frame is `{ts, kind, payload}`. The full variant list and every payload field are in the [events reference](/provium/reference/events.md). ## Replay and analysis ``` provium tests/ --save-events events.msgpack provium-coverage --from events.msgpack ``` The msgpack file is portable. You can analyse it on a different host than the one that produced it, or run multiple consumers against the same file: ``` provium-coverage --from events.msgpack > coverage.html my-custom-tool --from events.msgpack > metrics.json ``` ## Building your own consumer The protocol is `provium-protocol`'s `EventFrame` over length-prefixed msgpack frames (the same framing as the agent wire protocol). The framing helpers in `provium-protocol::frame` are exposed for reuse. A minimal Rust consumer: ```rust use provium_protocol::events::{Event, EventFrame}; use provium_protocol::frame::{read_frame, DEFAULT_MAX_FRAME_BYTES}; use std::fs::File; use std::io::BufReader; fn main() -> std::io::Result<()> { let f = File::open("events.msgpack")?; let mut r = BufReader::new(f); loop { let frame: EventFrame = match read_frame(&mut r, DEFAULT_MAX_FRAME_BYTES) { Ok(f) => f, Err(_) => break, // EOF or malformed; production code should distinguish }; match frame.event { Event::TestPassed(t) => println!("PASS {}::{}", t.path, t.name), Event::TestFailed(t) => println!("FAIL {}::{} — {}", t.path, t.name, t.reason), _ => {} } } Ok(()) } ``` For other languages, use any msgpack library that handles the length-prefix framing (e.g. read 4 bytes BE length, then read N bytes, then deserialise as msgpack). ## Live multiplexing for dashboards `--events-socket` is the right shape when you have a long-running dashboard: ``` # Provium runs in watch mode; events flow over the socket. provium tests/ --watch --events-socket /tmp/provium.sock # Dashboard connects once and stays connected across reruns. my-dashboard --from-socket /tmp/provium.sock ``` The socket persists across `--watch` iterations precisely so dashboards don't have to reconnect on every re-run. For non-dashboard pipelines (CI feeding a coverage tool), prefer `--events-stdout` — simpler, no socket-file management. ## Observability event guarantees The harness guarantees ordering and pairing invariants (exactly one `file_completed` per dispatched file, exactly one terminal event per started test, paired claim and fixture-build events) — the full list is in [events reference — Event guarantees](/provium/reference/events.md#event-guarantees). Consumers can rely on these to track in-flight state, e.g. counting "in-flight files" as `file_dispatched - file_completed`. ## Practical patterns ### "Why did this run take so long?" ``` provium tests/ --save-events events.msgpack # parse events.msgpack to find the longest file_completed.duration_ns ``` ### "Which fixtures rebuilt?" ``` provium tests/ --save-events events.msgpack # count fixture_build_started events ``` If many fixtures are rebuilding, something invalidated the cache — usually a kernel swap or a helper edit. ### "Spec coverage report" ``` provium tests/ --include-slow --coverage ``` `provium-coverage` reads `meta.spec` from each `test_passed` / `test_failed` event and produces a spec-section report. ### "What was the guest doing when this test failed?" The `test_failed` event includes `console_excerpt` — the last 4 KiB of every booted VM's console log captured at the failure moment. Useful when failures are kernel-side and reproducing locally is expensive. ## See also - [Events reference](/provium/reference/events.md) — every event, every field. - [Protocol version](/provium/reference/protocol-version.md) — wire-shape pinning. - [The CLI](/provium/running-tests/the-cli.md) — `--save-events`, `--events-stdout`, `--events-socket`, `--coverage`. --- # Fixtures and dependencies _Provium / Running tests_ > How fixtures are discovered, cached, keyed, locked, and evicted. When a fixture rebuilds. How to manage the cache from the CLI. Fixtures are pre-built VM (or whole-lab) snapshots that test files restore from instead of re-doing setup work. This page covers the cache model end-to-end. The exhaustive method reference is on [Lab](/provium/reference/lab.md) (`vm_fixture`, `lab_fixture`); the userdata is on [Snapshot](/provium/reference/snapshot.md). ## Discovery A fixture is any `*.fixture.lua` file under one of the `roots` directories listed in `provium.toml`. Test files reference fixtures by their test-root-relative path with the `.fixture.lua` suffix omitted: ``` tests/ fixtures/ base.fixture.lua -- referenced as "fixtures/base" cluster.fixture.lua -- referenced as "fixtures/cluster" uses-base.test.lua ``` ```lua -- in uses-base.test.lua: local vm = provium:vm_fixture("fixtures/base") ``` If the same name resolves under multiple roots, the first-listed root wins. Missing fixtures error at restore time with `fixture \`X\` not found in any test root`. ## Two fixture shapes ### Single-VM fixture (`vm_fixture`) The chunk ends with `return vm:snapshot()`: ```lua -- tests/fixtures/base.fixture.lua local vm = provium:vm("base", "peios"):boot() vm:run("seq 1000000 > /srv/corpus"):assert_ok() return vm:snapshot() ``` Cached as `.snap` — a single sparse zstd-compressed file. ### Lab fixture (`lab_fixture`) The chunk ends with `return provium:snapshot()`: ```lua -- tests/fixtures/cluster.fixture.lua local lan = provium:bridge("lan") local a = provium:vm("a", "peios"):boot() local b = provium:vm("b", "peios"):boot() lan:attach({a, b}) return provium:snapshot() ``` Cached as `.lab/` — a directory containing per-VM `.snap` files plus a `lab.json` index. ## What ends up in the cache key The cache key is a SHA-256 of, in order: 1. The fixture file's source bytes. 2. Every transitively-referenced fixture's key (so `vm_fixture("derived")` calling `vm_fixture("base")` invalidates when `base` changes). 3. Every `require()`d helper's source bytes (recursively — helpers that require other helpers fold in too). 4. The kernel and initrd identifier of EVERY profile in `provium.toml` (sorted by name for determinism). 5. Every external host-file declared with `vm:push_file("…", …)` or `lab:depends_on_file("…")` (path + mtime + size). See [External host-file deps](#external-host-file-deps). Edit any of those, and the next run rebuilds the fixture. This is intentional: - A fixture builder editing the file → rebuild. - A helper module the fixture requires gets edited → rebuild. - A different fixture the fixture references gets rebuilt → rebuild this one too. - A new kernel image is dropped in → every fixture rebuilds. Multi-profile cache-key folding means a kernel change on profile B invalidates fixtures even if they only ever boot under profile A. This is a deliberate over-invalidation: it's safer than serving a fixture built against a now-stale kernel. ## External host-file deps Fixtures routinely push host-side files — a freshly-built binary, a config template, a test corpus — into the guest. The cache key folds those host files in automatically so editing the file on the host invalidates the snapshot. ### `vm:push_file(host_path, guest_path, opts?)` Reads `host_path` on the host and writes its bytes to `guest_path` in the guest. The host file's identifier (path + mtime + size) is folded into the fixture's cache key by default — rebuild the binary, get a fresh fixture next run. ```lua -- tests/fixtures/uapi-ready.fixture.lua local vm = provium:vm("uapi", "peios"):boot() vm:push_file("../peios-uapi/target/release/peios-uapi", "/usr/bin/peios-uapi") vm:run("/usr/bin/peios-uapi --setup"):assert_ok() return vm:snapshot() ``` Relative `host_path` is resolved against the directory containing the fixture file (or the directory of the require'd helper that calls `push_file`), not the cwd at invocation time. That keeps things stable regardless of where `provium` was launched from. To opt out for a specific call — e.g. a large test corpus you don't want included in the key — pass `auto_dep = false`: ```lua vm:push_file("../big-corpus.tar", "/data.tar", {auto_dep = false}) ``` The opt-out is detected by static source scanning, so the `auto_dep = false` must be a literal in the call site. A variable like `{auto_dep = x}` does NOT opt out (the default is to track). ### `lab:depends_on_file(host_path)` Declares an external host-file as a fixture-cache dependency without doing any I/O. Useful when the fixture reads the file on the host side — e.g. a config template parsed in Lua, or a binary the build script invokes locally before pushing post-processed output. ```lua provium:depends_on_file("../templates/sshd_config.in") local rendered = render_template("../templates/sshd_config.in", {port = 2222}) vm:write_file("/etc/ssh/sshd_config", rendered) return vm:snapshot() ``` `provium:` is the root lab; sub-labs (`provium:lab("dc1"):depends_on_file(...)`) work too. All declarations on any lab in the fixture fold into the same fixture-level cache key. ### Limits of static scanning The fold is driven by scanning the fixture and its helpers for literal-string arguments to `push_file` and `depends_on_file`. Two cases fall outside that: - **Variable host paths**: `vm:push_file(my_path, "/foo")` doesn't fold (the scanner can't resolve `my_path`). Either inline the literal or follow up with an explicit `lab:depends_on_file("…")`. - **Generated paths**: `vm:push_file("build/" .. arch, "/bin/foo")` doesn't fold either. The fix is the same in both cases: add a separate literal `lab:depends_on_file("…")` declaration alongside the dynamic call. If a fixture's deps genuinely can't be expressed as literals, fall back to `provium fixture rebuild ` after host-side changes. ## Cache layout ``` ~/.cache/provium/fixtures/ c5e6f8….snap -- single-VM fixture snapshot (sparse, zstd) c5e6f8….lock -- per-key build lock abc123….lab/ -- lab-fixture directory lab.json a.snap b.snap abc123….lab.lock ``` Override the cache directory in `provium.toml`: ```toml [provium] cache_dir = "/var/cache/provium/fixtures" ``` Default: `~/.cache/provium/fixtures/`. ## Build flow When a test calls `provium:vm_fixture("name")`: ```mermaid flowchart TD Start[vm_fixture call] --> KeyComp[Compute key — source + deps + kernels] KeyComp --> Lookup{Key in cache?} Lookup -->|yes| Restore[Restore from cache] Lookup -->|no| Lock[Acquire build lock] Lock --> Recheck{Peer built it while we waited?} Recheck -->|yes| Restore Recheck -->|no| Build[Build under lock] Build --> Compress[Sparse + zstd] Compress --> Install[Atomic install] Install --> Restore Restore --> Return[Return VM] ``` Build under lock ensures only one process builds a given fixture even when 16 test files reference the same fixture. Other files queue on the lock and either restore from the freshly-installed cache (if the holder finished cleanly) or rebuild themselves (if the holder crashed). `fixture_build_started` / `fixture_build_done` events frame each build. `fixture_build_waiting` fires when a file queues behind a peer; the payload includes `held_by_file` so dashboards can show "file X is waiting on file Y to build fixture Z." ## Atomic install Lab fixtures use `renameat2(RENAME_EXCHANGE)` to swap the freshly-built `.lab/` with any existing one. If the kernel doesn't support the syscall, falls back to a sibling-rename + cleanup. Single-VM fixtures use plain `rename` after sparse + zstd compression. Both paths leave the cache in a coherent state — readers either see the old version or the new, never half-installed. ## Eviction LRU eviction runs at every `provium` startup, before tests dispatch: 1. Sum file sizes under `cache_dir`. 2. Sort entries by access time (atime). 3. Delete oldest until total is ≤ `cache_max_size`. Default `cache_max_size`: `20 GiB`. Override in `provium.toml`: ```toml [provium] cache_max_size = "100G" ``` Each successful restore bumps the entry's atime so frequently-used fixtures stay hot. Without the bump, relatime would let every entry's atime collapse together and eviction would degrade to filesystem order. ## Corrupt entries If a restore fails (decompression error, version mismatch not caught by the key), the harness: 1. Prints a warning to stderr naming the fixture. 2. Evicts the entry. 3. Falls through to the rebuild path. This is logged but not fatal — the next restore attempt builds fresh. ## CLI management ### `provium fixture list` ``` 23.45MiB vm c5e6f8… 78.12MiB lab abc123… 2 entries, 101.57MiB ``` Each entry shows size, kind (`vm` or `lab`), and the cache key. Useful for "what's hot in my cache?" ### `provium fixture build ` Force-build the named fixture. If the entry is already cached, prints `already built: ` and exits. ``` provium fixture build fixtures/base ``` Useful in CI to warm the cache before the test run. ### `provium fixture rebuild ` Evict and rebuild. Useful when you know the fixture should change but the harness's cache key didn't catch it (rare): ``` provium fixture rebuild fixtures/base ``` ### `provium fixture clean` Wipe the entire cache directory: ``` provium fixture clean ``` The next run rebuilds everything from scratch. Slow but safe. ### `provium fixture stale` List fixtures whose source / dep / kernel hash doesn't match any cached entry: ``` provium fixture stale tests/fixtures/base.fixture.lua 1 stale fixture(s) ``` Useful to check "what would the next `provium` run rebuild?" without running anything. ## Performance notes Fixture restores are fast — typical single-VM snapshot restore is < 1 second on warm caches because: 1. The snapshot file is sparse-zstd compressed at build time, so reading and decompressing is dominated by kernel buffer cache hits. 2. The harness uses `renameat2(RENAME_EXCHANGE)` for atomic install (no observable in-between state for readers). 3. Each successful restore bumps atime, so popular fixtures stay LRU-hot. Fixture builds are slow — they involve actually booting a VM, doing the setup, and snapshotting. Build them once per cache-key change and amortise across every test that references them. ## Common patterns ### "Make a clean baseline available everywhere" ```lua -- tests/fixtures/clean.fixture.lua local vm = provium:vm("clean", "peios"):boot() return vm:snapshot() ``` Then every test starts from a fresh boot without paying the boot cost: ```lua test("…", function(t) local vm = provium:vm_fixture("fixtures/clean") -- … fresh-boot semantics … end) ``` ### "Stack fixtures to amortise expensive setup" ```lua -- tests/fixtures/with-corpus.fixture.lua local vm = provium:vm_fixture("fixtures/clean") vm:run("seq 1000000 > /srv/corpus"):assert_ok() return vm:snapshot() ``` `with-corpus` is keyed off `clean`'s key — when `clean` rebuilds, `with-corpus` rebuilds too. When `clean` is a cache hit, `with-corpus` only pays the corpus-build cost. ### "Cache a multi-VM topology" ```lua -- tests/fixtures/two-node-cluster.fixture.lua local lan = provium:bridge("lan") local a, b = provium:vm("a", "peios"):boot(), provium:vm("b", "peios"):boot() lan:attach({a, b}) a:run("ip addr add 10.0.0.1/24 dev eth0 && ip link set eth0 up") b:run("ip addr add 10.0.0.2/24 dev eth0 && ip link set eth0 up") return provium:snapshot() ``` ```lua test("…", function(t) local cluster = provium:lab_fixture("fixtures/two-node-cluster") cluster.a:run("ping -c 1 -W 1 10.0.0.2"):assert_ok() end) ``` The whole topology — bridge, both VMs, partition state, attached disks, everything — is one cached unit. ### "Pre-warm the cache in CI" Add to your CI script: ```sh provium fixture build fixtures/clean provium fixture build fixtures/with-corpus provium fixture build fixtures/two-node-cluster provium tests/ # now every test sees cache hits ``` Without pre-warming, the first test to reference each fixture pays the build cost serially. ## What can invalidate the cache | Change | Invalidates | |---|---| | Edit `.fixture.lua` | That fixture only. | | Edit a helper that the fixture `require`s | The fixture and every fixture that references it. | | Edit a fixture that another fixture references via `vm_fixture`/`lab_fixture` | Both. | | New kernel or initrd image (any profile) | Every fixture. | | Change `[profiles.].kernel` or `.initrd` path | Every fixture. | | Edit a file declared with `vm:push_file` or `lab:depends_on_file` | The fixture (and any fixture that references it). | | `provium fixture rebuild` / `clean` | Per command. | Adding a new profile invalidates the cache (the new profile's kernel/initrd are folded in even if no test uses the new profile). This is by design. ## See also - [Lab reference](/provium/reference/lab.md) — `vm_fixture`, `lab_fixture`. - [Snapshot reference](/provium/reference/snapshot.md) — what fixture builders return. - [provium.toml reference](/provium/configuration/provium-toml.md) — `cache_dir`, `cache_max_size`. - [The CLI](/provium/running-tests/the-cli.md) — `provium fixture …` subcommands. --- # Pools and parallelism _Provium / Running tests_ > How the resource pool, per-file claims, PSI throttling, and CPU overcommit shape parallel test execution. How to tune for your hardware. Provium's dispatcher runs multiple test files in parallel against a shared resource pool. Each file declares what it needs (or accepts the per-file overhead default), and the dispatcher schedules as many as fit. This page covers how to think about that for your test corpus. ## The pool When `provium` starts, it builds one pool with two budgets: | Budget | Default | Override | |---|---|---| | Memory | 80 % of host RAM | `--mem 16G` | | vCPUs | host online CPUs | `--cpus 8` (then `--cpu-overcommit `) | The pool tracks total and available; every dispatch takes from available, every release returns. A `pool_state` event fires every second with `{used, available}` so dashboards see live utilisation. If you want to be conservative on a busy host: ``` provium tests/ --mem 8G --cpus 4 ``` If you have a beefy CI box and want maximum throughput: ``` provium tests/ --mem 64G --cpus 32 --cpu-overcommit 1.5 ``` `--cpu-overcommit` is clamped to `[0.5, 8.0]`. Default `1.0` (strict — no oversubscription). `1.5` or `2.0` allows oversubscription if the workload tolerates scheduling jitter. ## Per-file claims Each test file may declare what it needs: ```lua provium:claim({memory = "4G", cpus = 4}) test("…", function(t) end) test("…", function(t) end) ``` The claim is taken at file dispatch and released at file completion. A second `:claim` errors — `lab claim already held; one-shot per lab`. Without a claim, the dispatcher uses the per-file overhead default (50 MiB memory, 0 CPU). A file that boots 3 VMs at 1 GiB each but doesn't claim anything gets through the pool gate immediately and then might OOM the host because the overhead default is way under the actual demand. **Rule of thumb:** claim memory equal to the sum of expected VM memory budgets plus a small buffer; claim CPUs equal to the sum of expected VM vCPU budgets. ```lua -- 3 VMs × (2G memory, 2 CPUs each): provium:claim({memory = "7G", cpus = 7}) -- 6G VMs + 1G buffer; 6 vCPUs + 1 ``` ## File dispatch flow ```mermaid flowchart LR Disc[file_discovered] --> Wait{pool has room?} Wait -->|no| Block[file_blocked] Block --> Wait Wait -->|yes| Disp[file_dispatched] Disp --> Acq[claim_acquired] Acq --> Run[Run tests in file] Run --> Comp[file_completed] Comp --> Rel[claim_released] ``` Files queue at the pool until their reservation fits. The order is stable — the dispatcher takes files in discovery order, so a file that doesn't fit blocks subsequent files (FIFO). For now there's no priority or backfill; if you need a specific file to run first, list its path explicitly. ## PSI throttling On Linux hosts with pressure stall information (PSI) available, Provium spawns a pressure monitor at startup. It polls CPU and memory pressure once a second; when either crosses the threshold (10 % some-pressure averaged over 10 seconds), the dispatcher pauses new file dispatches until pressure drops. The threshold and poll interval are fixed — they are not configurable from the CLI. Files that are already in flight continue. The throttling only delays new dispatch. When PSI throttling is the cause of a `file_blocked`, the event's `reason` field is `psi_pressure` (versus `pool_full` for "the pool can't afford this file's claim"). ## KSM tuning Kernel Same-page Merging deduplicates identical pages across VMs. When you boot many VMs from the same kernel + initrd, KSM can reclaim significant memory. Provium tunes `/sys/kernel/mm/ksm/*` at startup unless `--no-ksm` is passed. The KSM tuning runs once per `provium` invocation, setting `run = 1`, `pages_to_scan = 1000`, and `sleep_millisecs = 20`. Best-effort: a one-line summary goes to stderr. ``` provium: ksm: tuned (3 knobs) ``` If your host is shared with non-Provium workloads, pass `--no-ksm` so Provium doesn't change global tuning. ## What blocks parallelism Several things prevent unlimited parallelism even when the pool has room: | Thing | Why | |---|---| | Fixture build lock | One process at a time per fixture key. Other files queue on `fixture_build_waiting`. | | Pool reservation | A file with a 16 G claim won't run alongside other big files until pool has 16 G free. | | PSI pressure | High CPU pressure pauses new dispatches. | | `--fail-fast` | Stops new dispatch after the first file failure. | Of these, fixture build lock is the most common surprise. If 16 test files all reference `fixtures/base` and the cache is cold, all 16 queue on the build lock; only one builds, the rest wait. Pre-warm with `provium fixture build fixtures/base` to amortise. ## Inspecting parallelism in a run The event stream gives you the full picture: ``` provium tests/ --save-events events.msgpack ``` Then walk the stream: - `file_discovered` events at the start tell you the universe. - `file_dispatched` events tell you what actually ran in parallel (count of in-flight = `file_dispatched - file_completed`). - `file_blocked` events with `reason` tell you why something queued. - `pool_state` events at 1 Hz give you a usage timeline. `provium-coverage` summarises this in its run report; for ad-hoc debugging, parse the msgpack with the snippet in [events and coverage](/provium/running-tests/events-and-coverage.md). ## Tuning patterns ### "I want to maximise throughput" ``` provium tests/ --cpus $(nproc) --mem $(awk '/MemTotal/ {printf "%dG", $2/1024/1024 - 4}' /proc/meminfo) ``` Use everything except 4 GiB of RAM and overcommit CPUs gently: ``` provium tests/ --cpu-overcommit 1.5 ``` ### "I want to be conservative on a shared host" ``` provium tests/ --cpus 4 --mem 8G --no-ksm ``` ### "I want to detect over-subscription" Watch the `pool_state` and `file_blocked` event streams. Frequent `file_blocked` with `reason = pool_full` means files are claiming more than the pool can serve in parallel — either the pool is too small or claims are too generous. Frequent `file_blocked` with `reason = psi_pressure` means the host is genuinely overloaded — reduce parallelism (`--cpus`, `--mem`) or move other workloads off the host. ### "I want to debug a slow run" ``` provium tests/ --save-events events.msgpack ``` Parse the msgpack for the longest `file_completed.duration_ns`. Then look at that file's `test_started` / `test_passed` events to see which test(s) are slow. Cross-reference with `vm_spawned` events to see how many VMs were involved. ### "I want to test for resource leaks" The pool's available budget should return to its full value after each `claim_released`. Watch `pool_state` over time — if available is drifting down monotonically, something is leaking. The most common culprits: - A test that exits via panic without releasing a claim — but the dispatcher releases on `file_completed` regardless of how it ended, so this shouldn't happen. - A bug in the harness — file an issue with the event stream. ## VMs vs files A common confusion: the pool tracks per-file resources, not per-VM. The dispatcher reserves the file's full claim at dispatch and holds it until file completion, regardless of how many VMs the file actually boots concurrently. ```lua -- This file claims 4G, but only ever has one VM live at a time: provium:claim({memory = "4G"}) test("a", function(t) local vm = provium:vm("v", "peios"):boot() vm:shutdown() -- VM gone, but claim still held end) test("b", function(t) local vm = provium:vm("v", "peios"):boot() end) ``` The claim doesn't release between tests. If you want fine-grained reservation, you'd need a smaller claim and rely on the dispatcher's per-file overhead — but the trade-off is potential OOM if the claim is too small for the actual peak. For the typical case, claim for the file's worst-case peak. ## See also - [The CLI](/provium/running-tests/the-cli.md) — `--mem`, `--cpus`, `--cpu-overcommit`, `--no-ksm`. - [Lab reference](/provium/reference/lab.md) — `lab:claim`. - [Events](/provium/reference/events.md) — `pool_state`, `file_blocked`, `claim_acquired`, `claim_released`. - [Fixtures and dependencies](/provium/running-tests/fixtures-and-dependencies.md) — fixture build lock as a parallelism limiter.