Workspaces

14 min read

A workspace lets you drive a whole collection of recipes as one unit. Instead of running pekit build in each recipe directory by hand, you declare which recipes belong together in a workspace.pekit.toml file and run a single pekit workspace build, which delegates that command to every member. This is how a distro-scale tree of packages — dozens or hundreds of recipes — is built, tested, packaged, and published in one go.

A workspace does three things:

  1. Discovers members from include/exclude globs (each member is a directory containing a pekit.toml).
  2. Shares configuration — [env], [wrap], and a distro-wide [policy] — down to every member.
  3. Schedules and delegates a single command across all members, with optional concurrency, fail-fast, and cross-member safety checks.

The workspace.pekit.toml file #

The workspace file is TOML and lives at the root of the tree it governs. Its schema is small — exactly five top-level keys are accepted, and anything else is rejected with unknown_key: include and exclude (globs selecting and removing member directories, relative to the workspace root), [env] and [wrap] (shared configuration contributed to every member), and [policy] (distro-wide derivation policy — see Policy). The key-by-key schema is in Supporting files.

include is mandatory: an omitted or empty include fails with missing_key ("workspace include is required"). Everything else is optional.

# workspace.pekit.toml at the distro root
include = ["core/*", "extra/**"]
exclude = ["extra/experimental/*"]

[env]
LC_ALL = "C"

[wrap]
command = ["nice", "-n", "10", "{{command}}"]

[policy.symbol_versions]
"libc.so.6" = "GLIBC_"

Member discovery #

include and exclude are glob patterns matched against the directory tree under the workspace root. Matching uses doublestar globbing, so ordinary wildcards (*, ?, character classes) work and ** matches across directory boundaries. Patterns are always workspace-root-relative.

Discovery proceeds as:

  1. Every exclude pattern is expanded and its matches recorded.
  2. Every include pattern is expanded. A match is kept only if it was not excluded and the matched directory contains a pekit.toml.

A matched directory with no pekit.toml is silently ignored — it is simply not a recipe, so it is not a member. This means you can point include at a broad glob (extra/**) and only the directories that actually hold recipes become members.

If discovery yields no members at all, the run fails with empty_workspace ("workspace has no members").

Locating the workspace #

When you run pekit workspace, pekit finds the workspace file by walking upward from the current directory looking for workspace.pekit.toml (the same upward search recipes use). You can override this with --workspace <path>, which accepts a path to a workspace.pekit.toml file, a directory containing one, or a remote locator (for example a github.com/... reference).

Member identity #

Each member has a stable id: its workspace-relative directory path, written with forward slashes (for example core/zlib or extra/tools/ripgrep). This id is the member's name everywhere it appears:

  • in diagnostics and error messages (member_failed, member_skipped);
  • in structured JSON events and in output-line prefixes;
  • in the publish-collision owner labels (see below);
  • in the end-of-run summary.

Members are processed in a deterministic order: ids are sorted lexicographically. With the default single-worker execution this is the exact order members run in; with concurrency it is the order in which work is dispatched.

Invocation #

pekit [global flags] workspace [workspace flags] <command> [args and flags]

workspace is itself a command, but it does no building of its own — it delegates another command — any of build, test, install, package, publish, clean, gen, verify, or lock — to every member. That delegated command and its arguments come after the workspace flags.

Strict flag placement #

The tail after workspace is parsed specially, and the placement rules are strict:

  • Workspace flags — --jobs N, --fail-fast and --journal <file> — must appear after workspace and before the delegated command. They are recognised only in this slot.
  • Command-level flags (version selection, --local, --env, --keyring, --refresh-source, --all, …) belong to the delegated command and must come after it. Placing one before the delegated command is an error with a hint: "workspace command flag <name> must appear after the delegated command."
  • Global flags (--dry-run, --quiet, --verbose, --json, --allow-unused, --workspace) may appear either before workspace or in the tail before the delegated command.
# correct: workspace flags before the command, command flags after
$ pekit workspace --jobs 4 --fail-fast build --version 1.2.3

# also correct: a global flag may precede workspace
$ pekit --dry-run workspace build

# error: --version is a command flag placed before the delegated command
$ pekit workspace --version 1.2.3 build
#   workspace command flag version selection must appear after the delegated command

Because the workspace flags are recognised only in the workspace slot, writing them after the delegated command hands them to the delegated command's own parser, which rejects them as an unknown_flag. Keep them in the workspace slot.

Both failures share the diagnostic code missing_workspace_command: it fires when a command flag precedes the delegated command, and when no delegated command follows the workspace flags at all ("workspace requires a delegated command").

Execution model #

Each member is run as an ordinary per-recipe invocation of the delegated command — exactly as if you had cd'd into that member and run pekit <command> ... there, with the delegated flags and any positional selectors carried through unchanged.

A failed member's explicitly incomplete target stage is cleared when that target is rebuilt on a later run. This applies under workspace isolation as well; it does not allow --no-build to reuse incomplete or unverified output. See Stages that did not finish.

Concurrency: --jobs #

--jobs N sets how many members run concurrently. N must be a positive integer; a missing value is missing_flag_value and a non-positive or non-numeric value is invalid_flag_value ("--jobs requires a positive integer"). The default is 1, i.e. members run one at a time, in dependency order.

Fail-fast: --fail-fast #

By default the workspace attempts every member even if some fail: it runs them all and then fails the overall command at the end if any member failed (workspace_failed, "N workspace member(s) failed").

With --fail-fast, the first member failure sets a stop flag so that no further members are started. Members that had not yet begun are recorded as skipped.

Resumable runs: --journal #

--journal <file> makes a long run resumable. Each member that succeeds is appended to the journal as it finishes, and the file is synced, so it survives a power loss straight afterwards. Repeating the same run with the same journal skips the members it lists (they count as skipped in the summary) and runs the rest. A skipped member still takes part in dependency order: the packages it defines carry their runtime dependencies, so a member installing one still waits for whatever those need. Members that failed, or never started, are not listed, so they run again.

To stop a run cleanly, create <file>.stop. At the next point where a member finishes, pekit starts nothing more, waits for the members still running, and exits with workspace_stopped. It removes the stop file as it honours it, and also removes one left over when a new run starts. Anything stopped by other means, such as a power-off, loses only the members that were running.

$ pekit workspace --jobs 3 --journal round2.journal publish --all --latest --locked
$ touch round2.journal.stop     # later: finish what is running, then stop
$ pekit workspace --jobs 3 --journal round2.journal publish --all --latest --locked

The journal's first line records the run it belongs to: the delegated command, version selection (including --locked), --all, --replace, --strict, --no-gates, --env, tags and selectors. A journal from a different run is refused with journal_mismatch, so one round's journal cannot skip another round's members. --jobs and --fail-fast are not part of it and may change between attempts. --dry-run reads an existing journal but never creates or writes one. The journal is opt-in; without it a workspace run behaves as before.

Dependency order #

For build, install, test, package and publish, pekit orders members so that a member producing a package runs before the members that install it. The aim is the best chance of a whole batch succeeding, even into an empty repository.

A member waits for another when the dependencies of its gate targets name a package the other member defines. Gate targets are the ones the command installs dependencies for: build targets for build and install, build and test targets for test, and build targets plus gated tests for package and publish (only build targets under --no-gates). Only the dependency set of the selected env's dependency_provider counts: under an env whose provider's names no member produces (a foreign distribution's packages, say), there are no edges and members are independent.

Runtime dependencies count too. When a member needs a package another member defines, it also waits for whatever that package's own runtime [dependencies] name, followed transitively through member packages, provided the package declares the same dependency_provider. Names no selected member defines are not ordered: they resolve from whatever the provider already has, as before.

Scheduling:

  • Among members whose waits are satisfied, lower ids start first; --jobs runs several at once.
  • A member starts even if one it waited for failed. It then resolves that dependency from what the provider already has.
  • A dependency cycle cannot be ordered. When nothing is runnable and nothing is running, pekit starts the blocked member with the fewest unfinished dependencies (lowest id on a tie) and carries on. There is no warning: building a real cycle from nothing needs a separately built seed, which tags let you select.

Ordering reads each member's effective recipe, resolving its source as the member's own run will. Under --dry-run only already-cached sources are visible, and --dry-run or --verbose reports each member's waits as workspace_order events. A member whose recipe cannot be resolved is not ordered, and reports its failure when it runs.

Selecting members by tag #

A recipe may label itself with tags:

# pekit.toml
tags = ["bootstrap"]

After the delegated command, --tag <tag> runs only members carrying any given tag, and --exclude-tag <tag> skips members carrying any given tag. Both repeat, apply after include/exclude, and fail with empty_workspace when nothing matches. Tags are catalogue policy, so only the member recipe's own tags count, never a delegated source's.

Tags and ordering together are how a distribution bootstraps from nothing: build the tagged seed in an environment that needs none of the catalogue, then everything else in dependency order, then the seed again natively, then everything else against the native seed. Here the workspace's default profile is the native one, and debian is the named profile used only for the seed. The later rounds publish versions that already exist, so they overwrite them with --replace, which is only for a repository nobody consumes yet:

$ pekit workspace publish --all --latest --tag bootstrap --env debian
$ pekit workspace publish --all --latest --exclude-tag bootstrap
$ pekit workspace publish --all --latest --tag bootstrap --replace
$ pekit workspace publish --all --latest --exclude-tag bootstrap --replace

Summary #

At the end of a run pekit emits a workspace_summary event tallying members as succeeded, failed, or skipped:

12 succeeded, 1 failed, 3 skipped

Succeeded and failed count members that actually ran; skipped covers members not started because of --fail-fast or a stop request, members already finished in the journal, and members skipped by selector planning (below). The overall command exits non-zero if any member failed.

Per-member strictness #

The delegated command and its flags are validated once, when the tail is parsed. Beyond that, strictness is evaluated per member, because members differ: one recipe may define a build target or a package selector that another does not.

  • Each member runs the delegated command against its own recipe. If a flag or selector cannot be honoured by that member's recipe, that member errors — its failure is reported (member_failed) and counted, and the other members are unaffected (unless --fail-fast is set).

  • --allow-unused relaxes this per member. When a run carries positional selectors (target names, or def:variant package selectors) together with --allow-unused, pekit plans which selectors apply to which member:

    • selectors that a member cannot satisfy are suppressed for that member (reported via workspace_selector_suppressed);
    • a member to which none of the requested selectors apply is skipped ("no requested selectors apply to member");
    • if a requested selector applies to no member at all, the whole run fails with missing_selector ("workspace selector … matches no selected member").

    --allow-unused also suppresses per-member unsupported version selection rather than failing that member.

Without --allow-unused, selectors are passed to every member verbatim and a member that does not recognise one fails in the usual way.

Cross-member publish collision detection #

When the delegated command is publish, two members could be configured to publish an artifact to the same destination — a mistake that would have them overwrite each other. Pekit catches this before publishing anything.

A preflight pass runs each member's publish planning in dry-run mode and reserves every resolved destination path in a shared registry. Destinations are keyed by path, each with an owner of the form <member-id>:<instance>. Reserving a path that a different owner already holds fails the entire run with publish_collision ("publish destination collision between … and …"). Reserving the same path for the same owner is idempotent, so a member does not collide with itself.

Because this is a preflight, a destination clash is reported up front, before any member has written to a publish target.

Shared configuration #

[env] and [wrap] #

[env] and [wrap] declared at the workspace level are shared defaults contributed to every member, layered together with each member recipe's own [env] and [wrap]. For how these compose with recipe- and environment-file values — and the precedence between them — see Environments and keyrings. The [wrap] command follows the same {{command}} placeholder rules as a recipe wrap. Keyrings behave similarly: a named --keyring on a workspace run resolves against the workspace root once, and the values are shared with every member.

Policy: symbol-version requirements #

The [policy] table carries distro-wide derivation policy. [policy.symbol_versions] is a map from a shared-library soname to the symbol-version token prefix whose tokens are commensurable with the providing package's version:

[policy]
symbol_capabilities = ["libstdc++.so.6"]

[policy.symbol_versions]
"libc.so.6" = "GLIBC_"
"libgcc_s.so.1" = "GCC_"

This governs which sonames receive a symbol-version floor when pekit derives a package's dependencies across the workspace — so that, for example, a binary using a GLIBC_2.34 symbol gains an appropriate lower bound on its libc dependency. An unknown sub-table under [policy] is rejected (unknown_key, "policy.… is not a known policy table"). For how derived dependencies and claims work, see Dependencies and claims.

symbol_capabilities derives every required and provided version node directly, including both GLIBCXX and CXXABI for libstdc++. Its token numbers are not GCC package versions. Publish capability-bearing runtimes before consumers.

Where to go next #

For the recipes a workspace's members are, read Anatomy of a recipe.

For how workspace [env] and [wrap] compose with a recipe's own, read Environments and keyrings.

Named env files are workspace defaults too. A selected <name>.env.pekit.toml at the workspace root applies to every member; a same-named file beside one recipe overlays it. This lets a workspace define shared build rungs without per-member symlinks.

For the full flag surface and remote locators, read Invocation and flags.

Related Content

Edit this page