6.3 Transition Causes

8 min read

Every transition carries a cause recording why it happened. peinit tracks both the current state and the cause of the most recent transition, and the cause determines restart eligibility, OnFailure behaviour, and what an administrator is told.

6.3.1 The taxonomy #

6.3.2 Restart eligibility #

Causes fall into four classes, and the class decides whether the restart policy is consulted at all.

Restart-eligible. peinit consults RestartPolicy and the budget. If a restart is allowed and the budget holds, the service goes to Backoff and then to Starting; otherwise Failed.

ProcessCrash, WatchdogTimeout, HealthCheckFailure, ReadinessTimeout, PreHookFailure, PreExecFailure, ParentSetupFailure.

Always-only. CleanExitRestart applies when a Simple service exits successfully and the policy is Always. It uses the same backoff and the same budget as a failure, which is what stops a daemon that exits cleanly in a tight loop from bypassing throttling entirely. It is never treated as a ProcessCrash, and both status and events make clear the process succeeded and was restarted only because the policy says so.

CleanExit is its non-restarting counterpart: straight to Inactive, consulting neither policy nor budget.

Budget-exempt. BindsToRecovery takes a service from Failed to Starting when its binding target returns. It is not subject to the policy or the budget, because the service did not fail on its own — it was stopped because its dependency went away.

Never-restart. peinit does not consult the policy at all: ExplicitStop, ExplicitReset, ShutdownWave, ConflictEviction, BindsToPropagation, ProcessUnkillable, RestartBudgetExhausted, ValidationError, CycleDetected, DependencyFailure, AssertionError, ConditionSkipped, TtyUnavailable, InternalError. Retrying cannot help with any of them.

6.3.3 OnFailure #

When a service enters Failed and its definition names an OnFailure service, peinit starts that service — with the exceptions below. OnFailure fires on entry to Failed, and Failed to Failed is not a transition, so it fires at most once per failure.

It does not fire for:

• ShutdownWave — no new service starts during shutdown, so a fallback would be both impossible and pointless.
• ValidationError, CycleDetected, DependencyFailure, AssertionError — these are definition or graph breakage rather than runtime degradation. A broken definition cannot meaningfully trigger a fallback, and the fallback would probably sit in the same broken graph.

It fires for everything else, including ProcessCrash, WatchdogTimeout, HealthCheckFailure, the startup failures, and RestartBudgetExhausted on a non-Critical service.

For a Critical service exhausting its budget, the reboot takes precedence and no fallback is started. The suppression asks whether a reboot is owed rather than inspecting ErrorControl itself, which is the same question §6.4 puts.

OnFailure is for graceful degradation — the main web interface fails, so start a minimal emergency endpoint. It is not for monitoring or alerting, which is eventd's job.

6.3.3.1 A handler that is already running #

If the named handler is already Active — or Reloading, on its way back to Active — the failure starts nothing. A running service has no state to transition into on being started again, and it is already doing the job the failure would have asked of it, so the start resolves as satisfied: no operation is created, and no loop-guard chain entry is recorded for a cascade that did not happen.

This is the same answer the control interface gives an administrator who starts an already-running service, and it is deliberately the same test. A handler in any other state is started normally — so a Oneshot handler that ran for an earlier failure runs again for a new one, and a handler that has since failed is restarted.

6.3.3.2 The loop guard #

An OnFailure handler can fail and carry its own OnFailure, so a misconfiguration where A's handler is B and B's is A could run forever. peinit bounds the chain originating from one failure two ways: it tracks the set of services already started as handlers for that failure and will not start one already in the set, and it will not follow the chain past a fixed depth of 16. When either trips, peinit records a peinit.on-failure.suppressed event naming which (outcome.reason cycle or max-depth), and stops.

A chain entry is retired two ways.

The handler stopped needing tracking. Completed, Inactive, Skipped or Abandoned — it ended, gave up, or became unkillable.

The handler arrived. It has been dependent-satisfying for its RestartWindow — the same window that resets a restart budget (§6.4). That is peinit's existing test for a service having genuinely taken over rather than merely come up, and it is what ends the originating failure: a later failure of that handler is a new one, starting a fresh chain.

The distinction matters in both directions. Without the second rule, a handler that runs healthily for days holds a slot forever, so a degradation path handing off through several healthy layers exhausts its own depth budget by working. But retiring on Active alone would break the guard entirely: Readiness=Alive reports Active the instant the process spawns, so two services naming each other as handlers would hand off forever, with no delay and no cost — an OnFailure start is not a restart and spends no restart budget. Only a handler that held is treated as having arrived.

6.3.4 The logging contract #

A transition into Failed, Skipped or Abandoned is written to the console, naming the service and its cause: what failed, and why. There is no per-transition event — a transition is visible through the peinit.job.* and peinit.operation.* events that carried it, not as a record of its own.

Cryptic failure messages are a defect. A reboot loop caused by a configuration error with an opaque message is the worst outcome the system has, and the cause taxonomy exists so that the "why" is never a guess.

Edit this page