Troubleshoot a service

12 min read

Collect the current state and cause before changing anything:

$ svctl status <service>
$ svctl list
$ svctl boot

Keep the job or operation ID and any warnings. Open Logs… for the service in Services Manager to inspect its main process, hooks, and health checks. For older attempts, query logs and events; live status is not a permanent history.

Use the symptom below to choose the next check. Avoid repeating a restart or changing permissions until you know which layer failed.

A service won't start #

status shows it Failed or Skipped. The cause says why:

A service keeps restarting #

The service flaps — up, down, up, down. This is restart policy doing its job, but it points at an unstable service.

• It eventually settles in Failed with RestartBudgetExhausted. It crashed RestartMaxRetries times faster than it could stay healthy for RestartWindow. The fix is the service, not the policy — read its logs for the crash. Raising the budget only delays the inevitable.
• It flaps forever and never exhausts the budget. Check whether it stays Active for the full RestartWindow between failures. Reaching Active briefly does not reset the budget. Inspect the health-check output too; a faulty check can restart a working service.
• It restarts on a clean exit you did not expect. RestartPolicy=Always restarts even successful exits (cause CleanExitRestart). If the service is meant to exit, use OnFailure instead.

The machine keeps rebooting #

Preserve the last console error and any service records before attributing repeated reboots to a service. If peinit reports a Critical service exhausting its restart budget, its ErrorControl=Critical policy makes peinit sync and reboot. Repeating that failure can produce a reboot loop; the symptom alone does not establish this cause.

The boot-attempt counter can break a sequence of boots that reach peinit's counter increment but never become confirmed: after N attempts (default 3), peinit enters Recovery mode. If each boot reaches the success grace before the Critical failure, its counter is reset and the reboot loop can continue without automatic Recovery. Check svctl boot while the normal control interface is available; it is unavailable in Recovery. Do not assume every reboot loop will stop after three attempts, or that peinit can deliver Recovery if it cannot start.

In Recovery, start with the failure reason on the console. Phase 2 services are skipped, so stored eventd history is not automatically queryable from that shell. Where a supported running eventd query service is available, use Find service logs and Find missing records to investigate earlier attempts. evctl queries that service; it does not open stored databases offline. Otherwise, retain the console evidence for the administrator or provider's recovery procedure.

Once the evidence identifies a Critical service failure, repair its cause or restore known-good configuration through a supported recovery path before rebooting. Disabling a required platform service can leave its dependents unable to start. If you need to intervene before the counter trips, boot with peios.recovery=1 on the kernel command line.

A service is Abandoned #

Abandoned means peinit SIGKILLed the process but it survived — stuck in uninterruptible kernel sleep (D-state), the signature of hung I/O (a dead NFS mount, a failing disk). Nothing in userspace can kill a D-state process.

• The underlying I/O fault is the real problem — investigate the storage or mount, not peinit.
• The service's cgroup is leaked and shows in warnings. A later start uses a fresh generational cgroup, so the new instance is unaffected.
• svctl reset <service> clears the Abandoned state and re-checks the cgroup: if the stuck process finally died, peinit cleans up; if not, it stays leaked and warns you. Leaked cgroups clear fully only on reboot.

I changed the config and nothing happened #

peinit works from an in-memory snapshot, and each field has a mutability class that decides when a change applies:

• Immutable at runtime (ImagePath, Type, Identity, RequiredPrivileges, ErrorControl) → takes effect only on restart.
• Apply on next start (dependencies, conditions, OnFailure) → next start or graph reload.
• Reloadable at runtime → next relevant operation. Timeouts and health-check policy can affect later checks; environment, arguments, working directory and process limits apply to a new process, not one already running.
• Hot-reloaded (ServiceSecurity) → next control request.

If a change is not landing, check its class first. For a wholesale re-read of every definition, run svctl reload-config — it rebuilds and validates the whole graph atomically and swaps it in only if validation passes (running services are untouched). And remember peinit pulls changes from registry notifications rather than having them pushed, so there can be a brief lag.

ACCESS_DENIED #

A control command returns ACCESS_DENIED. peinit ran AccessCheck on your token against the target's descriptor and the right was not granted:

• A service command needs the matching right in the service's ServiceSecurity descriptor (start needs SERVICE_START, restart needs both stop and start, …).
• shutdown and reload-config need SYSTEM_SHUTDOWN / SYSTEM_RELOAD_CONFIG in peinit's control descriptor.
• The check uses your effective identity at connect time — if you are impersonating, that is what is checked.

Under the default descriptors every denial is recorded by the kernel as a kacs.audit.access.checked event naming the caller, the target (its object kind is service, job or peinit-system) and the rights requested and granted. Walk it through Debugging a denial.

A job submission is refused #

A job submit — from svctl or from a service — comes back with an error rather than a job. The code says which door it hit:

A job that was accepted but never ran is not an error response: it is an ok response whose view is failed with a cause — parent_setup_failure (peinit could not prepare the launch) or pre_exec_failure (the program could not be executed as that identity, which is where "no such file" and "permission denied" land, because peinit does not check the image before accepting a submission).

A job's output has gaps #

If a submitter attached an output sink and sees gaps, look for a peinit.job.output.dropped event for the job in eventd: the sink was not being drained fast enough, and peinit dropped lines for the sink only rather than let it slow the job. Query eventd under the job’s GUID for the independently forwarded record. Sink loss does not itself drop that record, but eventd’s log path is also best-effort, so completeness is not guaranteed.

UNKNOWN_JOB #

The GUID is unknown or its retention has elapsed. A submitted job is retained for 60 seconds after it ends, then dropped; check eventd for any stored record. A known job that you lack permission to query returns ACCESS_DENIED. job list instead omits jobs you cannot query.

A dependent never started #

You started (or booted) a service, but something that depends on it never came up:

• A Requires/BindsTo dependent stays blocked until its target reaches a satisfying state (Active, Reloading, Completed, or Skipped). If the target is stuck in Starting or Failed, so is the dependent — fix the target.
• If a dependent got connection refused on boot, check the target's readiness setting and startup logs. With Readiness=Alive, peinit waited only for the process to exist, not to be serving. Use Readiness=Notify only if the program supports sending READY=1 from its tracked main process when ready; changing the setting does not add that support and can cause ReadinessTimeout. See the notification contract. If the program cannot signal readiness, have its service or package owner resolve the startup race. peinit's boot validation warning can identify an Alive-readiness dependency; it does not prove the cause of a refused connection.
• A Wants dependent waits for the optional target’s startup attempt to resolve, but starts even if the target failed. Use Requires when successful readiness is necessary.

Booted into Safe mode #

Safe mode selects Critical and SafeMode=1 services within the boot-triggered set. Those flags do not give a demand-only service a boot trigger. You land here when graph validation finds a cycle or unresolvable conflict involving a Critical service, or when peios.safemode=1 is on the kernel command line. The TCB is healthy; the configuration is broken.

Fix the graph: the logs name the cycle path (A → B → C → A) or the conflicting pair. Once the cycle is broken or the conflict resolved, a normal boot returns. You can start any service by hand from Safe mode in the meantime — it is only auto-start that is restricted.

Booted into Recovery mode #

Recovery mode gives you a SYSTEM shell on the console and skips Phase 2 services. It starts registryd only if the boot has not already attempted that start. You reach it when the boot counter hits N, when peios.recovery=1 is set, or when registryd failed in Phase 1 (Recovery is immediate; this boot attempt has already incremented the counter).

If the registry itself is failing:

1. Preserve peinit's console failure reason and the source's startup diagnostics. A failed registry operation does not prove the source process has exited.
2. Identify the installed registryd provider and version, its hive declarations and actual database paths. LCS and sources explains the default loregd provider and its diagnostics.
3. Establish whether the source is already active before any storage work. Recovery may retain a registryd started or attempted earlier in the boot. Do not launch a second source against the same hive files.
4. Obtain the installed provider's supported recovery procedure before attempting offline inspection or repair. The source-checked loregd implementation accepts hive declarations and has no offline repair switches. It does not create an automatic startup backup; its orphan cleanup is not corruption repair or backup restoration. Those references identify the checked revision. Verify what recovery copy actually exists and what it contains before planning a replacement.

When the source is serving the required registry operations, Backup and restore describes the supported privileged subtree workflow. That workflow depends on a working source; it does not repair an unavailable source. Restore replaces the target contents and descendants, including security descriptors, so review the target and backup before using it. Role definitions can re-supply service configuration but do not restore every registry value.

Recovery offers no TCB guarantee. Treat the unrestricted SYSTEM shell accordingly, and remember it needs console access (physical, IPMI, or serial); there is no remote recovery yet.

Where to start #

To read states and causes fluently, keep The service lifecycle handy.

For restart, backoff, and Critical-reboot behaviour, see Keeping services running.

For the commands and their outputs, see Controlling services.

Edit this page