8.5 Crash Recovery and Signals
8.5.1 After a crash #
An ungraceful termination — a segmentation fault, a kill, an out-of-memory kill — leaves four things true.
The ring buffers are unaffected. KMES writes regardless of consumer state, and events emitted while eventd was down accumulate there.
The databases are consistent. WAL mode guarantees committed transactions survive, and SQLite rolls back the in-flight batch on the next open.
There is a sequence gap. Events between the last committed batch and the crash were never persisted. On restart eventd derives its resume points from committed rows, sees the difference from the current ring buffer state, and writes a gap record (§2.5).
Socket-buffered data is gone. The kernel discards a socket receive queue on process exit, taking whatever logs and metrics were waiting. Acceptable by the loss model.
No manual recovery is needed and none is offered. eventd restarts, re-attaches, resumes draining, and records what was missed. The boot boundary logic recognises the restart from the committed rows themselves rather than from any flag written in advance (§3.7) — which is the point, since a crash is precisely the case where nothing was written in advance.
8.5.2 Signals #
| Signal | Behaviour |
|---|---|
SIGTERM | Begin graceful shutdown (§8.4). |
SIGINT | Begin graceful shutdown. |
SIGQUIT | Write a diagnostic dump to standard error, then begin graceful shutdown. |
SIGHUP | Re-read configuration from the registry (§8.3). |
Every other signal keeps its default behaviour.
8.5.3 The diagnostic dump #
SIGQUIT writes human-readable text to standard error before step 1 of
the shutdown sequence, so that it reflects the daemon's state while it
is still running rather than while it is tearing down. It includes at
least:
- the current boot ID
- the active shard count and the readable historical shard count
- the per-CPU last committed sequence numbers, derived from committed rows
- the current non-streaming and streaming query counts
- the metric series cache occupancy
- the last observed write error for each store, where one exists
The format is not a stable machine interface and its wording may change.
The set is chosen to answer the questions an operator has about a misbehaving eventd that a query cannot: how far behind the writers are, whether the series cache is thrashing (§5.3), whether query slots are exhausted (§6.5), and whether a store has been failing writes quietly. Standard error is the destination because peinit captures it, so the dump reaches the log store by the ordinary path — and reaches standard error directly when the log store is the thing that is broken.