2.1 The Pipeline
eventd is the primary consumer of the KMES ring buffers. Events travel from shared memory to a committed database row in four stages.
- Drain. One thread per CPU reads events from that CPU's ring buffer, following the lock-free read protocol PSPK specifies (§2.2).
- Detect. The drain thread compares each event's sequence number against the last it saw for that CPU. A jump means events were lost, and the loss becomes a gap record (§2.5).
- Hand off. The drain thread copies the event out of the mapped region and passes it to the writer thread that owns the shard it routes to (§2.3).
- Write. The writer thread accumulates events into a transaction and commits, sizing the batch to whatever throughput allows (§2.4).
Two principles govern the whole pipeline, and most of its behaviour follows from them rather than from anything specific to a stage.
2.1.1 The ring buffers are the only buffer #
eventd holds no large intermediate queue between KMES and SQLite. The handoff channel is bounded by the maximum batch size and nothing else accumulates.
When the writer falls behind, the channel fills; when the channel is full, the drain thread stops reading; when the drain thread stops reading, events accumulate in the ring buffer, which is exactly what a ring buffer is for. Backpressure propagates all the way back to the kernel, and the absorption capacity is the ring buffer's, which an administrator already sizes.
If the ring buffer also fills, KMES overwrites its oldest events, the drain thread notices the sequence jump when it resumes, and the loss is recorded (§2.5). That is the designed worst case: eventd loses events visibly rather than buffering without bound and dying.
The alternative — a large in-process queue — would move the same capacity into a place where losing it is invisible, where it competes with the page cache for memory, and where an out-of-memory kill takes the whole queue with no record that it existed.
2.1.2 Sharding scales writes linearly #
Each shard is a self-contained SQLite database with its own file, its own write-ahead log and its own writer thread. Shards share no write-path state, so the write path has no cross-shard lock, no shared counter and no coordination point (§2.3).
The consequence for the query path is that a shard means nothing to it. A shard database holds whatever CPUs happened to route to it in whatever eventd lifetime wrote it, so a query filtering by CPU reads every shard, and the query path never assumes a relationship between a shard and a CPU (§6.4).