eventd

eventd is the observability daemon: the single persistent sink for everything a Peios system records about itself. Events, logs and metrics all end in eventd, and every query for any of them is answered by it.

It is one of the platform daemons the service manager starts at boot, signed at TCB level, and it is Critical — a system that loses it loses its audit trail.

The three data types are genuinely different and eventd treats them differently at every layer.

Events are structured, typed records carrying identity stamps the kernel applied and an emitter could not influence. They arrive through KMES, in per-CPU shared-memory ring buffers, and eventd is their primary consumer. They are the audit and security telemetry, and losing one is a real failure — so the event path is the one with sequence numbers, gap detection, per-transaction durability, and a synthetic record written whenever anything is lost.

Logs are text: a line a program wrote, with light metadata attached. They arrive on a datagram socket, mostly from the service manager forwarding what it read from a service's standard output and standard error. Losing one is an inconvenience, and the ingestion path is designed around that tolerance rather than against it.

Metrics are numeric measurements over time — dense series rather than discrete occurrences. They arrive on a second datagram socket, pushed by whatever is doing the measuring. eventd is a sink, not a collector: it scrapes nothing and polls nothing.

1.1.1 The shape of the daemon #

Three ingestion paths, three storage engines, one query surface.

The event path runs one drain thread per CPU, each attached to one ring buffer, handing events over a bounded channel to a writer thread that batches them into a SQLite shard. Shards are independent — separate files, separate write-ahead logs, separate writer threads, no shared write-path state — so write throughput scales with the shard count (§2.3).

The log and metric paths each run a single thread that reads datagrams and writes them, to one database each. Neither contends with the event path.

Underneath all three sits a decision that shapes the event pipeline entirely: the KMES ring buffers are the only buffer. eventd holds no large intermediate queue. Events move from the ring buffer through a small bounded handoff straight into a transaction, and when the writer falls behind, backpressure propagates backwards until the ring buffer absorbs it — and when the ring buffer cannot, the loss is detected and recorded rather than hidden (§2.5).

Two subsystems adapt to the workload rather than being tuned for it. Adaptive indexing watches which fields queries filter on and maintains indexes for them, shedding those indexes under write pressure because throughput outranks query latency (§3.4). Adaptive rollups pre-compute the metric aggregations that are asked for often (§5.6).

Everything eventd holds is readable only through access checks that KACS performs, per event type, per log origin, per metric name, and per field within a record (§7).

1.1.2 What eventd is not #

It is not a log framework. eventd stores lines; it does not parse them, does not understand severity beyond a single error flag, and does not care whether the text happens to be JSON.

It is not a metric collector. Nothing in eventd reads /proc, scrapes an endpoint, or polls a service. Something else measures and pushes.

It is not a tracing system. Distributed tracing is out of scope entirely.

It is not the low-latency path to events. A consumer that needs events in microseconds attaches to the KMES ring buffers directly, as revstrm does. eventd sits above that transport, adds persistence and access control, and costs a batch commit interval in latency.

It is not the only KMES consumer, and holds no privileged position among them.

This is a Technical Reference Manual Proposal. eventd has not been written.

Everything in this manual is therefore a description of a design rather than of an artifact: the schemas, the thread structure, the algorithms, the constants and the failure behaviour are what eventd is specified to do, not observations of what it does. Nothing here has been checked against an implementation, because there is none to check against.

It is written in the indicative mood, exactly as a reference manual for existing software would be, and it makes no conformance demands. When the code lands, this document becomes eventd's Technical Reference Manual — corrected wherever the implementation and the design turn out to disagree, and with no change in kind. That is the whole reason for the proposal form: a design document that will be read as a manual is better written as one from the start.

The distinction a reader needs to keep is that a TRM's authority comes from the software and this one has none of that yet. Where a statement here is surprising, it is a design decision that has not survived contact with an implementation.

1.2.1 What is a contract and what is not #

eventd's three external interfaces are specified separately, in PSPU §3: the log ingestion channel, the metric ingestion channel, and the query channel with its query language. Those are contracts, binding on anything that speaks them, and they are normative where this manual is not.

This manual covers the other side of that boundary — how eventd fulfils them:

• how it consumes KMES and what it does when it falls behind (§2)
• what it stores, in what schema, and how it accelerates and expires it (§3, §4, §5)
• how a query in the language of PSPU §3 becomes an answer (§6)
• how access decisions are reached (§7)
• how it starts, stops, and behaves when something breaks (§8, §9)

Where a chapter touches the contract, it references PSPU §3 rather than restating it, and describes only what eventd adds.

The access control mechanism is here rather than in PSPU because it is behind the abstraction: a client sees only which records and fields it received (PSPU §3.28). The field GUID derivation in §7.3 is the one part of it that a third party may need to reproduce — an administrator writing a Security Descriptor computes the same GUIDs — and it is a candidate for promotion into a specification if that turns out to be a common thing to do.

1.2.2 Versions and constants #

Constants, configuration keys and catalogues are collected in the appendices at the end of the manual, so that a chapter's reference material does not interrupt the prose that explains it. Where an appendix defines a value, the body references it rather than repeating it.

Terms defined elsewhere are used with the same meaning and are not redefined here: event, header, payload, stamp, ring buffer, consumer, origin class and sequence number from the KMES chapters of the Peios Kernel TRM and PSPK; token, GUID, SID, Security Descriptor, ACL, ACE and privilege from the Peios Kernel TRM and PCDS; registry, hive, key, value and layer from the Peios Kernel TRM; producer, client, log record, metric sample, time series and concrete identifier from PSPU §3.2.

Syscall numbers and signatures for the kernel interfaces eventd calls — kmes_attach, kacs_open_peer_token, kacs_access_check and kacs_access_check_list — are in the Peios Kernel TRM's generated ABI appendices, §2.A and §3.A. This manual names them and does not repeat their numbers.

The following are specific to eventd.

Drain thread: one of the threads that reads from a per-CPU KMES ring buffer. There is exactly one per CPU (§2.2).

Writer thread: the sole writer to one event shard. There is exactly one per shard, and no other thread writes to that database (§2.3).

Shard: one of the independent SQLite databases the event store is split across, each with its own file, write-ahead log and writer thread. A shard is a write-path construct only; the query path treats the whole directory as one store (§2.3).

Active shard: a shard in the current configuration's numbering. Historical shard: a shard database left behind by a previous configuration, opened read-only and still queried (§3.3).

Handoff channel: the bounded queue between drain threads and a writer thread. It is a staging area for the current batch, not a buffer (§2.3).

Synthetic event: a record eventd generates itself and writes directly to a shard, bypassing KMES. Synthetic events carry no identity stamps and no sequence numbers, and are distinguished by a synthetic.-prefixed type (§2.6).

Gap record: the synthetic event recording that events were lost on one CPU, and which sequence numbers went missing (§2.5).

Event store directory: the directory holding every shard database and the metadata database. Metadata database: eventd-meta.db, the one database that is not a shard and survives shard reconfiguration (§3.5).

Desired index set: the global, priority-ordered list of fields eventd aims to have indexed across all shards. Material indexes: the indexes a given shard actually has, which converge toward the desired set when the shard is quiet and diverge from it under pressure (§3.4).

Shedding: dropping secondary indexes to protect write throughput (§3.4).

Rollup: a pre-computed metric aggregate for one series, one function and one time window (§5.6). Rollup registry: the global set of (function, window) pairs being pre-computed.

Series cache: the bounded in-memory map from series identity to series row, which keeps metric ingestion off SQLite in the common case (§5.3).

Logical live size: (page_count - freelist_count) × page_size for a SQLite database — the space actually holding data, excluding pages freed by deletion and available for reuse. Retention is enforced against this rather than against the file size (§3.6).

Quarantine: renaming a database SQLite has reported corrupt, aside from the path eventd uses, and creating an empty one in its place (§3.3).

eventd is not a port of anything. It occupies a position several existing systems also occupy, and differs from each of them in ways worth being explicit about. PSPU §3.C compares the wire contracts; this compares the systems.

1.4.1 Windows Event Log and ETW #

The closest structural match. Windows splits the job in two: Event Tracing for Windows delivers events from kernel and application providers, and the Event Log service persists and serves them. Peios splits it the same way, with KMES in the ETW position and eventd in the Event Log service's.

Held in common: kernel-mediated delivery with metadata the emitter cannot forge, a userspace service owning persistence, and access control by Security Descriptor on a named channel — which in eventd becomes a descriptor per event type pattern (§7.2).

Different: eventd unifies three data types where Windows separates them across the Event Log, ETL trace files and Performance Counters. eventd stores in SQLite rather than a proprietary binary format. And ETW's buffering is a kernel-managed trace session, where KMES exposes shared-memory ring buffers with a lock-free consumer protocol that eventd drains directly (§2.2).

1.4.2 journald #

journald is the systemd system journal: it captures service output and structured messages and stores them in an indexed binary journal.

Held in common: a single daemon for system-wide log ingestion, capturing standard output and standard error as records with metadata attached, stored in a binary format with indexes.

Different: journald is log-only, and a systemd system needs a separate stack for metrics. journald's access control is Unix file permissions and polkit, where eventd's is KACS Security Descriptors evaluated per record and per field (§7). And journald reads kernel messages from /dev/kmsg, a text interface with no identity, where eventd receives kernel events through KMES with the kernel's own identity stamps intact.

The similarity worth noting is that journald's storage is also its index, and its query surface is a matcher over fields rather than a language. eventd goes further in that direction — an actual query language, over all three data types — for the reason PSPU §3.C gives: computing on the collector's side is what lets access control constrain the computation.

1.4.3 Prometheus and OpenTelemetry #

Prometheus is a pull-based metrics system with a local time-series database; OpenTelemetry is a vendor-neutral collection framework spanning traces, metrics and logs.

eventd's metric store fills roughly the role of a local Prometheus TSDB, and takes the Prometheus data model — a name plus labels identifying a series, cumulative-bucket histograms — more or less wholesale (§5.2).

Different: eventd is pushed to rather than scraping. It implements no distributed tracing at all. And where OpenTelemetry's answer to the three-signal problem is a common collection framework in front of three backends, eventd's is one backend with one access control model and one query surface — which is the whole design bet.

1.4.4 Deliberately elsewhere #

eventd is the primary consumer of the KMES ring buffers. Events travel from shared memory to a committed database row in four stages.

1. Drain. One thread per CPU reads events from that CPU's ring buffer, following the lock-free read protocol PSPK specifies (§2.2).
2. Detect. The drain thread compares each event's sequence number against the last it saw for that CPU. A jump means events were lost, and the loss becomes a gap record (§2.5).
3. Hand off. The drain thread copies the event out of the mapped region and passes it to the writer thread that owns the shard it routes to (§2.3).
4. Write. The writer thread accumulates events into a transaction and commits, sizing the batch to whatever throughput allows (§2.4).

Two principles govern the whole pipeline, and most of its behaviour follows from them rather than from anything specific to a stage.

2.1.1 The ring buffers are the only buffer #

eventd holds no large intermediate queue between KMES and SQLite. The handoff channel is bounded by the maximum batch size and nothing else accumulates.

When the writer falls behind, the channel fills; when the channel is full, the drain thread stops reading; when the drain thread stops reading, events accumulate in the ring buffer, which is exactly what a ring buffer is for. Backpressure propagates all the way back to the kernel, and the absorption capacity is the ring buffer's, which an administrator already sizes.

If the ring buffer also fills, KMES overwrites its oldest events, the drain thread notices the sequence jump when it resumes, and the loss is recorded (§2.5). That is the designed worst case: eventd loses events visibly rather than buffering without bound and dying.

The alternative — a large in-process queue — would move the same capacity into a place where losing it is invisible, where it competes with the page cache for memory, and where an out-of-memory kill takes the whole queue with no record that it existed.

2.1.2 Sharding scales writes linearly #

Each shard is a self-contained SQLite database with its own file, its own write-ahead log and its own writer thread. Shards share no write-path state, so the write path has no cross-shard lock, no shared counter and no coordination point (§2.3).

The consequence for the query path is that a shard means nothing to it. A shard database holds whatever CPUs happened to route to it in whatever eventd lifetime wrote it, so a query filtering by CPU reads every shard, and the query path never assumes a relationship between a shard and a CPU (§6.4).

2.2.1 Attachment #

At startup eventd discovers the CPU count by calling kmes_attach with incrementing CPU identifiers from 0 until the call returns EINVAL. Each successful call returns one file descriptor for that CPU's ring buffer, and eventd maps each one. The mapping size is derived from the capacity value the call reports, as PSPK defines it.

Attachment requires SeSecurityPrivilege in the effective token, which is the privilege that grants an unfiltered view of every event on the system. eventd holds it because it is the party that then applies per-event access control on everything it stores (§7).

Discovering zero CPUs is a startup failure (§8.2). There is no configuration for the CPU count and no way to attach to a subset.

2.2.2 Drain threads #

There is one drain thread per CPU, and each reads exactly one ring buffer. A drain thread never reads another CPU's buffer, which is what makes per-CPU sequence tracking a thread-local variable rather than shared state.

Each thread follows the read protocol PSPK specifies:

• read_pos starts at tail_pos on first attachment, so eventd begins at the oldest surviving event rather than at the newest.
• The drain loop loads write_pos with acquire ordering, checks tail_pos for lapping, validates the event's structural integrity, and advances read_pos by event_size.
• After reading an event it re-reads tail_pos — the torn-read check — to detect that KMES overwrote the event while it was being copied.
• With nothing to read it uses the notification protocol, setting need_wake and waiting on the futex, rather than spinning.

2.2.3 Copying #

A drain thread copies the event — header and payload — into process-local memory before it advances read_pos.

Nothing derived from the mapped region ever reaches a writer thread. The region is producer-owned and KMES may overwrite any part of it the moment read_pos moves past, so a pointer handed across the channel would be a pointer into memory that another CPU is entitled to rewrite before the writer gets to it.

The copy is bounded by the event's own event_size, and the drain thread never reads beyond it.

2.2.4 Generation changes #

An administrator changing the ring buffer capacity causes KMES to replace the buffers, which it signals by changing the generation field. A drain thread checks it after each drain cycle, and on a change:

1. Records the sequence number of the last event it processed.
2. Calls kmes_attach again for its CPU, obtaining a descriptor for the resized buffer.
3. Maps the new buffer.
4. Unmaps the old one and closes the old descriptor.
5. Scans the new buffer for the first event whose sequence number exceeds the recorded one.
6. Resumes draining from there.

Each drain thread handles this independently. There is no barrier, no coordination and no shared state, because each attaches only to its own CPU's buffer — so a resize is a per-CPU event that happens to occur on every CPU at roughly the same time.

The scan in step 5 is what makes the transition lossless in both directions: no event is skipped, and none is written twice.

2.2.5 Sequence tracking #

Each drain thread holds the last sequence number it saw for its CPU. It serves gap detection (§2.5) and resumption after a generation change or a restart.

At startup, if committed rows already exist for the current boot, the resume point for each CPU is derived from those rows:

MAX(sequence)
WHERE boot_id = current_boot_id
  AND cpu_id = cpu
  AND sequence IS NOT NULL

across every readable event shard database, historical shards included. Historical shards can hold current-boot rows: an eventd restarted within one boot under a smaller shard count leaves its higher-numbered shards behind, and the events it wrote to them before the restart are still that boot's events.

The sequence IS NOT NULL clause excludes synthetic events, which have no sequence numbers (§2.6).

Committed rows are the authority. The metadata database holds sequence checkpoints and the shutdown event records the same numbers (§3.5, §8.4), but both are diagnostic: a checkpoint can be stale in exactly the case that matters, where eventd died between its last commit and its last checkpoint, and trusting it would mean re-reading events already stored or skipping a gap.

With no prior rows for the current boot, the tracker starts at 0. The first event on each CPU carries sequence number 1, so a tracker at 0 expects 1 and detects a gap correctly if the first event it sees is later.

2.3.1 The shard count #

Event writes are distributed across one to 256 independent SQLite databases. The count comes from StorageShards (§A); zero means "as many shards as there are CPUs", and is the default.

Two properties make a count perform well, and neither is enforced. A power of two lets routing use a bitwise AND rather than a modulo. A multiple of the CPU count distributes shards evenly across CPUs. The default satisfies the second by construction.

2.3.2 Assignment #

Shard-to-CPU assignment is computed once at startup and is fixed for the process lifetime.

For each CPU c, eventd assigns every shard j where j % cpu_count == c. If that produces nothing — which happens when there are fewer shards than CPUs — CPU c is instead assigned c % shard_count. Every CPU ends with at least one write path.

The three cases behave differently:

A drain thread owning several shards distributes its events round-robin, sending each successive event to the next shard it owns.

When the counts do not divide evenly, some CPUs carry one shard more than others, or some shards receive from one CPU more than others. The resulting imbalance is one shard's worth of throughput, which is negligible against the whole.

2.3.3 Shards are not a query-path concept #

Assignment is not persisted. A shard database accumulates events from whatever CPUs routed to it during whatever eventd lifetimes wrote it, so a single shard file may hold events from a different set of CPUs in different regions of its history.

The query path therefore assumes nothing: it reads every database in the directory, and a query filtering on cpu_id scans all of them (§6.4). Sharding is a write-path optimisation that the read path pays a fan-out for.

2.3.4 Writer threads #

Each shard has exactly one writer thread, and that thread is the only writer to that database. No other thread and no other connection writes to it, which is what makes the single-writer assumptions in §5.3 and §3.4 safe.

Drain threads never write to SQLite. When several drain threads share a shard they hand off concurrently, so the handoff channel is multi-producer and single-consumer.

2.3.5 The handoff channel #

Each writer thread has one bounded channel through which drain threads submit events. Its capacity does not exceed the maximum batch size (§2.4).

When the channel is full the drain thread stops reading from the ring buffer and waits. It does not drop events to relieve the pressure and it does not grow the channel. Events accumulate in the ring buffer instead, which is the designed path (§2.1):

writer slow → channel fills → drain pauses → ring buffer absorbs
            → KMES overwrites oldest if full → gap detected on resume

When the writer commits and the channel has room, the drain thread resumes immediately.

The channel is a staging area for one batch, not a second buffer. Its bound is the batch size precisely so that it cannot become one.

2.3.6 Lifecycle and reconfiguration #

Shard databases are created in the event store directory on first use. eventd never deletes or overwrites one left by a previous configuration: starting with fewer shards than exist leaves the excess in place, and the query path continues to read them (§3.3).

Changing StorageShards takes effect at the next restart. The configuration watch notices the change and eventd defers it rather than reassigning CPUs or creating shards while running (§8.3).

Shard count changes are expected to be rare — set once from the hardware profile, one for a small board and a multiple of the CPU count for a server, and then left alone. Live migration would mean rebalancing writer threads and channels while events are in flight, for a configuration change that happens once in a machine's life.

2.4.1 Transactions #

Each writer thread writes to its shard with explicit transactions: a BEGIN, one INSERT per event, a COMMIT. The commit is the durability boundary.

The database runs in WAL mode with synchronous = FULL, so every commit fsyncs the write-ahead log. This is the strictest of the three stores' settings, and the only one where per-transaction durability is bought at per-transaction cost — because an event may be an audit record and losing the last second of them to a power cut is a real loss.

2.4.2 Adaptive batch sizing #

The writer sizes each batch to balance throughput against how much sits uncommitted at any moment.

Throughput is always the priority. If eventd falls behind the emission rate, ring buffers fill and events are overwritten, which is irrecoverable; a shorter power-loss window is not worth that trade. The algorithm maximises resilience within the constraint that throughput is maintained, never against it.

1. When the first event is available, the writer opens a transaction and records the start time.
2. It reads available events from its drain threads and inserts them.
3. After each group of inserts it commits if any of these holds:
   • no assigned drain thread currently has an event available
   • the batch holds MaxBatchSize events
   • MaxBatchLatencyMs has elapsed since the first event entered it
4. Otherwise it keeps reading and inserting.
5. With nothing available and an empty batch, it sleeps until a producer wakes it.

The first condition is what makes the algorithm adaptive. Under light load the input drains immediately, so a batch of three events commits at once and the exposure window is microseconds. Under sustained load batches grow until they hit the size cap or the latency cap, whichever comes first, and the per-commit fsync is amortised across thousands of rows.

The writer chooses its own insert-group size, subject to a group never letting a batch exceed the size cap or stay open past the latency cap.

Both bounds are configuration (§A). The defaults are 10000 events and 100 milliseconds — the tightest latency of the three stores, for the same reason the durability setting is the strictest.

2.4.3 WAL checkpointing #

WAL mode accumulates log data until a checkpoint copies it back into the main database file. Under sustained writes the log grows.

Each writer triggers a checkpoint when its write-ahead log reaches WalCheckpointPages (§A), in SQLITE_CHECKPOINT_PASSIVE mode — checkpointing as much as it can without blocking readers. If a passive checkpoint cannot make progress because readers hold pages, the writer does not block: it keeps writing and retries after a later commit.

Checkpointing runs on the writer thread and briefly serialises with insert work, which is inherent to SQLite rather than a choice — a database cannot be checkpointed and written concurrently. Passive mode is the lightest option available, yielding immediately when readers hold pages, and the per-checkpoint cost is bounded by the threshold.

2.4.4 Prepared statements #

Each writer prepares its INSERT once at startup and reuses it for every row, which keeps SQL parsing and planning off the hot path entirely.

Every event carries a per-CPU, per-boot sequence number, and a drain thread knows what it last saw. That is the whole mechanism: an event whose sequence number exceeds the expected next one means the numbers in between belonged to events eventd never received.

2.5.1 Causes #

• Ring buffer overrun. KMES overwrote events before eventd read them. The most serious case: it means audit events were lost irrecoverably.
• Structural drops. KMES declined to write an event that exceeded its size limits.
• Downtime. Events emitted while eventd was not running.

All three appear identically at the consumer, which is why the record says what was lost rather than why.

2.5.2 Gap records #

On detecting a jump, the drain thread generates a gap record carrying:

• the CPU identifier
• the first missing sequence number, the last seen plus one
• the last missing sequence number, the revealing event's minus one
• the count of missing events
• the timestamp of the last event successfully processed on this CPU, where one is known
• the timestamp of the event that revealed the gap

The record is written into the shard database through the normal write path — handed to the same writer thread, batched with ordinary events, committed in the same transaction. It is not emitted through KMES, which would be circular: the mechanism for recording that the event transport lost something cannot depend on that transport.

The gap details are stored as a MessagePack map in the payload column (§3.2), and gap records are queryable exactly like any other event.

2.5.3 The CPU column #

A gap record populates cpu_id and leaves the other KMES header columns — sequence, origin_class, the identity GUIDs — null (§3.1).

cpu_id is populated deliberately, and it is the one place a synthetic event carries a header field. Without it, EVENTS WHERE cpu_id == 3 would return every event from CPU 3 except the record saying that events from CPU 3 went missing — which is the one record such a query most needs to return.

sequence stays null because a gap record has no place in the sequence: it describes numbers that were skipped, and giving it one of them would make it indistinguishable from the event that was lost. It is also what keeps gap records out of the resume-point derivation in §2.2.

2.5.4 Lapping #

If read_pos falls behind tail_pos, the consumer has been lapped and the PSPK read protocol advances it to tail_pos.

Lapping needs no special handling here. The next event read is the oldest survivor, its sequence number is far beyond what the thread expected, and ordinary gap detection records the difference. The lapping case and the restart case produce the same record by the same path.

Synthetic events are records eventd generates about itself. They are written straight to a shard database and never touch KMES.

They carry no KMES header: no identity stamps, no sequence number, no origin class. What they have is a wall-clock timestamp taken when eventd generated the record, and an event type string prefixed synthetic. which is what distinguishes them in the events table — no separate record-type column exists (§3.1).

2.6.1 When they are generated #

Payload schemas for all five are in §3.2.

Note what is absent: there is no synthetic event for malformed ingestion input. Log and metric datagrams arrive unauthenticated from arbitrary local processes, and emitting a durable record per bad datagram would hand every process an amplification primitive (PSPU §3.4). These five are conditions eventd observed about itself, not reactions to what it was sent.

2.6.2 Which shard #

CPU-specific synthetic events — gap records — go to the shard assigned to the CPU that generated them, handed to that writer thread alongside that CPU's ordinary events. A gap record travels with the events it describes.

Daemon-wide ones — startup, shutdown, configuration changes, storage errors — go to shard 0 when shard 0 is writable, and otherwise to the lowest-numbered writable active shard. If no shard is writable at all the event is skipped, with the failure logged to standard error.

A storage error is the case that needs the fallback. It describes a failure on one particular shard but is itself a daemon-wide notification, so it is not written to the failing shard unless that shard has since been replaced and is writable again (§9.2) — writing the record of a shard's failure into that shard would lose it exactly when it matters.

These events are infrequent enough that concentrating them on shard 0 costs nothing measurable in balance.

2.6.3 Storage and ordering #

Synthetic events live in the same shard databases as KMES events and participate in the same batching, the same retention and the same queries. Access control treats their types like any other (§7.2), so Machine\System\eventd\Security\Events\synthetic governs them.

They are ordered by their eventd-assigned timestamp and take no part in per-CPU sequence numbering.

That timestamp is when eventd noticed, not when the condition occurred. A gap record is stamped at detection, which may be long after the events it describes were overwritten — and after a restart, may be the first thing written in a new boot about events lost in the previous one.

Every shard database holds one events table.

3.1.1 Header fields are columns #

Every KMES header field is extracted into its own column rather than left inside the payload blob. That is what lets a predicate on process_guid or event_type become a SQL comparison rather than a decode of every candidate row, and it is what makes those fields indexable by ordinary column indexes (§3.4).

event_type is the sole discriminator between real and synthetic records. No record-type column exists, because the synthetic. prefix already partitions the type namespace and a second column would be a second thing to keep consistent.

3.1.2 The payload is not touched #

For a KMES event the payload column holds the bytes KMES delivered, unmodified. eventd does not decode them on the write path, does not re-encode them, and does not validate them beyond what the ring-buffer protocol already checked.

The payload is a MessagePack value whose schema belongs to the emitting subsystem, and eventd has no catalogue of those schemas. It decodes on the read path when a query needs a payload field (§6.1), which is also the only point at which the flattening rules of PSPU §3.22 apply.

Storing the bytes verbatim is also what keeps a payload field that collides with a header name recoverable: the value is suppressed from the query surface but remains in the blob.

3.1.3 Identity may be absent two ways #

effective_token_guid distinguishes two cases that would otherwise look alike. Null means the record is synthetic and never had an identity. The null GUID — sixteen zero bytes — means the record is a real KMES event whose identity was not available at emission time, because it was emitted before or outside a context that had one.

The distinction matters for audit: "eventd wrote this" and "the kernel emitted this and could not attribute it" are different facts.

3.1.4 Write-time indexes #

One index is created with the table:

• idx_events_timestamp on events(timestamp)

Time-range filtering is the foundational access pattern — nearly every query carries a SINCE — and it is the one index eventd never sheds, whatever the write pressure (§3.4). Every other index is the adaptive system's business.

3.1.5 Schema version #

Each shard holds a metadata table:

with two required entries: schema_version, and created_at as a UTC timestamp formatted YYYY-MM-DDTHH:MM:SSZ. The current version is in §B.

eventd checks the version at startup and applies the lifecycle rules of §3.3. It does not migrate: an unrecognised version is a startup failure for an active shard and an exclusion for a historical one. Migration is an administrative operation, deliberately not an automatic one — a daemon that silently rewrote an audit store's schema on first start after an upgrade would be doing the one thing an audit store must not do unattended.

Each of the five synthetic event types (§2.6) carries a MessagePack map in payload, with the schema below. These field names are stable query-language payload field names after flattening (PSPU §3.22), except where a value is a nested array or map, which flattening does not traverse.

3.2.1 synthetic.startup #

restart is the boot-boundary decision of §3.7 recorded as data, which makes "did eventd crash during this boot, and how often" answerable by query rather than by inference from gaps.

3.2.2 synthetic.shutdown #

Diagnostic only. Startup derives its resume points from committed rows, never from this payload (§2.2).

3.2.3 synthetic.gap #

cpu_id appears both here and in the cpu_id column (§2.5). The column is what a WHERE cpu_id == N predicate matches; the payload field is what a reader of the record sees without joining anything.

3.2.4 synthetic.config_change #

Values are rendered deterministically so that two eventd instances observing the same change record the same bytes: REG_SZ as the string in UTF-8, REG_DWORD and REG_QWORD as unsigned decimal without leading zeroes, REG_BINARY as lowercase hexadecimal, two digits per byte.

Everything is a string, including numbers, because the field is the same field for all five types and a query filtering WHERE key == "…" should not have to know which.

3.2.5 synthetic.storage_error #

error is diagnostic text and its wording is not stable. store and shard_index are the fields worth alerting on.

3.3.1 The event store directory #

Every shard database and the metadata database live in one directory, named by EventStorePath (§A). There is no compiled-in default: a missing or invalid value is a startup failure, and eventd writes event databases nowhere else.

eventd creates the directory if it is absent.

3.3.2 Naming #

Active shards are shard-NNNN.db, with the shard index zero-padded to four digits — the index assigned at startup, which is not a CPU number (§2.3).

Starting with more shards than exist creates the new ones. Starting with fewer leaves the excess in place: they become historical shards, are never deleted, and remain available to the query path.

3.3.3 Creation #

A shard database that does not exist is created with:

1. WAL mode, PRAGMA journal_mode=WAL
2. PRAGMA synchronous=FULL
3. the events and metadata tables (§3.1)
4. the idx_events_timestamp index
5. the schema_version and created_at entries

3.3.4 Opening an active shard #

1. Open in WAL mode.
2. Set synchronous to FULL.
3. Read and verify schema_version. Missing or unrecognised is a startup failure. No migration is attempted.
4. Verify structural integrity — the required tables and write-time indexes exist. Failing this, with SQLite reporting no corruption, is a startup failure.
5. If SQLite reports corruption while opening or verifying, quarantine and replace (below).

Steps 3 and 4 fail rather than repair because an active shard is required: eventd has no degraded mode that runs without one (§8.2).

3.3.5 Quarantine #

When SQLite reports corruption in a required store, eventd renames the database aside and starts a fresh one at the original path. The main database file and any matching -wal and -shm files are renamed with the suffix .corrupt.<timestamp_ns>, all three using the same suffix from one operation, and a new empty shard-NNNN.db is created.

If a target name is taken, eventd appends .N with the lowest positive integer that makes it unique — which happens when two quarantines land in the same nanosecond, and when a previous quarantine already used the name.

Quarantining rather than deleting is the point: the corrupt file is the only copy of whatever it held, recovering data from it is an administrative operation, and eventd attempts no automatic repair.

The corruption is logged and a synthetic.storage_error event is emitted once a shard is available to write it to (§9.2).

3.3.6 Historical shards #

A historical shard is never required for startup. If one has a missing or unrecognised schema version, fails structural verification, or cannot be opened read-only, eventd logs the error and excludes it from the query path for this run — it does not fail startup and does not quarantine it.

The asymmetry is deliberate. An active shard that will not open means eventd cannot do its job; a historical one that will not open means some old data is unreadable, which is a smaller problem than refusing to boot the audit daemon over it.

3.3.7 Query path discovery #

The query path opens every file in the directory matching shard-NNNN.db that has a recognised schema and passes structural verification — active and historical alike. It assumes no particular number of them.

It explicitly does not treat every .db file in the directory as a shard: eventd-meta.db is excluded by the naming pattern, along with anything else that happens to be there.

Each is opened with a read-only connection. Read-only connections in WAL mode do not contend with the writer's connection.

3.3.8 Concurrency #

Each shard has exactly one read-write connection, owned by its writer thread, and any number of read-only connections owned by query handlers. WAL mode permits concurrent readers alongside one writer without blocking either.

Writer threads never share connections. Each creates and owns its own for the process lifetime, which is what makes the prepared statement in §2.4 a per-thread object with no locking around it.

Secondary indexes make queries fast and writes slow. Which indexes are worth that trade depends on what a particular deployment actually queries, which varies between systems and over time, and which nobody wants to tune by hand.

eventd observes the queries and maintains the indexes they imply — subject throughout to the rule that throughput outranks query latency.

3.4.1 Three decoupled parts #

Query frequency counters. Query handlers increment a per-field counter when a field appears in a WHERE predicate. This is the write-heavy path — once per query, per predicate. Counters live in memory and are flushed periodically to the metadata database (§3.5), never to a shard, because they are global state that must survive shard reconfiguration.

Index policy. A periodic process reads the counters, applies the creation and removal thresholds, and computes the desired index set — an ordered list of fields, highest priority first. It runs every AdaptiveIndexPolicyIntervalMinutes (§A), and it is the only writer to the desired set.

Shard convergence. Writer threads read the desired set and move their material indexes toward it. They never read the counters and never write the desired set.

The separation exists so that the high-frequency counter updates never contend with the writer threads. The policy is the bridge between them and runs on the order of once an hour, so it is never a contention point.

The desired set is global — one list applying to every shard. Individual shards do not make independent decisions; they differ only in how far they have got.

3.4.2 Convergence #

A shard converges when it is quiet. When a writer thread has no pending events and its material indexes do not match the desired set, it takes one convergence action — creating the highest-priority missing index, or dropping the lowest-priority material index no longer wanted — then rechecks write pressure before considering another.

Creation uses CREATE INDEX IF NOT EXISTS; removal uses DROP INDEX IF EXISTS. Both run on the shard's writer thread, which is the only thread permitted to write that database (§2.3).

Index creation is cancellable. If drain threads detect rising write pressure during a build, they signal the writer to abort; the writer cancels the CREATE INDEX, SQLite rolls back the partial index cleanly, and the writer returns to event batches immediately. The abandoned build is retried at the next quiet period.

Cancellation responsiveness matters more than it looks. sqlite3_interrupt sets a flag checked at SQL VM opcode boundaries, and during B-tree construction for a large index the gap between checks can be tens of milliseconds — long enough to overrun a ring buffer at a high event rate. sqlite3_progress_handler(), registering a callback invoked every thousand opcodes that checks a cancellation flag and returns non-zero to abort, gives cancellation that tracks the pressure signal rather than lagging it.

Shards converge at their own pace. One under sustained pressure may lag the desired set indefinitely, and that is the correct outcome: it is prioritising throughput.

3.4.3 Shedding #

Under sustained pressure a shard drops indexes to cut per-insert cost.

Graduated shedding. If more than SheddingBatchPercent of a shard's batches within a SheddingWindowSeconds sliding window exceeded 75% of MaxBatchSize, the shard drops its lowest-priority secondary index — the one whose column has the lowest query frequency in the desired set. If pressure persists, the next-lowest goes, and so on. The check runs once per batch commit.

Emergency shedding. If a shard is at maximum batch size and its drain thread signals rising ring buffer pressure, it drops all secondary indexes at once. DROP INDEX is a metadata operation measured in milliseconds, so it is safe to do under pressure in a way that creation is not.

The pressure signal comes from the drain thread watching the gap between write_pos and its own read_pos. Exceeding EmergencySheddingBufferPercent of ring buffer capacity raises it. It is a distinct signal from the index-build cancellation one: this triggers shedding whether or not a build is in progress.

idx_events_timestamp is exempt. It is never shed at any pressure, because time-range queries are the access pattern everything else is built on and the store is unusable without it.

When pressure subsides, shedding reverses: the shard rebuilds toward the desired set under the same quiet-period scheduling and the same cancellability, highest priority first.

3.4.4 Candidates #

Any field that can appear in a WHERE predicate is a candidate.

Header columns. event_type, origin_class, cpu_id, effective_token_guid, true_token_guid, process_guid, boot_id. timestamp is always indexed and is not adaptively managed.

Payload fields. Any queryable flattened path that appears in a predicate is a candidate for an expression index. A path suppressed by the flattening rules of PSPU §3.22 — a top-level key colliding with a header field, a key that is not a valid segment, a duplicate path — never receives one, because it is not a query-language field at all.

The raw payload column never receives a plain column index. Indexing an opaque blob accelerates nothing.

3.4.5 Payload indexes are an optimisation, not an authority #

An expression index extracts a field from the payload on every insert and indexes a deterministic private key for it. The exact key bytes are internal to eventd and are not part of the storage contract.

eventd may implement these as SQLite expression indexes with deterministic extraction functions, as generated columns, or by any equivalent SQLite-backed means. What every mechanism has in common is that it implements the same field resolution and flattening as PSPU §3.22 — and that where the index cannot reproduce the query language's comparison semantics exactly, it is used only to narrow candidate rows, with the real predicate applied after the row is loaded (§6.3).

SQLite's native dynamic-type equality and ordering never substitute for the query language's string case folding, numeric comparison, binary comparison, array comparison, or null and missing-field handling. Getting a smaller answer faster is worthless if it is a different answer.

Rows where the field is absent, unqueryable or suppressed index as null, or are otherwise excluded in a way that preserves those semantics.

Payload indexes are otherwise ordinary members of the desired set, with the same priority ordering, shedding and convergence.

3.4.6 Naming #

Header column indexes are idx_events_<column> — idx_events_event_type, idx_events_process_guid.

Payload expression indexes are named from the field GUID (§7.3), to avoid both collisions and characters SQLite will not accept in an identifier:

idx_events_payload_<field_guid_hex>

where field_guid_hex is the UUID v5 field GUID for the flattened path, as 32 lowercase hexadecimal digits with braces and hyphens stripped. The path source.name yields idx_events_payload_ followed by the 32 hex digits of uuid_v5(EVENTD_FIELD_NAMESPACE, "source.name").

Deriving the name from the GUID rather than from the path means the same path always produces the same index name, and no path — however it is spelled — can produce a name that collides with another's or that SQLite rejects.

One database in the event store directory is not a shard: eventd-meta.db. It holds the state that is global to eventd rather than to any shard — the adaptive index and rollup state, diagnostic sequence checkpoints, and the administrative Security Descriptor — and it is the one database that survives shard reconfiguration untouched.

It is created on first startup if absent, opened in WAL mode with synchronous=NORMAL. It is written once per policy interval and read at startup, so per-transaction durability buys nothing: losing the last interval's counters costs some adaptation, not any data.

The query path excludes it explicitly, since it is in the same directory as the shards (§3.3).

3.5.1 Tables #

index_counters — query frequency per field (§3.4).

desired_indexes — the computed desired index set.

rollup_counters and desired_rollups — the same pair for metric rollups (§5.6), keyed by function_window, a composite of function name and window size such as avg_3600.

desired_rollups carries function_window and priority.

sequence_checkpoints — diagnostic only.

Primary key (boot_id, cpu_id). Startup resumption derives its points from committed event rows and never from this table (§2.2). The table exists so that an operator can see what eventd believed at shutdown and compare it with what the rows say — the two disagreeing is itself diagnostic.

meta — key-value.

with three required entries: schema_version (§B), created_at as a UTC YYYY-MM-DDTHH:MM:SSZ string, and admin_sd, a self-relative Security Descriptor governing administrative operations — the INDEX command above all (§7.2).

The default admin_sd grants SYSTEM and Administrators.

3.5.2 Concurrency #

One writer connection, owned by the index and rollup policy thread. No other thread opens the database read-write.

Query handlers write only to the in-memory counters; the policy thread flushes them at each interval. Writer threads and query handlers read the desired sets from memory, never from the database. Graceful shutdown writes sequence_checkpoints through the same connection, after policy activity has stopped (§8.4).

With a single writer and no other database-level access, SQLite's WAL mode is the whole of the concurrency control needed.

The policy thread checkpoints the write-ahead log at WalCheckpointPages in passive mode, and does not block when readers hold pages — the same rule as every other store (§2.4).

3.5.3 Recovery is cheap #

If the schema version is missing or unrecognised, or any required table or meta entry is missing or malformed, eventd logs an error and recreates the database from defaults.

This is the opposite of the rule for a shard, which fails startup (§3.3), and the difference is what is at stake. A shard holds the only copy of audit data. This database holds an optimisation policy, some diagnostics, and a descriptor with a known default — all of it reconstructible, none of it irreplaceable. Losing it costs the adaptation eventd had accumulated, and the counters begin refilling immediately.

The one thing recreation does lose is a customised admin_sd, which reverts to the default.

3.5.4 Startup #

1. Open eventd-meta.db, creating it if absent.
2. Verify the schema version and required meta entries; recreate from defaults on failure.
3. Load index_counters and rollup_counters into memory.
4. Load desired_indexes and desired_rollups into memory.
5. Load sequence_checkpoints, for diagnostics only.
6. Discover the material indexes in each shard from its schema and compare against the desired set.

eventd resumes convergence from wherever each shard happens to be. It neither drops nor rebuilds indexes at startup: a shard's material set is a fact to be observed, not a state to be restored.

Retention bounds disk growth. eventd deletes on two axes, age and size, and enforces both — an event goes when it exceeds either threshold.

The v0.23 model is deliberately minimal, and a later one is expected to support rules resembling queries: retain KACS events for ninety days, synthetic events for seven, userspace-origin events for fourteen; and to prune during ingestion rather than only in arrears. What is here is the least that prevents unbounded growth.

3.6.1 Age #

Rows are deleted from events where timestamp is older than EventRetentionDays (§A) from the current wall clock, until none remain. Each shard is processed independently, and the rule covers KMES events, synthetic events and gap records alike.

3.6.2 Size #

Size is measured as logical live size, not file size:

logical_live_bytes = (page_count - freelist_count) * page_size

from PRAGMA page_count, PRAGMA freelist_count and PRAGMA page_size, taken after attempting a passive WAL checkpoint. The event store's total is the sum across every shard.

Pages freed by retention do not count, because they are reusable by future inserts. Counting them would make retention chase its own tail: each deletion would free pages that still counted against the limit, prompting more deletion.

When EventRetentionMaxBytes is non-zero and the total exceeds it:

1. Identify every non-current boot ID present in the shards.
2. Order them by their newest event timestamp, oldest boot first.
3. Delete each of those boots entirely, across all shards, one boot at a time, until the total is within the limit or no non-current boots remain.
4. If still over, delete the oldest events of the current boot by timestamp, across all shards, until within the limit.

Size pressure prefers boot boundaries. Deleting a whole old boot removes a self-contained unit — its sequence numbers, its gap records and its startup event go together — and it preserves recent events across the boundary, which is what an operator investigating a reboot needs. Only when whole boots are exhausted does eventd start on the current one.

3.6.3 Running it #

Retention runs on a background thread of its own, never on a writer or drain thread, using a separate read-write connection per store. It processes the event store first, then the log store (§4.4), then the metric store (§5.5). The interval is RetentionCheckIntervalMinutes (§A).

WAL mode lets a reader run alongside a writer, but not a second writer, so the retention thread coordinates with each shard's writer thread — a shard-level mutex taken before writing, which the writer briefly yields to.

Deletion is batched. Each transaction deletes at most RetentionDeleteBatchRows and commits before the next, and between batches the retention thread releases the coordination primitive and rechecks writer pressure. A single unbatched DELETE over a month of events would hold a write transaction for as long as it took, blocking the writer thread and, behind it, the drain thread and the ring buffer.

3.6.4 Reclamation #

Deleting rows does not shrink a SQLite file. Freed pages are reused by later inserts, and reclaiming filesystem space needs VACUUM, which rewrites the whole database.

eventd never runs VACUUM automatically. Reclamation is an explicit administrative operation.

In steady state it is not needed. Where ingestion and retention run at comparable rates, the file settles at roughly the high-water mark of retained data and the freed pages are recycled without further growth — which is also why logical live size, rather than file size, is the right thing to measure.

Every record eventd stores — every event, every log line, every raw metric sample — carries a boot_id: a 16-byte GUID identifying the boot that produced it. peinit assigns it at each boot and eventd reads it at startup.

Derived metric rollups are the exception. They are boot-agnostic aggregates and carry no boot_id (§5.6).

3.7.1 What it is for #

Disambiguation. KMES per-CPU sequence numbers restart at zero each boot. Without a boot ID, sequence 42 from one boot is indistinguishable from sequence 42 from the next, and every gap calculation across a reboot would be wrong.

Lifecycle. Retention can delete a whole boot as a unit rather than scanning by timestamp, and a boot is the natural unit to delete: its events, its gaps and its startup record go together (§3.6).

3.7.2 Where it is stored #

The log store records it because log output can mean different things across boots — a service configured differently at boot time says different things.

The metric store records it per sample but does not make it part of series identity (§5.2). A time series stays continuous across a reboot, which is what a chart of CPU usage across a restart should show, and a query that wants one boot's worth filters for it explicitly (PSPU §3.25). Rollups stay boot-agnostic scalars, so a boot-filtered metric query is served from raw samples and never from a rollup.

3.7.3 Uniqueness #

Within one boot an event is uniquely identified by (cpu_id, sequence). Across boots, boot_id supplies the disambiguating dimension, and the triple (boot_id, cpu_id, sequence) is globally unique.

3.7.4 Detecting the boundary #

At startup eventd reads the current boot ID from peinit, then searches every readable event shard database — historical shards included — for committed rows carrying it.

No committed rows for this boot. This is the boot's first eventd start. eventd resets every per-CPU sequence tracker to 0, records the new boot ID for all subsequent writes to all three stores, and emits synthetic.startup with restart false.

Committed rows exist. eventd crashed and peinit restarted it within the same boot. eventd restores each CPU's tracker from the maximum non-null sequence for (boot_id, cpu_id) across every readable shard, with CPUs having no rows resuming at 0; continues writing under the existing boot ID; and emits synthetic.startup with restart true.

The two cases are distinguished by the data itself rather than by any flag eventd persisted. That is the point: a flag would have to be written at a moment eventd might not reach, and a crash is precisely the case where it did not.

Committed rows are the authority throughout. The metadata database's sequence checkpoints and the previous synthetic.shutdown payload record the same numbers, and both are diagnostic — neither is consulted for resumption (§2.2, §3.5).

One thread reads datagrams from the log socket and writes log records to the log store. It is independent of the event drain and writer threads, so log ingestion never contends with event ingestion.

The wire contract — socket type, datagram ceiling, record format, and exactly which malformations cost what — is PSPU §3.6 to §3.8. What follows is what eventd does with a record once it has one.

4.1.1 One thread does both jobs #

The log thread performs both the socket reads and the SQLite writes.

The consequence is direct: during a batch commit the socket is not being drained, and datagrams arriving in that window occupy the receive queue until it fills, after which the kernel discards them. The queue — SO_RCVBUF — is sized at four times the datagram ceiling, and it is the whole cushion.

Splitting into a reader and a writer with a bounded handoff — the shape the event path uses (§2.3) — would decouple them. It is not done, and the reasoning is that log loss is tolerable by design (PSPU §3.4), log volume is normally well below event volume, and the single-thread model avoids a handoff channel and its backpressure semantics entirely.

Where log throughput does become the constraint, sharding the log store the way the event store is sharded is the larger lever; splitting the thread only moves the stall.

4.1.2 Batching #

The writer batches on the same adaptive principle as the event writer (§2.4), with the socket receive queue as its input. A transaction opens when the first valid record is available and commits when any of these holds:

• no further datagram is immediately available in the receive queue
• the batch holds LogMaxBatchSize records
• LogMaxBatchLatencyMs has elapsed since the first record entered it

If a datagram yields more valid records than fit in the remaining space, the writer commits, then continues with the same datagram in a new transaction. A transaction never exceeds the size cap and never stays open past the latency cap — a batched datagram cannot smuggle a larger transaction past either.

The defaults (§A) are 5000 records and 500 milliseconds. The latency is five times the event writer's, because log loss on power failure is acceptable where event loss is not, and larger, less frequent transactions are more efficient at the moderate volumes logs normally run at.

4.1.3 Durability #

The log store runs in WAL mode with synchronous=NORMAL, not FULL.

NORMAL syncs at checkpoint time rather than at every commit. It is durable against process crashes — the write-ahead log survives — but not against power loss, where commits since the last checkpoint may be gone.

This is a deliberate divergence from the event store, and it is the single clearest expression of the hierarchy the whole daemon is organised around: events are sacred, logs are not. Paying an fsync per transaction to protect data whose loss is defined as acceptable would be paying for nothing.

4.1.4 Adding to the record #

eventd supplies the boot_id and, where the producer omitted timestamp, its own clock at receipt. Everything else is stored as given — message byte for byte (PSPU §3.8).

The log store is a single SQLite database — not a directory of shards. There is no sharding here: one ingestion thread produces the writes, so splitting the target would give a single writer several files to switch between rather than several writers working in parallel.

It holds one logs table.

The schema is deliberately narrow. A log record is text with light metadata: what produced it, whether it was an error, when, and optionally which execution it belongs to. There is no payload blob and no origin class, and the only identity-like field is the optional correlation key — which is not an identity at all, since origin is self-asserted and unverified (PSPU §3.28).

A program needing more structure than this emits events.

4.2.1 is_error is an integer here and a boolean there #

The column stores 0 or 1; the query language exposes a boolean, and accepts either WHERE is_error == true or WHERE is_error == 1 (PSPU §3.22). ERROR ONLY is sugar for the first.

4.2.2 Write-time indexes #

Three indexes are created with the table:

• idx_logs_timestamp on logs(timestamp) — time-range filtering, as everywhere.
• idx_logs_origin on logs(origin) — "show me logs from X", which is the dominant log query.
• idx_logs_job_id on logs(job_id) WHERE job_id IS NOT NULL — a partial index for "show me logs for job X". Partial because directly-submitted lines carry no correlation key, so only correlated lines are worth indexing.

The origin index costs write amplification beyond the timestamp index, and the cost is modest in practice: origin has low cardinality, tens of distinct names on a normal system, so its index pages stay in SQLite's page cache and insertion stays cheap. The trade is accepted deliberately — the two dominant log queries must not become full table scans.

4.2.3 No adaptive indexing #

The log store does not participate in adaptive indexing (§3.4). Its field set is closed and small, and the three write-time indexes already cover the access patterns; there is no space of candidate fields for a policy to discover.

The same follows for query frequency counters: log queries do not increment them (§6.5).

4.2.4 Schema version #

The log store holds a metadata table with the same two-column structure as a shard's (§3.1). Its version is in §B.

eventd checks it at startup and applies the lifecycle rules of §4.3, and does not migrate.

4.3.1 Path #

The log store is the file named by LogStorePath (§A) — a file path, unlike the event store's directory. There is no compiled-in default: a missing or invalid value is a startup failure.

eventd creates the file and any absent parent directories.

4.3.2 Creation #

A log store that does not exist is created with:

1. WAL mode, PRAGMA journal_mode=WAL
2. PRAGMA synchronous=NORMAL
3. the logs and metadata tables (§4.2)
4. the idx_logs_timestamp, idx_logs_origin and idx_logs_job_id indexes
5. the schema_version and created_at entries

4.3.3 Opening #

1. Open in WAL mode.
2. Set synchronous to NORMAL.
3. Verify the schema version. Missing or unrecognised is a startup failure; no migration is attempted.
4. Verify structural integrity — required tables and indexes present. Failing this, with SQLite reporting no corruption, is a startup failure.
5. On SQLite reporting corruption, quarantine and replace.

Quarantine works exactly as for a shard (§3.3): the database, -wal and -shm files are renamed with a shared .corrupt.<timestamp_ns> suffix, .N appended with the lowest positive integer if the name is taken, and a fresh empty log store is created at the configured path.

The log store is a required store. There is no degraded mode in which eventd runs without one (§8.2), which is why steps 3 and 4 fail startup rather than proceeding without logs.

4.3.4 Concurrency #

One read-write connection owned by the log writer thread, and any number of read-only connections owned by query handlers. WAL mode lets them run concurrently.

4.3.5 Checkpointing #

The log writer checkpoints when its write-ahead log reaches WalCheckpointPages (§A), in passive mode, and does not block if readers hold pages — it keeps writing and retries after a later commit.

Checkpointing matters more here than in the event store, because synchronous=NORMAL makes the checkpoint the durability boundary rather than merely a space-reclamation event: data committed since the last checkpoint is what a power cut takes (§9.5).

Log retention works exactly as event retention does (§3.6), on the same background thread, running after the event store and before the metric store. As there, the v0.23 model is an early simplification and both limits are enforced with the more aggressive one winning.

4.4.1 Age and size #

Rows older than LogRetentionDays (§A) are deleted from logs until none remain.

If LogRetentionMaxBytes is non-zero and the store's logical live size exceeds it, the oldest entries by timestamp are deleted until it is within the limit. Logical live size is the same measure as §3.6 — (page_count - freelist_count) * page_size after attempting a passive checkpoint — and freed pages do not count.

There is no boot-boundary preference here. Event size retention prefers to drop whole old boots because a boot is a self-contained unit of sequence-numbered records; a log line has no such structure, so oldest first is the whole rule.

4.4.2 The default is shorter than events' #

Fourteen days against the event store's thirty (§A).

Historical log data is worth less than historical audit data, and it is usually bulkier per unit of value. The metric store's default is longer than either at ninety days, for the opposite reason: a metric sample is tiny and trend data is worth more the further back it goes (§5.5).

4.4.3 Batching #

Deletion is batched at RetentionDeleteBatchRows per transaction, with a commit between batches. Between them the retention thread releases any writer coordination primitive and rechecks writer pressure.

The stall this avoids is the log ingestion thread's, and that thread is also the one draining the socket (§4.1) — so a retention pass holding a long write transaction would not merely delay writes, it would stop the socket being read and lose the datagrams that arrived meanwhile.

4.4.4 Reclamation #

VACUUM is never run automatically, as everywhere. Freed pages are recycled by later inserts and are excluded from the size measure.

One thread reads datagrams from the metric socket and writes samples to the metric store, independent of both the event and log paths. It has the same single-thread shape as the log writer, with the same consequence during a commit (§4.1).

The wire contract is PSPU §3.9 to §3.13.

5.1.1 Processing a record #

For each valid record:

1. Resolve the series from name and labels — and, for a histogram, bucket boundaries — through the in-memory series cache (§5.3). A series that does not exist is inserted into series with the record's type, and the cache is updated.
2. Check the type. If the record's type differs from the resolved series' type, the record is dropped silently. The type is set at creation and is immutable; a series never changes type.
3. Insert the sample into samples with the resolved series_id, the timestamp and the value. SQLite assigns samples.id, which is the deterministic tiebreaker among samples sharing a timestamp (§5.2). A histogram's data is encoded as the canonical MessagePack sample map and stored in histogram_data.

Step 2 is the failure that leaves no trace. A producer that changes a metric's type has silently stopped emitting it — every sample discarded, no event, no counter, nothing in any log — and the only symptom is a series that stopped advancing. The reason nothing is emitted is PSPU §3.4: ingestion is unauthenticated, and reacting to input at all is an amplification vector.

5.1.2 Out-of-order samples #

The writer stores a valid sample whose timestamp precedes samples already held for that series.

Producers batch, clocks step, and sweeps get retried. Refusing late samples would convert any of those into silent loss, so eventd accepts them and defines every ordering it performs over (timestamp, id) rather than over insertion order — which is what makes rollup computation and RATE evaluation deterministic regardless of arrival (§5.6, §6.2).

5.1.3 Batching #

The same adaptive algorithm as the event and log writers (§2.4), with the socket receive queue as input. A transaction opens at the first valid sample and commits when any of these holds:

• no further datagram is immediately available
• the batch holds MetricMaxBatchSize samples
• MetricMaxBatchLatencyMs has elapsed since the first sample entered it

A datagram yielding more samples than fit is split across transactions, the writer committing before continuing with the same datagram. Neither cap is ever exceeded.

The defaults (§A) are 5000 samples and 1000 milliseconds — the longest latency of the three writers. Metrics are typically sampled every fifteen seconds, so a one-second commit window accumulates a whole sweep's worth without any latency that a dashboard could notice. Under a burst, where a collection agent submits every core, disk and interface at once, the size cap is what forces a timely commit.

5.1.4 Durability #

WAL mode with synchronous=NORMAL, as the log store (§4.1). Metric loss on power failure is acceptable, so per-transaction fsync buys nothing.

The metric store is a single SQLite database, and unlike the event and log stores it is organised around series rather than records. Individual samples are appended to a series that already has an identity.

5.2.1 The series table #

5.2.1.1 The canonical label string #

Labels are sorted by key in unsigned UTF-8 byte order, each pair written key=value, and the pairs joined with commas: core=0,host=server1. The empty label set is the empty string.

No escaping is performed and none is needed, because ingestion rejects = and , inside a key or a value (PSPU §3.10). That prohibition exists precisely to make this encoding unambiguous, and it is the reason the constraint binds the producer rather than being handled internally.

5.2.1.2 The boundaries blob #

A fixed binary encoding, not MessagePack:

1. boundary_count, u32 little-endian
2. that many IEEE-754 f64 values, each little-endian, in the validated order the producer sent

It exists only to identify histogram series and to resolve hash collisions, and is never returned in a query result.

5.2.1.3 Hashes narrow, they do not decide #

label_hash and boundaries_hash are 64-bit FNV-1a over the exact bytes of the canonical string or blob, with offset basis 0xcbf29ce484222325 and prime 0x100000001b3. The high bit is cleared before storage, hash & 0x7fff_ffff_ffff_ffff, so the value always fits SQLite's signed INTEGER.

A lookup always verifies the full labels string, and for a histogram the full boundaries blob, after narrowing by hash. A hash is an index key, never an identity: two label sets that collide are still two series.

5.2.1.4 Uniqueness #

The table carries UNIQUE(name, labels, boundaries_hash).

For counters and gauges boundaries_hash is null, and SQLite treats nulls as distinct in a unique constraint — so the constraint does not enforce uniqueness for them. What does is the single-writer resolution logic (§5.3), which checks before inserting. The constraint is a defensive backstop against a future change that introduces a second write path, not the primary mechanism.

type is not part of the identity. A record resolving to an existing series with a different type resolves successfully and is then dropped for the mismatch (§5.1).

5.2.2 The samples table #

For a histogram, histogram_data is a canonical MessagePack map (PSPU §3.5) with exactly four keys: boundaries, an array of float64 in the producer's order; counts, an array of unsigned integers; total_count; and sum, a finite float64.

value is a placeholder for histogram rows and is never returned as a metric query value. Storing 0 rather than null keeps the column NOT NULL and keeps the row layout uniform.

Canonical encoding is required here because a stored sample map must be byte-stable: two equal histograms encode identically, which is what makes them comparable without decoding.

boot_id is per sample and is not part of the series identity, so a series stays continuous across a reboot (§3.7).

5.2.3 Ordering #

Query execution order within a series is always (timestamp, id) ascending, never insertion order alone. Duplicate timestamps are permitted and id gives them a stable order.

id is internal. It is never exposed as a query field, a result field, an access-control field, or a reserved label key (PSPU §3.28).

5.2.4 The rollups table #

The database also holds rollups, defined in §5.6. Rollups are boot-agnostic scalar aggregates and carry no boot_id, which is why a boot-filtered metric query is never served from one.

5.2.5 Write-time indexes #

• idx_samples_series_timestamp on samples(series_id, timestamp, id) — the dominant pattern is "samples for series X over range Y in deterministic order", and this one composite index serves the series lookup, the range filter and the (timestamp, id) ordering in a single scan.
• idx_series_name on series(name) — name lookups.
• idx_series_label_hash on series(label_hash) — series resolution on the ingestion path.
• idx_rollups_series_function_window on rollups(series_id, function, window_seconds, window_start).

5.2.6 Schema version #

A metadata table with the same structure as the other stores' (§3.1). Version 1 comprises series, samples, rollups and metadata; the current value is in §B. eventd checks it at startup, applies the lifecycle rules of §5.4, and does not migrate.

Every arriving sample must be turned into a series_id before it can be inserted. This happens once per sample on the single metric ingestion thread, so it is the hottest lookup in the daemon and the reason a cache exists at all.

5.3.1 Resolving #

1. Compute the canonical label string: sort by key in unsigned UTF-8 byte order, encode each pair key=value, join with commas. The empty label set encodes as the empty string. No escaping — ingestion has already rejected the delimiters (§5.2).
2. Hash it.
3. For a histogram, compute the canonical boundary blob and its hash from the validated, producer-supplied order. eventd never sorts boundaries. For counters and gauges both are null.
4. Look up series by name, label_hash, and for histograms boundaries_hash.
5. On a match, verify the full labels string, and for histograms the full boundary blob. If the record's type differs from the existing series' type, drop the record (§5.1). Otherwise use the existing series_id.
6. On no match, insert a new series row and use the new identifier.

A histogram whose boundaries changed takes step 6: it is a new series (PSPU §3.13). The old one keeps its historical samples and the new one starts accumulating.

5.3.2 The cache #

Resolution runs through a bounded in-memory cache mapping (name, canonical labels, boundaries hash, boundaries blob) to series_id. For counters and gauges the boundary components are absent.

A hit is a hash table lookup with no SQLite involvement. A miss costs one SELECT on name and label_hash, after which the result is inserted, evicting the least recently used entry if the cache is full.

The bound is MetricSeriesCacheSize (§A), default 50000, with LRU eviction. It bounds memory, not the number of series: the series table is uncapped and a new series is always created in the database. A system with a million series and a 50000-entry cache uses memory proportional to the cache, and at roughly 200 to 300 bytes an entry the default costs 10 to 15 MB.

The cache starts empty after a restart and is warmed on demand — within one collection cycle, typically fifteen seconds, every active series is cached. There is no pre-warming pass, because reading a million-row series table at startup to populate a 50000-entry cache would be work spent to discard most of its result.

5.3.3 Sizing it #

The cache is sized for the set of actively reporting series, and behaves badly below it.

Below that, LRU does not help, because every series is equally hot: each collection cycle evicts the overflow and reloads it, producing a fixed number of SELECTs every cycle, permanently. A system with 55000 active series and a 50000-entry cache incurs about 5000 cache misses every fifteen seconds, indefinitely.

This interacts badly with label cardinality (PSPU §3.10). Labels with unbounded values — request identifiers, user-supplied strings — grow the series table without limit, and once the active set exceeds the cache every cycle pays the eviction cost on the one thread that also drains the metric socket. The failure presents as metric loss, because the thread stops reading while it queries.

eventd does not defend against this and cannot: at the interface, a producer creating a genuinely new series is indistinguishable from one creating garbage, and every available defence would break a correct producer to inconvenience an incorrect one (PSPU §3.13).

5.4.1 Path #

The file named by MetricStorePath (§A). No compiled-in default; a missing or invalid value is a startup failure. eventd creates the file and any absent parent directories.

5.4.2 Creation #

1. WAL mode.
2. PRAGMA synchronous=NORMAL — the log store's reasoning, for the same reason: metric loss on power failure is acceptable (§4.1).
3. The series, samples, rollups and metadata tables (§5.2, §5.6).
4. Every write-time index.
5. The schema_version and created_at entries.

5.4.3 Opening #

1. Open in WAL mode with synchronous NORMAL.
2. Verify the schema version. Missing or unrecognised is a startup failure; no migration.
3. Verify structural integrity — required tables and indexes present, including rollups and its lookup index. Failing this, with SQLite reporting no corruption, is a startup failure.
4. On SQLite reporting corruption, quarantine and replace, exactly as for a shard (§3.3): matching -wal and -shm files renamed with the same .corrupt.<timestamp_ns> suffix, .N appended if the name is taken, and a fresh empty store created at the configured path.

The metric store is a required store; there is no degraded mode without one (§8.2).

After opening or creation the series cache is empty and fills on demand (§5.3).

5.4.4 Concurrency #

One read-write connection owned by the metric writer thread, and any number of read-only query connections. WAL mode permits both concurrently.

The single-writer property is load-bearing here in a way it is not for the other stores: series resolution checks for an existing row and then inserts, without a transaction spanning both, and only one writer makes that safe (§5.2).

5.4.5 Checkpointing #

The metric writer checkpoints at WalCheckpointPages (§A) in passive mode, and does not block when readers hold pages.

As with the log store, the checkpoint is the durability boundary under synchronous=NORMAL, not merely space reclamation (§9.5).

Metric retention runs on the same background thread as the other two, after the log store (§3.6, §4.4). Both limits are enforced and the more aggressive wins.

5.5.1 The pass #

1. Delete rows from samples older than MetricRetentionDays (§A) until none remain.
2. If MetricRetentionMaxBytes is non-zero and the store's logical live size exceeds it, delete the oldest samples by timestamp until it is within the limit.
3. Track every series_id whose samples were deleted by either step.
4. Delete every rollups row for those series.
5. Delete every series row with no remaining samples.

Logical live size is the same measure as §3.6.

Step 4 is not optional bookkeeping. A rollup is a pre-computed aggregate of raw samples, so a rollup outliving its inputs would be served to a query as an exact answer computed from data the store no longer has — and the query engine, finding a matching rollup, would never look at the raw samples to notice (§5.6). Rollups for the affected series can be recomputed later from whatever samples remain.

Step 5 removes the definitions of series nobody produces any more, which is the only mechanism that ever removes a series row. A series that stopped receiving samples persists until its last sample ages out — ninety days by default — so a burst of short-lived series from a high-cardinality producer stays in the table for that long.

5.5.2 The longest default #

Ninety days, against fourteen for logs and thirty for events (§A).

A metric sample is small and its value grows with age: a year of CPU utilisation is a capacity-planning input in a way that a year of log lines is not. A thousand series sampled every fifteen seconds produce about 5.7 million samples a day, which is a few hundred megabytes in SQLite.

5.5.3 Not yet: downsampling #

The v0.23 model deletes; it does not downsample. A later retention engine is expected to aggregate high-resolution data into lower-resolution rollups as it ages — per-second samples becoming five-minute averages after a week, hourly averages after a month — which is what makes long-term metric retention affordable while preserving the trend. The rollups table is the mechanism that would serve it, but nothing currently promotes raw samples into rollups as a retention action.

5.5.4 Batching and reclamation #

Batched at RetentionDeleteBatchRows per transaction, with the coordination primitive released and writer pressure rechecked between batches — the same rule and the same reason as §4.4, since the metric writer is also the thread draining the metric socket.

VACUUM is never run automatically. Freed pages are recycled and are excluded from the size measure.

Aggregate metric queries read raw samples. Over a large range that means scanning thousands or millions of rows to produce a handful of numbers, and the same numbers over and over.

Rollups pre-compute them. The principle is adaptive indexing's (§3.4): watch which query patterns recur, compute their results in the background, and serve from the results when they exist. What differs is that a rollup is an answer rather than an access path, so it has to be exactly the answer the raw samples would have given.

5.6.1 The rollups table #

A unique constraint and an index both cover (series_id, function, window_seconds, window_start).

Every row satisfies: sample_count greater than zero, value finite, and covered_ns greater than zero for RATE and DELTA and exactly zero for the other four.

sample_count and covered_ns are what make rollups composable. A window built from one sample is not as good as one built from sixty, and combining sub-windows correctly needs their weights — an average composes weighted by sample_count, a rate composes weighted by covered_ns. Without them a rollup could only serve a query whose window matched it exactly.

Rollups are per series and carry no boot_id. Cross-series aggregation composes per-series rollup rows and then applies the query's terminal aggregation across them.

5.6.2 What is not rolled up #

Percentiles. P50, P95 and P99 are not composable: the P95 of twelve five-minute P95 values is not the P95 of the hour. Percentile queries always compute from raw histogram samples.

A later revision could add histogram rollups storing merged bucket counts, computing percentiles from the rolled-up distribution — but that is a different storage model, not a row in this scalar table.

Histogram samples in scalar rollups. AVG, MIN, MAX and SUM roll up raw counter and gauge values only.

Non-window RATE and DELTA scalar aggregations. A stored RATE or DELTA row is a window-level rate, whereas a scalar aggregation over a transformed series operates on the per-pair values (PSPU §3.25). Serving one from the other would give a different answer, so these are not recorded and always fall back to raw samples.

5.6.3 The registry #

eventd maintains a global set of (function, window) pairs worth pre-computing, derived from query frequency exactly as the desired index set is (§3.4), with its counters in the metadata database (§3.5).

Each metric query with a rollup-eligible aggregation records a pair:

• AVG_OVER, MIN_OVER, MAX_OVER and SUM_OVER record AVG, MIN, MAX and SUM respectively, when no RATE or DELTA transform is present, with the query's window duration.
• With RATE or DELTA present alongside a window aggregation, the transform is the recorded function and the terminal aggregation is applied afterward to the per-series values. The window duration is again the query's.
• Scalar AVG, MIN, MAX and SUM over raw counter or gauge samples with a SINCE clause record the same function using AdaptiveRollupScalarWindowSeconds (§A) as the window — a scalar query has no window of its own, so a base window is chosen for it and composition covers the rest.

A pair crossing AdaptiveRollupCreateThreshold over the rolling window joins the registry; one falling below AdaptiveRollupDropThreshold leaves it. Both thresholds are lower than the indexing ones, because rollup computation is cheaper — it proceeds window by window rather than building a whole B-tree — and the speedup is larger, twenty-four rows instead of eighty-six thousand for a daily query at one-second resolution.

The registry is global: if hourly averages are queried often for anything, they are computed for every compatible series. Incompatible series are skipped.

5.6.4 Computation #

On a background thread, during low write activity. For each registry pair, the thread finds windows with raw samples but no rollup row, reads those samples, computes, and inserts.

Only completed windows. The current, still-accumulating window is never pre-computed and is always computed from raw samples at query time.

AVG, MIN, MAX and SUM take the raw counter or gauge values whose timestamps fall in the window. RATE and DELTA are computed for counter series only, and computation skips any series whose type the function does not fit.

RATE and DELTA use the same counter-window rule the query engine uses (PSPU §3.25): consecutive pairs in (timestamp, id) order whose later sample falls inside the window; the immediately preceding sample before the first in-window one as the baseline for the first pair, where it exists; pairs with non-positive elapsed time ignored. DELTA's value is the sum of reset-adjusted deltas, and RATE's is that divided by covered_ns in seconds. A window with no contributing inputs — or, for RATE, zero covered_ns — gets no row at all rather than a zero.

Computation is cancellable on rising write pressure and resumes later, through the same mechanism as adaptive index creation (§3.4).

5.6.5 Serving a query from rollups #

When a metric query carries a rollup-eligible aggregation, the engine checks for matching rollups. It uses them when a rollup covers the requested function, window size and time range and the query does not filter by boot_id — rollups are boot-agnostic, so a boot-filtered query cannot be answered from one (§3.7).

Partial coverage is handled rather than refused. Where rollups exist for the complete windows fully inside the effective range, the engine reads those and computes the remaining prefix or suffix from raw samples. That edge handling is required for exactness whenever a SINCE or UNTIL bound is not aligned to the window.

With no matching rollup, or a boot filter, or a percentile, or a non-window RATE or DELTA scalar aggregation, the query falls back to raw samples entirely. The result is identical either way — rollups are a transparent optimisation and never a different answer.

5.6.6 Composition #

A rollup window need not match the query window for composable functions. Smaller windows serve larger queries; the reverse never works.

AVG_OVER 1h is served from twelve five-minute AVG rollups by weighting each by its sample_count.

Counter resets are handled once, during computation: a stored RATE or DELTA already reflects reset-adjusted deltas, so composition operates on adjusted values and never has to consider a reset again.

For cross-series unbracketed window queries, composition happens per series first and the terminal aggregation is applied across series afterward. The weighting differs between two cases that look alike:

• Without a transform, AVG_OVER across series combines per-series AVG rollups weighted by sample_count, because the query means the average of every scalar sample value in the window.
• With RATE or DELTA, the terminal average is an unweighted mean of the per-series window values, because under PSPU §3.25 each series contributes at most one scalar per window, and weighting one-value contributions by their pair counts would silently favour the busiest series.

5.6.7 Retention and departure #

Rollup rows follow raw sample retention. When retention deletes samples it deletes the rollups for the affected series (§5.5).

When a pair leaves the registry, existing rows are not deleted. They remain available to queries until they age out through normal retention; only new computation stops. Deleting them would discard work already done in exchange for nothing — the rows are correct, and a query that can use one still can.

5.6.8 Persistence #

The registry and its counters live in the metadata database (§3.5) and survive restarts. Existing rollup rows are discovered in the table itself; eventd resumes computation from whatever state it finds, exactly as it resumes index convergence (§3.4).

A query arrives as one string (PSPU §3.15). Turning it into an answer has four phases before any data is read: parse, plan, authorize, execute.

6.1.1 Parsing #

The string is parsed into a syntax tree. The parser:

1. Identifies the mode from the first token — EVENTS, LOGS or METRIC.
2. Extracts the primary selector: a type pattern, a FROM list, or a metric name with an optional label selector.
3. Collects every clause, in whatever order they appear.
4. Validates that the clauses suit the mode — CONTAINING only in log mode, RATE only in metric mode, SELECT only where a result schema is not fixed.

Parse errors are returned immediately, before anything is opened, read or authorized. A malformed query costs a decode and a parse.

6.1.2 What can only fail later #

Some failures need data. The parser cannot know whether a metric name resolves to a counter or a histogram, how many series a selector matches, or whether the effective range exceeds the cross-type lookback limit — those depend on the store, so they surface at planning or execution time (PSPU §3.B).

The practical consequence is that the same query string can parse everywhere and fail on one machine: a metric selector matching one series on a two-core box matches two on a four-core one.

6.1.3 Planning #

Planning resolves what the query will actually touch:

• which concrete identifiers the data could carry — event types, log origins, metric names — because access control resolves per identifier and a broad selector authorizes nothing by itself (§7.4)
• which series a metric selector matches, and whether they are type-homogeneous
• which fields the query references, for both authorization and frequency accounting (§6.5)
• which stores are involved, including any cross-type source

The identifier discovery step is the expensive one and its cost is not bounded by the query. EVENTS SINCE 30d ago with no type pattern has to establish every distinct event type in that range before it can authorize anything, and whether that is an index scan or a table scan depends on whether event_type currently has an index — which is an adaptive decision that pressure may have reversed (§3.4).

6.1.4 When payloads are decoded #

Event payloads are stored as opaque MessagePack and are never decoded on the write path (§3.1). Decoding happens here, on the read path, and only where a query needs it: to evaluate a payload predicate, to build a flat result record, or to compute a payload expression index's key on insert.

At high result counts this dominates the query path. Constructing flat maps from thousands of events means decoding thousands of payloads and applying the flattening rules of PSPU §3.22 to each. Partial extraction is the lever: with a SELECT present, only the named paths need decoding, and without one a streaming decoder that emits flattened pairs avoids materialising the payload at all (§C).

6.1.5 Read connections #

Execution uses read-only SQLite connections, which in WAL mode do not block writer threads.

An event query opens one connection per shard database in the directory (§6.4). A log or metric query opens one. eventd supports concurrent queries up to its admission limit (§6.5), subject to the operating system actually having the descriptors and memory; where it cannot allocate what an admitted query needs, it fails that query rather than blocking a writer or exceeding the limit.

PSPU §3.21 requires every result order to be total and deterministic for a fixed set of stored records, so that SKIP and TAKE page reliably. This is how eventd achieves it.

6.2.1 The tiebreakers #

Where the query's explicit SORT keys — or the mode's default ordering — do not uniquely order two records, eventd appends internal keys until the order is total:

These are not query-language fields. They never appear in a result record, cannot be named in a SORT or a SELECT, and have no access-control identity (PSPU §3.28).

The shard index is the numeric identifier from the shard-NNNN.db filename. It appears in the event tiebreaker because rowids are per-database: two events in different shards can share a rowid, and without the shard index the pair would be genuinely unordered.

The metric tiebreaker includes name and labels because an unbracketed query merges several series into one output stream, where the timestamp alone does not separate rows from different series.

6.2.2 Why insertion order is never the answer #

samples.id and events.id break ties within one database, but neither is a substitute for the timestamp ordering they follow.

Metric samples may arrive out of timestamp order (§5.1), so insertion order and time order genuinely differ. Every metric computation — RATE's consecutive pairs, rollup window membership, cross-type interval construction — is defined over (timestamp, id) ascending precisely so that a late-arriving sample lands where its timestamp says it belongs rather than where it happened to be written.

Events are less prone to it, since a drain thread reads one ring buffer in order, but a shard receiving from several CPUs interleaves them arbitrarily and a clock step can invert two events from the same CPU.

6.2.3 Value ordering is not SQLite's #

Sorting, grouping and equality all use the query language's semantics, never the storage engine's dynamic-type rules (PSPU §3.20, §3.21).

The divergences are not edge cases. SQLite compares an integer and a real by converting; the query language compares them mathematically and exactly, including beyond binary64's exact integer range. SQLite orders text by byte; the query language folds ASCII case and then falls back to the original bytes only to break a fold-equal tie. SQLite has its own type-affinity ordering across storage classes; the query language fixes its own type order, nulls first, arrays last.

Where a SQL construct cannot reproduce those semantics, eventd uses it only to narrow candidates and applies the real comparison after loading the row (§6.3).

6.2.4 Canonical representatives #

A group whose members are equal under the language's rules but not byte-identical emits the smallest member rather than the first (PSPU §3.21).

Smallest rather than first is what makes the representative a property of the set. An event query merges results from every shard in an order that depends on which shard answered first, so "first" would be nondeterministic across runs of the identical query — which is exactly the property this chapter exists to prevent.

Events and logs are translated to SQL. Metrics are translated to SQL against series and samples. Clients never see any of it — the translation is entirely internal and carries no guarantees.

6.3.1 What translates directly #

Event header fields are columns, so a predicate on event_type, process_guid or cpu_id becomes a SQL WHERE comparison over an indexable column (§3.1).

Log fields are all columns; log mode has no payload and its field set is closed (§4.2).

Metric selection resolves names and labels through series — from the in-memory cache where possible — and reads samples for the range, ordered by the composite index that already provides (timestamp, id) (§5.2).

6.3.2 What does not #

Event payload predicates have no column. They become eventd-internal payload extraction predicates, and may use an adaptive payload expression index to narrow candidates (§3.4).

The rule governing every such translation is that SQL narrows, the query language decides. Where a SQL construct cannot reproduce a predicate's comparison semantics exactly, eventd uses it only to reduce the candidate set and then applies the real predicate after loading the row.

SQLite's native dynamic-type equality and ordering never substitute for the query language's ASCII case folding, exact numeric comparison, binary comparison, array comparison, or null and missing-field handling (§6.2). An index that returns a smaller set faster is useful; one that returns a different set is a wrong answer arriving quickly.

6.3.3 Where access control sits #

Access filtering is part of the logical execution, not a filter over the output (PSPU §3.18, §3.28). eventd may push authorization predicates down into SQL when the concrete identifier set is known at planning time, or read candidate rows and discard them before aggregating.

Which it does is a performance decision. What is fixed is that the externally visible result is identical to the one filtering-first would produce — aggregates, ordering and pagination included, since all three would otherwise leak the existence of rows the caller cannot read.

6.3.4 Aggregation #

Aggregation is pushed into SQL wherever the storage engine can express it, which is most of the time for simple grouping over columns and none of the time for grouping over payload paths whose comparison semantics SQL cannot reproduce.

For an event query the push-down matters twice over, because it also determines what crosses the shard boundary (§6.4): a shard returning per-group partial aggregates sends a result proportional to the group cardinality, where a shard returning rows sends a result proportional to the row count.

Event queries execute against every database in the event store directory — active shards and historical ones alike (§3.3). Log and metric queries touch one database each and need none of this.

Shards carry no meaning for the query path (§2.3). A shard holds whatever CPUs routed to it during whatever lifetimes wrote it, so there is no shard a query can skip on the basis of its contents, and a predicate on cpu_id scans all of them.

6.4.1 Merging #

How results combine depends on the query.

Non-aggregating queries. Each shard produces rows sorted by the effective sort key, tiebreakers included (§6.2), and the coordinator performs an N-way merge of the sorted streams.

With TAKE present, each shard returns at most SKIP + TAKE rows — or TAKE rows when there is no SKIP — and the coordinator applies SKIP and TAKE after merging. The total read is therefore at most (SKIP + TAKE) × shard_count, which is the price of not knowing in advance which shard holds the winning rows.

With TAKE absent, each shard streams every matching row until the query completes or times out.

Aggregating queries. Each shard computes a partial aggregate and the coordinator combines them:

AVG is the one that cannot be composed from its own output. Averaging per-shard averages weights each shard equally regardless of how many rows it held, so a shard is asked for the sum and the count and the coordinator divides once — the same reasoning that makes rollup composition carry sample_count (§5.6).

Pushing aggregation down bounds the coordinator's memory to the group key cardinality times the shard count, rather than to the total row count.

6.4.2 The unbounded case #

A non-aggregating query without TAKE has no implicit row limit. EVENTS SINCE 7d ago may match millions of rows, all of which pass through the merge.

The query timeout is the only backstop (§6.5). Streaming merged results to the client incrementally, rather than materialising the whole set before sending, is what keeps the memory cost proportional to the merge frontier instead of to the result.

6.4.3 Descriptors #

An event query opens a read-only connection per database, and each SQLite connection holds one or two descriptors for the database and its write-ahead log. With many historical shards this adds up quickly across concurrent queries.

Active shard writer connections stay open for the process lifetime and are not negotiable. Historical shard read connections are the pool worth bounding — opened when a query touches them, closed after a period of inactivity (§C).

6.5.1 Recording what was asked #

Every event query is recorded by the adaptive indexing system (§3.4). For each WHERE predicate:

• a header column reference increments that column's frequency counter
• a payload field reference increments that path's counter

Cross-type WHERE predicates are counted like any other, since they narrow the same data by the same fields.

This applies to event queries only. The log and metric stores have fixed write-time indexes and no candidate space for a policy to explore (§4.2, §5.2), so their queries increment nothing.

Counters are in-memory and are flushed to the metadata database at each policy interval (§3.5). Query handlers never write to that database directly, which is what keeps the once-per-query update off any lock a writer thread contends for.

Metric queries feed the parallel rollup registry counters instead (§5.6), which record (function, window) pairs rather than fields.

6.5.2 Concurrency #

eventd bounds concurrent queries — streaming and non-streaming together — at MaxConcurrentQueries (§A). Beyond it a query is rejected with an error rather than queued.

The per-query cost that limit is protecting is real: read-only SQLite connections, one per shard for an event query (§6.4), memory for the merge, and CPU for execution and payload decoding.

MaxStreamingQueries is enforced separately and is lower. A streaming query holds its resources for as long as its client stays connected, where an ordinary one holds them for at most a timeout, so the two populations need different bounds.

Both are global rather than per-caller. eventd cannot attribute connections to a caller beyond the token it holds, so one client can occupy every slot — and the interim protection is that queries and ingestion are separate channels, so exhausting the query side cannot exhaust ingestion (PSPU §3.3).

6.5.3 Timeouts #

Every query has a maximum execution time, QueryTimeoutMs (§A).

The clock starts once the request has been decoded and the caller's token obtained, and covers everything after: parsing, planning, access checks, cross-type pre-computation, SQL execution, merging, aggregation, pagination, projection, and transmitting the initial result set.

It bounds the initial result set only. A non-streaming query sends end before it expires; a streaming query sends watch. Past watch the stream is not time-limited, and what bounds it instead is MaxStreamingQueries, MaxDistinctStreamValues and backpressure (§6.6).

On expiry eventd cancels the query and sends an error. Any result messages already sent are discarded by the client, since no terminal message arrived (PSPU §3.16).

Cancellation has to reach two kinds of work. SQLite work is interrupted through sqlite3_interrupt or an equivalent progress-handler check — the same responsiveness problem as cancelling an index build (§3.4). Non-SQL work — MessagePack flattening, the cross-shard merge — checks the same deadline periodically, because a query can spend most of its time in neither the database nor the kernel.

Large scans over unindexed fields are the main timeout risk, and the adaptive indexing system reduces it over time by indexing whatever keeps being filtered on — which is also why a timeout is a signal worth watching rather than merely an error to retry.

The externally visible behaviour of a streaming query — what may be streamed, what applies during the watch phase, how DISTINCT streams behave, and when a slow client is dropped — is PSPU §3.27. This is the machinery underneath.

6.6.1 Commit generations #

eventd keeps a monotonic u64 commit generation counter for each streamable store: one for the event store as a whole, and one for the log store. Metric queries do not stream, so the metric store has none.

After a writer commits a batch it increments the counter for its store and wakes the streaming handlers waiting on it. A handler records the last generation it processed and waits until the counter exceeds it.

The event counter covers the whole store rather than one per shard. Several writer threads increment it, so a wake is "something committed somewhere" and a handler re-examines every shard it cares about — which is what it would have to do anyway, since a shard means nothing to the query path (§6.4).

The counter is process-local and never persisted; it has no meaning across a restart, and a streaming query does not survive one. On wraparound the next increment is treated as a wake for every handler and operation continues. At any commit rate a machine can sustain, wraparound of a 64-bit counter is not reachable.

6.6.2 Latency #

Delivery latency is bounded below by the commit interval of the store concerned, because a record is not streamable until it is committed.

Under light load the actual latency is lower, because the adaptive batcher commits as soon as its input drains rather than waiting out the cap (§2.4). Under sustained load it converges on the cap.

A consumer needing better than this is not served by eventd at all: the KMES ring buffers are the low-latency path, they are specified in PSPK, and attaching to them directly costs the per-event access control that eventd exists to apply (§7).

6.6.3 The DISTINCT seen set #

A DISTINCT stream holds a per-query set of the values it has already emitted, initialised from the initial result set and added to as new values appear (PSPU §3.27).

It is bounded by MaxDistinctStreamValues (§A), and exceeding the bound terminates the query with an error rather than evicting. Eviction would make the output wrong rather than merely truncated: a forgotten value would be re-emitted as newly seen, and "newly seen" is the entire meaning of the result.

The set is per query and in memory, which is why the bound exists and why it is separate from the general query concurrency limit — sixty-four streams each holding a hundred thousand values is a different memory profile from sixty-four ordinary queries.

6.6.4 Cross-type re-evaluation #

Pre-computed cross-type ranges describe the past and are discarded when the watch phase begins (PSPU §3.27).

A metric condition costs one index seek per batch: the selector has already been constrained to exactly one series, so finding the active sample at the batch's latest candidate timestamp is a single lookup on idx_samples_series_timestamp (§5.2).

An existence condition is evaluated per candidate record rather than per batch, because the centred window is relative to each record's own timestamp and a matching record may be near some of a batch and not the rest.

The per-batch metric evaluation is an approximation, and the reason it is acceptable is the ratio between the two intervals: a commit batch spans a fraction of a second and a metric sample fifteen, so every record in a batch normally maps to the same sample. At sub-second metric resolution it filters more coarsely, and records near a threshold crossing are included or excluded as a group.

6.6.5 Backpressure #

Backpressure is detected on the socket send buffer. When a result message cannot be sent because the buffer is full, the query is terminated immediately; eventd never blocks on the send.

Blocking would put a slow reader in the path of eventd's own work, and the write path is what would suffer. A streaming client is the lowest-priority consumer of eventd's time, and dropping it is the same principle as dropping an ingestion datagram (PSPU §3.4), applied on the way out.

eventd enforces access on the read path only, using KACS Security Descriptors and the KACS AccessCheck API. Every query is evaluated against descriptors that determine which records — and which fields within a record — the caller may see, and everything else is filtered out silently (PSPU §3.28).

7.1.1 eventd decides nothing itself #

eventd implements no access check logic of its own. Every decision is delegated to kacs_access_check and kacs_access_check_list, which run the full KACS AccessCheck pipeline: integrity checks, restricted token evaluation, confinement, conditional ACE evaluation, and the SACL audit walk.

What eventd contributes is the three things AccessCheck needs and cannot know — which descriptor applies (§7.2), which object type list describes the record (§7.3), and what the caller's token is (§7.4) — and then it acts on the verdicts.

The alternative would be reimplementing an access check algorithm that already exists, in a daemon that would then have to be kept in agreement with it forever.

7.1.2 Enforcement is at query time #

All events, logs and metrics are stored regardless of who will ever be allowed to read them, and two callers querying the same store see different results.

Three reasons make this the only workable arrangement:

• Audit integrity. An audit event has to be stored whether or not anyone can currently read it. Filtering at storage time would let a descriptor decide what gets recorded, which is the one thing an audit store must not permit.
• Descriptors change. An administrator can grant or revoke access retroactively, and only query-time evaluation makes that meaningful.
• Callers differ. Several principals with different access levels query the same store, and there is one copy of the data.

7.1.3 Caller identity #

When a client connects to the query socket, eventd obtains its token by calling kacs_open_peer_token on the connected descriptor. The token represents the peer's identity as captured at connection time.

If the call fails, eventd denies the query entirely. It has no fallback identification and no anonymous mode.

The snapshot property matters most for streaming queries, which may run indefinitely: a client whose group memberships change, or whose access is revoked, continues to be evaluated against the token it connected with until it disconnects (§7.5).

7.1.4 Rights #

The generic mapping passed to AccessCheck is in §B.

EVENTD_ADMINISTER is distinct from EVENTD_READ deliberately. Accelerating a field costs write throughput on every record thereafter, so INDEX is a way for a caller to degrade the system for everybody, and a caller permitted only to read has not been permitted to do that (PSPU §3.23).

7.1.5 What is not controlled here #

Event emission is KMES's: kmes_emit and kmes_emit_batch require SeAuditPrivilege, and eventd is not involved.

Log and metric ingestion has no per-record control at all. The Security Descriptor on each ingestion socket is the entirety of it (§7.6).

Access is defined on named patterns, each standing for a category of observability data and each carrying a descriptor. The three data types have independent pattern namespaces:

7.2.1 Matching #

A pattern matches by dot-delimited prefix. The pattern kacs matches the exact string kacs and any string beginning kacs. — and matches neither kacs_extended nor kacsfoo, because the dot is the hierarchy separator and not a mere character.

* is the wildcard default and matches everything.

Ingestion constrains origins and metric names to the identifier grammar (PSPU §3.7, §3.10), which is what keeps a producer from choosing a name containing the wildcard or a registry path separator — a name that would otherwise match a rule its producer was never meant to satisfy, or store its descriptor somewhere other than where the administrator who wrote it believes.

7.2.2 Resolution #

For a concrete identifier, eventd resolves the applicable descriptor by walking up the hierarchy:

1. Look for an exact match on the full identifier, kacs.access_denied.
2. Remove the last dot-separated component and look again, kacs.
3. Repeat.
4. Fall back to the wildcard, *.

The first match wins; a more specific pattern overrides a less specific one.

7.2.3 Storage #

Descriptors are registry values under the eventd security subtree:

Machine\System\eventd\Security\Events\*
Machine\System\eventd\Security\Events\kacs
Machine\System\eventd\Security\Events\kacs.access_denied
Machine\System\eventd\Security\Logs\*
Machine\System\eventd\Security\Logs\loregd
Machine\System\eventd\Security\Metrics\*
Machine\System\eventd\Security\Metrics\cpu

Each type's wildcard default is load-bearing. If a default is missing, eventd denies access to all data of that type: resolution that reaches the end of the hierarchy without a match is a denial, not a grant (PSPU §3.28).

Storing them in the registry rather than in eventd's own databases means the registry's access control protects them, and an administrator edits them with the ordinary registry tools rather than through eventd.

7.2.4 Defaults on first boot #

eventd creates the three wildcard keys if they do not exist:

The asymmetry reflects sensitivity. Events include security audit data and are restricted to administrators; logs and metrics are operational data and are readable by any authenticated user. An administrator can tighten either.

7.2.5 Conditional ACEs #

Descriptors on eventd security objects may carry conditional ACEs, and KACS evaluates them as it would anywhere.

eventd passes no eventd-specific local claims to AccessCheck: local_claims_ptr is null and local_claims_len is zero. Conditions referencing token claims that KACS itself supplies still evaluate; conditions referencing eventd-local claims observe them as absent.

A stable set of eventd-local claims — the event's type, the log's origin, the time of day — is a plausible later addition, and defining one is a commitment to keep those claim names meaningful thereafter.

7.2.6 The administrative descriptor #

INDEX (PSPU §3.23) is checked against admin_sd, held in the metadata database rather than the registry (§3.5), with EVENTD_ADMINISTER as the desired access. Its default grants SYSTEM and Administrators.

eventd refuses INDEX outright if no administrative descriptor exists, by the same fail-closed rule as the read path.

A descriptor can grant read access to some fields of a record and not others, using KACS object ACEs and object type lists. A caller authorized for a pattern but not for a field receives the records with that field absent — indistinguishable from a record that never carried it (PSPU §3.28).

7.3.1 Object type lists #

For each access check eventd builds an object type list: a two-level tree with the data type's root at level 0 and one field per node at level 1.

Level 0: root GUID for the data type
  Level 1: timestamp
  Level 1: event_type
  Level 1: cpu_id
  Level 1: origin_class
  Level 1: effective_token_guid
  Level 1: true_token_guid
  Level 1: process_guid
  Level 1: granted_access
  Level 1: target_sid
  Level 1: source.name

kacs_access_check_list returns a verdict per node, and eventd uses them to include or exclude each field.

The three root GUIDs — one for events, one for logs, one for metrics — are in §B.

7.3.2 Field GUIDs are derived, not registered #

A field's GUID is computed deterministically with UUID v5 (RFC 4122):

field_guid = uuid_v5(EVENTD_FIELD_NAMESPACE, field_name)

with the namespace UUID in §B, and field_name the field's query-language name as UTF-8.

There is no registry of field GUIDs and no allocation step. The same name always yields the same GUID, so an administrator writing a descriptor computes the GUID from the field name with the same algorithm that eventd will use when it builds the list. This is the one part of eventd's access control that a third party reproduces rather than merely consumes.

Derivation rather than registration is what makes payload fields tractable at all: event payload schemas belong to the emitting subsystems, eventd has no catalogue of them, and a new event type carrying a new field needs no registration anywhere before a descriptor can name it.

Which names are used:

Payload fields suppressed by flattening or by a header collision are not query-language fields, so they get no GUID (PSPU §3.22). Metric label keys cannot collide with the fixed metric fields, because ingestion rejects records whose labels do.

7.3.3 The GUID does not encode scope #

A field GUID names a field and nothing else. granted_access produces the same GUID whatever event type carries it.

Scoping comes from the descriptor hierarchy: an object ACE naming the granted_access GUID inside the descriptor for pattern kacs means "the granted_access field of KACS events". The same ACE in a different pattern's descriptor means the same field of that pattern's records.

7.3.4 Writing one #

An object ACE with no object type GUID applies to the root and therefore to every field. One with a field GUID applies to that field.

To grant a security team full read access to KACS events, and a monitoring team only the timestamp, type and CPU:

• Allow SecurityAdmins, EVENTD_READ, no object GUID
• Allow MonitoringTeam, EVENTD_READ, object GUID = timestamp
• Allow MonitoringTeam, EVENTD_READ, object GUID = event_type
• Allow MonitoringTeam, EVENTD_READ, object GUID = cpu_id

MonitoringTeam querying KACS events receives records containing exactly those three keys. Payload fields, identity GUIDs and the remaining header fields are absent.

7.3.5 Building the list per record #

The list is constructed from the fields actually present in the record being checked: the root node, then a level-1 node per field.

Event records contribute every header field plus every non-suppressed flattened payload field present in that particular payload. Different event types produce different lists, because they carry different payloads — and two events of the same type can too, since a payload is opaque MessagePack and nothing requires two of a type to agree.

Log records have a fixed field set — timestamp, origin, is_error, message, job_id, boot_id — so the list is the same for every log record.

Metric records contribute the five fixed fields plus the series' label keys, which vary per series.

Derived aggregate outputs — count, sum, avg, min, max — are omitted from the list entirely. They are not source fields, have no GUID and no access identity of their own, and are visible when the caller is authorized for the records and source fields they were computed from (PSPU §3.28).

The order in which eventd evaluates a query is fixed (PSPU §3.18), and access control is the third phase — before predicates, transforms, grouping, aggregation, ordering and pagination. What follows is the sequence within that phase.

1. Obtain the caller's token from the connection (§7.1). Failure denies the query.
2. Parse the query to establish its data sources and filters (§6.1).
3. Discover the concrete identifiers the query could touch — event type strings, log origin strings, metric name strings. A broad selector authorizes nothing by itself: EVENTS, EVENTS kacs.*, LOGS without FROM and METRIC cpu.* are each resolved identifier by identifier.
4. Resolve and check each discovered identifier: find its descriptor by hierarchical matching (§7.2), build the object type list for the fields the query references (§7.3), call kacs_access_check_list, and cache the verdicts for this (token, identifier, field set) (§7.5).
5. Apply root verdicts. An identifier whose root is denied is invisible: its records are excluded from the logical row set before aggregation, ordering, pagination and formatting. For a cross-type source, a denied identifier is treated as having no matching data.
6. Apply field verdicts to predicates. Where the query references a field in a predicate or shaping clause and a matching identifier does not grant it, that identifier's records contribute nothing — exactly as if its root had been denied. The query is not rejected.
7. Cross-type sources get the same treatment. A denied root, or a denied field needed to evaluate the condition, makes the condition evaluate as though no matching cross-source data existed.
8. Execute, with root filtering already part of the logical row set.
9. Re-resolve per result identifier. For each distinct concrete identifier in the resulting rows, resolve its descriptor, build the object type list with field GUIDs, call kacs_access_check_list with the token, the descriptor, EVENTD_READ, the list and an audit context naming the identifier, and cache the per-field results.
10. Shape each record. Look up the cached results for its identifier; exclude the record entirely if the root was denied; otherwise include it, and include each field only if its node was granted.

7.4.1 Which clauses count as referencing a field #

Step 6 applies to every clause that reads a value rather than merely displaying one:

• ordinary WHERE predicates
• metric label filters in a primary selector
• GROUP, COUNT BY, TOP N BY, SORT and DISTINCT fields
• event and log aggregation arguments — SUM, AVG, MIN, MAX
• metric transforms and terminal aggregations, all of which read the fixed value field: RATE, DELTA, P50, P95, P99, AVG, MIN, MAX, SUM, AVG_OVER, MIN_OVER, MAX_OVER, SUM_OVER
• a metric boot filter, which reads boot_id; and an explicit metric type predicate, grouping, sort or distinct, which reads type

SELECT is not on this list. It shapes output and is applied last, so a field it omits was still available to every earlier phase — and conversely, selecting a field the caller may not read removes the field, not the record.

7.4.2 Denial is silent, not fatal #

Step 6 excludes rather than rejects, and this is the decision most worth being explicit about, because rejecting is the more informative behaviour and that is exactly the objection to it.

A rejection would tell the caller that some identifier exists, matches its query, and carries a field it may not read — three facts about data it was not permitted to see, delivered by the mechanism meant to withhold them, and enumerable by trying queries and watching which ones fail. Excluding costs the caller a result narrower than it appears; rejecting costs the model the property it rests on.

Cross-source fields already worked this way (step 7); primary-source fields now match them.

7.4.3 Field authorization does not depend on presence #

A field's authorization is resolved from the name as written, against each concrete identifier, whether or not any record of that identifier actually carries it.

Payload fields vary between records of the same type, so a rule turning on presence would require the scan that authorization is meant to precede.

7.4.4 Internal values are not fields #

Row identifiers, series identifiers, ordering tiebreakers and series type checks are not query-language source fields unless the mode exposes them as fixed fields or the query names them (§6.2). Errors raised by internal checks never carry a denied field's value.

A metric result's value is a source field, because it is either a raw sample or a scalar derived from raw samples.

7.4.5 Filtering is part of the logical result #

Access filtering is not a presentation step. Aggregating, ordering or paginating over unreadable records would leak them through counts, through ordering, and through the gaps in pagination.

eventd may push authorization predicates into SQL when the identifier set is known, or read candidates and filter them before aggregating (§6.3). The externally visible result is identical to filtering first, and COUNT, COUNT BY, TOP N BY and every other aggregate reflect only what the caller may see.

7.4.6 The audit trail #

Every access check produces a KACS audit event through the SACL audit walk in the AccessCheck pipeline.

eventd passes an audit_context blob naming the security pattern being accessed — "events:kacs.access_denied", "logs:loregd" — so the audit trail records exactly which observability data was read, by whom, rather than merely that eventd performed a check.

Those audit events are themselves KMES events, which eventd consumes and stores, and which are governed by the synthetic-independent event patterns like any other. Reading the audit store is auditable.

An access check is a syscall through the full AccessCheck pipeline. A query returning ten thousand records cannot afford ten thousand of them, so results are cached — at two levels, plus the descriptor resolution underneath both.

7.5.1 Record-level #

When a pattern's descriptor contains no object ACEs, the check is a plain grant or deny on the root, and the result is cached per (token, pattern).

A query returning ten thousand events across twenty distinct event types performs at most twenty checks.

7.5.2 Field-level #

When the descriptor does contain object ACEs, the verdict depends on which fields the record carries, since different payloads produce different object type lists (§7.3). The result is cached per (token, pattern, field set).

In practice events of one type carry the same fields, so this is effectively one check per (token, event type). Log records have a fixed field set, so log queries reach one check per origin. Metric records vary by series label keys.

The pathological case is an event type whose payload fields differ from record to record, which produces a distinct field set — and a distinct cache entry, and a distinct syscall — for each shape encountered.

7.5.3 Descriptor resolution #

Resolving a pattern to a descriptor is itself cached, across queries rather than within one, since it costs a registry read and a hierarchy walk (§7.2).

eventd watches the security registry subtree and invalidates cached resolutions and cached check results when a descriptor changes. That is what makes a revocation take effect on the next query rather than at the next restart.

If the registry watch fails after startup, eventd discards the descriptor cache and operates fail-closed for new resolutions until the watch is re-established. A cache it cannot trust to be current is worse than none: continuing to serve from stale entries would make a revocation silently ineffective, and the failure would be invisible. This is a degraded state, not a failure — eventd keeps ingesting, and keeps answering queries for descriptors already resolved (§9.3).

7.5.4 During a stream #

Verdicts reached for a streaming query's initial result set are reused through the watch phase, with two exceptions.

A new concrete identifier appearing in a streamed batch — an event type or log origin not present in the initial results — is resolved and checked before the record or its distinct value is used. It has never been authorized, and inheriting a verdict from a sibling pattern would be a grant nobody made.

A descriptor change invalidates the cache as it does anywhere, and subsequent batches are re-checked against the new one.

The token is not re-examined. It was captured at connection (§7.1), so a client whose memberships change mid-stream continues under what it connected with, and a client whose access is revoked keeps receiving records until it disconnects. The bound on that exposure is the client's own connection lifetime, which for a dashboard may be days.

There is no per-record access control on the way in. This article records what stands in its place and what that leaves open.

7.6.1 Events #

Emission is KMES's business. kmes_emit and kmes_emit_batch require SeAuditPrivilege, and eventd is not in the path — it consumes what KMES delivers and applies no admission control of its own (§2.2).

The identity stamps on an event are the kernel's, captured from kernel state at the moment of the write, and an emitting process cannot set, influence or suppress them. That is what makes an event's process_guid evidence in a way that a log's origin is not.

7.6.2 Logs and metrics #

The Security Descriptor on each ingestion socket is the entirety of the write-path control (PSPU §3.3). There is nothing per record: no token is obtained, no identity is checked, and no field is verified.

The socket descriptor is therefore doing all the work, and it is worth being precise about what it does and does not do on Peios. An access decision is routed through the object's Security Descriptor, not through POSIX mode bits, so setting a mode on a socket pathname restricts nothing — and an inode created without a descriptor is denied to every caller, so binding a socket into a directory carrying no inheritable ACEs produces a socket that nothing, including the service manager, can reach. eventd establishes the descriptor on each socket before it begins receiving on it.

7.6.3 Origin and name are claims #

origin in a log record and name in a metric record are self-reported (PSPU §3.7, §3.10). Any process that can reach an ingestion socket can write under any origin or metric name it likes, including one belonging to another program.

The consequences are the obvious ones. A compromised service can inject log lines attributed to another service, manufacturing a plausible operational narrative. It can bury a real incident under noise attributed elsewhere. It can create metric series under a name a dashboard trusts.

Read-path descriptors limit who can see data written under a given identifier; they do nothing about who wrote it. eventd never presents a stored origin or metric name as evidence of provenance.

7.6.4 What this means for an operator #

The interim posture is that the ingestion sockets are a trust boundary that only separates "can reach eventd" from "cannot" — and on a system where every service logs, nearly everything is on the inside.

Two things follow. Data whose provenance must be trustworthy belongs in an event, where the kernel stamps the identity, rather than in a log where the producer asserts it. And read-path descriptors on the origins and metric names that matter are worth writing even so: they prevent an unauthorized reader from querying data written under a spoofed identifier, which is a smaller property than authenticity but not nothing.

eventd needs four subsystems before it can do anything.

eventd is a peinit-managed service, started after loregd is available — it cannot read a single configuration value without the registry, and it has no compiled-in defaults for the six paths it needs (§A).

8.1.1 The dependency that is not one #

KACS is needed to serve queries and not to ingest. The drain, write and retention paths never call it. That asymmetry is what lets eventd keep ingesting through a KACS outage while refusing every query (§9.3), and it is the right way round: losing the ability to read the audit store is recoverable, losing the events is not.

8.1.2 Ordering in the boot #

eventd is one of the platform daemons and is Critical: peinit reboots the system rather than continuing without it. It comes up after loregd and authd, and it stops before them on the way down — it is among the last services shut down, because everything else's shutdown is worth recording.

The window before eventd exists is real and peinit covers it by buffering service output until the log socket appears. Events emitted during that window are not lost either: they sit in the KMES ring buffers, and eventd's first drain after attaching reads them from tail_pos (§2.2).

Startup proceeds in seven phases, and either completes or fails entirely.

8.2.1 Phase 1 — Configuration #

1. Read every configuration key under Machine\System\eventd\. The six required keys are EventStorePath, LogStorePath, MetricStorePath, QuerySocketPath, LogSocketPath and MetricSocketPath; a missing or invalid one fails startup.
2. Read the optional keys and apply compiled-in defaults for those absent (§A).
3. Arm a persistent watch on the subtree, for runtime changes (§8.3).

8.2.2 Phase 2 — KMES attachment and shard sizing #

4. Discover the CPU count by calling kmes_attach with incrementing CPU identifiers from 0 until EINVAL. The call requires SeSecurityPrivilege in the effective token. Discovering no CPUs fails startup.
5. Map each per-CPU ring buffer.
6. Resolve the active shard count from StorageShards — the CPU count when it is 0, the configured value otherwise.
7. Compute the shard-to-CPU assignment (§2.3).

8.2.3 Phase 3 — Storage #

8. Open or create each active event shard: verify the schema version, open in WAL mode with synchronous=FULL, create tables and indexes if new, and quarantine on reported corruption (§3.3). Discover historical shards matching the naming pattern, and open those with a recognised schema read-only; exclude the rest from the query path.
9. Open or create the log store — schema verified, WAL, synchronous=NORMAL, quarantine on corruption (§4.3).
10. Open or create the metric store, likewise (§5.4). The series cache starts empty and fills on demand.
11. Open or create eventd-meta.db. Load the index and rollup counters and desired sets, load the sequence checkpoints for diagnostics only, and discover each shard's material indexes from its schema (§3.5).

8.2.4 Phase 4 — The boot boundary #

12. Read the current boot ID from peinit.
13. Search every readable event shard for committed rows carrying it.
14. No rows — the boot's first eventd start. Reset every per-CPU sequence tracker to 0 and record the new boot ID for subsequent writes.
15. Rows exist — a restart within the boot. Derive each CPU's resume point from the maximum committed sequence for (boot_id, cpu_id) across every readable shard; CPUs with no rows resume at 0 (§3.7).

8.2.5 Phase 5 — Sockets #

16. Create the query socket at QuerySocketPath. A stale pathname left by a crash is unlinked first if it is an AF_UNIX socket; if the path exists and is not a socket, startup fails.
17. Create the log socket at LogSocketPath, same rule.
18. Create the metric socket at MetricSocketPath, same rule.
19. Establish the Security Descriptor on all three, before any of them accepts or receives anything (§7.6).

The stale-socket rule distinguishes the two cases deliberately. Unlinking a leftover socket is recovery from eventd's own crash; unlinking a regular file at a configured path would be destroying something that is not eventd's, and a path pointing at the wrong thing is a configuration error worth failing on.

8.2.6 Phase 6 — Threads #

20. One drain thread per CPU, each beginning to read its ring buffer.
21. One writer thread per active shard.
22. The log ingestion thread.
23. The metric ingestion thread.
24. The retention thread.
25. The adaptive indexing and rollup policy thread.

8.2.7 Phase 7 — Ready #

26. Write and commit a synthetic.startup event recording the boot ID, the shard count and the per-CPU resume points (§3.2). The commit happens before readiness is signalled.
27. Signal readiness to peinit.

Committing before signalling is what makes the startup record trustworthy. A readiness signal sent before the commit could be followed by a crash that loses the record, leaving a boot in which eventd demonstrably ran and left no trace of having started.

8.2.8 Failure #

If any phase fails, eventd does not signal readiness. It logs the failure to standard error where standard error exists, and exits non-zero. peinit's restart policy decides what happens next.

Partial startup is not permitted. There is no degraded mode in which eventd runs without one of its three stores, or without KMES. It either completes the sequence or fails.

The all-or-nothing rule is a simplification. A later revision might allow log and metric ingestion to proceed with KMES unavailable, but that means partial-failure state to manage in every subsequent path — what a query against a store that was never opened does, what happens when the missing subsystem returns — and the failure mode it protects against is one peinit already handles by restarting.

eventd watches Machine\System\eventd\ and reacts to changes without restarting — for the settings that can be changed that way.

Notifications arriving during startup are queued and processed after readiness is signalled (§8.2). Applying a configuration reload to half-initialised state would mean every phase having to tolerate its inputs changing underneath it.

8.3.1 What applies immediately #

Tuning parameters: batch sizes and latencies for all three writers, retention periods and the delete batch size, adaptive index and rollup thresholds and windows, the adaptive scalar rollup window, the WAL checkpoint threshold, the query timeout, the cross-type window and lookback limit, and the query and streaming concurrency limits.

8.3.2 What waits for a restart #

The watch notices these changes and eventd defers them rather than attempting a live migration. Shard count changes in particular are expected once in a machine's life, and rebalancing writer threads while events are in flight is a large mechanism for a rare event.

8.3.3 What is neither #

Security Descriptors under Security\ are not configuration in this sense. The registry watch invalidates the descriptor cache and the next query resolves afresh (§7.5), so a grant or revocation takes effect immediately without anything being "applied".

8.3.4 Recording it #

eventd emits a synthetic.config_change event for every change applied at runtime, carrying the key name and the old and new values rendered deterministically (§3.2).

Invalid values are ignored and the previous value is retained — an administrator who types a batch size outside its range does not get a daemon that stops working, and the retained value is the one already in use rather than the compiled-in default. Unknown keys in the subtree are ignored entirely.

8.3.5 SIGHUP #

SIGHUP re-reads the configuration, equivalent to a watch notification (§8.5). It exists for the case where the watch itself is not delivering — a registry outage, or a watch that failed and has not re-armed (§9.3) — and gives an administrator a way to force the read rather than restarting the daemon.

When peinit signals a stop, eventd persists as much in-flight data as it can without blocking indefinitely.

8.4.1 The sequence #

1. Stop accepting. Unlink all three socket paths so no new client can reach them, and stop accepting query connections. Existing streaming queries are terminated with an error. The log and metric socket descriptors stay open.
2. Drain ingestion. Read and process the datagrams still in the log and metric receive queues, then close those descriptors. This is bounded by the queue size — four times the datagram ceiling — so it completes quickly.
3. Final event drain. Each drain thread performs one last drain cycle from its ring buffer.
4. Final commit. Every writer commits its current batch immediately, whatever its size. The log and metric writers do the same.
5. Record sequence state. Derive the per-CPU last committed sequence from committed rows — the same rule startup uses — and write it to sequence_checkpoints for diagnostics (§3.5).
6. Emit the shutdown event. Write synthetic.shutdown with the per-CPU sequences, using the daemon-wide shard assignment rule (§2.6). If no shard is writable, the event is skipped and the failure logged to standard error.
7. Close databases. Close every connection, writer and reader. SQLite checkpoints the write-ahead log automatically on close.
8. Unmap. Unmap every ring buffer and close the per-CPU descriptors.
9. Exit.

Steps 1 and 2 are deliberately split. Unlinking the pathnames stops new senders finding the socket while the descriptors stay open, so whatever is already queued is still readable — closing them at step 1 would discard the queue, which is the data most recently produced and therefore most likely to explain why the system is being stopped.

The final checkpoint in step 7 matters most for the log and metric stores, which run synchronous=NORMAL and whose durability boundary is the checkpoint rather than the commit (§4.1).

8.4.2 The timeout #

Shutdown is bounded by peinit's service stop timeout. If the sequence has not finished, eventd aborts and exits immediately.

What an aborted shutdown costs:

• Uncommitted event batches are lost. Those events remain in the KMES ring buffers and are available at the next start, provided they have not been overwritten by then.
• Uncommitted log and metric batches are lost, which is acceptable by design.
• The diagnostic sequence metadata may be stale. It does not matter: startup derives resume points from committed rows and detects the difference between those rows and the ring buffer state as an ordinary gap (§2.5).

Every consequence is one the restart path already handles, which is why aborting is safe rather than merely tolerable.

8.5.1 After a crash #

An ungraceful termination — a segmentation fault, a kill, an out-of-memory kill — leaves four things true.

The ring buffers are unaffected. KMES writes regardless of consumer state, and events emitted while eventd was down accumulate there.

The databases are consistent. WAL mode guarantees committed transactions survive, and SQLite rolls back the in-flight batch on the next open.

There is a sequence gap. Events between the last committed batch and the crash were never persisted. On restart eventd derives its resume points from committed rows, sees the difference from the current ring buffer state, and writes a gap record (§2.5).

Socket-buffered data is gone. The kernel discards a socket receive queue on process exit, taking whatever logs and metrics were waiting. Acceptable by the loss model.

No manual recovery is needed and none is offered. eventd restarts, re-attaches, resumes draining, and records what was missed. The boot boundary logic recognises the restart from the committed rows themselves rather than from any flag written in advance (§3.7) — which is the point, since a crash is precisely the case where nothing was written in advance.

8.5.2 Signals #

Every other signal keeps its default behaviour.

8.5.3 The diagnostic dump #

SIGQUIT writes human-readable text to standard error before step 1 of the shutdown sequence, so that it reflects the daemon's state while it is still running rather than while it is tearing down. It includes at least:

• the current boot ID
• the active shard count and the readable historical shard count
• the per-CPU last committed sequence numbers, derived from committed rows
• the current non-streaming and streaming query counts
• the metric series cache occupancy
• the last observed write error for each store, where one exists

The format is not a stable machine interface and its wording may change.

The set is chosen to answer the questions an operator has about a misbehaving eventd that a query cannot: how far behind the writers are, whether the series cache is thrashing (§5.3), whether query slots are exhausted (§6.5), and whether a store has been failing writes quietly. Standard error is the destination because peinit captures it, so the dump reaches the log store by the ordinary path — and reaches standard error directly when the log store is the thing that is broken.

Every other failure in this chapter costs something recoverable. This one does not, which is why the whole ingestion pipeline is shaped around delaying it.

9.1.1 Ring buffer overrun #

When events are emitted faster than eventd drains them, the per-CPU ring buffers fill and KMES overwrites its oldest entries.

• eventd sees it as a sequence gap on the affected CPU (§2.5).
• A synthetic.gap record is written, naming the missing range.
• Draining resumes from the oldest survivor at tail_pos.

The events are gone. No other copy exists, and a gap record is a tombstone rather than a recovery — it records what was lost and when, which is the most that can be offered.

Four mechanisms delay it:

The first three are eventd's and operate automatically. The fourth is KMES's and is the one an operator can enlarge for a workload that bursts predictably.

9.1.2 Query timeouts #

A query exceeding QueryTimeoutMs is cancelled and the client receives an error (§6.5). Read-only connections are released; nothing is lost, and the query simply did not finish.

Streaming queries are bounded only up to watch; past that the watch phase is not time-limited.

The main risk is a large scan over a field with no index, which adaptive indexing reduces over time by indexing whatever keeps being filtered on (§3.4). A timeout is therefore worth reading as a signal about the index set rather than only as an error to retry.

9.1.3 Ingestion backpressure #

When a log or metric socket's receive queue is full, the kernel discards the datagram. Neither the sender nor eventd is notified, and eventd does not count it.

This is by design and is not a failure to be tuned away (PSPU §3.4). The one operational note is that the queue is not being drained while a batch is committing, because the same thread does both jobs (§4.1) — so a burst arriving during a commit is the common case for log loss.

9.2.1 Disk full #

When the filesystem holding a store reaches capacity, SQLite writes fail.

After any write failure consistent with disk-full or quota exhaustion, eventd schedules an immediate retention run across every store whose retention is enabled, under the ordinary bounded-batch rules (§3.6, §4.4, §5.5). Retention is the only lever eventd has that frees space, and waiting up to an hour for the next scheduled pass would waste the window in which recovery is still cheap.

9.2.1.1 The event store #

A failed INSERT or COMMIT does not crash the writer thread.

The batch is lost, and it is not recoverable: those events were already consumed from the ring buffer, so KMES no longer has them. The writer records the per-CPU sequence ranges of the failed batch in an in-memory lost-batch list, and on its next successful commit emits synthetic.gap records for every accumulated range before writing new events.

That ordering matters. Emitting the gap records first means the store never contains events written after a loss without the record of the loss preceding them.

The writer also logs the failure to standard error immediately — including the CPU identifiers and sequence ranges — which peinit captures. That is the only visibility available while the disk is still full and the gap record cannot yet be written.

If eventd crashes before the disk recovers, the in-memory list dies with it. Nothing is silently lost even so: on restart, resume points are derived from committed rows, and the difference from the current ring buffer state is detected as an ordinary restart gap (§3.7). The record is coarser — one gap rather than several — but the loss is still recorded.

Meanwhile events accumulate in the ring buffers. If the disk stays full long enough, they overrun and additional loss occurs, detected by the same mechanism (§9.1).

9.2.1.2 The log and metric stores #

A failed commit loses the batch. The writer retries on the next one. Acceptable under the loss model, and no lost-batch accounting exists for either — there is nothing to reconcile against, since neither has sequence numbers.

9.2.2 Corruption #

Corruption from a hardware error, a filesystem bug, or an incomplete write during a kernel crash.

Detection at startup is structural: eventd verifies that the required tables and indexes exist. It does not run PRAGMA integrity_check, which scans the entire database and costs time proportional to its size — unacceptable for a large event store on every boot. Corruption that leaves the schema intact, such as a single bad page, is found later, at query or write time, when SQLite touches it.

At startup, when SQLite reports corruption in a required active store, eventd quarantines the files, creates a fresh empty database at the original path, and continues (§3.3). It logs the corruption and emits synthetic.storage_error once a shard is available to hold it.

At write time, eventd stops writing to the affected database, emits synthetic.storage_error if it can, quarantines and replaces the database, and resumes writes to the replacement.

At query time, a handler encountering corruption fails the affected query with an error. It does not return the rows it managed to read: partial data from a database SQLite has declared corrupt is indistinguishable from complete data, and silently under-reporting an audit query is worse than failing it.

A missing or unrecognised schema version is not corruption. It is not repaired and not migrated: a required store with one fails startup, and a historical shard with one is excluded from the query path (§3.3).

Recovering data from a quarantined file is an administrative operation. eventd never attempts automatic repair, and the quarantined file is the only copy of whatever it held.

Two of eventd's four dependencies can disappear after startup without stopping it, and in both cases the ingestion path survives while the query path does not. That asymmetry is deliberate: events that are not collected are gone, and queries that cannot be answered can be asked again.

9.3.1 The registry becomes unavailable #

If LCS or loregd goes away after eventd has started:

• eventd keeps its last known configuration. Changes are not applied until the registry returns.
• Descriptor lookups fall back to the cache. A pattern the cache does not hold is denied, fail-closed (§7.5).
• eventd keeps ingesting and keeps serving queries for descriptors it already resolved, indefinitely.
• When the registry returns, the watch fires and eventd re-reads.

This is a degraded state, not a failure. eventd does not exit, and it does not stop collecting.

The related case is the watch failing while the registry is otherwise reachable. eventd discards the descriptor cache and operates fail-closed for new resolutions until the watch is re-established (§7.5), because a cache it cannot trust to be current would make a revocation silently ineffective. SIGHUP forces a configuration re-read in the meantime (§8.3).

9.3.2 KACS becomes unavailable #

If KACS goes away after startup:

• kacs_open_peer_token fails on new query connections, so new queries are denied.
• kacs_access_check and kacs_access_check_list fail, so a query in progress that needs a fresh check is denied.
• Cached check results stay valid for the duration of the query that obtained them.
• Event ingestion is unaffected. Neither the drain nor the write path calls KACS (§8.1).
• Log and metric ingestion are unaffected.

eventd keeps collecting and cannot answer. Query service resumes when KACS does.

9.3.3 KMES #

There is no partial mode. eventd attaches to every per-CPU ring buffer at startup and fails to start if it cannot (§8.2); there is no subsequent state in which KMES is present but unusable, because the mapping is established once and the read protocol has no call that can fail afterwards. A ring buffer resize is handled as a generation change, not as a failure (§2.2).

9.3.4 peinit #

peinit supplies the boot ID at startup and manages the lifecycle. It has no runtime role once eventd is running, so there is no failure mode here — peinit going away means the system is going away.

9.4.1 Memory #

If the out-of-memory killer takes eventd, it is treated exactly as a crash (§8.5): the databases are consistent, the ring buffers are untouched, peinit restarts the daemon, and the gap is recorded.

Three things bound eventd's memory, and each is worth knowing because each has a configuration that governs it:

The handoff channels are deliberately not on that list. They are bounded by the batch size (§2.3), which is the entire point of the ring-buffers-are-the-only-buffer rule: the thing that would otherwise grow without limit under load is the thing that is fixed.

The consumer most likely to surprise is the series cache under a high-cardinality producer — not because the cache grows, since it is bounded, but because the series table does, and the cache then thrashes on the thread that also drains the metric socket (§5.3).

9.4.2 Query slots #

Reaching MaxConcurrentQueries or MaxStreamingQueries rejects new queries with an error rather than queueing them (§6.5). Both bounds are global, so one client can occupy every slot.

Ingestion is unaffected, because queries and ingestion are separate channels and separate threads. That separation is what makes query-side exhaustion an inconvenience rather than data loss.

9.4.3 Descriptors #

An event query opens a read-only connection per shard database, and each connection holds one or two file descriptors (§6.4). Many historical shards multiplied by many concurrent queries reaches a process limit faster than anything else eventd does.

Where eventd cannot allocate what an admitted query needs, it fails that query rather than blocking a writer or exceeding its own limit (§6.1).

9.4.4 Writer stalls #

Two things stall a writer thread briefly, and both are bounded by design.

An index build in progress. Drain threads detect rising write pressure and signal cancellation; the writer aborts the CREATE INDEX, SQLite rolls back the partial index, and event writing resumes immediately (§3.4).

Retention holding a write lock. The shard's writer blocks until retention releases it. Retention works in small bounded batches and releases the coordination primitive between them, and under sustained write pressure it delays the next batch long enough for a waiting writer to make progress (§3.6). Events accumulate in the ring buffer during the stall, which is the ordinary absorption path.

Neither stall loses anything by itself. Both contribute to ring buffer pressure, and prolonged enough, both end at §9.1.

Sudden power loss is the failure the three stores' durability settings were chosen against, and it is where the hierarchy the whole daemon is built on becomes visible as three different outcomes.

The event store loses only the in-flight batch — the events accumulated since the last commit, which the adaptive batcher keeps as small as throughput allows (§2.4). On restart this appears as an ordinary sequence gap and is recorded as one (§3.7).

The log and metric stores may lose everything committed since their last write-ahead log checkpoint, which under synchronous=NORMAL is the real durability boundary rather than the commit. How much that is depends on WalCheckpointPages and the write rate, and it can be considerably more than one batch.

The difference is bought and paid for deliberately. FULL costs an fsync on every commit, which at ten thousand events a batch is amortised and at three events a batch is not — and eventd commits small batches constantly under light load. The event store pays it because an event may be an audit record whose absence is itself the finding. The other two do not, because their loss is defined as acceptable and paying for durability they do not need would slow the paths that are most likely to be bursty.

Events are sacred; logs and metrics are important but not fundamental. Every durability decision in this manual is that sentence applied to a particular store.

9.5.1 What restart does #

Nothing manual. eventd starts, finds its databases consistent — WAL recovery is SQLite's, and an incomplete transaction is rolled back on open — derives its per-CPU resume points from the committed rows, and records the difference from the ring buffer state as a gap.

The one case that differs from a crash is that events emitted while the machine was off are gone from the ring buffers too, since those are memory. A gap after a power cut therefore covers the downtime as well as the uncommitted batch, and there is nothing anywhere that held those events.

Every key lives under Machine\System\eventd\. eventd ignores unknown keys in the subtree. An invalid value is ignored and the value already in use is retained, and eventd emits a synthetic.config_change event for every change actually applied (§8.3).

A.1 Required #

No compiled-in defaults. A missing or invalid value fails startup (§8.2).

A.2 SQLite storage #

A.3 Event ingestion #

A.4 Log ingestion #

A.5 Metric ingestion #

A.6 Adaptive indexing #

A.7 Index shedding #

A.8 Adaptive rollups #

A.9 Retention #

A.10 Querying #

A.11 Cross-type filtering #

A.12 The security subtree #

Read-path descriptors live under Machine\System\eventd\Security\ and are not configuration in the sense above (§7.2):

Machine\System\eventd\Security\Events\*
Machine\System\eventd\Security\Events\<pattern>
Machine\System\eventd\Security\Logs\*
Machine\System\eventd\Security\Logs\<pattern>
Machine\System\eventd\Security\Metrics\*
Machine\System\eventd\Security\Metrics\<pattern>

The administrative descriptor is not here; it is admin_sd in eventd-meta.db (§3.5).

A.13 When a change takes effect #

Wire-protocol constants — the framing, the ingestion limits, the query message ceiling — belong to the interfaces rather than to eventd and are in PSPU §3.A.

B.1 Access rights #

B.2 Generic mapping #

Passed to AccessCheck in the generic_read, generic_write, generic_execute and generic_all fields.

EVENTD_ADMINISTER is in GENERIC_WRITE and deliberately not in GENERIC_READ or GENERIC_EXECUTE (§7.1).

B.3 Field GUID namespace #

EVENTD_FIELD_NAMESPACE = {e7d3a1b0-5c2f-4e8a-9b1d-0a6f3c8e2d4b}

Field GUIDs are uuid_v5(EVENTD_FIELD_NAMESPACE, field_name) with field_name as UTF-8 (§7.3).

B.4 Data type root GUIDs #

The level-0 node of an object type list.

B.5 Field names #

Field GUIDs are computed from the algorithm, never hardcoded. The names they are computed from are these.

Event header fields. timestamp, cpu_id, sequence, origin_class, event_type, effective_token_guid, true_token_guid, process_guid, boot_id.

Log fields. timestamp, origin, is_error, message, job_id, boot_id.

Fixed metric fields. timestamp, boot_id, name, type, value.

Event payload fields use the flattened dot path (PSPU §3.22). Suppressed paths and paths colliding with a header name are not query-language fields and have no GUID.

Metric label keys use the key itself: core produces uuid_v5(EVENTD_FIELD_NAMESPACE, "core"). A label key can never be one of the five fixed metric field names, because ingestion rejects records whose labels collide with them.

B.6 Origin class #

The query language accepts these names as aliases (PSPU §3.23).

B.7 Synthetic event types #

Payload schemas are in §3.2.

B.8 Metric types #

Stored in series.type. The query language exposes the names, not the numbers (PSPU §3.22).

B.9 Rollup functions #

These name per-series rollup functions. Window aggregation keywords have no identifiers of their own: AVG_OVER, MIN_OVER, MAX_OVER and SUM_OVER map to AVG, MIN, MAX and SUM when no transform is present, and RATE and DELTA rollups carry covered_ns for exact composition (§5.6).

P50, P95 and P99 have no identifiers because percentiles are not composable and are never rolled up.

B.10 Log severity #

Stored as an integer in logs.is_error; exposed as a boolean by the query language, which accepts both forms (§4.2).

B.11 Series hashing #

FNV-1a, 64-bit, over the exact bytes of the canonical label string or the boundary blob.

The high bit is cleared so the value fits SQLite's signed INTEGER. Hashes narrow lookups; identity is always confirmed against the full string or blob (§5.2).

B.12 Schema versions #

An unrecognised version is never migrated. For a required store it fails startup; for a historical shard it excludes the shard from the query path; for the metadata database it recreates from defaults (§3.3, §3.5).

None of the following affects the storage format, the wire protocol, the query language, or any behaviour a client can observe. An implementation omitting all of them is complete. Each buys measurable throughput or latency with no behavioural trade-off, which is why they are collected here rather than described as design.

C.1 Arena allocation for event copies #

Drain threads copy events out of the ring buffer at rates reaching hundreds of thousands per second (§2.2). Using the system allocator for each variable-sized copy costs freelist bookkeeping, potential lock contention in a multi-threaded allocator, and occasional page faults when it asks the kernel for more.

A per-drain-cycle arena avoids all of it: allocate a block at the start of the cycle, hand out sequential chunks by bumping a pointer, and release the whole block once the batch has been handed off. Per-event allocation cost falls from tens of nanoseconds to one or two, and the latency spikes disappear with it.

C.2 Drain thread affinity #

A drain thread reading a per-CPU ring buffer benefits from running on the same NUMA node as that CPU. It is not necessary — the per-CPU design eliminates write contention regardless of where the consumer runs — but NUMA-local reads avoid cross-node traffic in the drain loop.

In the 1:1 case, pinning the drain thread to the CPU whose buffer it reads gives the best cache locality available: the pages are likely still in that CPU's L3 from the kernel's write.

C.3 Partial payload extraction #

Building flat result records means decoding payloads, and at thousands of rows that is the dominant query cost (§6.1).

Where a SELECT names specific payload paths, only those need extracting. A streaming MessagePack decoder that scans for the wanted keys and skips over everything else avoids materialising the payload at all, and for a payload with many fields where one or two are selected this cuts per-row CPU by an order of magnitude.

Without a SELECT, a streaming decoder emitting flattened key-value pairs still avoids building a full in-memory representation.

C.4 Prepared statement pooling #

Writer threads already prepare one INSERT each and reuse it (§2.4). Query handlers executing translated SQL benefit from the same treatment: a small LRU pool of prepared statements per read connection, on the order of fifty to a hundred, covers the case where an operator repeats similar queries and where a dashboard issues the same query on a timer.

SQLite caches the query plan in a prepared statement, so re-preparing identical SQL spends CPU on parsing and planning that was already done.

C.5 Batched socket reads #

The log and metric ingestion threads read datagrams one at a time by default. recvmmsg reads many in one kernel round trip — up to a thousand or so — and at a batch of 64 to 256 it cuts syscall overhead by that factor under sustained load, with no protocol or format change.

It also shortens the window in which the thread is not draining the socket, which is the window that loses datagrams (§4.1).

C.6 SQLite page cache tuning #

Each connection has a page cache, around 2 MB by default. For a shard writer connection it holds B-tree pages for the events table and its indexes, and under sustained writes with several adaptive indexes the hot working set can exceed the default — producing evictions and re-reads on the write path.

A reasonable heuristic is 2 MB plus 1 MB per active secondary index for a writer connection, and 512 KB to 1 MB for a read-only query connection, whose queries are short-lived.

C.7 Bounded shard connection pool #

The query path opens a read-only connection to every database in the event store directory, and each holds one or two descriptors (§6.4). With many historical shards this becomes the binding resource.

Active shard writer connections stay open for the process lifetime and are not candidates. Historical shard read connections are: opening them lazily when a query touches them, and closing them after a period of inactivity, bounds the descriptor count without affecting any result.