Storage Sharding

§2.3.1 Shard model #

eventd distributes event writes across one or more independent SQLite databases called shards. Each shard is a self-contained database with its own file, WAL, and writer thread. Shards share no write-path state.

The number of shards is configured via the StorageShards registry key under Machine\System\eventd\. The valid range is 1 to 256. A value of 0 means the shard count equals the CPU count (as reported by kmes_attach). The default is 0.

§2.3.2 Shard-to-CPU assignment #

Each shard is assigned to exactly one drain thread (and thus one CPU) at startup. The assignment is static for the lifetime of the eventd process.

The assignment maps shard j to CPU j % cpu_count. When the shard count equals the CPU count, each CPU has exactly one shard (the 1:1 case). When the shard count is less than the CPU count, multiple CPUs share a shard. When the shard count exceeds the CPU count, a CPU is assigned multiple shards and round-robins events across them.

When a drain thread is assigned multiple shards, it distributes events across its shards using round-robin assignment. Each event is routed to the next shard in sequence.

When the shard count does not divide evenly by the CPU count, some CPUs are assigned one more shard than others. The resulting write throughput imbalance is proportional to one shard's worth of throughput and is negligible in practice.

The shard-to-CPU assignment is not persistent across restarts. A shard database may contain events from different sets of CPUs across different eventd lifetimes. Shards are a write-path optimisation only -- the query path MUST NOT assume any relationship between a shard and a specific CPU. Queries that filter by CPU ID MUST scan all shards.

§2.3.3 Writer threads #

eventd MUST create one writer thread per shard. The writer thread is the sole writer to its shard's database. No other thread or connection writes to that database.

When multiple drain threads are assigned to the same shard (shard count < CPU count), they hand off event batches to the shard's writer thread. When a drain thread is assigned multiple shards (shard count > CPU count), it hands off events to the appropriate writer thread based on the round-robin assignment. The handoff mechanism is implementation-defined but MUST NOT introduce unbounded buffering. The drain threads MUST NOT write to SQLite directly.

§2.3.4 Shard lifecycle #

Shard databases are created in the event store directory on first use. eventd MUST NOT delete or overwrite existing shard databases from previous configurations. If eventd starts with a different shard count than the previous run, the previously written databases remain in the directory and are available to the query path.

Shard database filenames MUST encode sufficient information to identify the shard and distinguish active shards from historical ones. The naming convention is defined in the storage chapter.

§2.3.5 Reconfiguration #

Changing the StorageShards value requires an eventd restart to take effect. eventd MUST NOT dynamically reassign CPUs to shards or create new shards while running.