4.2 The Logs Table

5 min read

The log store is a single SQLite database — not a directory of shards. There is no sharding here: one ingestion thread produces the writes, so splitting the target would give a single writer several files to switch between rather than several writers working in parallel.

It holds the logs, log_origins and metadata tables.

ColumnTypeContents
idINTEGER PRIMARY KEYSQLite rowid, monotonic.
boot_idBLOB NOT NULL16-byte boot ID GUID in PCDS binary layout.
timestampINTEGER NOT NULLNanoseconds since the Unix epoch — the producer's value if it supplied one, otherwise eventd's clock at receipt.
originTEXT NOT NULLThe producer peinit associated with the output pipe — a service, or a hook, health check, reload command or job within one.
is_errorINTEGER NOT NULL1 for standard error or an explicitly marked error, 0 otherwise.
messageTEXT NOT NULLThe log text.
job_idBLOB16-byte correlation GUID when the producer supplied one; null otherwise.

The schema is deliberately narrow. A log record is text with light metadata: what produced it, whether it was an error, when, and optionally which execution it belongs to. There is no payload blob and no origin class. origin is broker-attested rather than a service's self-assertion: the log socket admits peinit and excludes service-logon tokens (§7.6). The optional correlation key is not an identity.

4.2.1 What an origin looks like #

An origin is one or two components (PSPU §3.7):

origin    := component | component "/" producer
producer  := component | component "[" [0-9]+ "]"
component := [A-Za-z0-9_][A-Za-z0-9_.-]*

One component is a service — loregd, jellyfin — and is what a main process's output carries. Two name a producer within a service: peinit tags a pre-start hook jellyfin/ExecStartPre[0], a reload command jellyfin/ExecReload, a health check jellyfin/HealthCheck, and a submitted job jobs/<guid> (peinit TRM §11.1). The bracketed index appears only after a slash, and there is never more than one slash.

Ingestion discards a record whose origin is anything else, before the record reaches a batch (§4.1). Nothing wider is accepted: * would impersonate the access-control wildcard, and a backslash, a quote or whitespace would reach into the registry path an origin's descriptor lives at (§7.2).

A program needing more structure than this emits events.

4.2.2 The origin catalogue #

log_origins has one column, origin TEXT PRIMARY KEY. The log writer inserts a previously unseen origin in the same transaction as its first log row (§4.1), so every committed row is discoverable immediately.

Query planning enumerates origins from this compact catalogue rather than running DISTINCT origin over the retained log rows (§6.1).

Catalogue entries deliberately survive ordinary retention. An origin may therefore remain discoverable after its final row expires; this safe superset avoids write work on the ingestion path.

Low-priority maintenance may remove an entry after proving through a read-only scan that no row refers to it, with the deletion submitted to the log writer. The writer rechecks NOT EXISTS in that transaction and removes the origin from its in-memory set only after commit, so a log arriving between planning and execution cannot become undiscoverable.

Catalogue pages count toward logical live size.

4.2.3 is_error is an integer here and a boolean there #

The column stores 0 or 1; the query language exposes a boolean, and accepts either WHERE is_error == true or WHERE is_error == 1 (PSPU §3.22). ERROR ONLY is sugar for the first.

4.2.4 Write-time indexes #

Three indexes are created with the table:

  • idx_logs_timestamp on logs(timestamp) — time-range filtering, as everywhere.
  • idx_logs_origin on logs(origin) — "show me logs from X", which is the dominant log query.
  • idx_logs_job_id on logs(job_id) WHERE job_id IS NOT NULL — a partial index for "show me logs for job X". Partial because ordinary service output may carry no correlation key, so only correlated lines are worth indexing.

The origin index costs write amplification beyond the timestamp index, and the cost is modest in practice: origin has low cardinality, tens of distinct names on a normal system, so its index pages stay in SQLite's page cache and insertion stays cheap. The trade is accepted deliberately — the two dominant log queries must not become full table scans.

4.2.5 No adaptive indexing #

The log store does not participate in adaptive indexing (§3.4). Its field set is closed and small, and the three write-time indexes already cover the access patterns; there is no space of candidate fields for a policy to discover.

The same follows for query frequency counters: log queries do not increment them (§6.5).

4.2.6 Schema version #

The log store holds a metadata table with the same two-column structure as a shard's (§3.1). Schema version 1 comprises logs, log_origins and metadata; its version number is in §B.

eventd checks it at startup and applies the lifecycle rules of §4.3, and does not migrate.

Edit this page