Logs and events

Start with Event Viewer to inspect a machine's logs, events and metrics. eventd receives these records, stores them and answers queries about the stored history you may read. If the records you expect are missing, follow Find missing records before changing settings or removing stored data.

Three kinds of record #

eventd keeps three kinds of record, which arrive by different routes and answer different questions.

Logs #

A log line is what a program wrote to its standard output or standard error, the two streams every program writes text to. Each line carries:

• where it came from, its origin: the service's name, such as sshd. Output from a service's hooks and health checks carries the service's name and the hook's, such as sshd/ExecStartPre[0]. A job someone started carries jobs/ and the job's ID.
• which stream it was written to. Programs are meant to write errors to standard error, but many write everything there.
• when it was written, and which run of the service (its job) wrote it.

Log delivery to eventd is best effort: under load, downstream buffering and forwarding can drop lines. A service can still slow down when it writes output faster than peinit reads it, because writes to its output pipe can block. See Service output and logging.

Events #

An event is a record with named fields. Every event has a type, such as peinit.job.started or kacs.audit.access.checked, and a source, which says what recorded it:

Every event also carries the process and token that caused it, and the boot it happened in. Its other fields depend on its type. A kacs.audit.access.checked event, for example, says who asked, for what, on what, and whether it was allowed.

Use events when you need the structured record of an operation or access decision. eventd stores the events it receives and records detected loss, but this is not a guarantee that every event survives: the kernel's event buffers are finite and can overwrite events before a consumer reads them. The kernel buffers do not persist across reboots; eventd provides stored history. Check loss reports as well as the events themselves when accounting for what happened. See Auditing and the kernel manual's event failure modes.

Metrics #

A metric is a named number that a program measures again and again, such as how many requests a server has answered or how full a disk is. Each measurement is a sample. A metric can have several series, one for each set of its labels: eventd.store.bytes has a series for each of eventd's stores, labelled store=logs, store=events and so on.

Every metric is one of three types:

• a counter only goes up, such as requests answered, so what matters is how fast it rises;
• a gauge goes up and down, such as bytes in use;
• a histogram is a spread of measurements, such as how long requests took, read by its percentiles.

A program publishes metrics only under names it has been allowed to. eventd publishes its own health as metrics named eventd.*. Like logs, metrics are best effort.

Who may read what #

Every query to eventd runs as the person who asked. eventd checks what they may read against its read policy, which grants reading by name: a log origin, an event type or a metric name.

What a person may not read is left out without comment. eventd never says that something was withheld, because saying so would itself tell them something. An empty answer therefore does not prove that nothing happened. Event Viewer also tries to read the policy and reports which names it permits, restricts, or cannot determine. A restriction notice does not prove that any hidden record exists; even unrestricted results cover only stored, queryable records. See what you may not see and the missing-record checks.

Where to look #

• On the desktop: Event Viewer shows events and logs, newest first, as they arrive, and charts metrics on dashboards of your own.
• In a terminal: evctl runs a query in eventd's query language and prints what comes back. See Using evctl.
• For incident review: Save incident event evidence keeps a bounded query result with its diagnostics, completion status and context.

An empty result does not establish that nothing happened. Work through these checks in order, keeping the observations as you go. Preserve relevant records before changing retention or cleaning up storage.

1. Check the service, time and filters #

• For a service, open Logs… from Services Manager and confirm the service name and the incident time. Match the job and boot when inspecting a record; a restart has a different job ID. Use Find service logs for the service-run workflow. A service using TTYPath writes to its terminal instead of having its output captured for eventd.
• Events and Logs start with the last 24 hours. Choose a range that includes the incident, then use Show older to page back. any time removes the time restriction; it cannot bring back removed records.
• Check From, Containing and Errors only for logs, or Type and Source for events. Remove unintended restrictions and press Enter or Apply for typed filters. Standard error is a stream, not proof that a line describes an error.

For exact controls and paging limits, see Event Viewer.

2. Read the visibility notice #

Check the bottom-right notice described in What you may not see. It reports what the read policy allows for names, not whether matching hidden records exist. Some logs may be hidden from you means that the policy itself could not be read. Even an unrestricted policy cannot establish complete history: the data must still have been captured, retained and available to the query.

Record the notice. If the required names are restricted or the policy cannot be determined, ask an administrator to check the relevant policy and records; do not change permissions just to make an empty result disappear.

3. Distinguish a failed query from an empty one #

Keep any access refusal, timeout or storage error. A failed query does not establish the absence of matching records. A query timeout means the query did not finish, not that ingestion lost data. After recording the error, a narrower time range can help isolate the problem.

In a terminal, retain evctl's diagnostics on standard error as well as its result. Its result-commitment rules explain why an error before the initial result completes invalidates that partial result. If live following stops, record the notice before using Show the newest to resume; that action alone does not establish what happened in the gap.

4. Check the retention settings #

Open Event Viewer's Settings, then Keeping records, and note the current values or defaults without changing them. Anyone may inspect these settings; changing them requires the separate settings permission, granted to administrators by default.

The documented defaults are 30 days for events, 14 for logs and 90 for metric samples. Events and logs have no size limit by default; metrics default to 1 GiB. These are retention limits, not a minimum guaranteed history. Both age and any enabled logical-live-size limit are enforced, so size pressure can remove younger records.

Event retention can remove a whole earlier boot, including its loss records. Logs and metrics instead lose their oldest entries or samples under size pressure; see log retention and metric retention. Present settings alone cannot establish which records were previously removed.

5. Look for loss or an unavailable store #

Look around the incident time for eventd.events.lost, eventd.store.quarantined, and eventd's own log errors. If available, its health metrics also show recorded loss, write errors and retention deletions.

• A loss record establishes detected loss, not complete accounting. Power loss can destroy the final volatile event batch without a loss record. Logs and metrics have weaker durability, and full socket queues can refuse their data without a health count.
• A quarantined store is replaced by an empty one. The quarantined files may be the only copy of the earlier records. Query-time corruption instead fails the affected query.
• An unreadable or invalid historical shard can be logged and excluded from queries for that run without preventing startup. A running eventd therefore does not establish that every old store is being queried.

These observations narrow the cause; absence of a loss or quarantine record does not prove that the history is complete.

6. Preserve evidence before cleanup #

Keep the service, job and boot identifiers, incident time range, filters, visibility notices, exact errors, and retention values. Use Copy on relevant Event Viewer rows to keep all fields, or retain completed evctl results together with their query and diagnostics. Follow Save incident event evidence for private output files, completion checks and a bounded capture.

Leave quarantined files intact and involve the administrator responsible for recovery. eventd does not automatically repair them. Do not delete, rename or edit database files as a diagnostic experiment, or retry a destructive operation before preserving the evidence it may remove.

Retention measures logical live size, not filesystem usage. Deleting records frees pages for reuse; it does not immediately shrink the database file, and eventd never automatically runs VACUUM. See Reclamation before planning space recovery; these checks are not a database-repair or live-maintenance procedure.

Use evctl to save a finite event query before changing configuration or cleaning up storage. Keep the result, diagnostics and capture context together so another authorized reviewer can tell what was queried and whether it finished.

This procedure saves the records available to your query. It does not recover missing records or establish a complete incident history.

1. Choose a private destination #

Work in an existing, writable directory you control, whose native security descriptor and new-file inheritance already restrict access to the intended readers. Event payloads and diagnostics may contain sensitive information. Check the destination policy before writing, using Managing file security and Inheritance. If the required policy is unknown, have the responsible administrator check it first. Do not rely on chmod or a shell umask to establish Peios-native privacy.

Choose new, unused filenames for both the result and diagnostics. The shell opens redirection targets before evctl runs: reusing a name can truncate previous evidence even if the query later fails. Do not target existing files, symlinks or pipes.

Allow space for both the result and evctl's temporary spool. The temporary directory selected by the process must also be suitable for sensitive data, writable, have free space, and support anonymous temporary files (O_TMPFILE). A writable output directory alone is not enough.

2. Record what you are capturing #

In your private incident notes, record:

• the host and the identity running the query;
• the incident time range and time zone, and when you start the capture;
• the exact query, output format, destination filenames and any nondefault query socket;
• the installed tool version, shown by evctl version;
• relevant service, job and boot identifiers, if known.

Choose the event names and time range for the incident. The example below asks for at most 50 readable kacs.* events since one hour ago. It is a bounded sample, not all events from that hour. A relative range also moves when you rerun it, which makes the capture time important.

The query runs with the caller's access. Do not widen permissions to make the result appear complete. For query options and formats, see Using evctl.

3. Save the result and diagnostics separately #

After selecting the private working directory and confirming that both names are unused, run the complete quoted query. Save the exit status immediately, before running another command:

evctl --format jsonl 'EVENTS kacs.* SINCE 1h ago TAKE 50' > incident-events.jsonl 2> incident-events.err
capture_status=$?
printf 'evctl exit status: %s\n' "$capture_status"

JSON Lines writes one JSON object per record. Standard error goes to the separate diagnostic file. Record the displayed status and the capture's finish time in the same notes, then inspect the diagnostics.

Leave STREAM out of a finite capture. Do not pipe the capture through head or another consumer that can close early: evctl treats a broken output pipe as success, so a zero pipeline status cannot prove that all query output was saved. Review or filter the saved file afterward.

4. Decide whether the capture completed #

For the direct-to-regular-file command above, evctl reports:

If the shell cannot open a redirection target, evctl may never run; record that shell error too. Treat any interrupted or otherwise nonzero attempt as failed.

evctl buffers initial records until the server completes the query. A server error before completion prevents those records from being published. However, copying the completed result to your file is not atomic: a destination write or flush failure can leave a partial file. Do not treat such a file as a completed query result, even if some lines look valid. Keep it labelled as a failed attempt with its diagnostics.

An empty output file alone is ambiguous. It can accompany a failed query, or a successful query with no visible matching records. Always check status and diagnostics. If the capture failed, inspect both destination and temporary-filesystem errors; do not delete eventd databases to make room.

5. Keep the limits with the evidence #

Retain the exact query and its TAKE limit, times, identity, version, status and diagnostics with the result. Read policy can omit records without notice, and retention, collection loss or an unavailable store can limit what remains queryable. Follow Find missing records when expected data is absent.

Keep the original capture private and unchanged while reviewing it. Check the contents and intended recipient before sharing; a result you may read is not automatically suitable for a wider audience.

Implementation reference #

The capture behavior above is backed by these paths in eventd commit 3df88bab845d962e866158896807eac2976b83e3. Check the installed version when applying it to another build.

• CLI options and output channels
• Query completion and publication
• Exit status and broken-pipe handling, with failure classifications
• Temporary spool and final copy

Event Viewer shows what eventd has recorded on the machine: the events of its kernel and programs, the metrics programs publish, and the logs of its services. It shows events and logs newest first, adds new ones at the top as they are recorded, and shows any record in full. It charts metrics on dashboards you arrange.

It asks eventd as you, so results include only records you may read. It also tries to read the policy to explain restrictions on which names you may see; those notices do not establish that hidden records exist.

Opening it #

• From the launcher: start Event Viewer. It opens on the events.
• From Services Manager: select a service and choose Logs…, in the details pane or on the row's right-click menu. Event Viewer opens on that service's logs, including its hooks' and health checks'.
• From a program: gxwi-event-viewer opens on the events, gxwi-event-viewer --metrics on your dashboards, gxwi-event-viewer --settings on eventd's settings, and gxwi-event-viewer --logs NAME on what NAME logged.

A window opened on one service or one type of event says so in its title, such as Event Viewer: sshd, and on the Metrics tab the title names the dashboard, so several windows can be told apart.

Events #

Events is the first tab. Each row is one event, with when it happened, its type and its source. Select one to see its fields.

To narrow them:

• Type takes an event type, such as kacs.audit.access.checked. A * stands for any part of a type, so peinit.job.* is every job event and *.rejected every refusal of that kind. It suggests the types that have been recorded.
• Source shows only events from programs, the kernel, security (KACS) or the registry (LCS).

Metrics #

Select Metrics, between Events and Logs. On the left is every metric you may read, with its type and latest value, or how many series it has. A series is one of a metric's sets of labels: eventd.store.bytes has a series for each store, for example. Hold the pointer over a metric to see each series' latest value. Find a metric narrows the list as you type.

On the right is a dashboard: a page of charts over one time range. Until you make your own, it is eventd's health, which charts how eventd itself is doing: what it stores, whether it is losing events, how full its stores are, and whether queries are being refused.

Charts #

Select a metric on the left to add a chart of it to the dashboard. A chart has a line for each series, a legend naming them, and the values up its left side. Hold the pointer over a chart to see every line's value at that moment.

A chart divides its range into windows, such as a minute each over the last hour, and shows one value in each. Select ⋯ on a chart to change what it shows:

• Show is what each value is. A counter, which only goes up, is shown as its rate per second at first, or as its change in each window, or as recorded. A gauge, which goes up and down, is shown as recorded. A histogram, a spread of measurements such as how long requests took, is shown as a percentile: the median, the 95th or the 99th.
• In each window, the picks the average, least, greatest or total of the values that fall in each window.
• Lines gives one for each series, or one for all of them together.
• Move earlier and Move later reorder the charts, and Remove takes one away.

A chart's right-click menu has the same choices.

Dashboards #

The list at the top left picks a dashboard. Beside it:

• Type in the name box and press Enter to rename the dashboard.
• The time range is 15 minutes, an hour, 6 hours, 24 hours or 7 days.
• New dashboard starts an empty one.
• Delete… deletes the dashboard shown, after asking.

Charts are read again every few seconds, more often over a shorter range. Refresh, or F5, reads everything again now.

Your dashboards are yours: they are kept in your part of the registry, at CurrentUser\Software\gxwi-event-viewer, and other people have their own. Delete your last one and eventd's health takes its place again.

Logs #

Select Logs, after Metrics. Each row is one line, with when it was written, where it came from and what it says. Lines written to standard error are shown in red.

To narrow them:

• From takes a service's name, and shows its lines and those of its hooks and health checks. It suggests the names that have logged.
• Containing shows only lines with that text in them, whatever the case of the letters.
• Errors only shows only lines written to standard error. Many programs write everything to standard error, so this narrows less than its name suggests.

The time range and applying a filter #

The Events and Logs tabs look back over the last 24 hours at first. The list beside the filters changes that, to 15 minutes, an hour, 7 days or any time. Searching for text and the suggestions cover the whole range, so a shorter range is quicker on a busy machine.

Typed filters apply when you press Enter or select Apply. Errors only, Source and the time range apply as soon as you change them, and so does anything typed and not yet applied.

Following new records #

While the Pause button shows a dot, new records appear at the top as eventd stores them, usually within a second. Select Pause to keep the list still while you read, and Resume to show the newest again and carry on following.

If following stops, because eventd restarted or the window fell too far behind a burst of records, Event Viewer says so above the list. Select Show the newest to start again.

Older records #

The list starts with the newest 200 records. Show older, at the bottom of the list, adds the 200 before them. When there are no more matching, readable records in the queryable stores for the time range, it says so. Even any time cannot retrieve records removed by retention or held in a store excluded from queries. See Find missing records.

A window holds at most 1,000 records. Past that, the newest are let go to make room and the window stops following. It says so at the top, with Show the newest to go back. Show older carries on as before.

A record in full #

Select a row to see all of it in the pane on the right: for a log line, its whole text, its stream, its job and its boot; for an event, every field, the standard ones first under plain names, with the field's own name beneath. Times are shown to the nanosecond, on this machine's clock.

Drag the pane's left edge to make it wider or narrower. The width is kept while the window is open. On a narrow screen the pane is put away.

• Show only… narrows the list to that record's source or type. A row's right-click menu does the same.
• Copy copies every field, one per line.

The Up and Down arrow keys move through the list, and F5 reads it again from the newest.

What you may not see #

eventd leaves out what you may not read without saying so. Event Viewer tries to read eventd's read policy as you and reports the permissions on names, at the bottom right of the window:

The Metrics tab says the same of metrics. By default everyone signed in may read every metric.

If eventd does not let you ask it anything at all, the list says so instead of showing nothing.

Settings #

Select Settings, the last tab, to see how eventd is set up and who may read what. Anyone may look. Only those allowed to change eventd's settings may change them, by default administrators; everyone else sees them read-only, with a line saying why.

eventd's settings #

The settings are in groups: Keeping records first, then taking in events, logs and metrics, answering queries, and storage. Each one says what it is, what it is set to or that it has its default, the values it may take, and whether a change applies at once or when eventd next starts. These descriptions are eventd's own, from its regman page, which regman shows in a terminal.

• Type a value and press Enter or Save to change one. A value outside its range is refused, with the range given, and nothing is changed.
• Use the default takes a setting back to its default.

eventd notices a change by itself and records it as an eventd.config.changed event, which the Events tab shows. The paths of its stores and sockets are shown but not changed here: eventd needs them to point at places made ready for it.

Who may read what #

eventd's read policy says who may read which events, logs and metrics. It is a list of patterns, each a name with the permissions for it:

• * covers everything of its kind that no other pattern covers.
• A name, such as sshd for logs or kacs for events, covers that name and every name under it, such as kacs.audit.access.checked.

The most specific pattern that matches decides. Each pattern says, in words, who may read it, who may only read some of its fields, and who may not. For metrics it also says who may publish them.

• Change… opens the pattern in the permissions editor. Changes apply from the next question anyone asks eventd.
• Add a pattern starts a new pattern for a name, with the permissions that apply to that name now, and opens it to be changed. Nothing changes until you change it there.
• Remove… takes a pattern away after asking. The broader pattern above it then decides. The * patterns cannot be removed, since eventd would let nobody read what they cover.

Changing eventd's indexes, at the end, says who may ask eventd to index a field.

For example, to keep sshd's logs from everyone but administrators, add a pattern sshd under Logs and, in the editor, take Read away from Authenticated Users. Anyone else then sees "Hidden from you: logs from sshd*" on the Logs tab.

In a terminal #

evctl asks eventd the same questions in its query language. The window's filters are its LOGS FROM, CONTAINING, ERROR ONLY, EVENTS <type> and SINCE clauses, and a chart is a METRIC query such as METRIC eventd.store.bytes[] SINCE 1h ago MAX_OVER 1m, so anything the window shows, evctl can print. See Using evctl.

Every record in the event stream is a type name and a set of fields. The record says that something happened — kacs.audit.access.checked, with outcome.success false and an access.granted of 0x120089 — but not what any of it means. Whether access.granted is the rights the check granted or the rights a handle was opened with, whether a missing trigger.ace is a bug or expected, which of two masks you are looking at: that knowledge lives with the software that writes the records. It lives in evman, the event stream's manual.

evman is to events what regman is to the registry. Give it an event type or a field and it tells you what that is. Reach for it whenever a record does not say enough on its own — before writing a query, a filter or a security descriptor over a field, and whenever a value surprises you.

evman, in one sentence #

evman <name> documents what an event type or field means — when it fires, what each field records, which values it takes — drawn from shipped documentation, never from the live event stream.

Looking up an event type #

Give evman an event type and it prints the event's card: how important it is, what decides whether it is recorded, a description, and every field it carries.

$ evman kacs.audit.access.checked

kacs.audit.access.checked                                      defined by kacs

  Tier         essential
  Gating       a matching SACL audit ACE, or the token's audit policy
  Cardinality  once per matching audit ACE, not once per access

The record that an access check completed, and what it decided. The most
common event on the system, and the one most investigations start from.
...

Fields
  subject                           group, from kernel
    subject.token.sid
    subject.token.groups
    ...
  object.kind                       required
  object.file.path                  when object.kind == file
  access.requested                  required
  access.granted                    required
      The mask this check granted.
  ...

Each field line says when the field is present — always, optionally, or only when another field has a given value — and an indented note says what the field means in this event, where that is narrower than its general meaning. Read those notes: the same field can mean subtly different things in different events, and the note is where that is written down.

Looking up a field #

Give evman a field and it prints the field's card: its type, the values it can take, a description, and every event type that carries it.

$ evman subject.token.type

subject.token.type                                           defined by kernel

  Type    str.enum
  Values  primary | impersonation
  Closed  true

Whether the effective token is a primary token or an impersonation token.
Needed beside subject.token.impersonation, because a primary token and an
Anonymous impersonation token both report level 0 there.

Carried by
  kacs.audit.access.checked  (via group subject)
  kacs.audit.handle.used     (via group subject)
  lcs.audit.key.opened       (via group caller)
  ...

Carried by is the reverse index: the event types whose records can hold this field. It is the list to check before you rely on a field across the whole stream — a query on subject.token.sid finds nothing in an event type that does not carry it.

Closed says whether the list of values is complete. A closed list never grows without the field changing; an open one may gain values, so anything you write over it should cope with a value it has not seen.

Some fields are carried in the record header rather than the payload — the time, the sequence number and who wrote the record. Their cards say so: they are on every event, and no payload repeats them.

Looking up a subtree #

Field names and event types are both dotted, general to specific. Give evman a prefix and it lists what lies beneath it, one summary line each:

$ evman kacs.audit

Event types
  kacs.audit.access.checked  The record that an access check completed, and…
  kacs.audit.handle.used     The record of what was done with a handle afte…
  kacs.audit.privilege.used  The record that a privilege contributed access…

$ evman subject.token

Fields
  subject.token.auth-id             The LUID of the logon session the effec…
  subject.token.groups              The group SIDs carried by the effective…
  subject.token.integrity           The integrity RID of the effective toke…
  ...

So you can start from a subsystem, or from a part of a record such as "who acted", and drill into any one entry.

A few names are not defined one by one. Any field can have a variant with a suffix such as -previous or -requested — policy.generation-previous is the generation that was in force before — and any thing can have a standard attribute such as count or size. evman resolves these to the field or thing they qualify and says which rule it applied.

Searching, when you do not know the name #

evman -k searches names and summaries — the event stream's apropos. Every term must match.

$ evman -k swap
buffer.capacity-previous    The per-CPU ring capacity before a capacity swap…
buffer.generation-previous  The ring generation before a capacity swap, nor…
kmes.buffer.swap.failed     The record that KMES could not resize its per-C…

evman documents meaning, not what happened #

evman tells you what an event or field means — never which events have occurred on this machine. It reads shipped documentation and never touches the event stream or the event store. Ask it about kacs.audit.access.checked and you learn when the event fires and what its fields hold; you do not learn that anything was denied today. That is a query against the event store, which is the Event Viewer's job, or eventd's query interface.

Where the documentation comes from #

evman has no built-in list of events. Each package ships documentation for the events it writes and the fields it defines, as files in /usr/share/evman/. Install a package and its events appear in the manual; remove it and they go.

The manual is also the definition. Every field any Peios component writes is defined in exactly one of these files, by the component that owns it, and a field defined nowhere is not a field anything should emit. That is what keeps one value from acquiring two names: a security descriptor that denies a field must not be evadable by writing the same value under another name. The rules are in The Catalogue.

(For the people who write these files, evman lint checks them against those rules. That is a packaging concern rather than an everyday one; see Writing evman pages.)

Where to go next #

• Logs and events — what the event stream is, and how events differ from logs.
• regman — the same idea for the registry, including the emission-policy keys that switch event types on and off.