3.25 Metric Queries
METRIC name[label_selector] [transform] [aggregation] [clauses…]
Metric mode evaluates rather than searches. A collector MUST reject
SELECT in metric mode: the result schemas are fixed (§3.22).
3.25.1 Selecting series #
The primary selector is a metric name, optionally followed by a label
selector in brackets. The name supports * with the same glob semantics
as an event type pattern (§3.23).
METRIC cpu.usage
METRIC cpu.*
The brackets — present, absent, or present and empty — decide how multiple matching series are handled, and this is the distinction that governs the rest of the mode.
No brackets — aggregate. Every matching series is combined into one result.
METRIC cpu.usage -- average across all cores
METRIC cpu.usage MAX -- maximum across all cores
Empty brackets — break out. Each series is returned separately.
METRIC cpu.usage[] -- latest value per core
METRIC cpu.usage[] SINCE 1h ago -- a time series per core
Filled brackets — select. Only series matching the label predicates.
METRIC cpu.usage[core="0"]
METRIC cpu.usage[core="0", host="srv1"]
METRIC disk.usage[device STARTS_WITH "sd"]
Label predicates are comma-separated and combined with AND. They use
the operators of §3.20, and within a label selector = is accepted as
equality alongside ==. Label keys are identifiers; values are
identifiers or quoted strings. An absent label resolves to null, so
[device IS NULL] selects the series that carry no device label.
3.25.2 Homogeneity #
After the name, the label selector, WHERE predicates and access
filtering have been applied, the remaining series MUST be of one
type. A collector MUST reject a selection spanning more than one type
at execution time, with an error asking for a narrower name or an
explicit WHERE type == ….
A selection resolving to zero series returns no records — a successful query with an empty result, not an error.
The rule exists because every function below is defined on one type. A selection mixing counters and gauges has no meaningful rate, and a selection mixing either with histograms has no meaningful value at all.
3.25.3 Function stages #
Function keywords execute in fixed stages regardless of where they were written:
- Transform —
RATE,DELTA,P50,P95orP99. Operates within each series independently and produces scalars. At most one per query. - Terminal aggregation — either a scalar aggregation (
AVG,MIN,MAX,SUM) or a window aggregation (AVG_OVER,MIN_OVER,MAX_OVER,SUM_OVER). At most one per query; specifying both a scalar and a window aggregation is a parse error.
The pipeline operates on scalars throughout. Counter and gauge samples
are already scalar; a histogram sample is not scalar until a percentile
function has been applied. A collector MUST therefore reject, at
execution time when the type is known, a query that resolves to a
histogram series without a percentile function, or that applies RATE,
DELTA, or any scalar or window aggregation directly to one.
Every output is a finite binary64. If any computation would produce NaN or an infinity, the query MUST fail with an error rather than returning it.
3.25.4 Transforms #
3.25.4.1 RATE and DELTA #
DELTA is the change between consecutive samples; RATE is that change
per second. Both apply only to counter series, and a collector MUST
reject them on a gauge or histogram at execution time.
Both use the same pair construction. Samples of one series are taken in
ascending timestamp order, with a deterministic tiebreaker among samples
sharing a timestamp. Each consecutive pair (s1, s2) whose s2 falls
inside the effective query range, and where s2 is later than s1,
produces one scalar at s2's timestamp. The immediately preceding
sample before the first in-range one MUST be used as s1 for the
first pair, when such a sample exists — without it the first point of
every range would be missing, and a chart would show a notch at the
start of every window.
The adjusted delta is s2 - s1 when the value rose, and s2 alone when
it fell, because a fall means the counter restarted from zero. RATE is
that adjusted delta divided by the elapsed seconds. A pair with
non-positive elapsed time contributes nothing.
METRIC http.requests.total SINCE 1h ago RATE
METRIC http.requests.total SINCE 1h ago DELTA
3.25.4.2 P50, P95, P99 #
Percentiles of histogram series only; a collector MUST reject them on a counter or gauge at execution time. Each histogram sample yields one value.
Evaluation is nearest-rank over the sample's cumulative counts: for
percentile q, compute rank = ceil(q × total_count), and take the
first boundary whose cumulative count is at least rank.
A sample with total_count == 0 yields no value. A sample whose rank
falls above the final cumulative count — meaning the percentile lies
in the overflow region above the highest boundary — also yields no
value, because the distribution does not record where in that region it
lies.
METRIC request.duration P95
METRIC request.duration[origin="loregd"] SINCE 1h ago P99
3.25.5 Scalar aggregations #
AVG, MIN, MAX and SUM reduce scalars to one value. They MUST NOT
be applied to a histogram series directly.
What they aggregate over depends on the brackets:
- Bracketed, so one result per series: over time, within each series.
- Unbracketed without
SINCE: over the latest transformed value of each matching series. The result timestamp is the greatest of the contributing timestamps. A series that cannot produce a value — aRATEwith fewer than two samples, say — contributes nothing. - Unbracketed with
SINCE: valid only when the selector resolves to zero or one series. More than one MUST be rejected with an error asking for a window aggregation.
METRIC cpu.usage AVG
METRIC http.requests.total RATE SUM
METRIC cpu.usage[] SINCE 1d ago AVG
METRIC cpu.usage[core="0"] SINCE 1h ago MIN
The unbracketed default aggregation, when no SINCE and no explicit
function is given, is AVG. No implicit scalar aggregation is added
when a window aggregation is present.
If nothing contributes to an aggregation, the query returns no record
for that output group. Otherwise the output timestamp is the greatest
contributing timestamp — for RATE and DELTA, the later sample of the
contributing pair.
3.25.5.1 Why unbracketed plus SINCE needs a window #
A collector MUST NOT synthesise a merged time series from samples that do not share timestamps.
Two series sampled at unrelated moments cannot be averaged point by point without inventing values between the points, and interpolation would make the collector responsible for a number nobody measured. A window aggregation supplies the common time grid explicitly, which is why it is required rather than assumed.
3.25.6 Window aggregations #
AVG_OVER, MIN_OVER, MAX_OVER and SUM_OVER take a duration and
produce one value per window. They require SINCE; a collector MUST
reject a window aggregation without one as a parse error.
Windows are fixed and aligned to Unix-epoch multiples of the duration — not to the query's start — so that the same window boundaries fall in the same places for every query. The output timestamp is the window start. Windows with nothing in them are omitted rather than emitted as null.
METRIC cpu.usage SINCE 1d ago AVG_OVER 1h
METRIC cpu.usage[] SINCE 1d ago AVG_OVER 5m
METRIC http.requests.total SINCE 1h ago RATE SUM_OVER 5m
METRIC request.duration P95 SINCE 1h ago AVG_OVER 5m
AVG and AVG_OVER are different keywords and a collector MUST NOT
treat them as synonyms: AVG produces one value for the range,
AVG_OVER one per window.
For raw and percentile-transformed values, a window contains the scalars whose timestamps fall inside it, and the function is applied to those. No interpolation is performed.
For RATE and DELTA with a window aggregation, each series first
produces at most one scalar per window: the window DELTA is the
sum of reset-adjusted deltas for pairs whose later sample is in the
window, and the window RATE is that divided by the elapsed seconds
those pairs covered. The preceding-sample rule applies to the first pair
of each window. The terminal aggregation then combines the per-series
window values — so RATE SUM_OVER 5m sums the series' five-minute
rates, and RATE AVG_OVER 5m averages them. Where the selector resolves
to exactly one series, all four window functions return that series'
window value.
Bracketed window queries keep labels in the result rows. Unbracketed ones omit them, unless the selector resolved to exactly one series.
3.25.7 Without SINCE #
With no SINCE, the query returns the latest value.
METRIC cpu.usage[core="0"]
METRIC cpu.usage[]
METRIC cpu.usage
"Latest" is per series, by timestamp with the deterministic tiebreaker.
For RATE and DELTA, it is the latest valid consecutive pair with
positive elapsed time; a series with no such pair returns nothing.
3.25.8 Boot filtering #
Samples carry boot_id but series continue across boots (§3.13). A
query MAY restrict to one boot:
METRIC cpu.usage[] WHERE boot_id == "550e8400-e29b-41d4-a716-446655440000"
A boot-filtered metric query MUST be evaluated from raw samples. A collector MUST NOT serve one from any pre-computed aggregate that is not itself partitioned by boot.
3.25.9 Results #
One record per raw sample; one per valid pair for RATE and DELTA,
timestamped at the later sample; one per histogram sample that yields a
percentile; one per window for window aggregations; one for a scalar
aggregation.
A histogram result carries only the percentile in value. The
boundaries, counts, total and sum are not returned by the query
language in this revision, in any mode.
{timestamp: 1714000000000000000, boot_id: "{550e8400-…}", name: "cpu.usage", type: "gauge", core: "0", value: 42.7}
{timestamp: 1714000300000000000, name: "cpu.usage", type: "gauge", core: "0", value: 39.8}
Without SORT, metric results are ordered by timestamp ascending
(§3.21) — the opposite of events and logs, because a metric result is
read as a series rather than as a list of occurrences.