6.8 The Flow layer
The Flow layer (Rules\Flow, the rung-2 design on PEI-598) is the same
rule atom, verbs and evaluation as the per-packet layers, applied to a
different unit: the flow, what conntrack tracks. It exists because
the per-packet evaluator has no early exit — collation and
side-effects-always mean every packet walks the whole forest, so an
Established → PASS rule saves nothing — and because flows carry facts
(a start time, relatedness, one day an owner) and effect units (a
connection, not a packet) that packets do not. The Packet layer becomes
the cheap filter in front; the decisions move here and run once per
connection.
6.8.1 One judgment per local endpoint #
peios_ntfe_flow_dispatch() (flow.c) is called at the IP seats:
inbound, at LOCAL_IN, for every packet the Packet layer passed;
outbound, at LOCAL_OUT, for every packet, before any Packet judgment
— the Packet layer judges outbound traffic at egress, after it (§6.2).
An untracked packet (snap->flow == NULL) has no flow to judge and is
accepted, NF_ACCEPT: inbound the Packet verdict stands, outbound the
Packet layer judges it next at egress. A
tracked packet reads its flow's sentence:
- a current sentence — its policy object and policy/context invalidation
generation are both current, and its expiry (if any) has not passed — is applied
without evaluation (
flow_cached); - otherwise the Flow forest is evaluated against the snapshot
(
peios_ntfe_policy_snapshot_eval(), counted injudgedandflow_judged), the outcome is written as the new sentence, an event is emitted (withREJUDGEDwhen a stale sentence was replaced,flow_rejudgedorflow_expiredsaying why), and the verdict is applied. A refusal goes out before the event, so the event can confess a degradation.
No Flow forest at all (generation 0, or no Flow key) is permissive,
counted, and caches nothing.
Flow dispatch holds one RCU policy snapshot while checking layer presence, preparing identity facts and evaluating. The sentence retains that policy object's generation as an internal stamp alongside the existing invalidation generation, which also changes when network context changes. Both must match before either endpoint's cached verdict can be reused. A packet already evaluating an old policy may finish; its old verdict cannot remain current after a new policy is published. The extra policy stamp does not change the flow-dump ABI.
On a cache miss, Flow refreshes Network.* from the active context table
for the packet's traversal interface after capturing the invalidation
generation. It clears the old fields first, so removing a context makes
those facts absent. Context publication swaps the table before advancing
the generation; the reader's acquire of that generation precedes its
context lookup. Thus a packet snapshot taken before publication cannot
leave an old network verdict cached under the new generation. A context
published after the captured generation makes the resulting sentence
stale on the next traversal, even if that evaluation already saw the new
facts. A current cached sentence needs no extra context lookup.
What the Flow forest judges is the flow view, not the packet's
snapshot: a Flow fact is one identical for every packet of the flow, so
ntfe_flow_view() builds it from the flow. A reply-direction packet's
addresses and ports are swapped back to the original tuple (and its
ICMP type replaced by the tuple's); the direction is the originator's,
recorded at the first judgment along with the interface, the VLAN, the
peer's MAC and the endpoints' identities (§6.9), so a re-judgment on a
reply sees exactly the facts the first judgment saw. (Found live before the fix: an inbound viewer flow
re-judged on its reply packet as out, and matched outbound-ok.) A
loopback flow's view takes the slot's direction. The refusal, when the
verdict is one, answers the packet in hand; the event describes the flow
as judged.
A normal flow has one local endpoint and one sentence, slot 0, written
at the originator's seat on the first packet; the reply direction, and
every later packet, reads it. Direction in the judgment is the
originator's side. A loopback flow has two local endpoints and two
sentences: the outbound one (slot 0) at LOCAL_OUT and the inbound one
(slot 1) at LOCAL_IN, both on the same first packet, and every packet
of it answers to the stricter of the two (DROP > REJECT(Refused) >
REJECT(Prohibited) > PASS). The comparison is made before any
refusal is sent: a REJECT overruled by the other end's DROP sends
nothing, and one overruled by the other end's stricter REJECT sends
that one's kind. Loopback-ness is the seat's device
(IFF_LOOPBACK, snap->loopback); a stale other-endpoint sentence is
not applied — it is that seat's to refresh when it next sees the flow.
6.8.2 The sentence #
struct peios_ntfe_sentence lives in NTFE's conntrack extension
(include/linux/peios_ntfe.h), two per flow: the invalidation generation
that judged (0 = empty), the generation of the policy object that judged,
expires_at (epoch seconds, 0 = never), the FNV-1a-64 hash
of the attributing rule's whole path (the outcome's attributed_hash:
the event's attributed text is cut short, the hash never is; the same
identity the tag and counter stores use for names, so the viewer
resolves it against the policy), the verdict and the reject kind. Alongside: start_secs, stamped when
conntrack created the entry (peios_ntfe_ct_ext_add()) — the Start.*
facts — and, from the first judgment, the interface, the direction and
whether the flow is loopback, for the dump.
NTFE shares conntrack extension storage with accounting and other kernel features. Its allocation sizes and offsets use 16-bit fields; the build-time capacity guard includes the NTFE payload and alignment padding for every allocation order. Adding another extension must not wrap the total length or overwrite an existing extension. These fields are kernel-internal and do not change the flow-dump ABI.
Writes take the flow's lock (ct->lock, _bh), zero the generation
first, write the fields, and publish the generation last with a release
store. Each slot also has a modification
sequence, advanced at the start and end of every write, including a
refresh under the same policy generation. The policy generation
identifies the policy; it cannot detect every cache update.
Reads are lock-free on the hook path and in the flow dump. They make one attempt: an active writer or a changed modification sequence makes the sentence read as absent. Readers neither spin nor acquire the flow's lock. On a cache miss the flow is evaluated as usual. Two packets of a new flow racing on two CPUs may both evaluate; the second sentence write wins (both judged the identities the first resolution recorded, §6.9), and the effects ran twice — the only place NTFE tolerates that, because the alternative is a lock on the fast path for a race that needs a flow's first two packets to arrive concurrently.
The cache holds the verdict only. Effects run at every evaluation of the
flow and never per packet. DROP and REJECT sentences persist for the
life of a flow conntrack has confirmed: a cached REJECT refuses every
subsequent packet of it. A DROP or REJECT on a flow's first packet
at a seat that stands before confirmation — LOCAL_IN for an inbound
flow, LOCAL_OUT for an outbound one — leaves nothing to persist:
nothing kills the entry, but its packet is dropped before conntrack
confirms it (the refusal is filed as the entry's reply, which confirms
nothing), so the unconfirmed entry dies with the packet, sentence and
all. A retransmitted SYN is then a fresh flow, judged again — effects
included — and refused again. So the cached refusal answers only on a
confirmed flow: one re-judged mid-life, or a loopback flow's inbound
end, judged at LOCAL_IN after POST_ROUTING confirmed the entry. A
Refused reset to a confirmed TCP flow also moves it to conntrack's
CLOSE state (nf_ct_set_closing()), whose short timeout ends it. A flow whose extension could not be allocated has nowhere to hold
a sentence and is evaluated on every packet (flow_uncached).
6.8.3 Staleness #
A sentence is stale when its invalidation generation is not the current
one (a policy published, or a network context published — §6.5 — both
advance the one counter), when the policy object that judged it is not
the current policy object, or when t_secs >= expires_at. Both are checked lazily, on the flow's next
packet — an idle flow past a policy change is killed when it next
speaks, or conntrack times it out; there are no timers and no walk of
the table at publication. Grandfathering was rejected: the registry must
not lie about what is enforced, and a REJECT rule must be able to
reject something already running. A refused packet of an
established TCP flow tears down both ends at once (§6.2): the end that
sent it is refused with the kind's story, and the packet, turned into a
reset, is sent on to the other end. Before the teardown existed, a
killed viewer stream froze at the local end while the silent host peer
waited for its own next packet to meet the cached REJECT sentence —
correct under the lazy law, and half a kill.
The expiry is the evaluation's expires_at (§6.4): the earliest moment
any live-time condition the judgment consulted would flip. A forest
with no time conditions never expires a sentence; Start.* conditions
never contribute, which is the point of them.
6.8.4 The flows dump #
PEIOS_NTFE_IOC_FLOWS walks conntrack's table the way ctnetlink does —
local_bh_disable(), each bucket under its nf_conntrack_locks lock,
original-direction entries of init_net that are neither expired nor
dying — and fills struct peios_ntfe_flow_rec per flow: conntrack's id,
family, protocol, the original tuple (ports, or ICMP id and type/code),
seen_reply/assured/related, the remaining lifetime, packet and
byte counts (NTFE turns conntrack accounting on at init — the
net.netfilter.nf_conntrack_acct sysctl, init_net.ct.sysctl_acct —
since it is conntrack's consumer now), and the extension: start time, first-judgment interface
and direction, loopback, both sentences, both ends' identities, and up
to eight tags by hash with the flow's total tag count.
Records are batched in kernel memory (32 to a batch) and copied to user
between passes over the buckets, never under a lock; a bucket holding
more flows than the batch has room for is walked again under its lock
from where the batch filled, rather than losing the rest. The walk counts every live flow it saw so
a short buffer is visible, and is best-effort against a table that
changes under it.
6.8.5 What was decided against #
- Backstop inheritance (the flow's verdict as the Packet layer's compiled-in default): unnecessary once Packet is the filter in front and passes tracked traffic by shipped configuration.
- Two layers, Inbound and Outbound: direction-blind rules are common and shadowing gives the direction-specific exceptions; the posture is a visible root rule; per-seat direction pruning is a future evaluator optimisation, invisible to authors.
- Effects on first judgment only: a kill by
REJECT, REPORT(n)must report; theCOUNTnoise is per generation, not per packet. - Identity in the sentence. The sentence caches the verdict; the identities the judgment read are recorded beside it (§6.9), not re-derived per packet.