Universal Directory
A self-hosted directory service in the Active Directory lineage.
Single-page view · as markdown
What is Universal Directory
Universal Directory / Getting started
Universal Directory (UD) is Peios' directory service: a hierarchical tree of named objects — organisational units, users, groups, and whatever else the schema is extended to describe — in the lineage of X.500 and Active Directory, built self-hosted from the ground up.
UD is in early development. What exists today is a RAM-resident directory engine and its server, udd.
What works today #
- A tree of objects whose identity is a GUID. Distinguished names (
CN=Jack,OU=Sales) and native paths (/Sales/Jack) are always derived, never stored. - A self-hosted, compositional schema. Attributes, facets, and classes are themselves objects in the tree, browsable like everything else and extensible at runtime through namespaced definitions.
- A multi-master convergent replication engine with a three-stage deletion lifecycle, per-atom merge, and deterministic conflict repair — proven across in-process node fleets under simulated chaos.
- Durable persistence. A restart recovers the directory from a snapshot-and-log footprint on disk, with acked-implies-durable semantics.
- A JSON API and an embedded debug console that expose everything the engine knows — per-value metadata, replication stamps, storage marks — in the browser.
What does not work yet #
Be equally clear about the gaps.
There is no authentication or authorisation on the JSON API. Anyone who can reach the port has full control of the directory. The listener binds to loopback.
The GIP fabric is partially built. GIP is UD's intra-domain network layer. Daemons discover the other machines in the directory, hold supervised QUIC link connections to them, and run authenticated end-to-end encrypted service channels over those links. The first real service is ud/msgs — machine-to-machine talk with honest delivery outcomes.
Directory replication does not ride the fabric yet. The replication engine still syncs only between in-process nodes. Wiring it to the network is the next arrival.
Treat udd as a development tool at this stage, not a deployable service.
Design decisions you can see in today's behaviour #
Even this early, a few load-bearing decisions shape everything UD does.
Identity is not location. Every object gets an immutable GUID at birth. Its DN and path are addresses computed on demand. Renaming or moving a container changes one object's record — every descendant's DN changes logically, for free, and references never notice, because references store GUIDs.
The data model is X.500-shaped from day one. DNs use real LDAP naming attributes (CN=, OU=), attribute and class names follow LDAP descriptor grammar, and DN values are escaped per RFC 4514. There is no bespoke model waiting for a compatibility layer to be bolted on.
Names are caseless from birth. Jack and jack cannot coexist as siblings. Case-insensitive but case-preserving uniqueness is baked into the tree's conflict detection rather than layered on later. See naming rules for the exact folding.
Composition, not inheritance. The schema has no class hierarchy at all. Classes are flat compositions of facets, and objects can gain facets individually or through class extensions. See the schema model.
Nothing is hidden. The schema is ordinary objects under /Configuration/Schema. Engine-level bookkeeping like facet attachment is stored as an ordinary attribute. The console shows unset attributes, system flags, dangling references, and per-value metadata. Full observability is a feature, not a debug mode.
Where to go next #
To create a domain and start the daemon, read Running udd.
For the object model — GUID identity, derived addresses, and the deletion lifecycle — read Objects and identity.
For the schema, which is where UD differs most from what you may expect, read The schema model.
Running udd
Universal Directory / Getting started
udd is the Universal Directory server daemon. It lives in the ud repository as the first crate of a Cargo workspace.
Creating a domain and running one are separate commands, deliberately. udd on its own never creates anything: a mistyped --data-dir, an unattached mount, or a half-finished restore must fail loudly rather than silently birth an empty directory that answers queries as if it were yours.
Create a domain #
cd ud
cargo run -- new
udd: created a domain in ud-data
udd: machine object /Domain Controllers/dc1
udd: networkAddress 10.0.0.5
udd: networkAddress dc1
udd: run it with `udd --data-dir ud-data`
udd new creates the data directory's footprint, materialises the domain, mints this machine's identity keypair, plants this machine's object in the tree, and exits. It refuses a directory that is not empty — whatever is in there, moving it aside is your call, not the daemon's.
| Option | Effect |
|---|---|
--data-dir <path> | Where to create the domain. Default ./ud-data. |
--name <name> | The machine object's name. Default: this machine's hostname, or DC1 if that is unusable. |
--gip-port <port> | Advertise this port in the machine's addresses instead of leaving the well-known GIP port implicit. |
--address <endpoint> | Advertise this address instead of detecting any. Repeatable. |
The machine object and its key #
The machine object is an ordinary computer object under /Domain Controllers, carrying the machine facet: its detected addresses and its public key (machineKey). Nothing about it is privileged — rename it, move it, edit its addresses in the console. Its placement is convention only; nothing in UD looks for domain controllers by path.
The private half of the keypair never enters the directory. It lives in the data directory's write-once machine-key file, beside the domain's generation files, and the whole data directory is created owner-only (mode 0700) because it holds a private key and the entire directory database. The key is this machine's GIP identity.
Check the detected addresses #
Address detection is a best effort, and you should check it. udd asks the kernel which local address it would use to reach the wider network — one per IP family, by opening a UDP socket and reading back its source address, sending no packets — and adds the hostname if it is a legal DNS name.
Behind NAT, inside a container, or on a multi-homed machine, that answer can be confidently wrong. This is why every value is printed at creation, overridable with --address, and editable afterwards like any other attribute.
Run it #
cargo run # or: cargo run -- --data-dir /srv/ud
udd: storage [ud-data] recovered: generation 2, 53 objects, 14 frames replayed
udd: debug console listening on http://127.0.0.1:5389
udd recovers the domain from the files and serves it. It never creates. Pointing it at a directory that holds no domain is an error that tells you to run udd new.
| Option | Effect |
|---|---|
--data-dir <path> | The directory holding the domain. |
--ephemeral | No disk at all. The identical persistence code path runs against an in-memory filesystem, with a fresh domain created at startup and everything vanishing at exit. |
--listen <addr> | The debug console and API address. Default 127.0.0.1:5389. |
--gip-bind <addr> | The GIP fabric's UDP bind. Default 0.0.0.0:5390; ephemeral daemons default to a free loopback port. |
--ephemeral is the one mode that creates implicitly, because a RAM directory has nothing to recover by definition.
The server serves everything from one listener: / is the debug console, a browser UI embedded in the binary, and /api/… is the JSON API. The address is currently a hardcoded constant — 127.0.0.1:5389, a mnemonic nod to LDAP's port 389 — and becomes configurable when udd grows more configuration.
GIP link connections are encryption-only: no trust rides them, so an open fabric port exposes nothing but a QUIC handshake. Trust lives one layer up. Every service conversation is a channel, end-to-end encrypted and mutually authenticated against the directory's keys, and the daemon admits only IK-authenticated channels naming a registered service from a machine whose key the directory holds.
The fabric #
A running daemon discovers every other machine in the directory — any alive computer object carrying a machineKey and a networkAddress, placement irrelevant — and holds a supervised QUIC connection to each: dialled on discovery, kept alive with keepalives, and redialled with exponential backoff when it dies.
Every dialled link is identified the moment it connects. The daemon opens a gip/ident channel over it: a payload-free conversation whose Noise handshake is the content, proving to the dialer that the far end holds the key the directory binds to that machine, and proving to the acceptor who dialled. A link whose ident fails — the machine at that address is not who the directory claims — is closed on the spot.
Identification is also what lets two daemons that have both dialled each other keep exactly one connection. The lower machine GUID's dial survives; the other side gracefully retires its own and holds the peer's connection instead. The console shows this as via inbound, the same state a NAT-ed peer's partner lives in permanently.
Watch all of it live in the console's fabric tab: peer states, RTT, byte counters, identified inbound connections named by machine, and an event stream narrating every dial, ident, tiebreak, loss, and retry.
ud/msgs #
The first real service riding the fabric is ud/msgs — machine-to-machine talk, sendable from the console's msgs tab or with POST /api/msgs/send.
It is deliberately talk, not email: online-only, no queueing, no storage. Its real job is to prove the fabric end to end. Every message is one short-lived authenticated channel over the held connection, and every send reports an honest fate — delivered with the round trip, failed when provably undelivered, or unconfirmed when the channel died between the send and the ack.
Try it in the two-DC harness: send both ways, then kill one daemon and watch the outcomes tell the truth.
The two-DC dev harness #
dev/two-dc.sh stands up a complete two-machine domain on localhost. It runs udd dev-fixture, which builds two data directories sharing one domain out-of-band: machine A creates the domain and performs the admission write for machine B — exactly the shape udd join will later perform over the wire — and B receives everything through the real pull machinery in-process. The script then launches both daemons on separate ports and prints their console URLs and PIDs.
Open both fabric tabs, kill one daemon, and watch the other back off and recover in real time.
When udd refuses to start #
A report of N torn bytes truncated after a crash is normal. It is unacknowledged residue of writes that were still in flight; nothing that was ever confirmed is affected.
A refusal to start is different. Recovery will not guess around damage, so udd stops and names the file and offset. Three cases have specific causes:
- The files are damaged. Restore the directory from a copy, or move it aside and create a new domain.
- The
machine-keyfile is missing, damaged, or belongs to a different node. A file copied from another machine's data directory must never let one machine wear another's identity. Restore the file from a backup of the same data directory, or move the directory aside and create the domain again. - The directory holds a node but no domain. A
udd newwas interrupted before the domain's birth was durable. The birth is a single all-or-nothing frame, so there is no half-built directory to salvage: move the directory aside and create it again.
A boot warning that the directory's machine object does not carry this machine's key means someone edited or deleted the object. That one is fixable in the console.
What to know before you start #
- There is no authentication. Anyone who can reach the port has full control of the directory. The listener binds to loopback only. Treat
uddas a local development tool. - The fabric does not carry directory data yet. Daemons hold real QUIC link connections and real services ride them, but replication between daemons arrives with the DRS-over-GIP work. The replication engine itself still syncs only between in-process nodes.
- One process per data directory. Nothing locks the directory yet; two daemons on the same files would interleave destructively.
What a fresh directory contains #
udd new materialises the root object, /Configuration, /Configuration/Schema, /LostAndFound, and /Domain Controllers, with the complete base schema — 48 definition objects, all flagged system and immutable — plus the one machine object for the machine you ran it on.
Everything else is yours to create, and survives restarts.
Objects and identity
Universal Directory / Concepts
Every entry in the directory — an OU, a user, a schema definition — is an object. UD's object model rests on one distinction it never blurs: identity versus address.
Identity: the GUID #
An object's identity is a GUID, assigned at creation and immutable for the object's entire life.
Everything that needs to point at an object stores that GUID, never a name or a path: a group's member values, a facet's attribute list, a class's composition. Renames and moves therefore never break references. Deleting an object leaves any remaining references visibly dangling at the dead GUID, rather than silently rebinding to whatever later takes the name.
Address: derived, never stored #
Internally the store is inode-shaped: a flat table of objects keyed by GUID, where each object records only its parent's GUID and its own bare name (Sales, Jack). The tree is an index over that table.
Both address forms are computed on demand:
- the distinguished name — leaf-first with typed components,
CN=Jack,OU=Sales, where each prefix comes from the object's class (dnPrefix) and values are escaped per RFC 4514; - the native path — root-first and untyped,
/Sales/Jack, used by the console, the JSON API's path resolution, and anywhere a human types an address. Path lookup is case-insensitive.
Because addresses are derived, renaming an OU is a single-object change. Every descendant's DN and path change logically, with zero writes, and nothing that references those descendants notices at all.
The tree has a single root object — empty name, class domainRoot — which never appears in DNs and cannot be renamed, moved, or deleted.
Names and sibling uniqueness #
Within one parent, names are unique under two deliberately strict rules.
Caseless, case-preserving. Names compare under Unicode canonical caseless matching, so Jack, jack, and a decomposed Jack are all the same name. The spelling you create is the spelling you see; the folding governs comparison and conflict only. See naming rules for the exact function.
Type-blind. Uniqueness ignores class entirely. An OU named Sales and a group named sales cannot be siblings. One container, one meaning per name.
Name validity is strict too — non-empty, no /, no control characters, no leading or trailing whitespace. Strict rules can be safely loosened later; a loose rule can never be tightened without breaking existing data.
Rename and move #
Rename and reparent are one operation, in the shape of LDAP's ModifyDN: change the name, the parent, or both, atomically.
The engine refuses:
- moves that would create a cycle, placing an object under itself or its own descendant;
- a destination where the caseless, type-blind name is already taken;
- any rename or move of the root, of system objects, or of schema objects — definitions are add and delete only — and any move into the schema subtree.
A case-only rename, jack → Jack, is always legal. It is the same caseless name, so it conflicts with nothing.
Delete: the three-stage lifecycle #
Deletion is soft and subtree-wide. Deleting an object tombstones it and every descendant in place, each with its own stamped existence transition.
A tombstone leaves the effective tree immediately — its name is free at once — but the full record survives, and every tombstoned object is individually restorable. Restoring is top-down: the parent must be alive and the name must still be free. It is an ordinary write, and a restore into a conflict simply refuses.
Beyond Deleted sit two harder states.
Shredded is security erasure, reachable directly from alive or as an escalation of a soft delete. The payload and even the name are destroyed; only a skeleton — GUID, parent GUID, stamps — remains. Every reference to the shredded GUID is physically scrubbed out of every holder, because erasure that leaves the object's GUID lingering in group member lists would fail its own purpose. The destruction reaches the node's files within a guaranteed window. There is no way back.
Purged means the tombstone is gone entirely. This is never manual: purge happens automatically once a tombstone's age passes maxTimeApart.
Aging is computed, not written #
Two domain-wide knobs, stored as attributes on /Configuration with defaults of 30 and 180 days, drive the automatic transitions. Past timeUntilShred a soft-deleted record hardens: the restore window closes and the flesh is scrubbed to a skeleton. Past maxTimeApart the skeleton purges.
Aging is a pure function of the object's existence stamp, the config windows, and the clock. Every node evaluates a tombstone's effective stage the same way, so nodes never race stamps about it. Local garbage collection merely catches physical storage up with what the aging function already answers.
The root and system objects are permanently protected, and schema definitions add their own unused-delete rules. Because references store GUIDs, deleting a referenced object never edits the referrers: their values dangle visibly at the tombstone until the target is shredded, at which point they are scrubbed.
System objects #
Objects created at domain creation carry a system flag: the root, /Configuration, /Configuration/Schema, /LostAndFound, /Domain Controllers, and every base schema definition.
System objects refuse renames, moves, and deletion — they are the fixed points the rest of the directory is built around — but their attributes are writable. The replication knobs live as attributes on /Configuration, and a description on the root is legal. Schema definitions are the exception, fully immutable via the schema-subtree rule.
Structural repairs are derived, never written #
Replication will merge writes that were each legal at their origin but collide when combined. UD repairs these as a view: the stored records stay exactly as written, and every node derives the same effective tree from the same atoms. No repair writes, no repair replication traffic.
Three rules, all favouring the earliest claim — the first legal write stands, because the later write is the one that caused the violation:
- Name conflicts. Two live siblings whose names fold equal: the earliest placement claim keeps the name, and the later one displays as
name CNF:<guid>. It remains resolvable and renameable, and its stored name is untouched. - Orphans. A live object whose stored parent is dead or missing derives into
/LostAndFound. Restoring the parent snaps it back automatically. - Cycles. Concurrent moves that form a loop break at the newest placement: the latest mover derives into
/LostAndFound, and the earlier move survives intact.
Admin operations target the effective view — rename the CNF loser, move the orphan out, as ordinary writes — and the repair dissolves the moment the underlying violation is gone.
Objects whose class definition has not arrived yet are ghosts: visible in the debugger, holding their sibling name, but refusing every originating write until the schema converges.
Replication stamps #
Every replicated fact about an object decomposes into merge atoms:
- its existence — creation, with class and the system flag riding along;
- its placement —
(parent, name)as one atom, so rename and move contend as a unit; - each attribute value.
Every atom carries a stamp: a version, an originating time, and an originating invocation id. Together these decide conflicts deterministically — higher version wins, then later timestamp, then origin — and a transfer certificate drives change enumeration. Removing a value leaves a stamped absence atom rather than a hole, so the removal itself can win merges.
For how those atoms travel and merge, read Replication.
The schema model
Universal Directory / Concepts
UD's schema is compositional. There is no class inheritance anywhere in the model, and never will be. The schema is built from three layers, each referencing the one below by GUID, with flat set-union as the only combining operation.
The three layers #
Attributes are defined globally. One definition per attribute — surname, member, expiry — carrying:
- its syntax:
string,integer,boolean,timestamp,reference,endpoint, orpublicKey; - whether it is single- or multi-valued;
- its matching rule,
caselessorexact, governing value comparison and set uniqueness; - the meta-attributes its values may carry (
legalMeta, below); - an aliases slot recording alternative wire names.
Facets reference attributes; they never define them. So two facets that both want description agree by construction, because there is only one description.
A facet is a named set of attribute references — the smallest unit of composition. securityPrincipal carries sid; person carries givenName, surname, email; membership carries member. Each reference in a facet can be flagged required, shown in the console as *. Facets cannot include other facets: composition is flat, so it is order-free and collision-free.
A class is a named set of facet references plus nominal properties. user is described + person + securityPrincipal, plus a dnPrefix of CN supplying its DN component type.
Classes carry zero attributes directly — a class needing unique attributes gets its own facet. A class is nominal: an object is a user, queryably and permanently, since class is immutable at birth, but its attribute surface comes entirely from facets.
An object's surface #
Which attributes an object may hold — its surface — is the union of three facet sets:
- its class's facets;
- facets grafted onto its class by class extensions, covered in modifying the schema;
- its own attached facets — facets added to this one instance.
Setting an attribute outside the surface is refused. The console renders the entire surface, set and unset attributes alike, grouped by the facet that provides them, so the surface is always visible rather than discovered by trial and error.
Attached facets are ordinary data: the engine-level attachedFacets attribute holds facet references on the object itself. Attaching a facet immediately widens the surface. Detaching is refused while any attribute only that facet provides still holds values — clear the values first, explicitly. Nothing is ever silently destroyed.
Values and per-value metadata #
Attributes are natively multi-valued: an attribute holds an ordered set of values, unique under the attribute's matching rule.
Each value can carry metadata: a generic map of meta-attribute name to values, with nothing hardcoded. Which meta keys are legal on which attribute is itself schema data — an attribute's legalMeta lists other attribute definitions, so meta values are typed and validated by the same machinery as everything else.
Two invariants hold:
Meta never forks identity. The value itself is the set key. member cannot contain the same GUID twice with different metadata; changing a value's meta is an update to that value, not a new value.
Meta is validated, not yet interpreted. The canonical example — a group membership with an expiry — is expressible today and fully validated: member permits the expiry meta key, a timestamp. Engine semantics for expiry, filtering expired values at read, are not yet active.
Self-hosting #
The schema is not a config file. It is objects in the directory, flat under /Configuration/Schema, one per attribute, facet, and class, browsable and inspectable through the same console and API as everything else. CN=surname,OU=Schema,OU=Configuration is a real DN.
The recursion grounds out cleanly: attributeDefinition, facetDefinition, and classDefinition are themselves classes, composed of facets like attributeSchema — and classDefinition is an instance of itself. The engine works from a cache compiled off these objects, but the objects are the truth.
The complete contents of the base schema — 26 attributes, 11 facets, 11 classes — are tabulated in the base schema reference.
Modifying the schema
Universal Directory / Concepts
UD's schema is writable at runtime. The rules are designed so that a directory can be extended for decades without accumulating the classic schema scar tissue: colliding names, orphaned definitions that can never be removed, and third-party leftovers polluting the core forever.
Namespaces #
All new definitions live in a namespace: a schemaNamespace OU created directly under /Configuration/Schema.
The flat set of unprefixed base definitions is reserved forever. Even the local organisation extends the schema through its own namespace — /Configuration/Schema/Homelab/, /Configuration/Schema/Acme/. Namespaces hold definitions only; they cannot nest.
Namespace names may be dotted, and reverse-DNS (com.acme.tools) is the convention for anything redistributable, which makes vendor collisions structurally unlikely rather than merely hoped against. Short names remain fine for org-internal extensions. gip and ud are permanently reserved, being GIP's service namespaces.
A namespaced definition's wire name is Namespace-name. A facet badging in the Acme namespace is addressed everywhere — API, console, attribute lookups — as Acme-badging. The hyphen is structural: it is the namespace separator, which is exactly why native definition names may not contain one. See naming rules. Collisions between namespaces are therefore structurally impossible, not conventionally unlikely.
Definitions are born whole, then frozen #
A definition is created complete, in one atomic call — object and attributes together. The API's create accepts an attributes map; the console's create modal generates a field per attribute of the chosen class. The engine fills in lifecycle: active and system: false itself.
Every schema write then passes a trial recompile of the whole schema. An attribute definition missing its syntax, an extension whose target is not a class, a dangling facet reference — any incoherence rejects the write atomically, with the compiler's message, as if it never happened.
Once created, a definition is immutable and immovable: no attribute edits, no renames, no moves. Add and delete are the only verbs. To fix a definition, delete it and recreate it.
The recreated definition is a new definition with a new GUID, so anything that referenced the old one dangles visibly rather than silently rebinding. This is the no-resurrection guarantee: a deleted definition's name can be reused, but its identity can never be impersonated.
The unused-delete rule #
A definition can be deleted only when provably unused.
| Kind | Blocked while… |
|---|---|
| attribute | any object holds values for it, any value uses it as a meta key, any attribute lists it in legalMeta, or any facet references it |
| facet | any class composes it, any extension grafts it, or any object has it attached |
| class | any instance exists — including restorable soft-deleted ones; shredded skeletons do not count — or any extension targets it |
| class extension | any instance of the target class holds values in attributes the extension exclusively provides |
Refusals enumerate their blockers by name: attribute still has values on: /Sales/Jack, facet is used by class Acme-badgeHolder. Teardown is a matter of reading the error.
Two shapes of teardown work:
- Dependency order — clear values, remove the extension, then the facet, then the attribute, then the namespace.
- Wholesale — deleting a namespace takes its entire contents down in one atomic subtree delete, and usage between the dying definitions themselves does not count. Only usage from outside the namespace blocks.
Definition deletion is the same soft delete as everywhere else. A deleted definition sits restorably in the morgue — restoring it recompiles the schema and revives its wire name, refused if the name has been retaken — then hardens to a skeleton when the restore window closes. Values stored under a definition die at its shred: data whose meaning can never return is not kept.
Class extensions #
A class extension grafts facets onto a class its author does not own. It is the mechanism for "our app needs a tracking number on every user" without touching the user class.
An extension is a definition of class classExtension, living in the extender's namespace, declaring a targetClass and the facets to add. From the moment it exists, every instance of the target class carries those facets in its surface. Delete the extension and the surface shrinks back — refused, per the table above, while instances still hold values only the extension legitimises.
The target class object is never modified. The extension is entirely the extender's property, living in their namespace, removable with their namespace.
Worked example: badges for users #
The full flow, as API calls. The console can do all of this through the create modal.
First the namespace:
POST /api/objects/{schema-ou}/children
{ "name": "Acme", "class": "schemaNamespace" }
An attribute, born whole:
POST /api/objects/{acme}/children
{ "name": "badgeColor", "class": "attributeDefinition", "attributes": {
"syntax": [{ "value": "string" }],
"singleValued": [{ "value": "true" }],
"matching": [{ "value": "caseless" }]
} }
A facet carrying it, then an extension grafting that facet onto user. Reference values are GUIDs; the console lets you type /paths instead:
POST /api/objects/{acme}/children
{ "name": "badging", "class": "facetDefinition", "attributes": {
"attributes": [{ "value": "{badgeColor-guid}" }]
} }
POST /api/objects/{acme}/children
{ "name": "userBadging", "class": "classExtension", "attributes": {
"targetClass": [{ "value": "{user-class-guid}" }],
"facets": [{ "value": "{badging-guid}" }]
} }
Every user in the directory now reports Acme-badging among its facets, and accepts:
PUT /api/objects/{some-user}/attributes/Acme-badgeColor
{ "values": [{ "value": "red" }] }
Replication
Universal Directory / Concepts
UD is multi-master and convergent: every node accepts writes, nodes exchange state by pulling from each other, and any two nodes that have seen the same writes hold the same directory — regardless of the order the writes arrived. There is no primary, no write lock, and no repair coordinator.
Today the engine replicates between in-process nodes. The test fleet includes a seeded chaos simulation that asserts byte-equal convergence under partitions, clock skew, power loss, and hostile restores, with every node running the real persistence path. A network transport for udd-to-udd sync is a later slice.
Everything on this page is engine behaviour you can observe through the debug console.
Nodes, invocations, and coordinates #
Every node has a permanent node id and an invocation id naming one incarnation of its database. The node keeps a monotonic local USN, an update sequence number, and the pair (invocationId, usn) is the globally meaningful coordinate of a write.
A node that restores from backup re-mints its invocation id and lets the counter run on. The new id keeps old coordinates truthful, which is what makes restore survivable at all.
Atoms and merge #
Replication moves merge atoms: existence, placement, each attribute value, and the shred latch. Each carries a stamp.
Merging is per-atom, last-writer-wins by the stamp's total order — version, then timestamp, then origin. It is deterministic, so every node picks the same winner.
Three disciplines keep the merge honest:
- An apply that changes nothing — identical value and stamp — writes nothing, so duplicates die on arrival instead of echoing around the mesh.
- An equal value under a stronger stamp adopts the stamp, because future tiebreaks must agree everywhere.
- Replicated applies are never re-validated. A write that was legal at its origin always applies, even if this node's schema has not caught up. Conflicts are repaired, never refused.
The one refusal is the anomaly case: the same GUID arriving with a different class or system flag is two histories wearing one identity. The record is refused, the journal records it, and the sync cursor stalls so the refusal repeats until an operator intervenes. The engine never guesses.
The pull protocol #
A node syncs by pulling: give me everything above my watermark.
Per partner, keyed by invocation id, it remembers a high-watermark in the partner's local USN space. Alongside travels its up-to-dateness vector — per origin invocation, the highest coordinate it holds — which the sender uses for dampening: atoms the puller already covers are stripped from the stream, so data learned via a third node is never re-sent.
Answers come in chunks. A chunk with nothing to say still advances the watermark. Vectors only grow at a cleanly completed stream, never mid-flight, where a gap could be silently claimed.
A brand-new node joins by pulling. A zero watermark and an empty vector are the entire initial sync: the root is discovered from the records themselves, the schema compiles as definitions arrive — incomplete ones wait in quarantine — and objects whose class has not landed yet are ghosts, visible in the debugger, refusing writes, dissolving on convergence.
The pull handshake compares root GUIDs first. Different root, different domain, sync refused.
Conflict repair is derived #
Writes that were each legal at their origin can collide when merged. UD repairs these as a view, never as a write: colliding sibling names keep the earliest claim and CNF-display the later one, while orphans and broken move-cycles derive into /LostAndFound. Repairs dissolve the moment an ordinary write removes the violation.
Same atoms, same repair, on every node, with zero repair traffic.
Deletion that replicates #
The three-stage lifecycle is built for merge.
Tombstones propagate as stamped existence transitions. A shred additionally sets a latch — a set-once atom that can never lose a race — so no concurrently written restore or edit can resurrect destroyed data anywhere. Aging, meaning the restore window and the purge horizon, is computed from stamps and the replicated config knobs, never raced via writes.
A node whose watermark predates knowledge its partner has already purged is refused with full resync required: wipe, re-mint, pull from zero, rather than silently missing deletes.
Rollback self-defence #
A node restored from a snapshot without re-minting — the classic VM-revert disaster — is detected from protocol evidence: a partner claiming this node's own stream beyond its own counter.
That claim takes either of two forms, and UD watches for both: an up-to-dateness vector entry, or a watermark. The bookmark survives even when the only transfer was an abandoned stream, where vectors never absorb. Evidence flows both ways, since requests carry the puller's watermark and chunks carry the sender's bookmark of the puller's stream back.
On detection the node latches. It re-mints its invocation, re-certifies its holdings into the new stream so nothing it wrote gets swallowed by dampening — with persistence, publishing the salvage as a live-store snapshot — suspends originating writes, and alarms. One clean pull then heals it.
Detection is strongest at first contact, and first contact is why the protocol has probes: a zero-record hello carrying only status. Applying chunks consumes local USNs, and every consumed USN burns a unit of evidence margin, so a restored node probes every reachable peer before applying or originating anything.
Watching it happen #
The console's header shows the node's identity, counter, and vectors. The journal tab shows every merge that discarded concurrent input, every refusal, every derived repair, and every rollback event.
Multi-master's classic sin is that lost updates are silent. UD's are not.
Persistence
Universal Directory / Concepts
UD is RAM-resident by design. Every read and every merge runs against memory, and the disk exists for exactly one reason: so a node's state survives restarts.
This is the posture the directory world already runs in when healthy, made explicit. Active Directory's sizing guidance is RAM greater than or equal to the whole database; OpenLDAP's LMDB is a memory-mapped file; etcd caps its store at what mmap keeps resident. Scale-out comes from replication partitioning the tree across DCs, never from paging within one.
Generations: snapshot plus write-ahead log #
On disk lives a chain of generations. snap-N is the complete sorted state at the start of wal-N, and the WAL holds every change since as state deltas — the post-state of each touched atom, never logical operations, so replaying an old log can never depend on the code that wrote it.
One originating operation is one CRC-checksummed frame, and a subtree delete travels whole. One applied replication chunk is one frame. Every file opens with a self-checksummed, format-versioned header.
The encodings are hand-written, not derived. The disk format is a compatibility surface, and no struct rename may silently change bytes.
Booting is blind replay #
Load the newest valid snapshot, apply every later WAL frame with no merge logic and no clock reads, then recompute everything derived — the tree index, back-references, the compiled schema, structural repairs — exactly as a replication apply would.
A torn tail on the newest WAL is crash residue: nothing beyond it was ever acknowledged, and it truncates exactly. Damage anywhere else is corruption, and udd refuses to start rather than guess. That includes a torn sealed generation, valid frames after damage, and frames after a generation's closing seal.
The durability contract #
Nothing becomes visible to the outside before it is durable.
- A mutating API call returns only after its frame is fsynced. Concurrent writers share flushes through group commit, and the fsync never runs under the store lock.
- Protocol speech is durable-gated. Served records, chunk bookmarks, and advertised up-to-dateness claims never reference state a crash could erase. The deadly edge is a partner that dampens what you claim and can never resend it after your crash.
- A failed fsync is fatal, never retried. The daemon stops rather than acknowledge state the disk does not hold, and restart recovers to the last durable point.
Local reads may see RAM a few milliseconds ahead of the disk. No replication evidence ever does.
Background compaction #
Snapshots are produced by rebuilding from the sealed files — the recovery code path run ahead of time on a background thread — never by serialising the live store.
Rotation seals the active WAL, and the seal is durable before a successor exists. The rebuild replays the sealed chain, publishes the next snapshot atomically (write-temp, fsync, rename, fsync the directory), and only then deletes the generations it supersedes.
Recovery is therefore production-exercised: every compaction is a rehearsal of boot. Any mutation that forgot to write its delta shows up as a rebuild that disagrees with the live store, an invariant the test suite asserts continuously.
The one exception is the rollback-salvage snapshot, which serialises the live store because the salvage exists only in RAM.
Erasure reaches the disk #
Shredding destroys a payload in memory, but its bytes linger in older WAL frames and snapshots until compaction deletes those files.
That gap is managed as an explicit erasure debt. Every destructive transition starts a clock — a shred, a GC hardening, or an applied replicated shred, since the obligation follows the data rather than the command — and UD guarantees the destroyed payload has left the node's files within two minutes of the acknowledgment. In practice compaction kicks immediately; the window exists so a burst of shreds coalesces into one rebuild.
The debt is never written down. At boot it is re-derived from the replayed frames themselves, due immediately, so a node that dies mid-obligation finishes it promptly on restart.
The claim is deliberately scoped to the filesystem. What remains in unallocated blocks or an SSD's wear-levelled cells is the platform's concern, as it is for every directory product.
Crashes, restores, and the sim #
The persistence layer is torture-tested through a deterministic in-memory filesystem — torn writes at chosen bytes, failed fsyncs, power-loss simulation — and by the same chaos simulation that proved the merge.
Every simulated node runs on a faux disk through the real recovery path, crashes mid-anything, compacts under load, and is restored from forked disk images, the honest VM-revert. The fleet is still required to converge byte-for-byte at clear skies. A thin suite repeats the full lifecycle against a real filesystem.
The debug console
Universal Directory / The console
The debug console is the browser UI served at udd's root URL.
It is deliberately a debug console. It shows everything the engine knows — GUIDs, system flags, unset attributes, per-value metadata, dangling references, replication stamps, durability marks — and every mutation the engine supports is reachable from it. Nothing in UD is console-only: the console is a client of the same JSON API you can drive with curl.
The header shows the node's replication identity — node id, invocation id, current local USN — and its durability state: the durable USN beside the RAM one, with a ⚠ marking a RAM-ahead moment, the active generation, and the WAL's size. Alarms surface here too, in red: a dead WAL, rollback quarantine, an overdue erasure debt. Hovering carries the full GUIDs and the boot report.
The tree #
The left pane is the directory tree, loaded lazily. Expand a node with its ▸ toggle, click a row to select it. Each row shows the object's effective name and a badge with its class. The schema is right there in the tree — browse /Configuration/Schema like any other subtree.
Objects under a derived repair carry a ⚠ marker; hover for which repair it is — a CNF display name, an orphan relocated into /LostAndFound, a broken move cycle. Ghosts, objects whose class definition has not reached this node, get a dashed ghost badge. The details pane spells the repair out beside the stored facts: name conflict — displays as "sales CNF:…" (stored: sales).
Moving objects is drag-and-drop: drag a row onto the object that should become its new parent. The console deliberately does not second-guess the server. An illegal move — a cycle, a name conflict at the destination, a protected object — is refused by the engine and surfaces in the error banner at the top.
The context menu #
Right-clicking a tree row opens the object's operations menu.
create child… opens a modal with a name field, a class dropdown, and a generated field for every attribute the chosen class's surface allows. Multi-valued attributes get one-value-per-line textareas. Reference attributes accept a native path (/Configuration/Schema/user) or a raw GUID. When creating a facet definition, appending * to a line of attributes marks that reference required. This is also how schema definitions are authored, since definitions must be created complete.
rename… renames in place; identity and references are untouched.
facets… shows checkboxes for every facet. The class's own facets and extension-grafted facets are shown locked, marked (class) and (extension); the rest toggle instance attachment. Detaching a facet whose attributes still hold values is refused by the engine.
delete soft-deletes the object and its whole subtree into the morgue, with confirmation.
shred is security erasure of the subtree, with a much scarier confirmation: payload destroyed, references scrubbed, no way back.
System objects offer only create child, and the engine refuses even that inside the schema subtree except for namespaces and definitions.
The details pane #
Selecting an object shows its identity card: GUID, class with a (system) marker where applicable, effective facets, DN, path, parent, plus the existence and placement stamps and the node-private usnChanged index.
Below it is the attribute table, which renders the object's entire surface, not just what is set:
- Attributes are grouped by the facet that provides them, in surface order; required attributes carry
*. - Unset attributes show as
(unset)— the surface is the form. - Each attribute has an inline
✎editor: one value per line, empty saves as deletion. Metadata on values you do not touch is preserved. - Reference values render as
◈ /resolved/pathwith the raw GUID on hover. A reference whose target no longer exists shows in red as◈ (dangling …). - Per-value metadata renders as chips beside the value, such as
expiry = 2027-06-01T00:00:00Z. - Every value carries its atom stamp in dim text —
v2 · time · origin…, with the transfer certificate and local USN on hover. Removed values linger as struck-through absence atoms stamped with their removal: replication residue, honestly shown. - Engine-level data (
attachedFacets) appears under an(engine)group, read-only, managed through the facets modal. Anything set outside the current surface still renders under(outside surface), because stored data is never hidden. - A referenced by section lists every holder pointing at this object, from the back-reference index, clickable.
Inspecting a tombstone via the morgue shows a stage pill — stored and effective stage — with an inline restore button while the window is open.
The morgue #
The morgue tab lists every tombstone the node holds: name (or a GUID stub for shredded skeletons), class, stage, the transition time, and the last known path. Stage shows deleted → shredded when apply-time aging has outrun physical GC.
Fleshy soft-deletes offer restore and shred; everything offers inspect. The header's gc button runs local garbage collection on demand, hardening tombstones past the restore window and purging those past maxTimeApart. The node-identity tooltip carries the current purge horizon.
The schema view #
The schema tab switches from the tree to a dedicated schema-management page — a faster surface over the same API for authoring and tearing down definitions during testing:
- Kind-grouped tables of every definition — namespaces with definition counts, attributes, facets, classes, extensions — with base definitions marked
(base)and locked. - Purpose-built creation forms per kind: selects for namespace, syntax, cardinality, and matching; checkbox pickers for
legalMeta, facet attribute lists with a per-item required toggle, class facet composition, and extension targets. No GUIDs or paths to type. - Delete buttons on user definitions, riding the engine's unused-delete rules. A refused deletion surfaces its reason, with the blockers named, in the error banner.
- A quarantined table whenever definitions are excluded from the effective schema because their dependencies have not replicated in yet, each with its reason.
Everything the schema view does can still be done manually through the tree. Definitions remain ordinary objects under /Configuration/Schema.
The storage view #
The storage tab is the persistence layer's honest guts: how the store was booted (recovered from which snapshot, how many frames replayed, torn bytes truncated), the durable-versus-appended byte and USN marks, the generation files table with every snapshot and WAL, its size, and the active one marked, the outstanding erasure debt with its deadline, and the background rebuild's status and last outcome.
Two debug buttons drive compaction by hand: rotate now seals the active WAL, opens the next generation, and kicks a rebuild; rebuild snapshot kicks a rebuild alone. In normal life compaction runs itself, on size and on erasure obligations.
The fabric view #
The fabric tab is the GIP fabric drawn as a picture, live, auto-refreshing while visible.
At the top is this machine's identity strip: object GUID, key fingerprint, UDP bind, uptime, and a peers-connected badge.
Below it, one card per peer machine, each showing the link itself: a wire between "you" and the peer that flows with animation while the connection is up, its direction labelled our dial or their dial — the one-connection tiebreak's outcome made visible. Each card also carries the state badge (connected green, via inbound teal, dialing amber, backoff red with a live retry countdown and the last error), the address in use, RTT with a sparkline of recent samples, wire bytes both ways, MTU, congestion window, connection uptime, and a redial button. Live channels riding the connection appear as chips on the card, showing service name, direction, age, and bytes. Inbound connections that no channel has authenticated on yet appear as dashed ghost cards marked unverified.
Below the mesh sit two live logs. Recent channels lists every channel that ran — including gip/ident's split-second conversations and failed opens — with service, direction, peer, duration, bytes, and a coloured outcome. The fabric event stream narrates every discovery, dial, ident, tiebreak (superseded, adopted-inbound), loss, and retry. The header's fabric n/m counter warns with ⚠ whenever a desired peer is down.
The msgs view #
The msgs tab is machine-to-machine talk over the fabric — the first real GIP service, ud/msgs, and deliberately talk, not email: online-only, no queueing, no storage, just a RAM ring of the last 200 records, gone on restart.
The compose bar holds a peer dropdown built from the fabric's targets, showing name and live link state, a text field, and send, which blocks until the message's fate is known. Below it is the chat log: received rows on the left, sent rows on the right, each sent row carrying its outcome badge — delivered green with the round-trip time, unconfirmed amber, failed red — with the honest detail on hover.
The three outcomes are the recipient-ACK protocol told straight. Delivered means the peer recorded the message before acking. Failed means provably not delivered. Unconfirmed is the Two-Generals residue: sent, then the channel died or timed out before the ack, reported as exactly that and never rounded to either certainty.
Every message is one short-lived channel, so sends also flash through the fabric tab's channel chips and recent-channels log.
The journal #
The journal tab lists the node's replication observability ring, because multi-master honesty means lost updates are never silent. It records every merge that discarded concurrent input, every refused record or pull (class mismatches, cross-domain attempts, over-horizon watermarks), every repair the derived view produced while merging, and every rollback detection event. Kinds are colour-coded; the newest entries are last.
Errors and refresh #
Engine refusals appear in the red banner with the API's error code and message — name-in-use: an object with that name already exists here. The console never pre-filters what the server would allow.
The refresh button reloads the tree, re-expands to your selection, and refetches the schema, picking up definitions created since the page loaded.
The JSON API
Universal Directory / Reference
Everything udd can do is exposed as JSON under /api. The debug console is a client of this API and nothing more.
Requests and responses are application/json. There is no authentication: anyone who can reach the port has full control of the directory.
Object payload #
Endpoints that return an object use one shape:
refPath appears only on values of reference-syntax attributes: the target's current native path, or null if the reference dangles.
Every merge atom — existence, placement, each attribute value — carries a stamp, the replication record that decides merges:
absent lists absence atoms: values (or, for a single-valued attribute, the whole attribute — key: null) that were removed and persist as stamped residue so the removal can win future merges. Rewriting an attribute with an unchanged value does not restamp it; multi-valued value sets are unordered and render in canonical (set-key) order.
Objects under a derived repair additionally carry conflicted / orphaned / cycleBroken flags, plus effectiveName (the CNF display name) and effectiveParent where they differ from the stored facts; dn and path always render the effective view. Child summaries carry name (effective), ghost, and the three repair flags.
Endpoints #
| Method & path | Purpose |
|---|---|
GET /api/node | This node's replication identity and sync state: {nodeId, invocationId, usn, hasRoot, purgeHorizon, morgueSize, rollbackLatched, erasureDebt, storage, boot, machine, utd, watermarks} — storage carries the durability marks (generation, durable vs appended bytes and USNs, dead flag), boot the boot report, machine this machine's identity ({object, machineKey} — the machine object's GUID and public key). |
GET /api/journal | The node's observability journal (newest last, capped, survives restarts): {at, kind, detail} — merges that discarded concurrent input, refused records and pulls, derived repairs, rollback events. |
GET /api/storage | The storage view: generation files with sizes, durability marks, rotation threshold, erasure debt, background-rebuild status and last outcome. |
GET /api/fabric | The fabric view: this machine's identity and GIP bind, every discovered peer with its link state (idle/dialing/connected/viaInbound/backoff), live QUIC stats (RTT, congestion window, lost packets, MTU, wire bytes), inbound connections (each carrying machine {guid, name} once identified via gip/ident, null while unverified), channels (live channels: connection, peer, service, direction, age, bytes) and recentChannels (a ring of finished channels and failed opens with duration, bytes, and outcome), the supervisor's config, and a ring of timestamped fabric events. |
POST /api/fabric/redial/{guid} | Debug: clear a peer's backoff and dial it now. |
GET /api/msgs | The machine-talk log, oldest first (RAM only — capped at 200 records, gone on restart): each record is {atMs, direction (in/out), peerGuid, peerName, text, outcome}, where sent records carry outcome delivered (+rttMs), failed (+detail), or unconfirmed (+detail), and received records carry received. {enabled: false} on a node without a fabric. |
POST /api/msgs/send | Send one ud/msgs message and block for its outcome (bounded: 5 s channel-open + 10 s ack). Body: {"to", "text"} — to is a machine GUID or a caseless machine name resolved against the fabric's targets; text is 1–16384 bytes. HTTP errors are for request problems only (422 unknown-recipient, 422 bad-text, 503 no-fabric); a send that ran returns 200 carrying the finished record, including failed and unconfirmed outcomes — those are data, not transport errors. |
POST /api/storage/rotate | Debug: seal the active WAL, open the next generation, kick a background rebuild. |
POST /api/storage/rebuild | Debug: kick a background snapshot rebuild (no rotation). |
GET /api/root | The root object. |
GET /api/morgue | Every tombstone: {id, name, class, stage, effectiveStage, at, lastPath}. |
POST /api/gc | Run local GC now (harden aged tombstones, purge expired ones); returns the new horizon. |
POST /api/objects/{id}/restore | Restore a soft-deleted object (top-down; refuses conflicts and elapsed windows). |
POST /api/objects/{id}/shred | Security-erase (subtree-wide from alive, or escalate an existing soft delete). |
GET /api/objects/{id} | One object by GUID. |
GET /api/objects/{id}/children | Child summaries, in caseless name order: {id, name, class, system, hasChildren}. |
POST /api/objects/{id}/children | Create a child. Body: {"name", "class", "attributes"?} — attributes maps attribute name → value list and is applied atomically with the create (definitions are born whole). Returns 201 + the object. |
PATCH /api/objects/{id} | Rename and/or move (ModifyDN shape). Body: {"name"?, "parent"?}, parent by GUID. |
PUT /api/objects/{id}/attributes/{name} | Replace an attribute's values wholesale. Body: {"values": [{"value", "meta"?}]}. An empty values deletes the attribute. Returns the updated object. |
DELETE /api/objects/{id} | Soft-delete the object and its whole subtree (restorable tombstones). Returns 204. |
GET /api/path and GET /api/path/{path} | Resolve a native untyped path (caseless) to its object: /api/path/Sales/Jack. Bare /api/path is the root. |
GET /api/schema | The compiled schema summary (below). |
Attribute names in URLs and bodies are caseless and accept wire names (Acme-badgeColor); the stored spelling is always the schema's canonical one.
Every mutating endpoint acknowledges only after the write is durable — a 200/201/204 means the disk holds it. Attribute values are capped at 1 MiB each (large blobs are a future side-store's job, not an atom's).
The schema summary #
GET /api/schema returns the compiled schema, sorted by name:
quarantined lists definitions excluded from the effective schema — replication can deliver a facet before its attributes. Compilation is total: it never fails, it quarantines what cannot join yet (with the reason) and lets it join automatically when its dependencies arrive. References to tombstoned definitions compile out instead (the facet keeps working minus its dead member). Originating schema writes are still strictly gated: a local write that would grow the quarantine is refused with schema-violation.
Errors #
Failures return {"error": "<code>", "message": "<human text>"} with:
| Code | Status | Meaning |
|---|---|---|
bad-object-id | 400 | Path segment is not a GUID. |
no-such-object | 404 | Object (or path/reference target) does not exist. |
no-root | 503 | A joined store that has not yet received its first sync has no root. |
name-in-use | 409 | Sibling name taken (caseless, type-blind). |
invalid-name | 422 | Name fails validity rules (or definition-name grammar). |
unknown-class | 422 | No class by that name. |
class-not-instantiable | 422 | Class is marked non-instantiable (domainRoot, configuration). |
unknown-attribute | 422 | No attribute definition by that name. |
attribute-not-permitted | 422 | Attribute is outside the object's surface. |
single-valued | 422 | Multiple values for a single-valued attribute (or meta). |
syntax-violation | 422 | Value fails its syntax (bad integer/boolean/timestamp, non-GUID or dangling reference). |
illegal-meta-key | 422 | Meta key unknown or not in the attribute's legalMeta. |
duplicate-value | 409 | Value set contains duplicates under the matching rule. |
unknown-facet | 422 | attachedFacets value is not a facet definition. |
facet-has-values | 409 | Facet detach / extension delete refused while exclusively-provided attributes hold values. |
already-deleted | 409 | Mutation aimed at a tombstone (restore first, or leave the dead be). |
ghost-object | 409 | The object's class isn't in this node's schema yet; originating writes refuse until sync converges. |
rollback-quarantine | 409 | The node detected it is running rolled-back (restored) state: its invocation was re-minted and originating writes are suspended until a completed pull heals it. |
not-restorable | 409 | Restore refused: wrong stage, elapsed window, or parent not alive. |
root-is-protected | 409 | The root cannot be deleted, renamed, or moved. |
system-object-protected | 409 | System objects refuse moves and deletion (their attributes are writable). |
invalid-schema-placement | 422 | Wrong class/parent combination in or around the schema subtree. |
schema-definition-immutable | 409 | Definitions are add/delete only. |
definition-in-use | 409 | Unused-delete rule refused the deletion; the message enumerates the blockers by name. |
schema-violation | 422 | Trial recompile failed; the write was rolled back (message carries the compiler's reason). |
would-create-cycle | 409 | Move would place an object under itself or a descendant. |
The base schema
Universal Directory / Reference
The base schema is the set of system definitions every directory is born with: 26 attributes, 11 facets, and 11 classes — 48 objects in all, flat under /Configuration/Schema and immutable forever.
All new definitions live in namespaces; the unprefixed name space tabulated below is reserved. The base definitions are minted at domain creation by the first node, as its ordinary originating writes. There are no well-known GUIDs — a joining node receives everything by sync.
Attributes #
| Attribute | Syntax | Values | Matching | Aliases | legalMeta |
|---|---|---|---|---|---|
description | string | multi | caseless | ||
displayName | string | single | caseless | ||
givenName | string | single | caseless | ||
surname | string | single | caseless | sn | |
email | string | multi | caseless | mail | |
sid | string | single | exact | objectSid | |
member | reference | multi | exact | expiry | |
expiry | timestamp | single | exact | ||
syntax | string | single | caseless | ||
singleValued | boolean | single | exact | ||
matching | string | single | caseless | ||
legalMeta | reference | multi | exact | ||
aliases | string | multi | caseless | ||
lifecycle | string | single | caseless | ||
system | boolean | single | exact | ||
attributes | reference | multi | exact | required | |
required | boolean | single | exact | ||
facets | reference | multi | exact | ||
attachedFacets | reference | multi | exact | ||
dnPrefix | string | single | caseless | ||
targetClass | reference | single | exact | ||
instantiable | boolean | single | exact | ||
maxTimeApart | integer | single | exact | ||
timeUntilShred | integer | single | exact | ||
networkAddress | endpoint | multi | caseless | ||
machineKey | publicKey | multi | exact |
Notes: description is multi-valued (following the LDAP RFCs where they and Active Directory disagree). attachedFacets is engine-level — legal on every object regardless of surface, and the vehicle for instance facet attachment. Aliases are recorded on definitions but not yet used for lookup. expiry and required exist to serve as meta keys (on member and attributes respectively). instantiable (unset = true) marks classes users may instantiate. maxTimeApart and timeUntilShred are the domain's lifecycle knobs, in seconds, held on the /Configuration object itself: timeUntilShred is the restore window (default 30 days — past it a soft-deleted record hardens to a skeleton), maxTimeApart the purge horizon and future partition contract (default 180 days — past it tombstones purge entirely). networkAddress holds a machine's network endpoints — multi-valued because a DNS name, an IPv4, and an IPv6 address are legitimately concurrent addresses of one machine.
The endpoint syntax #
networkAddress uses the endpoint value syntax: host[:port], where the host is a DNS name, an IPv4 literal, or a bracketed IPv6 literal ([2001:db8::1]:6000), and an omitted port designates the well-known GIP port, 5390 (GIP — the General Interlink Protocol — is UD's intra-domain machine-to-machine protocol; the syntax ships today, the protocol does not yet). The grammar is deliberately strict, because a loose syntax can never be tightened once values persist:
- no schemes (
quic://…) and no whitespace anywhere; - DNS names are ASCII (IDN arrives as punycode), dot-separated labels of letters, digits, and interior hyphens, ≤ 63 bytes per label and ≤ 253 total, with no trailing dot;
- anything whose last label is all digits must be a valid IPv4 literal —
999.1.1.1is refused rather than silently treated as a DNS name — and leading zeros in octets (historical octal) are refused; - IPv6 must be bracketed, zone ids (
%eth0) are refused; - an explicit port is 1–65535 with no leading zeros.
Values compare caselessly as opaque strings: DC1.corp and dc1.CORP are the same set member, but dc1.corp and dc1.corp:5390 are two distinct values even though they designate the same place — set matching is literal; interpreting endpoints is the dialer's job.
The publicKey syntax #
machineKey uses the publicKey value syntax: <algorithm>:<hex>, where the only algorithm today is x25519 and the payload is exactly 64 lowercase hex characters (32 bytes) — one canonical spelling per key, compared exactly. This is the machine's GIP identity key: udd new mints the keypair at enrollment, keeps the private half in the data directory's machine-key file, and records the public half here, replicated like any other fact. The attribute is multi-valued so a future key rotation can hold old and new keys through an overlap window. The algorithm prefix is deliberate agility: a future signing key gets a new prefix, not a new syntax.
Facets #
| Facet | Attributes (* = required) |
|---|---|
described | description, displayName |
person | givenName, surname, email |
securityPrincipal | sid * |
membership | member |
schemaDefinition | aliases, lifecycle *, system * |
attributeSchema | syntax *, singleValued *, matching *, legalMeta |
facetSchema | attributes |
classSchema | facets, dnPrefix, instantiable |
extensionSchema | targetClass *, facets * |
replicationConfig | maxTimeApart, timeUntilShred |
machine | networkAddress, machineKey |
Required-ness is enforced today only where compilation demands it (schema definitions must be born complete); enforcement for ordinary objects is pending.
machine is the surface for anything reachable on the network; both attributes are deliberately optional on it — a pre-staged machine that has not yet enrolled has neither an address nor a key. The computer class composes it, and udd new creates one computer object for the machine it runs on. (The class is not called machine because definitions are siblings under /Configuration/Schema, where names are unique regardless of kind — the facet holds that name.)
Classes #
| Class | dnPrefix | Instantiable | Facets |
|---|---|---|---|
domainRoot | — (never in a DN) | no | described |
organizationalUnit | OU | yes | described |
user | CN | yes | described, person, securityPrincipal |
group | CN | yes | described, securityPrincipal, membership |
attributeDefinition | CN | yes | described, schemaDefinition, attributeSchema |
facetDefinition | CN | yes | described, schemaDefinition, facetSchema |
classDefinition | CN | yes | described, schemaDefinition, classSchema |
schemaNamespace | OU | yes | described |
classExtension | CN | yes | described, schemaDefinition, extensionSchema |
configuration | CN | no | described, replicationConfig |
computer | CN | yes | described, machine |
The self-hosting anchor: the classDefinition object under /Configuration/Schema is an instance of itself; every other definition's kind resolves from there.
An instantiable class must carry a dnPrefix (compile-enforced); non-instantiable classes are engine-created only. /Configuration is the singleton configuration instance — its class composition is what makes the replication knobs writable there, and future config areas graft on via classExtension. Besides /Configuration/Schema, the base tree also contains /LostAndFound, a system OU born with the domain: the destination replication repairs derive orphans and broken cycles into (the domain root is a replication head from birth). It also contains /Domain Controllers, the default home for machine objects — a convention, not structure: nothing in UD locates domain controllers by path, and a machine object is free to live anywhere.
Naming rules
Universal Directory / Reference
Object names #
An object's bare name must be:
- non-empty and at most 255 bytes,
- free of control characters,
- free of
/(the native path separator), - without leading or trailing whitespace.
Everything else — spaces, Unicode, =, , — is legal; DN composition escapes what needs escaping. These rules are deliberately strict: loosening a validity rule later is safe, tightening one is a data migration.
Caseless comparison (pinned) #
Sibling uniqueness, attribute-name lookup, path resolution, and caseless-matched value sets all compare under Unicode canonical caseless matching (Unicode §3.13): two strings are equal iff
NFD(casefold(NFD(a))) == NFD(casefold(NFD(b)))
So Jack = jack, STRASSE = Straße (ß case-folds to ss), and composed é = decomposed e + ́. Stored spellings are always preserved; the folding governs comparison only. This function is pinned: changing it once data persists would shift uniqueness invariants underneath stored data.
Distinguished names #
DNs compose leaf-first from typed components: CN=Jack,OU=Sales. The component type is the object's class's dnPrefix; the value is the bare name escaped per RFC 4514 — a backslash before any of " + , ; < > \ anywhere, before a leading space or #, and before a trailing space. The root object contributes nothing; its DN is the empty string.
Native paths #
The untyped address form: / followed by bare names joined with /, root-first — /Sales/Jack, resolving case-insensitively. The root's path is /. Because / is banned in names, paths are unambiguous.
Definition names and wire names #
Schema definition names (attributes, facets, classes, extensions) follow a stricter grammar than object names: a letter followed by letters or digits — camelCase by convention, and no hyphens, because the hyphen is the namespace separator. Namespace names additionally allow dots between segments (each segment a letter followed by letters or digits): Acme and reverse-DNS com.acme.tools are both legal, and reverse-DNS is the convention for anything redistributable — it makes accidental collisions between vendors nearly impossible. A namespaced definition's wire name is Namespace-name (Acme-badgeColor, com.acme.tools-badgeColor) — the single hyphen is unambiguous because neither side may contain one. Base definitions are unprefixed, and that unprefixed space is reserved forever.
The namespace names gip and ud (caselessly) are permanently reserved: they are GIP's service namespaces (gip/relay, ud/drs), and service names share the schema's namespace universe.
dnPrefix values follow RFC 4512 keystring grammar (a letter, then letters, digits, or hyphens).
CNF display names #
When replication merges two live siblings whose names fold equal, the earliest placement claim keeps the name and the later one displays as <stored name> CNF:<guid> (laddering (2), (3)… past pathological collisions). This is a derived view, not a rename: the stored name is untouched, the CNF form resolves in paths, and an ordinary rename dissolves it.
The reserved descriptor x #
The name x (caselessly) is permanently reserved and can never be defined as an attribute, facet, class, extension, or namespace name. UD reserves it as its untyped-lookup descriptor so that no schema element can ever collide with that use.