Concepts

Every entry in the directory — an OU, a user, a schema definition — is an object. UD's object model rests on one distinction it never blurs: identity versus address.

Identity: the GUID #

An object's identity is a GUID, assigned at creation and immutable for the object's entire life.

Everything that needs to point at an object stores that GUID, never a name or a path: a group's member values, a facet's attribute list, a class's composition. Renames and moves therefore never break references. Deleting an object leaves any remaining references visibly dangling at the dead GUID, rather than silently rebinding to whatever later takes the name.

Address: derived, never stored #

Internally the store is inode-shaped: a flat table of objects keyed by GUID, where each object records only its parent's GUID and its own bare name (Sales, Jack). The tree is an index over that table.

Both address forms are computed on demand:

• the distinguished name — leaf-first with typed components, CN=Jack,OU=Sales, where each prefix comes from the object's class (dnPrefix) and values are escaped per RFC 4514;
• the native path — root-first and untyped, /Sales/Jack, used by the console, the JSON API's path resolution, and anywhere a human types an address. Path lookup is case-insensitive.

Because addresses are derived, renaming an OU is a single-object change. Every descendant's DN and path change logically, with zero writes, and nothing that references those descendants notices at all.

The tree has a single root object — empty name, class domainRoot — which never appears in DNs and cannot be renamed, moved, or deleted.

Names and sibling uniqueness #

Within one parent, names are unique under two deliberately strict rules.

Caseless, case-preserving. Names compare under Unicode canonical caseless matching, so Jack, jack, and a decomposed Jack are all the same name. The spelling you create is the spelling you see; the folding governs comparison and conflict only. See naming rules for the exact function.

Type-blind. Uniqueness ignores class entirely. An OU named Sales and a group named sales cannot be siblings. One container, one meaning per name.

Name validity is strict too — non-empty, no /, no control characters, no leading or trailing whitespace. Strict rules can be safely loosened later; a loose rule can never be tightened without breaking existing data.

Rename and move #

Rename and reparent are one operation, in the shape of LDAP's ModifyDN: change the name, the parent, or both, atomically.

The engine refuses:

• moves that would create a cycle, placing an object under itself or its own descendant;
• a destination where the caseless, type-blind name is already taken;
• any rename or move of the root, of system objects, or of schema objects — definitions are add and delete only — and any move into the schema subtree.

A case-only rename, jack → Jack, is always legal. It is the same caseless name, so it conflicts with nothing.

Delete: the three-stage lifecycle #

Deletion is soft and subtree-wide. Deleting an object tombstones it and every descendant in place, each with its own stamped existence transition.

A tombstone leaves the effective tree immediately — its name is free at once — but the full record survives, and every tombstoned object is individually restorable. Restoring is top-down: the parent must be alive and the name must still be free. It is an ordinary write, and a restore into a conflict simply refuses.

Beyond Deleted sit two harder states.

Shredded is security erasure, reachable directly from alive or as an escalation of a soft delete. The payload and even the name are destroyed; only a skeleton — GUID, parent GUID, stamps — remains. Every reference to the shredded GUID is physically scrubbed out of every holder, because erasure that leaves the object's GUID lingering in group member lists would fail its own purpose. The destruction reaches the node's files within a guaranteed window. There is no way back.

Purged means the tombstone is gone entirely. This is never manual: purge happens automatically once a tombstone's age passes maxTimeApart.

Aging is computed, not written #

Two domain-wide knobs, stored as attributes on /Configuration with defaults of 30 and 180 days, drive the automatic transitions. Past timeUntilShred a soft-deleted record hardens: the restore window closes and the flesh is scrubbed to a skeleton. Past maxTimeApart the skeleton purges.

Aging is a pure function of the object's existence stamp, the config windows, and the clock. Every node evaluates a tombstone's effective stage the same way, so nodes never race stamps about it. Local garbage collection merely catches physical storage up with what the aging function already answers.

The root and system objects are permanently protected, and schema definitions add their own unused-delete rules. Because references store GUIDs, deleting a referenced object never edits the referrers: their values dangle visibly at the tombstone until the target is shredded, at which point they are scrubbed.

System objects #

Objects created at domain creation carry a system flag: the root, /Configuration, /Configuration/Schema, /LostAndFound, /Domain Controllers, and every base schema definition.

System objects refuse renames, moves, and deletion — they are the fixed points the rest of the directory is built around — but their attributes are writable. The replication knobs live as attributes on /Configuration, and a description on the root is legal. Schema definitions are the exception, fully immutable via the schema-subtree rule.

Structural repairs are derived, never written #

Replication will merge writes that were each legal at their origin but collide when combined. UD repairs these as a view: the stored records stay exactly as written, and every node derives the same effective tree from the same atoms. No repair writes, no repair replication traffic.

Three rules, all favouring the earliest claim — the first legal write stands, because the later write is the one that caused the violation:

• Name conflicts. Two live siblings whose names fold equal: the earliest placement claim keeps the name, and the later one displays as name CNF:<guid>. It remains resolvable and renameable, and its stored name is untouched.
• Orphans. A live object whose stored parent is dead or missing derives into /LostAndFound. Restoring the parent snaps it back automatically.
• Cycles. Concurrent moves that form a loop break at the newest placement: the latest mover derives into /LostAndFound, and the earlier move survives intact.

Admin operations target the effective view — rename the CNF loser, move the orphan out, as ordinary writes — and the repair dissolves the moment the underlying violation is gone.

Objects whose class definition has not arrived yet are ghosts: visible in the debugger, holding their sibling name, but refusing every originating write until the schema converges.

Replication stamps #

Every replicated fact about an object decomposes into merge atoms:

• its existence — creation, with class and the system flag riding along;
• its placement — (parent, name) as one atom, so rename and move contend as a unit;
• each attribute value.

Every atom carries a stamp: a version, an originating time, and an originating invocation id. Together these decide conflicts deterministically — higher version wins, then later timestamp, then origin — and a transfer certificate drives change enumeration. Removing a value leaves a stamped absence atom rather than a hole, so the removal itself can win merges.

For how those atoms travel and merge, read Replication.

UD's schema is compositional. There is no class inheritance anywhere in the model, and never will be. The schema is built from three layers, each referencing the one below by GUID, with flat set-union as the only combining operation.

The three layers #

Attributes are defined globally. One definition per attribute — surname, member, expiry — carrying:

• its syntax: string, integer, boolean, timestamp, reference, endpoint, or publicKey;
• whether it is single- or multi-valued;
• its matching rule, caseless or exact, governing value comparison and set uniqueness;
• the meta-attributes its values may carry (legalMeta, below);
• an aliases slot recording alternative wire names.

Facets reference attributes; they never define them. So two facets that both want description agree by construction, because there is only one description.

A facet is a named set of attribute references — the smallest unit of composition. securityPrincipal carries sid; person carries givenName, surname, email; membership carries member. Each reference in a facet can be flagged required, shown in the console as *. Facets cannot include other facets: composition is flat, so it is order-free and collision-free.

A class is a named set of facet references plus nominal properties. user is described + person + securityPrincipal, plus a dnPrefix of CN supplying its DN component type.

Classes carry zero attributes directly — a class needing unique attributes gets its own facet. A class is nominal: an object is a user, queryably and permanently, since class is immutable at birth, but its attribute surface comes entirely from facets.

An object's surface #

Which attributes an object may hold — its surface — is the union of three facet sets:

1. its class's facets;
2. facets grafted onto its class by class extensions, covered in modifying the schema;
3. its own attached facets — facets added to this one instance.

Setting an attribute outside the surface is refused. The console renders the entire surface, set and unset attributes alike, grouped by the facet that provides them, so the surface is always visible rather than discovered by trial and error.

Attached facets are ordinary data: the engine-level attachedFacets attribute holds facet references on the object itself. Attaching a facet immediately widens the surface. Detaching is refused while any attribute only that facet provides still holds values — clear the values first, explicitly. Nothing is ever silently destroyed.

Values and per-value metadata #

Attributes are natively multi-valued: an attribute holds an ordered set of values, unique under the attribute's matching rule.

Each value can carry metadata: a generic map of meta-attribute name to values, with nothing hardcoded. Which meta keys are legal on which attribute is itself schema data — an attribute's legalMeta lists other attribute definitions, so meta values are typed and validated by the same machinery as everything else.

Two invariants hold:

Meta never forks identity. The value itself is the set key. member cannot contain the same GUID twice with different metadata; changing a value's meta is an update to that value, not a new value.

Meta is validated, not yet interpreted. The canonical example — a group membership with an expiry — is expressible today and fully validated: member permits the expiry meta key, a timestamp. Engine semantics for expiry, filtering expired values at read, are not yet active.

Self-hosting #

The schema is not a config file. It is objects in the directory, flat under /Configuration/Schema, one per attribute, facet, and class, browsable and inspectable through the same console and API as everything else. CN=surname,OU=Schema,OU=Configuration is a real DN.

The recursion grounds out cleanly: attributeDefinition, facetDefinition, and classDefinition are themselves classes, composed of facets like attributeSchema — and classDefinition is an instance of itself. The engine works from a cache compiled off these objects, but the objects are the truth.

The complete contents of the base schema — 26 attributes, 11 facets, 11 classes — are tabulated in the base schema reference.

UD's schema is writable at runtime. The rules are designed so that a directory can be extended for decades without accumulating the classic schema scar tissue: colliding names, orphaned definitions that can never be removed, and third-party leftovers polluting the core forever.

Namespaces #

All new definitions live in a namespace: a schemaNamespace OU created directly under /Configuration/Schema.

The flat set of unprefixed base definitions is reserved forever. Even the local organisation extends the schema through its own namespace — /Configuration/Schema/Homelab/, /Configuration/Schema/Acme/. Namespaces hold definitions only; they cannot nest.

Namespace names may be dotted, and reverse-DNS (com.acme.tools) is the convention for anything redistributable, which makes vendor collisions structurally unlikely rather than merely hoped against. Short names remain fine for org-internal extensions. gip and ud are permanently reserved, being GIP's service namespaces.

A namespaced definition's wire name is Namespace-name. A facet badging in the Acme namespace is addressed everywhere — API, console, attribute lookups — as Acme-badging. The hyphen is structural: it is the namespace separator, which is exactly why native definition names may not contain one. See naming rules. Collisions between namespaces are therefore structurally impossible, not conventionally unlikely.

Definitions are born whole, then frozen #

A definition is created complete, in one atomic call — object and attributes together. The API's create accepts an attributes map; the console's create modal generates a field per attribute of the chosen class. The engine fills in lifecycle: active and system: false itself.

Every schema write then passes a trial recompile of the whole schema. An attribute definition missing its syntax, an extension whose target is not a class, a dangling facet reference — any incoherence rejects the write atomically, with the compiler's message, as if it never happened.

Once created, a definition is immutable and immovable: no attribute edits, no renames, no moves. Add and delete are the only verbs. To fix a definition, delete it and recreate it.

The recreated definition is a new definition with a new GUID, so anything that referenced the old one dangles visibly rather than silently rebinding. This is the no-resurrection guarantee: a deleted definition's name can be reused, but its identity can never be impersonated.

The unused-delete rule #

A definition can be deleted only when provably unused.

Refusals enumerate their blockers by name: attribute still has values on: /Sales/Jack, facet is used by class Acme-badgeHolder. Teardown is a matter of reading the error.

Two shapes of teardown work:

• Dependency order — clear values, remove the extension, then the facet, then the attribute, then the namespace.
• Wholesale — deleting a namespace takes its entire contents down in one atomic subtree delete, and usage between the dying definitions themselves does not count. Only usage from outside the namespace blocks.

Definition deletion is the same soft delete as everywhere else. A deleted definition sits restorably in the morgue — restoring it recompiles the schema and revives its wire name, refused if the name has been retaken — then hardens to a skeleton when the restore window closes. Values stored under a definition die at its shred: data whose meaning can never return is not kept.

Class extensions #

A class extension grafts facets onto a class its author does not own. It is the mechanism for "our app needs a tracking number on every user" without touching the user class.

An extension is a definition of class classExtension, living in the extender's namespace, declaring a targetClass and the facets to add. From the moment it exists, every instance of the target class carries those facets in its surface. Delete the extension and the surface shrinks back — refused, per the table above, while instances still hold values only the extension legitimises.

The target class object is never modified. The extension is entirely the extender's property, living in their namespace, removable with their namespace.

Worked example: badges for users #

The full flow, as API calls. The console can do all of this through the create modal.

First the namespace:

POST /api/objects/{schema-ou}/children
{ "name": "Acme", "class": "schemaNamespace" }

An attribute, born whole:

POST /api/objects/{acme}/children
{ "name": "badgeColor", "class": "attributeDefinition", "attributes": {
    "syntax":       [{ "value": "string" }],
    "singleValued": [{ "value": "true" }],
    "matching":     [{ "value": "caseless" }]
} }

A facet carrying it, then an extension grafting that facet onto user. Reference values are GUIDs; the console lets you type /paths instead:

POST /api/objects/{acme}/children
{ "name": "badging", "class": "facetDefinition", "attributes": {
    "attributes": [{ "value": "{badgeColor-guid}" }]
} }

POST /api/objects/{acme}/children
{ "name": "userBadging", "class": "classExtension", "attributes": {
    "targetClass": [{ "value": "{user-class-guid}" }],
    "facets":      [{ "value": "{badging-guid}" }]
} }

Every user in the directory now reports Acme-badging among its facets, and accepts:

PUT /api/objects/{some-user}/attributes/Acme-badgeColor
{ "values": [{ "value": "red" }] }

UD is multi-master and convergent: every node accepts writes, nodes exchange state by pulling from each other, and any two nodes that have seen the same writes hold the same directory — regardless of the order the writes arrived. There is no primary, no write lock, and no repair coordinator.

Today the engine replicates between in-process nodes. The test fleet includes a seeded chaos simulation that asserts byte-equal convergence under partitions, clock skew, power loss, and hostile restores, with every node running the real persistence path. A network transport for udd-to-udd sync is a later slice.

Everything on this page is engine behaviour you can observe through the debug console.

Nodes, invocations, and coordinates #

Every node has a permanent node id and an invocation id naming one incarnation of its database. The node keeps a monotonic local USN, an update sequence number, and the pair (invocationId, usn) is the globally meaningful coordinate of a write.

A node that restores from backup re-mints its invocation id and lets the counter run on. The new id keeps old coordinates truthful, which is what makes restore survivable at all.

Atoms and merge #

Replication moves merge atoms: existence, placement, each attribute value, and the shred latch. Each carries a stamp.

Merging is per-atom, last-writer-wins by the stamp's total order — version, then timestamp, then origin. It is deterministic, so every node picks the same winner.

Three disciplines keep the merge honest:

• An apply that changes nothing — identical value and stamp — writes nothing, so duplicates die on arrival instead of echoing around the mesh.
• An equal value under a stronger stamp adopts the stamp, because future tiebreaks must agree everywhere.
• Replicated applies are never re-validated. A write that was legal at its origin always applies, even if this node's schema has not caught up. Conflicts are repaired, never refused.

The one refusal is the anomaly case: the same GUID arriving with a different class or system flag is two histories wearing one identity. The record is refused, the journal records it, and the sync cursor stalls so the refusal repeats until an operator intervenes. The engine never guesses.

The pull protocol #

A node syncs by pulling: give me everything above my watermark.

Per partner, keyed by invocation id, it remembers a high-watermark in the partner's local USN space. Alongside travels its up-to-dateness vector — per origin invocation, the highest coordinate it holds — which the sender uses for dampening: atoms the puller already covers are stripped from the stream, so data learned via a third node is never re-sent.

Answers come in chunks. A chunk with nothing to say still advances the watermark. Vectors only grow at a cleanly completed stream, never mid-flight, where a gap could be silently claimed.

A brand-new node joins by pulling. A zero watermark and an empty vector are the entire initial sync: the root is discovered from the records themselves, the schema compiles as definitions arrive — incomplete ones wait in quarantine — and objects whose class has not landed yet are ghosts, visible in the debugger, refusing writes, dissolving on convergence.

The pull handshake compares root GUIDs first. Different root, different domain, sync refused.

Conflict repair is derived #

Writes that were each legal at their origin can collide when merged. UD repairs these as a view, never as a write: colliding sibling names keep the earliest claim and CNF-display the later one, while orphans and broken move-cycles derive into /LostAndFound. Repairs dissolve the moment an ordinary write removes the violation.

Same atoms, same repair, on every node, with zero repair traffic.

Deletion that replicates #

The three-stage lifecycle is built for merge.

Tombstones propagate as stamped existence transitions. A shred additionally sets a latch — a set-once atom that can never lose a race — so no concurrently written restore or edit can resurrect destroyed data anywhere. Aging, meaning the restore window and the purge horizon, is computed from stamps and the replicated config knobs, never raced via writes.

A node whose watermark predates knowledge its partner has already purged is refused with full resync required: wipe, re-mint, pull from zero, rather than silently missing deletes.

Rollback self-defence #

A node restored from a snapshot without re-minting — the classic VM-revert disaster — is detected from protocol evidence: a partner claiming this node's own stream beyond its own counter.

That claim takes either of two forms, and UD watches for both: an up-to-dateness vector entry, or a watermark. The bookmark survives even when the only transfer was an abandoned stream, where vectors never absorb. Evidence flows both ways, since requests carry the puller's watermark and chunks carry the sender's bookmark of the puller's stream back.

On detection the node latches. It re-mints its invocation, re-certifies its holdings into the new stream so nothing it wrote gets swallowed by dampening — with persistence, publishing the salvage as a live-store snapshot — suspends originating writes, and alarms. One clean pull then heals it.

Detection is strongest at first contact, and first contact is why the protocol has probes: a zero-record hello carrying only status. Applying chunks consumes local USNs, and every consumed USN burns a unit of evidence margin, so a restored node probes every reachable peer before applying or originating anything.

Watching it happen #

The console's header shows the node's identity, counter, and vectors. The journal tab shows every merge that discarded concurrent input, every refusal, every derived repair, and every rollback event.

Multi-master's classic sin is that lost updates are silent. UD's are not.

UD is RAM-resident by design. Every read and every merge runs against memory, and the disk exists for exactly one reason: so a node's state survives restarts.

This is the posture the directory world already runs in when healthy, made explicit. Active Directory's sizing guidance is RAM greater than or equal to the whole database; OpenLDAP's LMDB is a memory-mapped file; etcd caps its store at what mmap keeps resident. Scale-out comes from replication partitioning the tree across DCs, never from paging within one.

Generations: snapshot plus write-ahead log #

On disk lives a chain of generations. snap-N is the complete sorted state at the start of wal-N, and the WAL holds every change since as state deltas — the post-state of each touched atom, never logical operations, so replaying an old log can never depend on the code that wrote it.

One originating operation is one CRC-checksummed frame, and a subtree delete travels whole. One applied replication chunk is one frame. Every file opens with a self-checksummed, format-versioned header.

The encodings are hand-written, not derived. The disk format is a compatibility surface, and no struct rename may silently change bytes.

Booting is blind replay #

Load the newest valid snapshot, apply every later WAL frame with no merge logic and no clock reads, then recompute everything derived — the tree index, back-references, the compiled schema, structural repairs — exactly as a replication apply would.

A torn tail on the newest WAL is crash residue: nothing beyond it was ever acknowledged, and it truncates exactly. Damage anywhere else is corruption, and udd refuses to start rather than guess. That includes a torn sealed generation, valid frames after damage, and frames after a generation's closing seal.

The durability contract #

Nothing becomes visible to the outside before it is durable.

• A mutating API call returns only after its frame is fsynced. Concurrent writers share flushes through group commit, and the fsync never runs under the store lock.
• Protocol speech is durable-gated. Served records, chunk bookmarks, and advertised up-to-dateness claims never reference state a crash could erase. The deadly edge is a partner that dampens what you claim and can never resend it after your crash.
• A failed fsync is fatal, never retried. The daemon stops rather than acknowledge state the disk does not hold, and restart recovers to the last durable point.

Local reads may see RAM a few milliseconds ahead of the disk. No replication evidence ever does.

Background compaction #

Snapshots are produced by rebuilding from the sealed files — the recovery code path run ahead of time on a background thread — never by serialising the live store.

Rotation seals the active WAL, and the seal is durable before a successor exists. The rebuild replays the sealed chain, publishes the next snapshot atomically (write-temp, fsync, rename, fsync the directory), and only then deletes the generations it supersedes.

Recovery is therefore production-exercised: every compaction is a rehearsal of boot. Any mutation that forgot to write its delta shows up as a rebuild that disagrees with the live store, an invariant the test suite asserts continuously.

The one exception is the rollback-salvage snapshot, which serialises the live store because the salvage exists only in RAM.

Erasure reaches the disk #

Shredding destroys a payload in memory, but its bytes linger in older WAL frames and snapshots until compaction deletes those files.

That gap is managed as an explicit erasure debt. Every destructive transition starts a clock — a shred, a GC hardening, or an applied replicated shred, since the obligation follows the data rather than the command — and UD guarantees the destroyed payload has left the node's files within two minutes of the acknowledgment. In practice compaction kicks immediately; the window exists so a burst of shreds coalesces into one rebuild.

The debt is never written down. At boot it is re-derived from the replayed frames themselves, due immediately, so a node that dies mid-obligation finishes it promptly on restart.

The claim is deliberately scoped to the filesystem. What remains in unallocated blocks or an SSD's wear-levelled cells is the platform's concern, as it is for every directory product.

Crashes, restores, and the sim #

The persistence layer is torture-tested through a deterministic in-memory filesystem — torn writes at chosen bytes, failed fsyncs, power-loss simulation — and by the same chaos simulation that proved the merge.

Every simulated node runs on a faux disk through the real recovery path, crashes mid-anything, compacts under load, and is restored from forked disk images, the honest VM-revert. The fleet is still required to converge byte-for-byte at clear skies. A thin suite repeats the full lifecycle against a real filesystem.