DWE

Developer Workflow Embeddings are dedicated development paths built into core Peios software: the tools carry first-class support for developing against them, rather than being poked at from the outside by whatever a developer can improvise.

Its first and most general component is dwed, a service that gives you a persistent, maximally-privileged way to talk to a machine that is already running. Everything below is about that.

The problem it solves #

Debugging a live system means asking it one question, reading the answer, and asking a better one. The loop is only as good as how quickly you can go round it.

Without something like dwed, driving a Peios machine from outside means a serial console: boot the machine, feed it a script, read what comes back, and start again. Three things about that hurt more than they look:

• Nothing survives. Each run is a fresh boot, so every question has to be planned in advance and packed into one script. A question that occurs to you halfway through the output cannot be asked without starting over.
• The output is one stream. A console interleaves what you typed, what the shell echoed, and what the program wrote to stdout and stderr — with no reliable way to pull them apart afterwards. Exit codes are not carried at all unless the script prints them itself.
• You cannot be SYSTEM. A console session is a logon session belonging to a person. The most privileged thing a machine has to offer is not reachable through it at all.

dwed exists because the machine is already persistent. What was missing was anything willing to talk to it in between questions.

What it gives you #

A socket into a running machine that answers structured requests as SYSTEM:

• stdout and stderr come back separately, as raw bytes, with a real exit status.
• Commands run directly, not through a shell, so there is no quoting layer between what you meant and what ran.
• Work can be detached — started now, collected several connections later. The job outlives the connection that started it, which is what makes an investigation spanning hours possible.
• Files move in and out whole and binary-safe, rather than through base64 improvised into a shell pipeline.

The privilege #

dwed is started by peinit as an ordinary service with Identity = SYSTEM. It constructs no tokens of its own; the privilege arrives entirely from that one line in its service definition. Asked on a running machine, its token reports:

user          Local System (S-1-5-18)
type          Primary
logon_type    5 (Service)
privileges    36, all enabled — including SeCreateToken, SeTcb,
              SeAssignPrimaryToken, SeDebug, SeBackup, SeRestore,
              SeImpersonate, SeLoadDriver, SeSecurity, SeAudit

This is the same privilege peinit itself holds, deliberately. A full SYSTEM account is not reachable from a console at all, and reaching one is the single capability that makes dwed worth having over a serial login.

It is also why the rest of this page is about containment.

The security posture #

dwed does not authenticate its peer, and cannot.

The transport is vsock, which crosses a hypervisor boundary between two separate kernels. A guest kernel can be told a peer's context id, but it cannot attest anything about who is behind it — those are claims, not attestation, and no amount of work inside the guest changes that. Peios' identity model rules AF_VSOCK out as a carrier of process identity for exactly this reason.

So authentication is the job of whatever surrounds the machine — the host it runs on, the network it sits behind — and never of dwed.

Three things follow, and all three are load-bearing:

It is never published as a package. The peios-dwe package does not exist in the public repository and never will. It reaches a machine only inside a dedicated peios-dwe ISO, so it cannot arrive anywhere by way of an ordinary install.

Distribution is the only real control. The usual advice — "do not install it in production" — does not apply cleanly, because the machines DWE is wanted on are production in every sense except intent. There is no honest way to enforce the distinction from inside the software. Keeping it out of the repository is what stops it turning up somewhere by accident.

Installing it is not enough to start it. A package may ship a service definition but may not start it: the definition sits inert in the vendor seed library until an image names it in [registry] autoapply. For dwed, that opt-in is the moment a machine becomes remotely ownable — so it is a decision the image makes explicitly, not a consequence of a package being present.

What DWE is not #

It is not a test harness. Provium covers deterministic, repeatable testing of a whole system, from initramfs through to network interaction, and does it far better than anything built on dwed could. Tests belong there.

DWE is for the case Provium cannot serve: a machine that is already running, already misbehaving, and needs to be asked questions nobody thought to write down in advance. When a Provium test fails for a reason that is not obvious, DWE is how you go and look.

It is not a general remote-administration tool. There is no session model, no pty, no terminal multiplexing, and no plan for any of them until something concrete needs one. dwed is a way to ask a running machine questions, and the machine — not the connection — is the thing that persists.

Next #

• Driving a machine — booting with a vsock device, and the dwe command.
• The DWE protocol — the wire format, for building against it directly.

This page assumes an image built with the peios-dwe package and its service seed applied. If you are not sure, What DWE is explains why both are needed and why neither is the default.

Give the machine a transport #

dwed binds a vsock listener at boot, but a guest cannot conjure the transport itself — the hypervisor has to give it a vsock device. Under QEMU that is one flag:

-device vhost-vsock-pci,guest-cid=3

The context id is how the host addresses this guest. Any value of 3 or above works; QEMU refuses to start if another running guest already holds the one you picked, so concurrent machines need distinct ids.

In the Peios tree, make boot-dwe is make boot with that device attached:

cd dist/prod
make boot-dwe              # or: make boot-dwe DWE_CID=4

Leave it running. Unlike a scripted boot, the point is that the machine stays up.

Point the client at it #

The dwe client takes its target from --target or from DWE_TARGET:

export DWE_TARGET=vsock:3:4820        # a VM by context id
export DWE_TARGET=tcp:10.0.0.5:4820   # or over the network

The port defaults to 4820 and can be left off. Confirm you have the machine you think you have:

$ dwe info
dwed            0.1.0
protocol        1
boot id         79880a4b-e3d0-4992-bea2-eb56f3839711
uptime          193s

The boot id is worth reading. It changes on every boot, so it is how you tell "the machine rebooted under me" from "my connection dropped" — two situations that otherwise look identical and mean very different things.

Run something #

$ dwe exec -- ls -l /system
$ dwe exec --cwd /tmp -- ./probe

Everything after -- is the argument vector, executed directly. There is no shell, so nothing re-interprets your quoting, globs your arguments or splits them on spaces. When you want a shell, ask for one:

dwe exec -- sh -c 'for p in /proc/[0-9]*; do cat $p/comm; done'

Two behaviours matter more than they look:

The exit status is yours. dwe exec exits with the guest command's status, so ordinary shell chaining works:

dwe exec -- test -f /etc/hostname && echo "it is there"

A command killed by a signal exits 128+N, matching shell convention, so it is distinguishable from one that merely failed.

The streams stay apart. The guest's stdout and stderr are written to your stdout and stderr, still separated, as raw bytes:

$ dwe exec -- ls /nonexistent 2>errors.txt
$ cat errors.txt
ls: cannot access '/nonexistent': No such file or directory

Move files #

dwe pull /var/log/peinit.log ./peinit.log   # out of the guest
dwe pull /etc/hostname                      # or straight to stdout
dwe push ./probe.sh /tmp/probe.sh           # into the guest
echo "hello" | dwe push - /tmp/greeting     # from stdin

Transfers are binary-safe and whole-file. Encoding is handled inside the protocol, so you never have to improvise base64 through a shell pipeline to get a binary out intact.

Work that outlives the connection #

This is the part that makes a long investigation possible. --detach returns a job handle immediately, and the job keeps running when the connection closes:

$ dwe exec --detach -- make -C /src world
1

Come back whenever — a minute later, an hour later, over as many separate connections as you like:

$ dwe jobs
1      running      make -C /src world

$ dwe output 1
[... everything so far ...]

$ dwe output 1 --follow          # or watch it live
$ dwe signal 1 15                # or stop it

Reading only what is new #

Polling a job repeatedly with plain dwe output re-reads everything from the start. To pick up where you left off, ask for the offsets and pass them back:

$ dwe output 1 --offsets
[... output ...]
--since-stdout 4096 --since-stderr 128

$ dwe output 1 --since-stdout 4096 --since-stderr 128
[... only what arrived since ...]

The cursor is yours to keep rather than something dwed tracks. It has no sessions and cannot tell two callers apart, so a server-side cursor would have two people polling the same job eating each other's output.

When it does not answer #

cannot reach vsock:3:4820 — the machine is not running, has no vsock device, or is using a different context id. Check the QEMU command line for vhost-vsock-pci.

The connection opens but nothing answers — dwed is not running in the guest. Its service definition was shipped but never applied: an image has to name dwed-service.reg in [registry] autoapply for anything to start. On the console, look for peinit: service dwed started.

protocol mismatch — dwe and dwed are from different builds. The wire version is checked rather than guessed at, so this is reported instead of being allowed to misparse. Rebuild both.

Nothing at all, and the machine is wedged — dwed goes down with the machine it is debugging. Below that line the serial console is still the tool.

dwed speaks newline-delimited JSON: one JSON object per line in each direction, with each response carrying the id of the request it answers.

JSON rather than a compact binary encoding is a deliberate trade. When dwed itself is the thing misbehaving, the protocol has to stay drivable by hand — and being able to type at it and read what comes back is worth more than the bytes a binary framing would save:

$ nc 10.0.0.5 4820
{"id":1,"op":"info"}
{"id":1,"ok":{"reply":"info","protocol_version":1,"dwed_version":"0.1.0",...}}

Framing #

A request is an object with an id and an op, plus that op's arguments inline:

{"id": 7, "op": "exec", "argv": ["ls", "-l", "/system"]}

id is chosen by the client and echoed back untouched. Requests on one connection are answered in order; concurrency comes from opening more connections, or from detaching work.

A response carries the same id and exactly one of ok or error:

{"id": 7, "ok": {"reply": "exec", "exit": 0, "stdout": "…", "stderr": "", "truncated": false}}
{"id": 7, "error": {"code": "no_such_job", "message": "job 9 is unknown"}}

The reply field names the shape of the payload. It is there because several replies carry the same fields — exec and job.output both have stdout, stderr and an optional exit — and a reader should never have to guess which one it is holding from shape alone.

A request that does not parse is still answered, with id: 0 and a bad_request error. Silence would leave a client waiting on an id that is never coming, which presents as a hang — the one symptom hardest to tell apart from the bug being investigated.

Bytes #

Every field carrying payload bytes is base64: file contents, and captured stdout/stderr alike.

Output is base64 rather than a JSON string because a guest command's output is not guaranteed to be valid UTF-8, and a tool that mangles a binary is worse than one that refuses it. Clients decode and write raw bytes back out, so the encoding never reaches whoever is driving the tool.

Operations #

exec #

Run a command. Synchronous unless detach is set.

Replies exec with exit (null if signalled), signal, stdout, stderr and truncated; or job with a job handle when detach was set.

Both output streams are captured concurrently. A child that fills one pipe while the other goes undrained would otherwise deadlock, and "the command hung" is the least useful thing a debugging tool can report.

job.list #

No arguments. Replies job_list with a jobs array of {job, argv, running, exit, signal, started}.

Jobs outlive the connection that created them. They do not outlive dwed itself.

job.output #

Replies job_output with stdout, stderr, stdout_next, stderr_next, running, exit, signal and truncated.

The *_next values are what to pass as since_* on the following call. Cursors are the caller's to keep: dwed has no sessions and cannot distinguish two callers, so a server-side cursor would have concurrent readers consuming each other's output. An offset past the end is clamped rather than rejected.

job.signal #

{job, signal} — deliver a signal to a running job. Replies done. A job that has already exited gives not_running.

file.read #

{path} — replies file_read with bytes and mode.

file.write #

{path, bytes, mode?} — replies done. mode is applied after writing.

info #

No arguments. Replies info:

boot_id is the field worth using. It lets a client tell "the machine rebooted under me" from "my connection dropped" — two situations that look identical from the socket and mean entirely different things.

Errors #

errno is carried separately from the message so a client can match on the cause rather than parse English.

Transports #

vsock is the default, on port 4820. It needs no networking in the guest — which matters, because a machine whose networking is part of what broke is squarely one of the cases DWE exists for. dwed binds VMADDR_CID_ANY: a guest does not reliably know its own context id, and does not need to.

TCP is available on the same port but is opt-in (dwed --tcp 0.0.0.0:4820). Listening by default would hand the machine to anyone who can route to it, which is more than starting a service should quietly do given there is nothing behind it.

The protocol is identical over either.

Versioning #

PROTOCOL_VERSION is bumped whenever any wire type changes shape. A client compares it against info and reports a mismatch rather than misparsing a response it half understands.

There is no negotiation and no compatibility window. Both halves ship from one repository and are built together; a version check is there to give a clear error, not to bridge a gap.