# DWE

---

# What DWE is

_Peios / Developing for Peios / DWE_

> Developer Workflow Embeddings — development paths built into core Peios software. Its first component, dwed, is an unauthenticated SYSTEM control surface that lets you drive a running machine from outside, across as long as an investigation takes.

**Developer Workflow Embeddings** are dedicated development paths built into core Peios software: the tools carry first-class support for developing against them, rather than being poked at from the outside by whatever a developer can improvise.

Its first and most general component is **`dwed`**, a service that gives you a persistent, maximally-privileged way to talk to a machine that is already running. Everything below is about that.

> [!CAUTION]
> `dwed` performs **no authentication of any kind**. Anything that can reach its socket owns that machine as `SYSTEM`. It exists only on a dedicated development ISO and must never be present on a machine you would not hand to a stranger.
>
> This is the design, not a gap to be closed later. See [The security posture](#the-security-posture).

## The problem it solves

Debugging a live system means asking it one question, reading the answer, and asking a better one. The loop is only as good as how quickly you can go round it.

Without something like `dwed`, driving a Peios machine from outside means a serial console: boot the machine, feed it a script, read what comes back, and start again. Three things about that hurt more than they look:

- **Nothing survives.** Each run is a fresh boot, so every question has to be planned in advance and packed into one script. A question that occurs to you halfway through the output cannot be asked without starting over.
- **The output is one stream.** A console interleaves what you typed, what the shell echoed, and what the program wrote to `stdout` and `stderr` — with no reliable way to pull them apart afterwards. Exit codes are not carried at all unless the script prints them itself.
- **You cannot be `SYSTEM`.** A console session is a logon session belonging to a person. The most privileged thing a machine has to offer is not reachable through it at all.

`dwed` exists because the machine is already persistent. What was missing was anything willing to talk to it in between questions.

## What it gives you

A socket into a running machine that answers **structured** requests as `SYSTEM`:

- `stdout` and `stderr` come back **separately**, as raw bytes, with a real exit status.
- Commands run **directly**, not through a shell, so there is no quoting layer between what you meant and what ran.
- Work can be **detached** — started now, collected several connections later. The job outlives the connection that started it, which is what makes an investigation spanning hours possible.
- Files move in and out **whole and binary-safe**, rather than through `base64` improvised into a shell pipeline.

## The privilege

`dwed` is started by peinit as an ordinary service with `Identity = SYSTEM`. It constructs no tokens of its own; the privilege arrives entirely from that one line in its service definition. Asked on a running machine, its token reports:

```text
user          Local System (S-1-5-18)
type          Primary
logon_type    5 (Service)
privileges    36, all enabled — including SeCreateToken, SeTcb,
              SeAssignPrimaryToken, SeDebug, SeBackup, SeRestore,
              SeImpersonate, SeLoadDriver, SeSecurity, SeAudit
```

This is the same privilege peinit itself holds, deliberately. A full `SYSTEM` account is not reachable from a console at all, and reaching one is the single capability that makes `dwed` worth having over a serial login.

It is also why the rest of this page is about containment.

## The security posture

`dwed` does not authenticate its peer, and **cannot**.

The transport is [vsock](/peios/developing-for-peios/dwe/protocol.md), which crosses a hypervisor boundary between two separate kernels. A guest kernel can be told a peer's context id, but it cannot *attest* anything about who is behind it — those are claims, not attestation, and no amount of work inside the guest changes that. Peios' identity model rules `AF_VSOCK` out as a carrier of process identity for exactly this reason.

So authentication is the job of whatever surrounds the machine — the host it runs on, the network it sits behind — and never of `dwed`.

Three things follow, and all three are load-bearing:

**It is never published as a package.** The `peios-dwe` package does not exist in the public repository and never will. It reaches a machine only inside a dedicated `peios-dwe` ISO, so it cannot arrive anywhere by way of an ordinary install.

**Distribution is the only real control.** The usual advice — "do not install it in production" — does not apply cleanly, because the machines DWE is wanted on are production in every sense except intent. There is no honest way to enforce the distinction from inside the software. Keeping it out of the repository is what stops it turning up somewhere by accident.

**Installing it is not enough to start it.** A package may ship a service definition but may not start it: the definition sits inert in the vendor seed library until an image names it in `[registry] autoapply`. For `dwed`, that opt-in is the moment a machine becomes remotely ownable — so it is a decision the image makes explicitly, not a consequence of a package being present.

## What DWE is not

**It is not a test harness.** Provium covers deterministic, repeatable testing of a whole system, from initramfs through to network interaction, and does it far better than anything built on `dwed` could. Tests belong there.

DWE is for the case Provium cannot serve: a machine that is *already* running, already misbehaving, and needs to be asked questions nobody thought to write down in advance. When a Provium test fails for a reason that is not obvious, DWE is how you go and look.

**It is not a general remote-administration tool.** There is no session model, no pty, no terminal multiplexing, and no plan for any of them until something concrete needs one. `dwed` is a way to ask a running machine questions, and the machine — not the connection — is the thing that persists.

> [!NOTE]
> `dwed` is a phase-2 service, so a boot that breaks before phase 2 has no DWE at all. It also goes down with a machine that wedges completely. Both are accepted limits: below that line, a serial console is still the tool.

## Next

- [Driving a machine](/peios/developing-for-peios/dwe/driving-a-machine.md) — booting with a vsock device, and the `dwe` command.
- [The DWE protocol](/peios/developing-for-peios/dwe/protocol.md) — the wire format, for building against it directly.

---

# Driving a machine

_Peios / Developing for Peios / DWE_

> Booting a Peios machine with a vsock device, pointing the dwe client at it, and running commands, moving files and detaching long work that outlives the connection.

This page assumes an image built with the `peios-dwe` package and its service seed applied. If you are not sure, [What DWE is](/peios/developing-for-peios/dwe/what-dwe-is.md) explains why both are needed and why neither is the default.

## Give the machine a transport

`dwed` binds a vsock listener at boot, but a guest cannot conjure the transport itself — the hypervisor has to give it a vsock device. Under QEMU that is one flag:

```sh
-device vhost-vsock-pci,guest-cid=3
```

The context id is how the host addresses this guest. Any value of 3 or above works; QEMU refuses to start if another running guest already holds the one you picked, so concurrent machines need distinct ids.

In the Peios tree, `make boot-dwe` is `make boot` with that device attached:

```sh
cd dist/prod
make boot-dwe              # or: make boot-dwe DWE_CID=4
```

Leave it running. Unlike a scripted boot, the point is that the machine stays up.

> [!NOTE]
> `/dev/vhost-vsock` is owned by `root:kvm`, so membership of the `kvm` group is enough — this does not need root.
>
> On a guest booted *without* a vsock device, `dwed` reports that it has no transport and exits cleanly. That is not an error and does not restart-loop.

## Point the client at it

The `dwe` client takes its target from `--target` or from `DWE_TARGET`:

```sh
export DWE_TARGET=vsock:3:4820        # a VM by context id
export DWE_TARGET=tcp:10.0.0.5:4820   # or over the network
```

The port defaults to 4820 and can be left off. Confirm you have the machine you think you have:

```console
$ dwe info
dwed            0.1.0
protocol        1
boot id         79880a4b-e3d0-4992-bea2-eb56f3839711
uptime          193s
```

The **boot id** is worth reading. It changes on every boot, so it is how you tell "the machine rebooted under me" from "my connection dropped" — two situations that otherwise look identical and mean very different things.

## Run something

```console
$ dwe exec -- ls -l /system
$ dwe exec --cwd /tmp -- ./probe
```

Everything after `--` is the argument vector, executed **directly**. There is no shell, so nothing re-interprets your quoting, globs your arguments or splits them on spaces. When you want a shell, ask for one:

```sh
dwe exec -- sh -c 'for p in /proc/[0-9]*; do cat $p/comm; done'
```

Two behaviours matter more than they look:

**The exit status is yours.** `dwe exec` exits with the guest command's status, so ordinary shell chaining works:

```sh
dwe exec -- test -f /etc/hostname && echo "it is there"
```

A command killed by a signal exits `128+N`, matching shell convention, so it is distinguishable from one that merely failed.

**The streams stay apart.** The guest's `stdout` and `stderr` are written to *your* `stdout` and `stderr`, still separated, as raw bytes:

```console
$ dwe exec -- ls /nonexistent 2>errors.txt
$ cat errors.txt
ls: cannot access '/nonexistent': No such file or directory
```

## Move files

```sh
dwe pull /var/log/peinit.log ./peinit.log   # out of the guest
dwe pull /etc/hostname                      # or straight to stdout
dwe push ./probe.sh /tmp/probe.sh           # into the guest
echo "hello" | dwe push - /tmp/greeting     # from stdin
```

Transfers are binary-safe and whole-file. Encoding is handled inside the protocol, so you never have to improvise `base64` through a shell pipeline to get a binary out intact.

## Work that outlives the connection

This is the part that makes a long investigation possible. `--detach` returns a job handle immediately, and the job keeps running when the connection closes:

```console
$ dwe exec --detach -- make -C /src world
1
```

Come back whenever — a minute later, an hour later, over as many separate connections as you like:

```console
$ dwe jobs
1      running      make -C /src world

$ dwe output 1
[... everything so far ...]

$ dwe output 1 --follow          # or watch it live
$ dwe signal 1 15                # or stop it
```

### Reading only what is new

Polling a job repeatedly with plain `dwe output` re-reads everything from the start. To pick up where you left off, ask for the offsets and pass them back:

```console
$ dwe output 1 --offsets
[... output ...]
--since-stdout 4096 --since-stderr 128

$ dwe output 1 --since-stdout 4096 --since-stderr 128
[... only what arrived since ...]
```

The cursor is yours to keep rather than something `dwed` tracks. It has no sessions and cannot tell two callers apart, so a server-side cursor would have two people polling the same job eating each other's output.

> [!IMPORTANT]
> Job state lives in memory. A restart of `dwed` restores the listener but loses every job and everything it had captured. Output is capped at 16 MiB per stream, after which the tail is dropped and the reply says so — pipe a genuinely large job to a file in the guest and `pull` it instead.

## When it does not answer

**`cannot reach vsock:3:4820`** — the machine is not running, has no vsock device, or is using a different context id. Check the QEMU command line for `vhost-vsock-pci`.

**The connection opens but nothing answers** — `dwed` is not running in the guest. Its service definition was shipped but never applied: an image has to name `dwed-service.reg` in `[registry] autoapply` for anything to start. On the console, look for `peinit: service dwed started`.

**`protocol mismatch`** — `dwe` and `dwed` are from different builds. The wire version is checked rather than guessed at, so this is reported instead of being allowed to misparse. Rebuild both.

**Nothing at all, and the machine is wedged** — `dwed` goes down with the machine it is debugging. Below that line the serial console is still the tool.

---

# The DWE protocol

_Peios / Developing for Peios / DWE_

> The dwed wire protocol — newline-delimited JSON over vsock or TCP, the request and reply shapes for every operation, error codes, and the transport and versioning rules.

`dwed` speaks **newline-delimited JSON**: one JSON object per line in each direction, with each response carrying the `id` of the request it answers.

JSON rather than a compact binary encoding is a deliberate trade. When `dwed` itself is the thing misbehaving, the protocol has to stay drivable by hand — and being able to type at it and read what comes back is worth more than the bytes a binary framing would save:

```console
$ nc 10.0.0.5 4820
{"id":1,"op":"info"}
{"id":1,"ok":{"reply":"info","protocol_version":1,"dwed_version":"0.1.0",...}}
```

## Framing

A request is an object with an `id` and an `op`, plus that op's arguments inline:

```json
{"id": 7, "op": "exec", "argv": ["ls", "-l", "/system"]}
```

`id` is chosen by the client and echoed back untouched. Requests on one connection are answered in order; concurrency comes from opening more connections, or from detaching work.

A response carries the same `id` and exactly one of `ok` or `error`:

```json
{"id": 7, "ok": {"reply": "exec", "exit": 0, "stdout": "…", "stderr": "", "truncated": false}}
{"id": 7, "error": {"code": "no_such_job", "message": "job 9 is unknown"}}
```

The `reply` field names the shape of the payload. It is there because several replies carry the same fields — `exec` and `job.output` both have `stdout`, `stderr` and an optional `exit` — and a reader should never have to guess which one it is holding from shape alone.

A request that does not parse is still answered, with `id: 0` and a `bad_request` error. Silence would leave a client waiting on an `id` that is never coming, which presents as a hang — the one symptom hardest to tell apart from the bug being investigated.

### Bytes

Every field carrying payload bytes is **base64**: file contents, and captured `stdout`/`stderr` alike.

Output is base64 rather than a JSON string because a guest command's output is not guaranteed to be valid UTF-8, and a tool that mangles a binary is worse than one that refuses it. Clients decode and write raw bytes back out, so the encoding never reaches whoever is driving the tool.

## Operations

### `exec`

Run a command. Synchronous unless `detach` is set.

| Field | | |
|---|---|---|
| `argv` | required | Program and arguments. Executed directly — **not** through a shell. |
| `cwd` | optional | Working directory. |
| `env` | optional | Extra environment, as `[name, value]` pairs, on top of the service's own. |
| `stdin` | optional | Bytes written to the child's stdin, which is then closed. |
| `detach` | optional | Return a job handle immediately instead of waiting. |

Replies `exec` with `exit` (null if signalled), `signal`, `stdout`, `stderr` and `truncated`; or `job` with a `job` handle when `detach` was set.

Both output streams are captured concurrently. A child that fills one pipe while the other goes undrained would otherwise deadlock, and "the command hung" is the least useful thing a debugging tool can report.

### `job.list`

No arguments. Replies `job_list` with a `jobs` array of `{job, argv, running, exit, signal, started}`.

Jobs outlive the connection that created them. They do not outlive `dwed` itself.

### `job.output`

| Field | | |
|---|---|---|
| `job` | required | The handle. |
| `since_stdout` | optional | Resume stdout from this byte offset. |
| `since_stderr` | optional | Resume stderr from this byte offset. |

Replies `job_output` with `stdout`, `stderr`, `stdout_next`, `stderr_next`, `running`, `exit`, `signal` and `truncated`.

The `*_next` values are what to pass as `since_*` on the following call. Cursors are the caller's to keep: `dwed` has no sessions and cannot distinguish two callers, so a server-side cursor would have concurrent readers consuming each other's output. An offset past the end is clamped rather than rejected.

### `job.signal`

`{job, signal}` — deliver a signal to a running job. Replies `done`. A job that has already exited gives `not_running`.

### `file.read`

`{path}` — replies `file_read` with `bytes` and `mode`.

### `file.write`

`{path, bytes, mode?}` — replies `done`. `mode` is applied after writing.

### `info`

No arguments. Replies `info`:

| Field | |
|---|---|
| `protocol_version` | The version `dwed` speaks. |
| `dwed_version` | Its own release. |
| `boot_id` | Distinguishes one boot from the next. |
| `uptime` | Seconds since boot. |

`boot_id` is the field worth using. It lets a client tell "the machine rebooted under me" from "my connection dropped" — two situations that look identical from the socket and mean entirely different things.

## Errors

| `code` | |
|---|---|
| `bad_request` | Malformed JSON, an unknown op, or empty `argv`. |
| `io` | An underlying system call failed. Carries `errno`. |
| `no_such_job` | No job with that handle. |
| `not_running` | The job has already exited. |

`errno` is carried separately from the message so a client can match on the cause rather than parse English.

## Transports

**vsock** is the default, on port 4820. It needs no networking in the guest — which matters, because a machine whose networking is part of what broke is squarely one of the cases DWE exists for. `dwed` binds `VMADDR_CID_ANY`: a guest does not reliably know its own context id, and does not need to.

**TCP** is available on the same port but is **opt-in** (`dwed --tcp 0.0.0.0:4820`). Listening by default would hand the machine to anyone who can route to it, which is more than starting a service should quietly do given there is nothing behind it.

The protocol is identical over either.

> [!CAUTION]
> Neither transport authenticates anything. Whatever reaches the socket owns the machine as `SYSTEM`. See [the security posture](/peios/developing-for-peios/dwe/what-dwe-is.md#the-security-posture).

## Versioning

`PROTOCOL_VERSION` is bumped whenever any wire type changes shape. A client compares it against `info` and reports a mismatch rather than misparsing a response it half understands.

There is no negotiation and no compatibility window. Both halves ship from one repository and are built together; a version check is there to give a clear error, not to bridge a gap.