Enrolling a host for cross-frontend access

A box is enrolled when any frontend in the fleet can select its rows, attach to one, and type — without that frontend having an account, a key, or a route to the box. The route is: frontend → its own ssh child to the hub → the hub's socket for that box → one ssh from the hub → sotd stdio-bridge on the box → that box's daemon. The hub is the only machine that ever ssh's in. The hosts.toml grammar is under "hosts.toml" in Configuration files; this page is the procedure.

Everything here is run by an operator, once per box. Nothing in it is automatic, and every step has a check that fails loudly — "it looked right" is not one of them.

What the hub ends up holding

The hub must be a Linux host running systemd --user: every artifact below is a user unit, and apply and refresh call systemctl. sotd topology apply --yes, run on the hub, writes and enables one listener per enrolled host:

File (in ~/.config/systemd/user/)	What it is
sot-host-relay-<host>.socket	the listener, at the path sotd topology relay-sockets prints; Accept=yes, SocketMode=0600
sot-host-relay-<host>@.service	one instance per accepted connection, the connection as its stdin/stdout
sot-host-relay-<host>@.service.d/zz-sot-relay-command.conf	the command: ssh … <host> sotd stdio-bridge (no argument: that box's own daemon), set last among the files of that directory
sot-host-relay-<host>.socket.d/topology.conf	the host pin ConditionHost=|<hub> and ConditionHost=|<hub>.*: a home the fleet shares shows the unit to every host, so it must start on the hub only

Two more pins are not per host: sot-relay-tunnel@.service.d/topology.conf starts every reverse tunnel on the hub only, and sotd.service.d/topology.conf starts the daemon only on the hosts declared daemon or frontend.

All four files and both pins are generated by the hub's daemon: when it runs as sotd.service it rewrites them at each start (a daemon started by hand rewrites nothing until sotd topology refresh) whenever their text differs from what the running sotd generates, and sotd topology refresh does the same by hand. Each start also reloads systemd and restarts every enabled relay socket, so a step that failed at one start is redone at the next; a running bridge is not interrupted. A hand edit to them is lost. A per-host override is an Environment= line in a drop-in of its own beside the service (any name but topology.conf, which is apply's own). The command is the hub's: systemd applies drop-ins in file-name order, so the hub's drop-in sets the command last among the files of the hub's own unit directory, and an ExecStart= in an earlier-sorting one has no effect. Name your own Environment= drop-ins so they sort before zz-sot-relay-command.conf (for example local.conf): refresh renames any .conf there that sorts after it to .conf.retired, and says so. A drop-in in another systemd directory that overrides the command is reported by refresh as a failure, not fixed.

# ~/.config/systemd/user/sot-host-relay-<host>@.service.d/local.conf
[Service]
# not on the ssh PATH:
Environment=SOT_RELAY_SOTD=/home/<user>/.local/share/sot/bin/sotd
# alias differs from the declared host name:
Environment=SOT_RELAY_TARGET=<ssh alias>

For a Windows host, ssh starts cmd.exe in the profile folder and the Windows install does not put sotd on PATH, so give its path in that host's own drop-in:

[Service]
Environment=SOT_RELAY_SOTD=AppData\\Local\\sot\\bin\\sotd.exe

The host sits in the unit's name, not after an @, because a templated socket with Accept=yes hands its service the connection id and not the socket's instance (systemd issue #19071) — a per-host prefix is the only spelling in which the service knows which box to ssh to.

The checklist

Each step's check must pass before the next one is worth doing.

1. ssh, from the hub, non-interactively. ssh -o BatchMode=yes <host> sotd --version prints a version. A prompt, a host-key question or a "command not found" here is the whole enrollment failing early: nothing downstream runs attended. If sotd is found interactively but not this way, note its full path for the SOT_RELAY_SOTD override above rather than editing shell profiles.

2. sotd runs as a user service on the box, started by the OS, not by a launcher, surviving with nobody logged in. On the box: systemctl --user show -p WantedBy --value sotd.service (it names default.target where the boot drop-in is loaded; is-enabled prints static on every host, so it checks nothing), systemctl --user cat sotd.service (its pin drop-in names this host), loginctl enable-linger where that applies, and sotd status answers. An instrument box wants this regardless of anything on this page.

3. The bridge answers. From the hub, send one hello frame through ssh -o BatchMode=yes <host> sotd stdio-bridge and get the daemon's own reply, byte for byte, with nothing else on stdout. The bridge closes when its input ends, so keep the input open for a few seconds:

   proto="$(ssh -o BatchMode=yes <host> sotd --version | grep -oE 'protocol [0-9]+' | grep -oE '[0-9]+')"
   uid="$(ssh -o BatchMode=yes <host> id -u)"
   { printf '{"v":1,"id":1,"kind":"req","op":"hello","payload":{"client_id":"enrol-check","last_seen_revision":0,"protocol":%s,"app_version":"enrol-check","host":"enrol-check","os_user":"uid:%s"}}\n' "$proto" "$uid"; sleep 5; } \
     | ssh -o BatchMode=yes <host> sotd stdio-bridge

   A good reply is one JSON line whose payload carries the daemon's app_version and host. With the daemon stopped, the same command must exit nonzero at once, one line on stderr, nothing on stdout — an honest failure, not a hang.

4. One ssh login per bridge connection. The generated service runs a plain ssh per accepted connection with connection sharing switched off (-o ControlMaster=no -o ControlPath=none -o ControlPersist=no), because sharing hung every bridge on one Windows host. A command-line option wins over ~/.ssh/config, so a Control* block there for the host does not reach the relay. A connection lives as long as the frontend session that opened it, so the login is paid once per session.

5. Update ownership is safe to move. Declaring daemon = true makes the box's own backend the owner of its updates instead of its frontend's self-update. Confirm the box's backend is healthy and at the expected version before step 6, not after.

6. Declare it, one box at a time. In the hub's canonical ~/.config/sot/hosts.toml:

   [host.<host>]
   daemon = true
   frontend = true   # only if it runs a frontend of its own

   Then on the hub: sotd topology apply (a dry run — read it) and sotd topology apply --yes. It must enable exactly one sot-host-relay-<host>.socket and disable nothing you did not expect. It also enables sot-relay-tunnel@<host> (the reverse tunnel for messaging) unless the box is a frontend; expect that line. On every other box: sotd topology sync. A box that shares the hub's home reads the new sotd.service pin at its next boot, or after systemctl --user daemon-reload there; a reload starts nothing, so a box newly declared to run a daemon then needs systemctl --user start sotd.

7. Verify end to end. On the hub, sotd status reports that box's real rows through unix:<runtime>/sot-host-<host>.sock. From another machine, relaunch the frontend: the host appears in Hosts mode, its rows in Sessions mode, attaching to one gives a read-write pty.

Reading a failure

• The socket file exists and the box is down. Expected, and not a contradiction: systemd owns that socket, so it is present whether or not anything is behind it. The frontend renders the host unreachable and grays its last-known rows. Only a completed hello ever means "up".

• The host stays unreachable and its row's status text shows an ssh error. The hub has no listener for the box: step 6 was not applied on the hub, or apply ran while the box was not yet declared.

• sotd topology apply refuses. It only runs on the hub, and says so, naming the hub. Run it there.

• The bridge instance fails per connection. journalctl --user -u 'sot-host-relay-<host>@*' has ssh's own line: a refused key, an unknown host key, or sotd: command not found (step 1's check, and the SOT_RELAY_SOTD override).

• An ExecStart= in a relay drop-in does nothing. The hub's zz-sot-relay-command.conf comes after it and sets the command (a later-sorting .conf is renamed to .retired by refresh; an override from another systemd directory is reported as a failure); systemctl --user cat 'sot-host-relay-<host>@.service' shows the order.

• Every remote host went unreachable at once. That is the hub, not a per-host fault: peers reach every enrolled box through it.

Removing a box

Delete its [host.<host>] block on the hub, sotd topology apply --yes (which disables the listener and the box's sot-relay-tunnel@<host>, and removes the generated files), and sotd topology sync elsewhere. Nothing needs doing on the box itself — the hub was the only side holding a route — unless it shares the hub's home and runs a daemon from there: then stop it on the box with systemctl --user stop sotd.service and run systemctl --user daemon-reload (enable and disable change no file for sotd.service: its boot start is the shared drop-in default.target.d/sotd.conf, and the pin and hosts.toml decide which hosts start it). The relay sockets and tunnels are not covered: never disable those.