Enrolling a host for cross-frontend access

A box is enrolled when any frontend in the fleet can select its rows, attach to one, and type — without that frontend having an account, a key, or a route to the box. The route is: frontend → its own ssh child to the hub → the hub's socket for that box → one ssh from the hub → sotd stdio-bridge on the box → that box's daemon. The hub is the only machine that ever ssh's in; ADR 0048 is the decision, this page is the procedure.

Everything here is run by an operator, once per box. Nothing in it is automatic, and every step has a check that fails loudly — "it looked right" is not one of them.

What the hub ends up holding

sotd topology apply --yes, run on the hub, writes and enables one listener per enrolled host:

File (in ~/.config/systemd/user/)	What it is
sot-host-relay-<host>.socket	the listener, at the path sotd topology relay-sockets prints; Accept=yes, SocketMode=0600
sot-host-relay-<host>@.service	one instance per accepted connection, the connection as its stdin/stdout
sot-host-relay-<host>@.service.d/zz-sot-relay-command.conf	the command: ssh … <host> sotd stdio-bridge (no argument: that box's own daemon), set last among the files of that directory
sot-host-relay-<host>.socket.d/topology.conf	the host pin ConditionHost=|<hub> and ConditionHost=|<hub>.*: a home the fleet shares shows the unit to every host, so it must start on the hub only

Two more pins are not per host: sot-relay-tunnel@.service.d/topology.conf starts every reverse tunnel on the hub only, and sotd.service.d/topology.conf starts the daemon only on the hosts declared daemon or frontend.

All four files and both pins are generated by the hub's daemon: it rewrites them at each start whenever their text differs from what the running sotd generates, and sotd topology refresh does the same by hand. Each start also reloads systemd and restarts every enabled relay socket, so a step that failed at one start is redone at the next; a running bridge is not interrupted. A hand edit to them is lost. A per-host override is an Environment= line in a drop-in of its own beside the service (any name but topology.conf, which is apply's own). The command is the hub's: systemd applies drop-ins in file-name order, so the hub's drop-in sets the command last among the files of the hub's own unit directory, and an ExecStart= in an earlier-sorting one has no effect. Name your own Environment= drop-ins so they sort before zz-sot-relay-command.conf (for example local.conf): refresh renames any .conf there that sorts after it to .conf.retired, and says so. A drop-in in another systemd directory that overrides the command is reported by refresh as a failure, not fixed.

# ~/.config/systemd/user/sot-host-relay-<host>@.service.d/local.conf
[Service]
Environment=SOT_RELAY_SOTD=/home/<user>/.local/share/sot/bin/sotd   # not on the ssh PATH
Environment=SOT_RELAY_TARGET=<ssh alias>                            # alias ≠ declared host name
Environment=SOT_RELAY_SOTD=AppData\\Local\\sot\\bin\\sotd.exe   # a Windows host: ssh starts cmd.exe in the profile folder, and the Windows install does not put sotd on PATH

The host sits in the unit's name, not after an @, because a templated socket with Accept=yes hands its service the connection id and not the socket's instance (systemd issue #19071) — a per-host prefix is the only spelling in which the service knows which box to ssh to.

The checklist

Each step's check must pass before the next one is worth doing.

1. ssh, from the hub, non-interactively. ssh -o BatchMode=yes <host> sotd --version prints a version. A prompt, a host-key question or a "command not found" here is the whole enrolment failing early: nothing downstream runs attended. If sotd is found interactively but not this way, note its full path for the SOT_RELAY_SOTD override above rather than editing shell profiles.

2. sotd runs as a user service on the box, started by the OS, not by a launcher, surviving with nobody logged in. On the box: systemctl --user show -p WantedBy --value sotd.service (it names default.target where the boot drop-in is loaded; is-enabled prints static on every host, so it checks nothing), systemctl --user cat sotd.service (its pin drop-in names this host), loginctl enable-linger where that applies, and sotd status answers. An instrument box wants this regardless of anything on this page.

3. The bridge answers. From the hub, pipe one hello frame through ssh -o BatchMode=yes <host> sotd stdio-bridge and get the daemon's own reply, byte for byte, with nothing else on stdout. With the daemon stopped, the same command must exit nonzero at once, one line on stderr, nothing on stdout — an honest failure, not a hang.

4. One ssh login per bridge connection. The generated service runs a plain ssh per accepted connection with connection sharing switched off (-o ControlMaster=no -o ControlPath=none -o ControlPersist=no), because sharing hung every bridge on one Windows host. A command-line option wins over ~/.ssh/config, so a Control* block there for the host (an older version of this page said to add one) no longer reaches the relay and can be deleted. A connection lives as long as the frontend session that opened it, so the login is paid once per session.

5. Update ownership is safe to move. Declaring daemon = true makes the box's own backend the owner of its updates instead of its frontend's self-update. Confirm the box's backend is healthy and at the expected version before step 6, not after.

6. Declare it, one box at a time. In the hub's canonical ~/.config/sot/hosts.toml:

   [host.<host>]
   daemon = true
   frontend = true   # only if it runs a frontend of its own

   Then on the hub: sotd topology apply (a dry run — read it) and sotd topology apply --yes. It must enable exactly one sot-host-relay-<host>.socket and disable nothing you did not expect. On every other box: sotd topology sync. A box that shares the hub's home reads the new sotd.service pin at its next boot, or after systemctl --user daemon-reload there; a reload starts nothing, so a box newly declared to run a daemon then needs systemctl --user start sotd.

7. Verify end to end. On the hub, sotd status reports that box's real rows through unix:<runtime>/sot-host-<host>.sock. From another machine, relaunch the frontend: the host appears in Hosts mode, its rows in Sessions mode, attaching to one gives a read-write pty, and typing at the box itself takes the pen back.

Reading a failure

• The socket file exists and the box is down. Expected, and not a contradiction: systemd owns that socket, so it is present whether or not anything is behind it. The frontend renders the host unreachable and greys its last-known rows. Only a completed hello ever means "up".

• A frontend logs has no relay socket on hub '<hub>'. The box is in the plan but the hub has no listener for it — step 6 was not applied on the hub, or apply ran while the box was not yet declared.

• sotd topology apply refuses. It only runs on the hub, and says so, naming the hub. Run it there.

• The bridge instance fails per connection. journalctl --user -u 'sot-host-relay-<host>@*' has ssh's own line: a refused key, an unknown host key, or sotd: command not found (step 1's check, and the SOT_RELAY_SOTD override). Usage: sotd stdio-bridge --label <label> there means that host runs 0.6.6-rc1 to rc9.8, whose bridge needs an argument the hub no longer passes: upgrade the host.

• An ExecStart= in a relay drop-in does nothing. The hub's zz-sot-relay-command.conf comes after it and sets the command (a later-sorting .conf is renamed to .retired by refresh; an override from another systemd directory is reported as a failure); systemctl --user cat 'sot-host-relay-<host>@.service' shows the order. An older no-mux.conf can be deleted.

• Every remote host went unreachable at once. That is the hub, not a per-host fault: peers reach every enrolled box through it now.

Removing a box

Delete its [host.<host>] block on the hub, sotd topology apply --yes (which disables the listener and removes both generated unit files), and sotd topology sync elsewhere. Nothing needs doing on the box itself — the hub was the only side holding a route — unless it shares the hub's home and runs a daemon from there: then stop it on the box with systemctl --user stop sotd.service and run systemctl --user daemon-reload (enable and disable change no file for sotd.service: its boot start is the shared drop-in default.target.d/sotd.conf, and the pin and hosts.toml decide which hosts start it). The relay sockets and tunnels are not covered: never disable those.