Fleet operations¶

Owners: docker-compose-repo (read at ba6cee7), fleet-ops (read at 06280d9), and pi-devbox (the baked pi-devbox-environment and pi-devbox-firstboot skills, at 47b4506)

The fleet is every machine that runs the pi-devbox image, plus the services they share. The machines include personal Macs, a work laptop and a cloud VM, on both arm64 and amd64. Most of them are asleep most of the time. The fleet is run so that a machine that wakes after weeks can catch up on its own.

One stack per device¶

Each device has its own directory in docker-compose-repo with:

• a compose file: which image tag to run, which host directories to mount, and which named volumes to keep;
• an environment file with that device's settings and credentials. Only an age-encrypted copy is committed;
• a README for anything specific to the machine.

Some settings must be declared per machine because nothing can safely infer them. The git commit identity, for example, differs between work and personal machines. The repository says which is which.

Updating a device means pulling the image and recreating the container:

docker compose pull
docker compose up -d --force-recreate

A device on a floating tag such as latest-studio gets each new release on its next pull. After a recreate, the sanity check in the pi-devbox repository confirms the new image is running and the volumes came back.

What the fleet shares¶

flowchart LR
  hub["Docker Hub<br/>the image"] --> dev["Each device's<br/>container"]
  common["docker-compose-repo<br/>common/pi"] --> dev
  dev --> palace["Central palace"]
  dev --> gitea["Gitea<br/>repos and CI"]

• The image, from Docker Hub.
• Shared harness files. docker-compose-repo's common/pi directory holds the fleet extensions and the shared model list. Each device's checkout of it is mounted read-only into the container and linked in at start. A fix there reaches a device with a git pull and a /reload, without a new image.
• The central palace: memory, diaries, the knowledge graph and the mailbox (see Memory).
• Gitea, for the repositories and their CI.

First-boot acceptance¶

A release is built and smoke-tested in CI, but CI cannot see how an image behaves on a real device, with that device's volumes, configuration and credentials. So each device runs an acceptance round the first time it boots a new release.

Three tiers, and which ones a device owes¶

A device that settles a tier publishes a claim: a broadcast keyed on exactly what it verified, such as the release tag and source commit, or the per-arch image digest. The next device looks up the two claims for its own release and architecture and runs only what is still unsettled.

Three rules keep this safe:

• Derive the release; never predict it. The device reads the release from its own build manifest. Releases have been tagged and never published, so a predicted version can be wrong.
• Validate a claim before honouring it. A device skips a tier only if the claim's keys match its own first-hand readings. One stale claim must not switch off checking across the fleet.
• The device tier always runs. The same image reads differently on different devices, and both readings can be correct, because they depend on each device's configuration.

A chain that survives sleep¶

Each device's round ends by answering its own ask and then planting the ask for the next round. The new ask stays dormant until the release in the build manifest changes or the container is recreated. A restart does not count, because only a recreate rewrites the file the ask watches. A device that has never run a round plants its own first ask, so nobody has to keep a list of devices. Results also update the knowledge graph, such as which device runs which release.

Releases¶

A release is a version tag on pi-devbox. The changelog must name every component the build will change, and CI checks that. The ci-release-watcher skill watches the CI run to the end and verifies that the expected tags reached the registry. Each device is then recreated on the new release when it is next in use.

Reaching the host and the LAN¶

At start, the container writes an SSH configuration of its own, because the mounted ~/.ssh is read-only:

• an alias, host, for the Docker host itself;
• on hosts where Docker runs inside a virtual machine (Docker Desktop on a Mac), a jump through the host to reach machines on its local network;
• connection sharing, so a dozen SSH calls in a few minutes reuse one connection. That avoids tripping intrusion protection or a NAT connection limit on the far end.

Commands use it explicitly, as ssh -F ~/.ssh-local/config <name>, because agent shells do not load interactive aliases.

Coordinating work between devices¶

Work that one device needs from another goes through the palace's mailbox as a directed ask. For a change that every device must make, such as aligning a setting, the sender writes one ask per device on a shared correlation id. Each device applies the change, verifies it and replies with a terminal status. The reply says what changed, so the thread doubles as the record. See the coordination log.

Where the inventory lives¶

This site describes mechanisms. The fleet's specifics live in fleet-ops (private): which host does what, addresses, tunnels, backup targets and schedules, and a register of every credential seen in the fleet. The register records fingerprints only, never values. The split follows fleet-ops' own rule: if a paragraph would still be true for someone else's fleet, it is mechanism and belongs in a public repository.

Sources¶

• docker-compose-repo README.md (developer machines, git identity, secrets)
• fleet-ops README.md
• pi-devbox README.md ("Stop / recreate / update", "Post-recreate sanity check")
• the baked skills pi-devbox-firstboot and pi-devbox-environment, and the ci-release-watcher skill from skillset