Fleet
Configuration keys, lifecycle states, reconcile classes and the fleet and oci tasks
Fleet
Money first. A fleet rents machines that bill by the hour until they are destroyed, busy or not. Three brakes stand in front of every purchase: the spend limits (an hourly rate and a daily spend, checked before the provider is called, and a limit of 0 refuses everything), a confirmation on every purchase, and the reaper, which destroys only machines tagged for your own fleet. Set the limits before anything else.
Reference for a fleet's configuration, the states a machine passes through, what
reconcile reports, and every task in the fleet and oci families. For the
workflow — binding a fleet to a hub, buying a machine, watching it, reaping it —
see Rent machines to run the work.
Configuration
A hub keeps its fleet in its own settings. fleet-ctl keeps one in its state
file. Both read the same configuration.
| Setting | Hub screen | fleet-ctl fleet configure |
What it is |
|---|---|---|---|
| Provider | Provider | --provider |
Who the machines are rented from. This build offers hetzner |
| API token | API token | --token-stdin, per run |
The provider's token. A hub stores it sealed to its own identity; fleet-ctl never stores it |
| Region | Region | --region |
Where machines are bought, as fsn1 |
| OS image | OS image | --os-image |
The provider image a machine boots, as debian-12 |
| Coordinator | Coordinator | --coordinator |
Where a new machine reports in, as host:port. Behind a router, the router's public address and the forwarded port |
| Coordinator port | Coordinator port | — | The port this hub listens on while it runs a fleet: UDP for comms and TCP for the hub bundle. Default 18426. Applies when saved, with no restart |
| Coordinator identity | — | --coordinator-nid, --coordinator-key |
The hub's node id and public key, which a new machine trusts. A hub fills both from its own identity |
| Executor image | Executor image | --image |
The image a machine runs, as registry/repository:tag. Empty: the machine fetches this hub, as a bundle, from this hub |
| Job types | Job types | — | The job types this hub sends to its machines, and every machine takes. The node setting batch/remote/forwarded-types, comma separated, for example opal:node.probe. Applies with no restart |
| Workload | — | --workload |
A name for what the machines run. Default executor |
| SSH keys | SSH keys | --ssh-key, repeatable |
Key names at the provider that a person may log in with. Usually none |
| Spend limit per hour | Spend limit per hour | --max-hourly |
The most the fleet may bill per hour |
| Spend limit per day | Spend limit per day | --max-daily |
The most the fleet may spend per UTC day |
| Currency | Currency | --currency |
The currency of both limits. Default EUR |
The fleet can list machines and read the catalog with only a provider and a
token. It refuses to buy until the region, OS image and coordinator are set, and
the screen says which are missing. A hub can change every setting except the
token with the fleet.configure task, so a hub with no screen can run a fleet.
The pool
A pool keeps the fleet between a floor and a ceiling with no one at the screen. It is part of the fleet's record, so it survives a restart.
| Key | Fleet screen | fleet.pool |
What it is |
|---|---|---|---|
| Floor | Pool: at least | minWorkers |
Machines the fleet keeps, whatever the demand: a standing farm |
| Ceiling | at most | maxWorkers |
Machines the fleet may buy for waiting jobs |
| Idle drain | drain idle after | idleDrain, in seconds |
How long a machine above the floor may hold no job before it is drained and destroyed. Default 10 minutes |
| Type | — | type |
The machine type the floor buys. Empty: the cheapest the region sells for the workload |
The floor and the ceiling the same is a fixed farm. A floor of 0 is pure demand. A floor and a ceiling of 0 is no pool: machines are bought by hand. The machines that the pool bought are then drained and destroyed, each when its jobs finish. A machine bought by hand stays.
Once a minute the pool's loop compares the fleet with the pool. It buys what the waiting jobs need, up to the ceiling, and what the floor lacks. It drains a machine above the floor that has been idle for the idle drain, the longest idle first, and every machine above the ceiling. A drained machine with no job left is destroyed. Every purchase passes the spend limits, and nothing is bought while the fleet is paused.
The loop buys only with the permission fleet.autoscale. Grant it on the Permissions screen.
Nobody has it by default, because the loop spends money with no one there to approve each
machine. Without it, the pool is shown, and fleet.list, fleet.cost and fleet.pool say that the
loop does not run and why.
A machine the fleet buys runs in Drone mode: it exists to run jobs and offers its whole
capacity. It keeps that mode across a restart, in the node setting batch/node/mode.
How a machine joins
A machine has nobody beside it, so it joins by itself:
- At purchase, the fleet makes a one-time join token and keeps only its hash. The bootstrap that the provider hands the machine carries the token, the coordinator's address, and the coordinator's public key.
- With no executor image set, the machine boots plain Debian and fetches the hub bundle from the coordinator, over plain HTTP on the coordinator port. It checks the bundle's SHA-256, which the bootstrap carries, before it unpacks it. The coordinator serves the bundle only while a machine is waiting to join.
- The machine runs
hub --head-less --join-file /run/octomy/fleet.json. The file is on/run, a memory filesystem, so the token never reaches the disk through it. It makes its own identity, so its private key never leaves it. It sends a join request with the token, encrypted to the coordinator's key. - The coordinator accepts the token once, records the machine's identity, and trusts its key. The token is spent. A refused request gets no answer.
- The two hubs connect as any two paired nodes do. The machine deletes the join
file, and the copies of its user data that cloud-init kept on the disk. Its
executor reports in, and the machine is
Joined.
A token expires after one hour. A machine that has not joined by then never will, and reconcile reports it as stillborn.
The router. The coordinator port must reach the hub from the internet: forward it, UDP and TCP, from the router's public address to the hub.
The hub bundle. A hub builds the bundle of its own binary when it is missing
or older than the binary, into fleet/bundles/ in its data directory, and names
it hub-linux-<architecture>.tar.gz. A machine of an architecture with no bundle
is refused before the provider is called. To rent arm64 machines from an x86
hub, put an arm64 hub's bundle there.
Where a hub stores it. Three keys in the hub's settings: fleet/config (the
configuration above, as JSON, no secret in it), fleet/provider-token (the token,
sealed to the node — a copy of the settings file does not give it away), and
fleet/record (every machine the fleet has bought, the spend ledger and the
limits). A token another node sealed, or one stored unsealed, is refused, and the
screen asks for it again.
The life of a machine
| State | Bills | Means |
|---|---|---|
Requested |
no | The fleet decided to buy it and wrote it down. No answer from the provider yet |
Provisioning |
yes | The provider accepted and is building it |
Booting |
yes | Running, and not reported in |
Pulling |
yes | Fetching its image |
Joined |
yes | Its executor reported to the coordinator. Jobs reach it |
Draining |
yes | Takes no new work. Finishes what it holds |
Terminating |
yes | Destroy was called, and the provider may still be billing |
Gone |
no | The provider no longer lists it |
Reclaimed |
no | The provider took it back, as it may with a spot machine |
A machine that bills and has not joined after twenty minutes gets a warning in the fleet activity, and so does a reclaimed one. Hetzner bills every started hour, and the fleet's own count rounds the same way.
What reconcile reports
Reconcile compares what the provider lists under your fleet's tag with what the fleet has written down.
| Finding | Means | What happens |
|---|---|---|
| Orphan | The provider has a machine with your tag that the fleet has no record of | Reported. Destroyed after 30 minutes when you reap, or at once with Destroy |
| Vanished | The fleet has a machine the provider no longer lists | Marked Gone |
| Stillborn | A machine of yours whose executor never reported | Reported. Destroyed after 20 minutes when you reap — but only by a hub, which can see its executors |
| Ghost | An executor reports, and its machine is gone | The executor is forgotten |
| Foreign | A machine with another fleet's tag, or none | Reported by name. Never destroyed, at any age |
| Reclaimed | The provider took the machine back | Marked Reclaimed. Its work returns to the queue when its leases expire |
fleet-ctl cannot see executors, so it reports a stillborn machine and never reaps
one: from the command line, a busy machine and a dead one look the same.
The fleet tasks
Every button in the fleet activity runs one of these, as a plan or a model would, so a click meets the same tier and the same permission prompt.
| Task | Tier | Parameters | Does |
|---|---|---|---|
fleet.list |
operational | — | Every machine, with its state, cost so far and executor id. The pool, how many machines it holds, and whether its loop runs |
fleet.instance |
operational | id |
One machine in full, with every state it passed through |
fleet.cost |
operational | — | The hourly rate and today's spend, each beside its limit, and the pool |
fleet.demand |
operational | — | The work waiting for a machine, and what the strategy would buy |
fleet.reconcile |
operational | — | Names every difference. Destroys nothing |
fleet.scale |
fleet-spend | count 1–16, type, spot |
Buys count of type, or up to count of what the strategy proposes. Asks every time |
fleet.drain |
secured | id |
The machine takes no new work and finishes what it holds. It still bills |
fleet.destroy |
secured | id, discard |
Destroys a machine. Refuses one with work out unless discard is set |
fleet.reap |
secured | — | Destroys every orphan and stillborn machine of this fleet past its grace period |
fleet.pause |
secured | paused |
Stops buying. Running machines keep running and keep billing |
fleet.pool |
fleet-spend | any of minWorkers, maxWorkers, idleDrain, type |
Sets the pool, only the keys given. With none, answers the pool. It decides what the loop may spend, so it needs the owner |
fleet.configure |
fleet-spend | any of provider, region, osImage, coordinator, listenPort, image, workload, sshKeys, maxHourly, maxDaily, currency, jobTypes, and for a container host containerEngine, containerRepository, containerCa, containerCertificate, containerKey |
Changes only the settings given, keeps the token, and binds again with no restart. It can raise the spend limits, so it needs the owner. containerKey is sealed to the hub and never stored with the other settings |
fleet.run |
operational on named machines; fleet-spend with count |
program (or image on a container host), and instances or count and type; optionally arguments, files, timeout, outputLimit |
Starts a program on each machine: a run set, one row per machine. With count it buys the machines first, asks every time, and each run starts when its machine answers |
fleet.run.status |
operational | set or run, poll |
One row per run: its state, and for a finished run its exit code, signal and the ends of its output |
fleet.run.collect |
operational | run, path, out, maxBytes |
Copies one file the run wrote in its folder to out, and answers its digest |
The tiers. operational needs an active session on an active node. secured
needs a file-scoped session. fleet-spend needs the owner, because a purchase
starts a charge that outlives the session; no standing approval covers the next
one.
Running a program
fleet.run reaches a machine over SSH, spoken by the hub itself: no ssh
program runs. The first connection is already verified. When the fleet buys a
machine it makes the machine's host key, keeps the public half in the
machine's record, and hands the pair to the machine in its first-boot
configuration, so any other key is refused, by name. The fleet logs in with a
key of its own, made once and stored sealed to the hub.
On the machine, everything lives under /var/lib/octomy-runs. A program is
staged by its digest into store/, so the same program is never sent twice,
and each run gets a folder in runs/ with its stdout, stderr and
result.json. A small supervisor that the hub ships beside itself,
fleet-run-supervisor, starts the program with its arguments exactly as given
(no shell reads them), stops it at its timeout, and records how it ended.
| A run's state | Means |
|---|---|
waiting |
Its machine was bought for the set and does not answer yet. It starts when the machine does, or fails after 20 minutes |
starting, running |
On the machine |
finished |
Ended: an exit code, or a signal, or stopped at its timeout |
failed |
Never started. The row says which step refused and why |
lost |
Its machine went away mid-run, or the run's supervisor died. Final, never a hang |
One machine that fails never fails the set: a set of twenty reports twenty rows.
On a container host
A fleet can be bound to a container host instead: a machine running Docker
Engine, reached through its API on port 2376 with mutual TLS. Set
containerEngine (as https://203.0.113.9:2376), containerRepository (the
registry the host pulls from, as 203.0.113.9:5000/octomy/runs),
containerCa, containerCertificate and containerKey (all PEM) with
fleet.configure. Clear containerEngine to go back to SSH. The fleet uses
one or the other, never both.
There, the program is an image. Push it to the repository with
fleet-ctl image push, which takes --tls-cert, --tls-key and --tls-ca
for a registry that asks for a client certificate, and name its digest as
image in fleet.run. The host pulls an image only once: a second run of the
same digest starts at once. Each run is a container named octomy-run-<run>,
with its input files in /run. The fleet keeps the newest 1 000 finished runs;
when an older one leaves, its container is removed, as a run's folder is on a
machine.
The engine's init runs as the container's first process, so a program's
signals end it as they would on a machine: an abort reads as exit 134, as it
does over SSH. The engine reports exit codes only, so a signal shows as 128
plus its number.
The oci tasks
| Task | Tier | Parameters | Does |
|---|---|---|---|
oci.inspect |
operational | layout and ref, or reference |
Reads an image's manifest, config and layer digests |
oci.build |
operational | layout, ref, root, platform, and optionally prefix, entrypoint, cmd, env, baseLayout, baseRef |
Builds an image from a folder into a local layout. The same files give the same digest. No network |
oci.push |
secured | layout, ref, reference |
Pushes an image from a layout to an image registry |
oci.copy |
secured | source, target |
Copies an image between image registries, skipping what the target holds |
fleet-ctl
fleet-ctl fleet <verb> drives a fleet from the command line, with its state in
fleet-ctl.ini in the config folder, or wherever --state points.
| Verb | Calls the provider | Does |
|---|---|---|
configure |
no | Writes the configuration and the limits |
list |
no | The record, as JSON |
cost |
no | The rate, today's spend and the total |
pause, resume |
no | Stops or restarts buying |
catalog |
yes | The types sold, where, and at what price |
scale |
yes | Says what it would buy. With --yes, buys --count of --type |
refresh |
yes | Follows the provider and counts the cost |
destroy |
yes | Destroys --id. --discard for one with work on it |
reconcile |
yes | Reports every difference. --reap destroys orphans past --grace-minutes |
Every verb that calls the provider needs the token on standard input with
--token-stdin. Every run appends one line of what the fleet costs to
fleet-ctl.ini.usage.log beside the state file.