Check a worker farm with the debug workloads

A worker farm fails in ways a single machine never shows: a worker crashes mid-job, a link goes quiet, an answer arrives malformed or too late. The debug workloads make each of those faults on demand, on a real worker, so you can see that the farm survives it before real work meets it.

They are ordinary batch jobs of type debug.*. Every hub runs them and offers them to the coordinator it works for, in release builds too.

Before you start

The scope batch.debug. Nobody holds it by default. Grant it on the Permissions screen to the agent or person who runs the checks. A debug job can end a worker's process on purpose, so treat the grant like a key.
A worker. A machine the fleet bought, or a hub you paired. For the faults that act on a link (debug.drop, debug.corrupt mode late), the job must run on a worker, not on the coordinator.

Submit a debug job

Use batch.debug.submit. batch.job.submit refuses a debug.* type, so the scope cannot be skipped.

{"task": "batch.debug.submit",
 "args": {"type": "debug.sleep", "params": {"seconds": 30}, "where": "remote"}}

where: "remote" sends the job to a worker, never to this node.
where: "local" keeps it on this node.
Leave where out to let the dispatcher choose.

Then follow it with batch.job.get until it finishes.

Which job proves what

Type	What it does	Parameters	What a pass proves
`debug.echo`	Returns its inputs as its outputs, and its params as values	none	The artifact plane both ways, byte for byte
`debug.produce`	Writes an output of a chosen size and content	`bytes` (any size that fits on the worker's disk), `seed`	Outputs reach the coordinator, and a large one resumes
`debug.sleep`	Waits, reporting progress each second	`seconds`	Leases renew while a job runs
`debug.burn`	Loads the worker's CPU, memory and disk	`cpu` (cores), `memoryMiB`, `diskMiB`, `seconds`	Admission: the worker refuses more than it offers
`debug.crash`	Ends the worker's process	`mode`: `segfault`, `abort`, `exception`, `exit`; `code`; `once` (crash on the first placement only, so the run placed again finishes)	The coordinator places the job again
`debug.hang`	Stops, and ignores cancel	`mode`: `busy`, `deadlock`, `sleep`, `gui`; `seconds`; `once` (hang on the first placement only)	Watchdogs, and lease expiry for `gui`
`debug.fail`	Fails	`mode`: `once`, `n`, `always`; `n`; `retryable`	Retry and the error text, end to end
`debug.corrupt`	Answers wrongly	`mode`: `result` (a malformed result), `late` (a result after the lease expired), `hash` (output bytes that do not match their hash), `truncate` (fewer bytes than declared); `seconds`	The coordinator refuses each by name, and handles a late result
`debug.drop`	Breaks the link	`mode`: `session` (closes it), `silence` (stops talking); `seconds`. With `bytes` and `seed`, it first writes an output that size, and breaks the link `times` times (at most 10), one every `everySeconds`, while the output travels	Re-placement, the session coming back, and a large output that resumes
`debug.replay`	Builds a recorded fault again	`case`	A fixed fault stays fixed

Every duration is capped at one hour. A burn asks for the CPU and memory it burns, so the worker's own admission refuses a burn larger than its capacity.

Replay a recorded fault

Each fault the farm has shown in a real run is a named case. List them:

{"task": "batch.debug.cases", "args": {}}

Each case names the fault, what it builds and what proves the fix. Run one on a worker:

{"task": "batch.debug.submit",
 "args": {"type": "debug.replay", "params": {"case": "stuck-exact-proxy"}, "where": "remote"}}

A case that passes finishes done. A case that fails says the fault is back, and why.

What a worker offers

A worker runs only the job types it offers, and refuses the rest when the coordinator asks. Read or change the list with batch.node.offer on the worker:

{"task": "batch.node.offer", "args": {"types": "opal:node.probe,construct.mesh"}}

This list is apart from the types a node sends to its workers (batch/remote/forwarded-types). A machine the fleet buys sets its offered types from the fleet's job types when it joins. The debug workloads are always offered on a hub.