Run a long test under hub-watchdog

hub-watchdog is a small subprocess supervisor that wraps any OctoMY node binary, polls its heartbeat, and kills it if it goes silent. Use it for long unattended runs (overnight ablations, week-long tier-3 sweeps) where a deadlock would otherwise burn hours.

What it watches

Three independent liveness signals are emitted by each node binary into <base-dir>/process-health/:

  • heartbeat — written every 1 s by a dedicated thread inside the node. Its mtime tells you when the node last made progress at all.
  • heartbeat.lock — held for the lifetime of the node via QLockFile (flock(2) semantics). Released by the OS on any exit path including SIGKILL.
  • The "mta" (main-thread alive) field inside the heartbeat body — written by a 1 Hz timer on the Qt event loop. If the event loop deadlocks (e.g. the original incident's crash-handler hang), this stops advancing even though the heartbeat thread keeps writing.

hub-watchdog polls all three. The truth table:

Lock held? Heartbeat fresh? mta fresh? Diagnosis
yes yes yes healthy — keep watching
yes yes no event loop wedged (GUI deadlock or crash-handler hang) — escalate
yes no — alive but the dedicated thread froze too — escalate
no — — process exited (clean or crashed) — reap status, exit

Basic usage

hub-watchdog \
    --base-dir ~/.local/share/OctoMY/hub \
    --max-stale-ms 120000 \
    --initial-grace-ms 10000 \
    -- hub --opal --close-on-completion

Everything after -- is the wrapped child command. The watchdog forks/execs it, forwards stdout/stderr, and polls every 10 s.

Flags

Flag Default Purpose
--base-dir <path> required Per-node data directory. The watchdog composes <base-dir>/process-health/{heartbeat,heartbeat.lock} from it.
--max-stale-ms <N> 120000 Heartbeat-file mtime age beyond which the child is considered hung.
--max-mta-stale-ms <N> 30000 Main-thread pulse staleness threshold. The "mta" field in the heartbeat is updated by a 1 Hz timer on the Qt event loop. If it stops advancing (e.g. event-loop deadlock), the watchdog escalates even though the heartbeat file itself stays fresh. Set 0 to disable.
--initial-grace-ms <N> 10000 Window after launch during which a missing heartbeat is normal (startup time).
--poll-ms <N> 10000 How often to check the three liveness signals.

Exit codes

Code Meaning
0 Child exited cleanly with exit code 0.
anything other than 137/139 Child's own exit code.
137 Watchdog had to SIGKILL the child.
139 Internal watchdog error (bad args, child failed to spawn).

In a CI pipeline you can re-launch on exit 137 to recover from a hang. The next launch picks up any .rlpol.checkpoint sidecar BanterAutoTest left behind, so you don't restart from scratch.

Constellation children

Each constellation child has its own --base-dir (a per-personality folder). To watch a specific child:

hub-watchdog --base-dir "~/.local/share/OctoMY MockNode/Alice" -- agent --base-dir "~/.local/share/OctoMY MockNode/Alice"

Multiple watchdog instances on the same machine do not collide because each watches a different base-dir.

What the watchdog does NOT do

  • Restart on its own. Recovery policy is up to the caller (a shell loop, CI, a systemd service). The watchdog only kills.
  • Distinguish "audio chain stuck" from "everything stuck". Use the system.watchdog.status OPAL task while the process is still alive to differentiate.
  • Catch SIGSTOP-induced silence directly. A kill -STOP <pid> freezes the heartbeat thread; the watchdog sees stale heartbeat and escalates after max-stale-ms. That is correct behavior for an unresponsive child.

Reading a crash marker

When a node binary crashes, its signal handler writes a durable marker file at <base-dir>/process-health/crash-marker using only async-signal-safe primitives (open(2), write(2) — no allocation). On next launch, hub-watchdog (or ProcessHealth::start() for direct launches) rotates the marker to crash-marker.<ts>.txt and emits a [HUB-EVENT] {"type":"crash_marker_found",...} line so the prior crash is visible immediately.

The marker is a plain key=value text file:

ts_epoch_ms=1718040123456
signal=6
signal_name=SIGABRT
pid=12345
tid=12350
rip=0x7f8a1234abcd
fault_addr=0x0
frame=0x7f8a1234abcd
frame=0x55a1c89def01
frame=0x55a1c89dee30
exe=/usr/local/bin/hub

To symbolize the raw frame addresses, copy them into the addr2line hint that the crash handler prints to stderr at crash time. Example:

addr2line -e /usr/local/bin/hub -f -C -p 0x7f8a1234abcd 0x55a1c89def01 0x55a1c89dee30

The output gives you function names + file:line locations. If the binary was built with -O0 -g (debug build at default/install-root/...), addr2line produces full symbolic traces. Stripped release binaries give addresses-to-shared-library mappings only.

Rotated marker files accumulate at <base-dir>/process-health/crash-marker.<ts>.txt — they are post-mortem artifacts, not state, so you can delete them at will once you've extracted what you need.

See also

In this section