Audio recovery and the watchdog system

OctoMY's audio pipeline runs underneath everything that talks acoustically: BanterLab, Banter chat, ablation tests. When the audio pipeline silently stops delivering samples, every higher-level subsystem keeps running on stale input, often for hours, before anyone notices. The watchdog system exists to make that failure mode loud and recoverable.

What can go wrong with audio

The motivating incident: an overnight ablation test launched at 21:00, intending 20 000 iterations. Eleven hours later the harness reported 19 999 of 20 000 iterations "complete" — but every iteration after #3000 was tainted, decoded zero bytes, and the audio bus was silent. The reinforcement-learning controller had been training on noise for ~8 hours.

The trigger that night was a stuck codec probe pinning four hardware-encoder threads at 100 % CPU. The same starvation pattern can be produced by any spinning thread (a driver bug, a misbehaving plugin, a runaway script). The structural lesson is what matters: every safety net in the system before this work used a QTimer on the GUI thread, so a single starvation event collapsed all of them simultaneously.

What the watchdog does

Each node binary now runs three independent monitors, all driven by dedicated OS threads (not Qt timers):

  • Audio liveness watchdog. Inside each AudioInputNode and AudioOutputNode, a dedicated thread polls the sample counter every 250 ms. If the counter hasn't advanced for 3 s, the watchdog rebuilds the underlying audio device. This survives GUI-thread starvation because the polling thread is scheduled by the OS, not the Qt event loop.
  • Thread runaway detector. A process-wide singleton (ThreadRunawayDetector) walks /proc/self/task/* once per second on Linux, computing per-thread CPU usage. Any thread sustaining > 90 % CPU for ≥ 5 s emits a threadRunaway signal; ≥ 30 s emits processHealthDegraded. The detector itself runs on its own dedicated thread.
  • Heartbeat + lock. A ProcessHealth singleton writes a timestamped heartbeat file every second to <node-folder>/process-health/heartbeat and holds a QLockFile next to it. An external hub-watchdog tool can read both — fresh heartbeat means "making progress", held lock means "process alive". If the process crashes or is killed, the OS releases the lock automatically (flock(2) semantics).

What you see when something fails

Three escalation paths surface in BanterLab and run logs:

  1. Liveness rebuild. When the audio liveness watchdog rebuilds a device, you'll see BanterAutoTest: stream stalled silently … rebuilding QAudioSource (rebuild #N) in the log and a [HUB-EVENT] {"type":"liveness_rebuild",...} event line. The rebuild is automatic — no operator action needed.
  2. Iteration tainted. Iterations where the rebuild didn't fully recover get an AUDIO_LOST[Nticks] tag in the run log. Tainted iterations are retried up to 3 times before being accepted into the reinforcement-learning feed.
  3. Run aborted. If three consecutive iterations exhaust retries into AUDIO_LOST, BanterAutoTest saves the current RL policy to a .rlpol.checkpoint sidecar and emits aborted("AUDIO_LOST"). BanterLab shows ⚠ Test aborted: AUDIO_LOST and stops. The next run can resume from the checkpoint instead of starting from scratch.

Two heartbeats

Audio is not the only thing that can stop. The Qt event loop itself can deadlock — most dramatically when a crash handler tries to write a stack trace and gets stuck inside the libc allocator. To detect this, every node binary now emits two independent liveness signals:

  • The dedicated-thread heartbeat at <base-dir>/process-health/heartbeat. This is what survived the original incident: the worker thread keeps writing whatever else is going on.
  • A 1 Hz "main-thread alive" pulse (the "mta" field inside the same heartbeat file). This is updated by a QTimer on the Qt event loop — if the event loop dies, this stops advancing even though the heartbeat file keeps appearing fresh.

hub-watchdog checks both. A healthy node has both fresh. A node where the event loop has deadlocked has fresh heartbeat but stale mta — exactly the case that the original incident left undetected. See Diagnose an audio stall for the full decision table.

Crashes are also handled gracefully now: the signal handler uses only async-signal-safe primitives (no malloc, no dlopen at signal time), writes a durable marker file at <base-dir>/process-health/crash-marker, and exits cleanly. On the next launch, hub-watchdog rotates the marker to a timestamped filename and emits a [HUB-EVENT] crash_marker_found line so operators see prior crashes immediately. See Run with hub-watchdog for the marker schema and addr2line workflow.

What the operator can do

Most of the time, nothing — the system recovers itself. When it doesn't:

See also

In this section