Diagnose an audio stall
You've noticed BanterLab is running but reporting all tainted iterations, or the live audio bus is silent. This guide walks through what to check, in order.
0. First check: heartbeat vs main-thread
If you're connected to hub-watchdog output, look for a [HUB-EVENT] {"type":"main_thread_stale",...} line. If present, the Qt event loop has deadlocked — every audio symptom downstream of that is secondary. Skip directly to "what wedged the event loop" (often an audio device callback, a codec probe hang, or a crash-handler hang).
If you have access to the raw heartbeat file at <base-dir>/process-health/heartbeat, the table below maps the two ages to the most-likely cause:
| heartbeat mtime | "mta" field age | Diagnosis |
|---|---|---|
| < 2 s | < 2 s | event loop AND watchdog thread both healthy |
| < 2 s | > 5 s | event loop wedged — GUI deadlock or crash-handler hang |
| > 30 s | — | process probably crashed; check for crash-marker.<ts>.txt |
| < 2 s | absent | older binary; main-thread pulse not enabled |
Note: the "mta" field is absent on builds older than the audio-recovery-watchdog Phase 11 landing. If you don't see it at all, that's fine — proceed with the standard checks below.
1. Check the watchdog status
From the chat overlay or the CLI:
hub --opal 'system.watchdog.status'
The result contains three useful fields:
runaway_count— how many threads in this process are currently held above the 90 % CPU threshold for ≥ 5 s. Nonzero means something is eating CPU.heartbeat_age_ms— how long since the heartbeat thread last wrote the heartbeat file. Should be < 2000 ms when the node is healthy.base_dir— confirms which per-node folder this hub is using (relevant under constellation mode).
If runaway_count > 0, list the offending threads:
hub --opal 'system.thread.list-runaway'
You get tid, comm (thread name), cpu_percent, and duration_ms per hot thread.
2. Read the run log
Search the log for [HUB-EVENT] lines. They are structured JSON and grep cleanly:
grep -E '\[HUB-EVENT\]' <run.log>
Common events:
| Event type | Means |
|---|---|
liveness_rebuild |
Audio liveness watchdog rebuilt a device. Automatic recovery. |
audio_lost |
An iteration was tainted with AUDIO_LOST. Retries pending. |
thread_runaway |
A specific thread sustained > 90 % CPU. |
escalation |
Three consecutive AUDIO_LOST iterations — run aborted. |
checkpoint_save |
RL policy auto-saved to <rlpol-path>.checkpoint. |
checkpoint_resume |
Resumed from a previous checkpoint at start. |
3. Force an audio device rebuild
If the chain seems stuck but the watchdog hasn't caught up:
hub --opal 'banter.audio.rebuild-force node=Bob'
Omit the node arg to rebuild all node audio chains. This is destructive in the sense that it tears down and re-opens the audio device — any in-flight frame is lost — but recoverable.
4. Resume from a checkpoint
If BanterAutoTest aborted with AUDIO_LOST, the policy was saved to <rlpol-path>.checkpoint. To resume:
- Confirm the file exists:
ls -la <test-set>.rlpol.checkpoint - Launch BanterLab in the same test-set context.
- On start, BanterAutoTest checks whether the checkpoint is newer than the named
.rlpol. If yes and the action-space fingerprint matches, it loads the checkpoint automatically and continues from where it stopped. - If the action space has changed (you edited parameters in the test set), the load is declined and a warning is logged:
BanterAutoTest: checkpoint fingerprint mismatch …. Fall back to running from scratch with the named.rlpol.
5. Force-save the current checkpoint
If you want to capture the in-progress policy before stopping a run:
hub --opal 'banter.checkpoint.save-force'
This writes a [HUB-EVENT] {"type":"checkpoint_save_requested",...} line. The active BanterAutoTest (if any) picks up the request on its next iteration boundary.