RL Convergence Detection

The BanterLab RL convergence detector watches a learning run's score stream and decides when training has plateaued. When all four criteria below pass simultaneously, the current test ends early, its policy file is saved with a per-test sidecar name, and the runner advances to the next test in the set.

This page describes the four criteria, the per-codec defaults, the JSON override schema, and the status strip rendered above the test-set table.

The four criteria

Convergence requires all four checks to pass on the same iteration. Failure of any single check on a subsequent iteration revokes the convergence verdict and re-arms the detector for the next plateau.

Best-score plateau

max(score[-W:]) ≤ max(score[-3W:]) + epsilonBest

No new best score in the last W iterations beats the previous 2W window by more than epsilonBest. Robust to RL exploration noise because each comparison is on the window maximum, not the window mean.

Decode-rate stationarity

stddev(decodeRate_perBucket[-K:]) < deltaDecode

Decode success ratio is averaged into 100-iter buckets; the standard deviation across the last K buckets is the trend metric. A regression that recovers within one or two buckets does not fail this check (a four-bucket window absorbs short dips).

Rolling-mean slope

linregSlope(rollingMean(score, W=100)[-500:]) < sigmaSlope

The slow drift that the per-window best-score check misses. A gradually-rising mean — even when no single window sets a new peak — keeps this criterion violated.

Minimum-iter guard

iterationsSeen ≥ max(minIters, dims × dimMultiplier)

dims is the number of parameters in Learning sweep mode. SimpleBandit needs roughly 25 iterations per learning dimension before its priors carry meaning. Without this floor, the first ~50 iterations can satisfy the other three by accident.

Per-codec defaults

Codec windowW epsilonBest kBuckets deltaDecode sigmaSlope minIters dimMultiplier
FSK 200 0.005 4 0.03 0.0001 500 25
MCFSK 250 0.005 4 0.03 0.0001 600 30
DPSK 300 0.005 5 0.04 0.00008 1000 40
CSS 200 0.005 4 0.03 0.0001 500 30
OFDM 400 0.005 6 0.04 0.00008 1500 50

An unknown codec falls back to the FSK row rather than raising an error — useful when prototyping a new codec before its row is added.

Per-test override

Each test row may carry a "convergence" block that overrides any subset of the defaults:

{
  "name": "rl_ofdm_3k",
  "codec": "OFDM",
  "convergence": {
    "windowW": 300,
    "minIters": 1500,
    "dimMultiplier": 40
  }
}

Fields omitted from the override inherit the per-codec default. The block is propagated through the test-set defaults / per-test diff merge, so a shared override block in defaults is picked up by every test row.

A test-set-level "autoAdvanceOnConvergence": false flag suppresses the auto-advance: the detector still fires, the DB row is still written, but the runner stays on the current test until max iterations. Useful for ablation runs.

Status strip

A single-row strip is rendered above the test-set table while a learning-mode run is active. Five fields, left to right:

Iter 1487/3000 │ Best 0.972 (peak @ 1320) │ Decode 94.5 % (K=4) │ ●●●○ Conv 3/4 │ ETA ~210
Field Meaning
Iter counter Current iteration / total iterations
Best Highest score seen so far, with the iteration on which it was set
Decode Mean decode success rate across the last K buckets
Dots Four-criterion status (in declared order). Filled dot = passing, open dot = violated. Trailing Conv X/4 mirrors the dot count for low-DPI displays
ETA Linear extrapolation from rolling-mean slope. Shows — when the slope is below the noise floor

A transition message appears to the right of the ETA when the detector fires: Converged @ 1702, saving….

Database

When the detector fires, the per-session row in the BanterAutoTest result DB receives two values:

Column Meaning
convergence_iter Iteration on which all four criteria first passed (or the operator force-advanced). NULL when the run reached max iters without convergence
convergence_reason Compact JSON snapshot of the four criteria's pass/fail state, peak iter, peak score, and decode-rate window mean at trigger time

The columns are additive (ALTER TABLE migration). Existing DBs from before this feature open unchanged and report NULL for both columns until a new run lands.

Policy sidecars

A test-set named rl_all_codecs_3k.json containing five test rows rl_fsk_3k, rl_mcfsk_3k, rl_dpsk_3k, rl_css_3k, rl_ofdm_3k produces five sidecar files alongside the test-set on disk:

rl_all_codecs_3k.json
rl_all_codecs_3k__rl_fsk_3k.rlpol
rl_all_codecs_3k__rl_mcfsk_3k.rlpol
rl_all_codecs_3k__rl_dpsk_3k.rlpol
rl_all_codecs_3k__rl_css_3k.rlpol
rl_all_codecs_3k__rl_ofdm_3k.rlpol

Test names with characters outside [A-Za-z0-9_-] are sanitised to _ so the resulting filename is always a legal POSIX path. A subsequent run reloads each row's sidecar at the start of that row, warm-starting the policy.

When a test-set carries only a single row (or the row has an empty name), the detector falls back to the test-set-level filename rl_all_codecs_3k.rlpol — the legacy path used before per-test sidecars existed.

OPAL tasks

Task Purpose
banter.policy.list For each test row in the active test-set: name, derived sidecar path, whether the file exists, fingerprint match status
banter.policy.delete Remove a per-test sidecar. Refuses to delete files outside the test-set's directory
banter.convergence.status Live detector state — per-criterion pass/fail with values vs thresholds, current iter, peak score, decode-rate snapshot

The banter.policy.save and banter.policy.load tasks predate this feature and continue to target the current row's sidecar when a test-set is active.

In this section