RL convergence detection

The BanterLab RL convergence detector watches a learning run's score stream and decides when training has plateaued. When all four criteria below pass simultaneously, the current test ends early, its policy file is saved with a per-test sidecar name, and the runner advances to the next test in the set.

This page describes the four criteria, the per-codec defaults, the JSON override schema, and the status strip rendered above the test-set table.

The four criteria

Convergence requires all four checks to pass on the same iteration. Failure of any single check on a subsequent iteration revokes the convergence verdict and re-arms the detector for the next plateau.

Best-score plateau

max(score[-W:]) ≤ max(score[-3W:]) + epsilonBest

No new best score in the last W iterations beats the previous 2W window by more than epsilonBest. Robust to RL exploration noise because each comparison is on the window maximum, not the window mean.

Decode-rate stationarity

stddev(decodeRate_perBucket[-K:]) < deltaDecode

Decode success ratio is averaged into 100-iter buckets; the standard deviation across the last K buckets is the trend metric. A regression that recovers within one or two buckets does not fail this check (a four-bucket window absorbs short dips).

Rolling-mean slope

linregSlope(rollingMean(score, W=100)[-500:]) < sigmaSlope

The slow drift that the per-window best-score check misses. A gradually-rising mean — even when no single window sets a new peak — keeps this criterion violated.

Minimum-iter guard

iterationsSeen ≥ max(minIters, dims × dimMultiplier)

dims is the number of parameters in Learning sweep mode. SimpleBandit needs roughly 25 iterations per learning dimension before its priors carry meaning. Without this floor, the first ~50 iterations can satisfy the other three by accident.

Per-codec defaults

Parameter FSK MCFSK DPSK CSS OFDM
windowW 200 250 300 200 400
epsilonBest 0.005 0.005 0.005 0.005 0.005
kBuckets 4 4 5 4 6
deltaDecode 0.03 0.03 0.04 0.03 0.04
sigmaSlope 0.0001 0.0001 0.00008 0.0001 0.00008
minIters 500 600 1000 500 1500
dimMultiplier 25 30 40 30 50

An unknown codec falls back to the FSK column rather than raising an error — useful when prototyping a new codec before its column is added.

Per-test override

Each test row may carry a "convergence" block that overrides any subset of the defaults:

{
  "name": "rl_ofdm_3k",
  "codec": "OFDM",
  "convergence": {
    "windowW": 300,
    "minIters": 1500,
    "dimMultiplier": 40
  }
}

Fields omitted from the override inherit the per-codec default. The block is propagated through the test-set defaults / per-test diff merge, so a shared override block in defaults is picked up by every test row.

A test-set-level "autoAdvanceOnConvergence": false flag suppresses the auto-advance: the detector still fires, the DB row is still written, but the runner stays on the current test until max iterations. Useful for ablation runs.

Status strip

A single-row strip is rendered above the test-set table while a learning-mode run is active. Five fields, left to right:

Iter 1487/3000 │ Best 0.972 (peak @ 1320) │ Decode 94.5 % (K=4) │ ●●●○ Conv 3/4 │ ETA ~210
Field Meaning
Iter counter Current iteration / total iterations
Best Highest score seen so far, with the iteration on which it was set
Decode Mean decode success rate across the last K buckets
Dots Four-criterion status (in declared order). Filled dot = passing, open dot = violated. Trailing Conv X/4 mirrors the dot count for low-DPI displays
ETA Linear extrapolation from rolling-mean slope. Shows — when the slope is below the noise floor

A transition message appears to the right of the ETA when the detector fires: Converged @ 1702, saving….

Database

When the detector fires, the per-session row in the BanterAutoTest result DB receives two values:

Column Meaning
convergence_iter Iteration on which all four criteria first passed (or the operator force-advanced). NULL when the run reached max iters without convergence
convergence_reason Compact JSON snapshot of the four criteria's pass/fail state, peak iter, peak score, and decode-rate window mean at trigger time

The columns are additive (ALTER TABLE migration). Existing DBs from before this feature open unchanged and report NULL for both columns until a new run lands.

Policy sidecars

A test-set named rl_all_codecs_3k.json containing five test rows rl_fsk_3k, rl_mcfsk_3k, rl_dpsk_3k, rl_css_3k, rl_ofdm_3k produces five sidecar files alongside the test-set on disk:

rl_all_codecs_3k.json
rl_all_codecs_3k__rl_fsk_3k.rlpol
rl_all_codecs_3k__rl_mcfsk_3k.rlpol
rl_all_codecs_3k__rl_dpsk_3k.rlpol
rl_all_codecs_3k__rl_css_3k.rlpol
rl_all_codecs_3k__rl_ofdm_3k.rlpol

Test names with characters outside [A-Za-z0-9_-] are sanitised to _ so the resulting filename is always a legal POSIX path. A subsequent run reloads each row's sidecar at the start of that row, warm-starting the policy.

When a test-set carries only a single row (or the row has an empty name), the detector falls back to the test-set-level filename rl_all_codecs_3k.rlpol — the legacy path used before per-test sidecars existed.

OPAL tasks

Task Purpose
banter.policy.list For each test row in the active test-set: name, derived sidecar path, whether the file exists, fingerprint match status
banter.policy.delete Remove a per-test sidecar. Refuses to delete files outside the test-set's directory
banter.convergence.status Live detector state — per-criterion pass/fail with values vs thresholds, current iter, peak score, decode-rate snapshot

The banter.policy.save and banter.policy.load tasks predate this feature and continue to target the current row's sidecar when a test-set is active.

In this section