Run a multi-codec test-set with auto-advance

This page walks through running a test-set that trains an RL policy on each of the five codecs (FSK, MCFSK, DPSK, CSS, OFDM) in turn, with the convergence detector advancing automatically between tests. By the end you will have five policy sidecar files, one per codec, and a result database with the convergence iteration recorded for each.

Prerequisites

  • A built hub binary (qbs build -f project.qbs -p hub).
  • A test-set file with at least one test row whose parameter set includes one or more Learning-mode parameters. The reference test-set is test/testBanter/specs/tier4_rl/rl_all_codecs_3k.json.

Walkthrough

1. Launch BanterLab with the test-set

hub --activity BanterLabActivity \
    --open test/testBanter/specs/tier4_rl/rl_all_codecs_3k.json

The BanterLab window opens, the test-set table populates with five rows (one per codec), and the iteration counter sits at 0/3000 waiting for you to start.

2. Start the run

Click Start test set in the toolbar. The first row (rl_fsk_3k) becomes active, the parameter table snaps to the FSK defaults, and a single-row status strip appears between the toolbar and the test-set table:

Iter 0/3000 │ Best — │ Decode — │ ○○○○ Conv 0/4 │ ETA —

3. Watch the strip evolve

As the run progresses, fields update on every iteration:

  • Iter climbs from 0 toward 3000.
  • Best fills in as soon as the first non-zero score lands, with the iteration at which the peak was set.
  • Decode appears once the first 100-iter bucket completes.
  • Dots flip from open ○ to filled ● as each criterion's window saturates and its threshold is met. The trailing Conv X/4 text mirrors the dot count.
  • ETA displays a rough iteration count for the next 0.05 score gain — or — while the rolling-mean slope is below the noise floor.

A typical FSK run hits all four criteria around iter 1700 ± 100. When that happens, the strip briefly displays Converged @ 1702, saving…, the policy is saved to rl_all_codecs_3k__rl_fsk_3k.rlpol next to the test-set file, the session row in the result DB is stamped with convergence_iter = 1702 and a convergence_reason JSON snapshot, and the runner advances to rl_mcfsk_3k.

4. Repeat for every row

The MCFSK / DPSK / CSS / OFDM rows behave the same way, each producing its own sidecar:

rl_all_codecs_3k.json
rl_all_codecs_3k__rl_fsk_3k.rlpol
rl_all_codecs_3k__rl_mcfsk_3k.rlpol
rl_all_codecs_3k__rl_dpsk_3k.rlpol
rl_all_codecs_3k__rl_css_3k.rlpol
rl_all_codecs_3k__rl_ofdm_3k.rlpol

OFDM typically takes the longest (it has the highest minIters floor at 1500 and the largest dimension multiplier).

5. Verify in the result database

The result DB lives at default/banter-results/rl_all_codecs_3k__<git>.db. With the SQLite CLI:

sqlite3 default/banter-results/rl_all_codecs_3k__*.db \
    "SELECT config_name, convergence_iter, total_runs FROM sessions ORDER BY id;"

For a successful run you should see five rows, each with a non-NULL convergence_iter strictly less than total_runs. A NULL convergence_iter means that test ran out the full 3000 iterations without satisfying all four criteria — usually because the rolling-mean slope stayed above sigmaSlope or the decode-rate stddev never settled.

Operator overrides

Force the next row early

If a test is clearly stuck — operator inspection shows the policy has plateaued at a low score but one criterion is preventing convergence — the strip's Force advance button (planned, see implementation status) skips to the next row and writes convergence_reason = {"forced":true} to the DB.

While the button is not yet available, the same effect is achieved by stopping and restarting the run from the next row.

Disable auto-advance for ablation

Set "autoAdvanceOnConvergence": false at the top level of the test-set JSON. The detector still fires and records convergence_iter, but the runner stays on the current test until max iterations. Use this when you want to compare "with convergence cutoff" vs "ran to completion" without restarting.

Re-run a single row

Delete the row's sidecar — either with the OPAL task or directly on disk:

rm rl_all_codecs_3k__rl_ofdm_3k.rlpol

The next run of the test-set starts that row from a fresh policy. Other rows continue to warm-start from their preserved sidecars.

Troubleshooting

Symptom Likely cause Fix
Detector never fires; all four dots filled, but Conv 4/4 text shows the moment then dots flicker back to 3 Decode-rate stddev oscillating around deltaDecode. The detector requires all four passing on the same iteration Raise deltaDecode to 0.05 in the row's "convergence" block
Run converges at iter ~200 with a low best score minIters floor is too small for the dimension count Increase minIters or dimMultiplier in the row's "convergence" block
Mismatched fingerprint on sidecar reload The test row's learning-param set changed between runs (added/removed/renamed a Learning param) Delete the sidecar to force a fresh start, or revert the param change
In this section