Run tests

Run the OctoMY™ test suites with kiln test, read their results, and debug one suite on its own. To add a suite, see Write a test.

Pro Tip

Run tests with -v2 for verbose output - it shows each check as it runs, making it easier to spot where failures occur. Use -functions to list all available test functions before running.

---

How the test system works

Each suite is one folder, test/test<Name>/, with one Qt Test class and one test.qbs. About 1 100 suites exist.

// test/testMyClass/test.qbs
Test {
    localPath: path
    category: "unit"
    expectedPassed: 5
}

The Test { } item and its properties are in the qbs reference. The properties that matter most:

• category says what the suite needs to run, and so whether it is built at all. The default build compiles unit, integration, widget and opal. It leaves out legacy, interactive, hardware, perf, live_audio and live_net. A suite in a category that is left out is not compiled and not run. See the category table.
• expectedPassed is the exact number of passed slots. A run with more or fewer fails, so a deleted case cannot go unseen. initTestCase and cleanupTestCase count as slots.
• shared makes the suite a member of the shared test binary, testsLogical. Most purely logical suites are members. They link into one program instead of one program each, and kiln test runs them as parallel worker processes. See The shared binary.

One program runs the suites everywhere: kiln test. It holds each suite to its contract (its timeout, its platform and its pass count) on a developer's machine, in the lint gate and on the build farm.

Choose which categories to build

# Only unit tests
qbs build project.testCategories:'["unit"]'

# The default set, plus the benchmarks
qbs build project.testExcludeCategories:'["legacy","interactive","hardware","live_audio","live_net"]'

# Only the suites that run without a display
qbs build project.headlessTestsOnly:true

kiln test runs only the suites that the build compiled. A run that keeps a category that the build left out finds no programs for it, and reports those suites as NOT BUILT. Build them first, for example with project.testExcludeCategories.

---

Run the suites

kiln is a program of the tree. A build compiles it by default and installs it as <build dir>/install-root/usr/local/bin/kiln, so the build that made the suites also made the runner, from the same commit. The commands below write kiln for that program, and default for the build folder.

--prebuilt <build dir> runs the programs that a qbs build left in that folder, and builds nothing. kiln decides the features with the Qt installation that the qbs build used, so you do not need qt.qmake in .kiln.local.

# Every built suite, as CI runs them
kiln test --prebuilt default --profile ci

# Every built suite, with no profile
kiln test --prebuilt default

Choose a profile

A profile names one kind of run. It is a [profile=<name>] section of the root .kiln, and it sets the keys of the run: which suites, the timeout cap, the result files, the display and the number of parallel suites. An option on the command line wins over the profile.

# The profiles that this checkout declares, one per line
kiln test --list-profiles

A profile that does not exist fails the run with exit code 3. It never falls back to the default, because a typing error would then run the wrong set and report a pass. A .kiln.local can add profiles of its own for one machine, and --list-profiles lists those too.

The GPU suites need software Vulkan and an X server. The script prepares the host (Mesa lavapipe and Xvfb), then runs kiln test --profile gpu:

integration/run-gpu-integration-tests.sh

The key reference of each profile is in the kiln reference.

Run one suite or a few

Give the folder of each suite. A suite that you name runs whatever its category.

# One suite
kiln test --prebuilt default test/testCommsChannel

# Several suites
kiln test --prebuilt default test/testCommsChannel test/testCryptHash

# One category, or several, comma-separated
kiln test --prebuilt default --category unit

# Only the guard suites
kiln test --prebuilt default --guard-only

# Give arguments to every suite
kiln test --prebuilt default --category unit -- -maxwarnings 0

--category and --guard-only replace the whole selection of a profile.

A member of the shared binary runs in testsLogical, even when you name it alone. kiln test never runs the member's own program, which a former build can have left in the folder and which can be stale.

Run suites in parallel

# 8 suites at the same time
kiln test --prebuilt default --profile ci --jobs 8

# Half the cores
kiln test --prebuilt default --profile ci --jobs auto

# A random order from seed 1
kiln test --prebuilt default --profile ci --jobs auto --shuffle 1

--jobs <n> runs n suites at the same time, each in its own process. A suite that needs a display, or that declares exclusive: true in its test.qbs, runs alone after the others. Without --jobs, the suites run one at a time, and the ci profile does not set a number either.

--shuffle <seed> runs the suites in a random order from that seed. tests.json keeps the seed, so a failure that depends on the order repeats with the same --shuffle. The same seed gives the same order on every host.

Keep the results

# Write tests.json and tests.junit.xml, with each suite's log
kiln test --prebuilt default --report /tmp/test-results

# Keep each suite's output only
kiln test --prebuilt default --log-dir /tmp/test-logs

Without --report, a profile that names result files, such as ci, writes them into test-results/ of the build folder, never into the checkout. The schema of tests.json is in the kiln reference.

Read the result

Each suite gets one line when it ends, so the order of the lines is the order of completion:

The last line counts every list: how many suites kept their contract, failed, timed out, were not built, were skipped by category, and were left out by the selection. Exit code 0 means that every selected suite kept its contract, and 1 means that one did not.

Run as the build farm does

kiln test --prebuilt default --options build-request.json --report /tmp/test-results --log-dir /tmp/test-results/logs

--options reads a build request: one JSON object with the build options by name, as the farm's build form writes it. kiln test reads only its testProfile. --profile on the command line wins over the request, and the profile none runs no suite. The format is in the kiln reference.

Run the guard suites with the lint gate

integration/lint/run-all.sh --with-tests

The gate builds the guard suites and runs kiln test --profile gate. When a guard suite is a member of the shared binary, the gate builds testsLogical too.

Run on Windows

On Windows, a suite finds the Qt DLLs through PATH. kiln test puts the bin folder of the build's Qt installation first on PATH for each suite, so you do not need to set it. A widget suite needs the interactive desktop, and an SSH session has none. Run the sweep through the wrapper, from the checkout root:

powershell -ExecutionPolicy Bypass -File integration\win\kiln-test-desktop.ps1 `
    -BuildDir C:\path\to\build -OutDir C:\path\to\results -KilnArgs '--profile','windows' -Wait

The wrapper starts kiln test through a scheduled task in the logged-in user's session. -Stop with the same -OutDir ends the run and its suites.

---

The shared binary

The suites with shared: true compile into one program, testsLogical, and are linked once. Their own programs are not built by default. kiln test runs testsLogical as several worker processes, each with a shard of the members, and judges each member against its own contract. In tests.json, the field binary names the program that a suite ran in.

Isolation does not change between the two forms. Before each member, the shared binary sets the working folder, the application name and the data folder again, so that each member resolves the folders that its own program resolves. If a worker crashes, kiln test marks the member that ran as FAIL, runs that member again alone, and runs each member that did not start in a process of its own. A member that passes alone and fails in the shared binary is FAIL, "passed alone, failed in the shared binary": the member leaves state behind, or finds state that another member left.

Debug one member alone

The shared binary takes two options of its own:

# The members, one per line
./default/testsLogical.*/testsLogical --list

# One member, by its suite name, with arguments for QTest after --
./default/testsLogical.*/testsLogical --run testCryptHash -- -v2

# Several members, in this order
./default/testsLogical.*/testsLogical --run testCryptHash,testCryptMacKdf

# One case of one member under the debugger
gdb --args ./default/testsLogical.*/testsLogical --run testCryptHash -- testAbcDigest

--run takes suite names, separated by commas, and runs them in that order in one process. Each member's output stands between a ##TSR begin <suite> line and a ##TSR end <suite> exit=<code> line. The process exits with the number of members that failed. The arguments after -- go to every member, as a standalone suite takes them.

To work on one member with the short build loop, build its own program by name. It then has its own main, as before it joined:

qbs build -p testCryptHash
./default/testCryptHash.*/testCryptHash --headless

---

Run one suite program directly

# Find the program, then run it from the project root
./default/testCommsChannel.*/testCommsChannel --headless

# A suite that needs a real platform
./default/testViewportParity.*/testViewportParity --display

Run test programs from the project root. --headless uses the offscreen platform, and --display uses the real one. When a suite crashes, set QTEST_DISABLE_STACK_DUMP=1. Then a crash ends the run, and does not hang it while QTest starts a debugger. kiln test sets it for every suite.

QTest also stops each test function after QTEST_FUNCTION_TIMEOUT milliseconds, and the default is 300 seconds. The suite's timeout does not change this value when you run the program yourself. For a function that runs longer, set the variable:

QTEST_FUNCTION_TIMEOUT=900000 ./default/testBanterPerformanceSweep.*/testBanterPerformanceSweep --headless

kiln test sets QTEST_FUNCTION_TIMEOUT for every suite to the suite's timeout plus 10 seconds. A value that you set in the environment of kiln test stays.

OctoMY test options

OC_TEST_MAIN takes these options out of the command line before Qt Test reads it. A suite takes no environment variable: a value it needs comes in as an option.

Through qbs, put the options after --, for example qbs run -p testChatBubbleDelegate -- --write-golden.

Write golden files

Some suites compare a result with a file in the source tree. Give such a suite --write-golden to write that file from the current code instead. The same option serves every suite:

./default/testChatBubbleDelegate.*/testChatBubbleDelegate --headless --write-golden
git diff test/testChatBubbleDelegate/golden/

Read the diff before you commit the files. A golden file that a defective build wrote makes the defect the new reference. Until 2026-10-08 each suite had its own environment variable for this (OC_RECORD_GOLDEN, CVP_WRITE_GOLDEN, CPF_WRITE_FIXTURES). They are gone.

Keep test windows off your desktop

A suite opens real windows when it needs a real platform, when you give it --display, or when it is not headless-capable and you run it without --headless. On a desktop, each window takes the keyboard and mouse focus. The test display moves all of them to a virtual X server instead. On a development machine, the test display is the rule for UI tests: install it once, and keep it running.

# Once per machine: install and start the test display (a systemd user unit)
mkdir -p ~/.config/systemd/user
ln -sf "$PWD/integration/test-display/octomy-test-display.service" ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now octomy-test-display

# Watch it in a window, if you like
integration/test-display/show-test-display.sh

When the test display runs, each suite says TestArgs: windows go to the test display :N. Programs that a suite starts, such as a hub, follow it there. GPU suites keep their Vulkan backend on it. Give a suite --desktop to show its windows on your own display for one run. Without the service, nothing changes.

A suite that starts one of our programs as a child, such as a hub, must give the child QT_QPA_PLATFORM=offscreen through ChildEnvironment, as testConstructCli and testMonitorE2E do. On a Wayland desktop a Qt child otherwise picks the Wayland platform and opens its window on your desktop, whatever DISPLAY says.

The view from show-test-display.sh is read-only. It shows the test display a few times a second and never sends your keyboard or mouse to it.

If test windows appear on your desktop again, the harness found no live test display. Check it:

systemctl --user status octomy-test-display
cat "$XDG_RUNTIME_DIR/octomy-test-display"     # the display, such as :1
xdpyinfo -display "$(cat "$XDG_RUNTIME_DIR/octomy-test-display")" | head -1

If the service is stopped, or xdpyinfo cannot open the display, restart it with systemctl --user restart octomy-test-display. The harness accepts a display that answers on its socket file or on its Linux abstract socket. Until 2026-10-06 it tried only the file, so a deleted socket file sent every window back to the desktop without a message.

---

Qt Test options

A suite program takes Qt Test's own options. Common ones:

# Verbose output
./default/testCommsChannel.*/testCommsChannel -v2

# Run one test function
./default/testCommsChannel.*/testCommsChannel testSendReceive

# List the test functions
./default/testCommsChannel.*/testCommsChannel -functions

---

Test output

Successful test

********* Start testing of TestCommsChannel *********
Config: Using QtTest library 6.8.3, Qt 6.8.3
PASS   : TestCommsChannel::initTestCase()
PASS   : TestCommsChannel::testConnect()
PASS   : TestCommsChannel::testSendReceive()
PASS   : TestCommsChannel::testDisconnect()
PASS   : TestCommsChannel::cleanupTestCase()
Totals: 5 passed, 0 failed, 0 skipped
********* Finished testing of TestCommsChannel *********

Failed test

********* Start testing of TestCommsChannel *********
PASS   : TestCommsChannel::initTestCase()
FAIL!  : TestCommsChannel::testSendReceive() Compared values are not the same
   Actual   (received): "hello"
   Expected (expected): "world"
   Loc: [TestCommsChannel.cpp(45)]
PASS   : TestCommsChannel::cleanupTestCase()
Totals: 3 passed, 1 failed, 0 skipped
********* Finished testing of TestCommsChannel *********

kiln test reads the Totals: line and each FAIL! line. Each FAIL! line becomes an entry of failedCases in tests.json, with the case, the file, the line and the message.

---

Test coverage

For detailed coverage instructions, see Code Coverage.

Quick start

# Build with coverage (clang or gcc)
qbs build config:debug project.enableCoverage:true

# Run tests
./debug/testFoo.*/testFoo

LLVM (clang):

llvm-profdata merge -sparse *.profraw -o merged.profdata
llvm-cov show ./debug/testFoo.*/testFoo -instr-profile=merged.profdata \
    -format=html -output-dir=coverage

GCC (gcov/lcov):

lcov --capture --directory . --output-file coverage.info
genhtml coverage.info --output-directory coverage

View report

xdg-open coverage/index.html

Note

Coverage is mutually exclusive with sanitizers. If both are enabled, sanitizers take precedence.

---

Trusting a green

A check that cannot fail looks exactly like a check that passed. Both print nothing and both exit zero. Before you trust a green result, confirm that the check is able to go red.

This section records two real cases from this repository. Each one wasted days. Each one was cheap to catch.

Shape 1: a skip that names the wrong cause

A test skipped with this message:

SKIP : No offscreen RHI: scene at Y=0.9 did not load

The message blames the graphics environment. The graphics environment was not the cause. The test fixture wrote a scene file at format version 1, and the persistence layer reads version 3 only. The load failed, and the catch-all skip reported the failure as a missing GPU.

Three sibling assertions were red at the same time. A specification recorded "render pending an RHI" for six days on the strength of that one string.

Rule: when a skip names an environment, run the same case where that environment is present.

Here that meant one command:

# The sanctioned GPU runner: Xvfb, the xcb platform, and Mesa lavapipe.
integration/run-gpu-integration-tests.sh

The test failed there in the same way. A GPU was present, so the message was false, and the real cause became obvious within the hour.

Shape 2: a check that never ran

An icon drift guard reported nothing and looked green. The guard had never executed. Qbs prunes a rule when nothing consumes its output, and the first wiring produced an output that nothing consumed.

There was no skip message to interrogate. There was no error, no warning, and no subsystem to blame. Silence was the only symptom.

Rule: do not accept the green until you have seen the red.

Here that meant one edit and two builds. Break the input, prove the build goes red, then restore it:

F=src/libs/libres/resources/icons/app/reload.svg
printf '\n' >> "$F"

# Assert that the write reached the disk. See the warning below.
git diff --quiet -- "$F" && echo "PROBE DID NOT APPLY" && exit 1

qbs build -p libres
#   DRIFT: app/reload.svg differs from the sheet
#   Checked: 373, Drifted: 1, Errors: 0
#   exit 1

git checkout -- "$F"
qbs build -p libres
#   Checked: 373, Drifted: 0, Errors: 0
#   exit 0

The cost is one edit and two builds. No GPU, no second machine, and no special environment. A guard that you have never seen fail is a guard you cannot cite.

Choose a probe with nothing to match. icon-export --check compares the whole file byte for byte against freshly generated content, so any byte drifts it. Appending one newline therefore needs no knowledge of the file. Editing a named attribute would work today and rot tomorrow.

A probe can fail to apply, and that looks like a pass.

The first attempt at the proof above used sed 's/<svg /<svg id=...'. That pattern matches nothing in these files. The master sheet declares the SVG namespace by prefix, so every exported icon opens <svg:svg , not <svg . The same surprise makes a naive grep '<defs' find nothing in them.

The file never changed. The build reported Drifted: 0 and exit 0, and that green looked exactly like a passing guard. The probe had become the thing this section warns about: a check that could not fail.

Keep the git diff --quiet line even with a probe that cannot miss. It answers a different question now — not "did my pattern match" but "did the write reach the disk" — and it costs one line.

When you cannot produce the environment

Sometimes no runner you can reach has the thing the skip names. The first rule then gives you nothing, and you are back to trusting the message.

Make the skip condition measured, not caught.

A caught error cannot tell you which error it caught, so it can always report the wrong reason. A measured condition is the observation itself, so it cannot misattribute.

The orientation test in testConstructVisionSource shows the difference. It renders one scene twice, with the subject above and below the origin. It skips when the two frames come back identical, because identical frames prove that the subject never rendered. The skip states a fact that the test measured.

This changes which number carries the evidence:

Always quote the skip count beside the pass count. Under the offscreen platform a Vulkan case skips silently, and the suite still reports success.

---

Troubleshooting

A new suite does not build

qbs finds a new suite folder only after a resolve that runs the probes again:

qbs resolve --force-probe-execution

Check the category too. A suite in a category that the build leaves out is not built, and no error says so.

A suite is NOT BUILT, or has two programs

The run found no program for a suite of a built category. Build it, or build the whole tree, and run again. For a member of the shared binary, the program that is missing is testsLogical.

A suite can also FAIL with "2 programs, … the run cannot tell which one is current". The build folder then holds a stale copy of the suite's program. kiln test runs neither, because a stale program gives a wrong verdict with no error. Delete the stale one, or build into a fresh folder.

A suite hangs

# Stop it after 60 seconds, and print a stack instead of hanging in the crash handler
QTEST_DISABLE_STACK_DUMP=1 timeout 60 ./default/testMyClass.*/testMyClass --headless

kiln test already stops a suite at its timeout, and sets QTEST_DISABLE_STACK_DUMP=1 for each suite. It also gives QTest's per-function watchdog the suite's timeout, so a suite that ends with "Test function timed out" under kiln test used more than its full timeout in one function. A crash in a suite with threads can look like a hang when you run the program yourself, because QTest's crash handler starts a debugger.

Warnings disappear

QTest drops every warning after the 2 000th. Pass -maxwarnings 0 to keep them all:

./default/testMyClass.*/testMyClass -maxwarnings 0

The shared binary passes -maxwarnings 0 to every member, because QTest counts warnings for the whole process.

A member fails only in the shared binary

The verdict is FAIL, "passed alone, failed in the shared binary", or the nightly profile reports two different verdicts. The member and another one share state that two processes never shared: a singleton, a cache, a pending event, or a thread. Run the member alone and in a pair to find the partner:

./default/testsLogical.*/testsLogical --run testMyClass
./default/testsLogical.*/testsLogical --run testOtherClass,testMyClass

Fix the shared state in the code that keeps it, not in the test.

A test fails only sometimes

Find the cause. Do not hide the failure with QEXPECT_FAIL or a retry. The usual causes:

• A fixed wait (QTest::qWait, a sleep) for something that takes longer on a loaded machine. Wait for the condition instead, with QTRY_VERIFY, QSignalSpy::wait or test::pumpUntil.
• State from an earlier case, or from a singleton. --shuffle with a few seeds shows an order dependence.
• A fixed path that two suites write at the same time. --jobs shows it.

Debug a failing test

# Run one function under the debugger
gdb --args ./default/testMyClass.*/testMyClass --headless testFailingFunction

# In gdb:
(gdb) run
(gdb) bt  # Backtrace on crash

For a member of the shared binary, see Debug one member alone.

---

Test organization

test/
├── testCommsChannel/      # One suite: its class and its test.qbs
├── testAssociate/
├── shared/testsLogical/   # The shared test binary
├── fixtures/              # Data that several suites share
├── templates/             # The template for a new suite
└── tests.qbs              # Finds every suite folder

• Test class: TestClassName
• Test file: test/testClassName/TestClassName.cpp
• Test functions: testFeatureName()

---

See also

• Write a test: the steps for a new suite.
• Kiln reference: the kiln test options, the profile keys and the tests.json record.
• Qbs reference: the Test.qbs properties and the categories.
• Test harnesses: the shared helpers that suites use.