Write a test

Add a new test suite to OctoMY™ in nine steps, from the choice of category to the first record of its run. Each step names the reference page that holds its details, so that a new suite fits the system from its first commit.

This guide assumes that you can build the tree with qbs. To run suites that already exist, see Run tests.

1. Choose the category

Choose the category by what the suite needs to run, never by the subsystem that it tests. The category decides whether the default build compiles the suite at all.

The suite…	Category
needs nothing: pure logic, no device, no display, no network	`unit`
drives several components together, unattended and headless	`integration`
constructs real widgets on the `offscreen` platform	`widget`
drives OPAL task plans	`opal`

The other categories (interactive, hardware, live_audio, live_net, perf, legacy) are not built by default, so a suite in one of them never runs in CI. The most demanding case sets the category of the whole suite: put a pure case of a demanding suite into a unit suite of its own. Moving a suite into a category that is not built needs the owner's approval.

Reference: Test categories.

2. Copy the template

Copy test/templates/testTemplate/ to a new folder, and rename the files, the class and the include guard to match the folder:

cp -a test/templates/testTemplate test/testMyFeature
cd test/testMyFeature
mv TestTemplate.hpp TestMyFeature.hpp
mv TestTemplate.cpp TestMyFeature.cpp
sed -i 's/TestTemplate/TestMyFeature/g; s/TESTTEMPLATE_HPP/TESTMYFEATURE_HPP/g' \
    TestMyFeature.hpp TestMyFeature.cpp

The result is one folder, test/test<Name>/, with one class, Test<Name>:

// test/testMyFeature/TestMyFeature.hpp
#ifndef TESTMYFEATURE_HPP
#define TESTMYFEATURE_HPP

#include "test/Common.hpp"

class TestMyFeature : public QObject
{
	Q_OBJECT
private slots:
	void test();
};

#endif
// TESTMYFEATURE_HPP

// test/testMyFeature/TestMyFeature.cpp
#include "TestMyFeature.hpp"

void TestMyFeature::test()
{
}


OC_TEST_MAIN(test, TestMyFeature)

End the source in OC_TEST_MAIN, or in OC_TEST_MAIN_GUILESS for a suite that shows no widget, never in QTEST_MAIN. The macro handles --headless and --display, pins the working folder to the project root, and sets up the platform. integration/lint/check-test-main.sh rejects QTEST_MAIN.
Give every helper function and type of the suite internal linkage: put it in an anonymous namespace, or make it static. Two suites in the shared binary that define the same external name fail the link, or worse, share one class with no error.
In the header comment of the class, say what the suite proves, and name the defect or report that caused it.

The Test item adds the sources of the folder, the libraries and Qt Test. qbs finds the new folder only after a resolve that runs the probes again:

qbs resolve --force-probe-execution

3. Declare the contract

The contract is a few lines of test.qbs. kiln test holds the suite to it on every run.

// test/testMyFeature/test.qbs
Test {
    localPath: path
    category: "unit"
    expectedPassed: 7
}

timeout: declare it when a case needs more than the default 60 s. Leave room for a loaded machine.
expectedPassed: the exact number of passed slots, initTestCase and cleanupTestCase included, for every suite whose count is a property of the code. A new case raises the number in the same commit.
expectedMinimumPassed: only when the count depends on the host, for example on a display. Write both counts in a comment beside it. A suite declares this or expectedPassed, never both.
requiresRealPlatform: true: the suite creates a QRhi and cares which backend it gets. intentionallyOffscreen: "<reason>" instead says why OpenGL on offscreen is the correct backend.
platformSensitive: true: the result can differ by platform. That is a suite that tests a platform branch, file paths, processes or IPC, an RHI backend, fonts or packaging. The Windows leg runs only these suites.
exclusive: true: the suite must run alone under kiln test --jobs. See A suite that must run alone.
property bool guardTest: true: the suite protects a rule that the whole tree depends on, and runs in the lint gate. A guard suite also declares a count.

Reference: Test.qbs properties, and the kiln keys that the properties become in the kiln reference.

A suite that must run alone

kiln test --jobs runs suites at the same time. Declare exclusive: true when a suite cannot share the machine with other suites:

The suite loads every core, or measures whether threads still get CPU time under load. The load of other suites makes its timing cases fail, and its own load slows the suites beside it.
The suite binds a fixed port, or is a heavy benchmark.

// test/testRecoveryEndToEnd/test.qbs
Test {
    localPath: path
    category: "integration"
    // Loads every core to 95 %, then checks that the watchdog still runs.
    exclusive: true
}

The runner starts an exclusive suite after the parallel part of the run, with no other suite beside it. Write the reason in a comment above the line. testRecoveryEndToEnd is the example: at a load average of 88 its runaway case took 56 s and the suite hit its 60 s limit. Alone, the suite takes 23 to 30 s. A longer timeout does not correct this kind of failure, because the load of the other suites, not the code, makes the case slow. A suite that needs a display runs alone without this declaration.

4. Choose the harness

Before you write a fake, a fixture or a helper, look in the catalog of shared test harnesses. Where a harness exists, use it. Never write a private copy: the next defect then gets fixed in one copy and not in the other.

When a second suite needs a helper of yours, move it to a shared place in the same commit as the second use: src/libs/libtest/test/ for a general helper, or src/libs/lib<X>/<x>/testing/ for a helper that knows one library. A shared harness gets a suite of its own, which tests the harness.

Reference: Test harnesses.

5. Read fixtures and write to a temporary folder

Read a data file through test::utility::fixturePath():

#include "test/Utility.hpp"

const QString path = test::utility::fixturePath(QStringLiteral("scene-v3.json"));
QVERIFY2(!path.isEmpty(), "the fixture is missing");

It looks in the suite's own folder, test/<suite>/, then in test/fixtures/, both under the project root, so the path depends neither on the working folder nor on the compiler. A name that is in neither place gives an empty path and a warning that names both places. Do not use QFINDTESTDATA or __FILE__. A fixture that two suites read lives in test/fixtures/.

Write only into a temporary folder: a QTemporaryDir, or test::TemporaryTree from test/TemporaryTree.hpp, which writes files by relative path. Never write into the repository or into a fixed path such as /tmp/name: two suites that run at the same time then write the same file.

6. Wait for conditions, not for time

Do not wait a fixed time for a result, with a sleep or QTest::qWait. Wait for the condition itself:

QTRY_VERIFY(worker.isDone());
QTRY_COMPARE(model.rowCount(), 3);

QSignalSpy spy(&client, &Client::connected);
QVERIFY(spy.wait(5000));

QVERIFY(test::pumpUntil([&] { bus.step(); }, [&] { return node.isReady(); }, 5000));

test::pumpUntil is in test/Pump.hpp. Code whose logic depends on time takes a clock as an argument, and the test drives a fake clock. A fixed wait is right only where the case measures the passage of time itself; say so in a comment.

A random case takes its seed from a constant and prints it, so a failure repeats. A case needs no network beyond loopback.

7. Prove the case red before green

Run each new case against the code before the fix, and see it fail. A case that was never red proves nothing.

Assert the defect, not the remedy that you expect. Another fix can be right.
Assert several effects of one branch, so that one missed event does not look like a defect, or a pass.
When a case skips, the skip names a condition that the case measured. Quote the skip count beside the pass count.

Reference: Trusting a green.

8. Join the shared binary

A purely logical suite joins the shared binary, testsLogical, with one line in its test.qbs:

Test {
    localPath: path
    category: "unit"
    expectedPassed: 7
    shared: true
}

Write the line as the literal shared: true: the build reads it from the text. The suite can join when all of these hold:

Its category is unit, integration or opal.
It is headless and does not need a real platform.
It shows no window, creates no QRhi, and opens no audio device, socket, serial port or child process. A loopback mock is allowed.
It reads files only through fixturePath(), and writes only into a temporary folder (step 5).
It waits for conditions, not for time (step 6).
Its helpers have internal linkage (step 2).
Its test object's destructor completes, and it leaves no thread, timer or pending event behind.
It gives the same verdict alone and in the shared binary, in two orders.
Its test.qbs holds its contract and the libraries it depends on, and nothing else: no compiler option, include path, condition, file list, asset bundle, or dependency on a product that is not a library. The shared binary compiles every member with one set of settings.

The build refuses a member that breaks rule 2 or rule 9. Check rules 7 and 8 by running it alone, then among the other members in two orders:

qbs build -p testsLogical
./default/testsLogical.*/testsLogical --run testMyFeature
kiln test --prebuilt default --category unit --jobs auto --shuffle 1
kiln test --prebuilt default --category unit --jobs auto --shuffle 2

The nightly profile repeats this check for every member: it runs each one shared and alone, and fails a member whose two verdicts differ.

A member's own program is no longer built by default. qbs build -p testMyFeature still builds it, for the short loop while you work on the suite.

Reference: The shared binary, and the shared property in the qbs reference.

9. Run it and read the record

Build the suite, then run it under its contract:

qbs build -p testMyFeature        # or -p testsLogical for a member
kiln test --prebuilt default test/testMyFeature --report /tmp/my-feature

Read the line of the suite, not only the exit code. PASS testMyFeature: 7 passed, 0.4s says how many slots passed. A FAIL line says why: a failed case, a count that differs from expectedPassed, a crash, or every case skipped. /tmp/my-feature/tests.json holds the same verdict, each failed case with its file, line and message, and the name of the log in logs/.

Before you commit, run the suites around yours too, for example its category, with --jobs and a --shuffle seed. A suite that passes alone and fails beside others shares a path or a singleton with them.

Reference: Read the result, and the tests.json record.