Write a test

Add a new test suite to OctoMY™ in nine steps, from the choice of category to the first record of its run. Each step names the reference page that holds its details, so that a new suite fits the system from its first commit.

This guide assumes that you can build the tree with kiln, the default build path (Start here). qbs, the second path, builds the same suites; With qbs at the end gives its commands. To run suites that already exist, see Run tests.

1. Choose the category

Choose the category by what the suite needs to run, never by the subsystem that it tests. The category decides whether a sweep of kiln test, and the build of CI, compiles the suite at all.

The suite… Category
needs nothing: pure logic, no device, no display, no network unit
drives several components together, unattended and headless integration
constructs real widgets on the offscreen platform widget
drives OPAL task plans opal
builds network namespaces, or needs the network beyond loopback live_net

The other categories (interactive, hardware, live_audio, perf, legacy) are not built by default, so a suite in one of them never runs in CI. The most demanding case sets the category of the whole suite: put a pure case of a demanding suite into a unit suite of its own. Moving a suite into a category that is not built needs the owner's approval.

If every case needs something that only some targets provide, such as a Direct3D 12 device, add requiresCapabilities: ["d3d12"] to test.qbs. Name the capability, never the operating system. A target without it does not build the suite. This switches the suite off on the other targets, so it needs the owner's approval too.

Reference: Test categories, Test capabilities.

2. Copy the template

Copy test/templates/testTemplate/ to a new folder, and rename the files, the class and the include guard to match the folder:

cp -a test/templates/testTemplate test/testMyFeature
cd test/testMyFeature
mv TestTemplate.hpp TestMyFeature.hpp
mv TestTemplate.cpp TestMyFeature.cpp
sed -i 's/TestTemplate/TestMyFeature/g; s/TESTTEMPLATE_HPP/TESTMYFEATURE_HPP/g' \
    TestMyFeature.hpp TestMyFeature.cpp

The result is one folder, test/test<Name>/, with one class, Test<Name>:

// test/testMyFeature/TestMyFeature.hpp
#ifndef TESTMYFEATURE_HPP
#define TESTMYFEATURE_HPP

#include "test/Common.hpp"

class TestMyFeature : public QObject
{
	Q_OBJECT
private slots:
	void test();
};

#endif
// TESTMYFEATURE_HPP

// test/testMyFeature/TestMyFeature.cpp
#include "TestMyFeature.hpp"

void TestMyFeature::test()
{
}


OC_TEST_MAIN(test, TestMyFeature)
  • End the source in OC_TEST_MAIN, or in OC_TEST_MAIN_GUILESS for a suite that shows no widget, never in QTEST_MAIN. The macro handles --headless and --display, pins the working folder to the project root, and sets up the platform. The lint check test-main, which testTestHygiene runs, rejects QTEST_MAIN.
  • Give every helper function and type of the suite internal linkage: put it in an anonymous namespace, or make it static. Two suites in the shared binary that define the same external name fail the link, or worse, share one class with no error. A header that two suites include declares no mutable static variable at namespace scope: in the shared binary, two members can then read two copies of it. The lint check header-statics rejects one.
  • In the header comment of the class, say what the suite proves, and name the defect or report that caused it.

The Test item adds the sources of the folder, the libraries and Qt Test. The next kiln build or kiln test finds the new folder with no other step. On the qbs path, the next build finds it too: the probes that list the suites run again when a folder under test/ is added, moved or removed.

3. Declare the contract

The contract is a few lines of test.qbs. kiln test holds the suite to it on every run.

// test/testMyFeature/test.qbs
Test {
    localPath: path
    category: "unit"
    expectedPassed: 7
}
  • timeout: declare it when a case needs more than the default 60 s. Leave room for a loaded machine.
  • expectedPassed: the exact number of passed slots, initTestCase and cleanupTestCase included, for every suite whose count is a property of the code. A new case raises the number in the same commit.
  • expectedMinimumPassed: only when the count depends on the host, for example on a display. Write both counts in a comment beside it. A suite declares this or expectedPassed, never both.
  • requiresRealPlatform: true: the suite creates a QRhi and cares which backend it gets. intentionallyOffscreen: "<reason>" instead says why OpenGL on offscreen is the correct backend.
  • platformSensitive: true: the result can differ by platform. That is a suite that tests a platform branch, file paths, processes or IPC, an RHI backend, fonts or packaging. The Windows leg runs only these suites.
  • requiresCapabilities: ["d3d12"]: every case needs a capability that only some targets provide. Name the capability, never the operating system. A target without it does not build the suite, and kiln test reports it SKIPPED. See Test capabilities.
  • exclusive: true: the suite must run alone under kiln test --jobs. See A suite that must run alone.
  • property bool guardTest: true: the suite protects a rule that the whole tree depends on, and runs in the lint gate. A guard suite also declares a count.

Reference: Test.qbs properties, and the kiln keys that the properties become in the kiln reference.

A suite that must run alone

kiln test --jobs runs suites at the same time. Declare exclusive: true when a suite cannot share the machine with other suites:

  • The suite loads every core, or measures whether threads still get CPU time under load. The load of other suites makes its timing cases fail, and its own load slows the suites beside it.
  • The suite binds a fixed port, or is a heavy benchmark.
// test/testRecoveryEndToEnd/test.qbs
Test {
    localPath: path
    category: "integration"
    // Loads every core to 95 %, then checks that the watchdog still runs.
    exclusive: true
}

The runner starts an exclusive suite after the parallel part of the run, with no other suite beside it. Write the reason in a comment above the line. testRecoveryEndToEnd is the example: at a load average of 88 its runaway case took 56 s and the suite hit its 60 s limit. Alone, the suite takes 23 to 30 s. A longer timeout does not correct this kind of failure, because the load of the other suites, not the code, makes the case slow. A suite that needs a display runs alone without this declaration, unless each job of the run has its own Xvfb, as in the ci profile. Then it runs beside the others.

4. Choose the harness

Before you write a fake, a fixture or a helper, look in the catalog of shared test harnesses. Where a harness exists, use it. Never write a private copy: the next defect then gets fixed in one copy and not in the other.

When a second suite needs a helper of yours, move it to a shared place in the same commit as the second use: src/libs/libtest/test/ for a general helper, or src/libs/lib<X>/<x>/testing/ for a helper that knows one library. A shared harness gets a suite of its own, which tests the harness.

Reference: Test harnesses.

5. Read fixtures and write to a temporary folder

Read a data file through test::utility::fixturePath():

#include "test/Utility.hpp"

const QString path = test::utility::fixturePath(QStringLiteral("scene-v3.json"));
QVERIFY2(!path.isEmpty(), "the fixture is missing");

It looks in the suite's own folder, test/<suite>/, then in test/fixtures/, both under the project root, so the path depends neither on the working folder nor on the compiler. A name that is in neither place gives an empty path and a warning that names both places. Do not use QFINDTESTDATA or __FILE__. A fixture that two suites read lives in test/fixtures/.

Keep a fixture in one of these two places. kiln test --os android copies only these two places to the device, into the tests-runner's own files folder, and gives the suite the copy as its project root with --project-root. A file at any other place in the checkout is not on the device. If the copy fails, kiln says so, and the suite runs without its fixtures: then fixturePath() gives an empty path and a warning, and the suite fails with its own message.

Write only into a temporary folder: a QTemporaryDir, or test::TemporaryTree from test/TemporaryTree.hpp, which writes files by relative path. Never write into the repository or into a fixed path such as /tmp/name: two suites that run at the same time then write the same file.

6. Wait for conditions, not for time

Do not wait a fixed time for a result, with a sleep or QTest::qWait. Wait for the condition itself:

QTRY_VERIFY(worker.isDone());
QTRY_COMPARE(model.rowCount(), 3);

QSignalSpy spy(&client, &Client::connected);
QVERIFY(spy.wait(5000));

QVERIFY(test::pumpUntil([&] { bus.step(); }, [&] { return node.isReady(); }, 5000));

test::pumpUntil is in test/Pump.hpp. Code whose logic depends on time takes a clock as an argument, and the test drives a fake clock. A fixed wait is right only where the case measures the passage of time itself; say so in a comment.

A random case takes its seed from a constant and prints it, so a failure repeats. A case needs no network beyond loopback.

A speed budget

Do not assert a wall time, such as "the extraction takes less than 250 ms". The same code is slower on a busy machine, and the case fails with no change to the code. Assert a ratio to the reference work instead:

#include "referencework/ReferenceWork.hpp"

const referencework::RelativeTime t = referencework::relativeTime([&]() { text = htmlToText(html); });
qInfo().noquote() << t.describe();
QVERIFY2(t.ratio < kRatioBudget, qPrintable(t.describe()));

relativeTime() times a short piece of fixed work and your work in turns, on one thread, and gives the best ratio of three pairs. The ratio is of the thread's CPU times, so the time that the thread waits for a CPU does not count. Where the system gives no CPU time for a thread, the ratio is of wall times. A busy machine slows both, so the ratio stays. A slower algorithm raises it, so the case still fails. Measure the budget on the reference host, and say in a comment how you measured it. The suite depends on libreferencework. The ratio cancels the load on the CPUs. It does not cancel contention for a disk, a GPU or a compositor. See the harness catalog and the kiln reference.

7. Prove the case red before green

Run each new case against the code before the fix, and see it fail. A case that was never red proves nothing.

  • Assert the defect, not the remedy that you expect. Another fix can be right.
  • Assert several effects of one branch, so that one missed event does not look like a defect, or a pass.
  • When a case skips, the skip names a condition that the case measured. Quote the skip count beside the pass count.

Reference: Trusting a green.

8. Join the shared binary

A purely logical suite joins the shared binary, testsLogical, with one line in its test.qbs:

Test {
    localPath: path
    category: "unit"
    expectedPassed: 7
    shared: true
}

Write the line as the literal shared: true: the build reads it from the text. The suite can join when all of these hold:

  1. Its category is unit, integration or opal.
  2. It is headless and does not need a real platform.
  3. It shows no window, creates no QRhi, and opens no audio device, socket, serial port or child process. A loopback mock is allowed.
  4. It reads files only through fixturePath(), and writes only into a temporary folder (step 5).
  5. It waits for conditions, not for time (step 6).
  6. Its helpers have internal linkage (step 2).
  7. Its test object's destructor completes, and it leaves no thread, timer or pending event behind.
  8. It gives the same verdict alone and in the shared binary, in two orders.
  9. Its test.qbs holds its contract and the libraries it depends on, and nothing else: no compiler option, include path, condition, file list, asset bundle, or dependency on a product that is not a library. The shared binary compiles every member with one set of settings.

The build refuses a member that breaks rule 2 or rule 9. Check rules 7 and 8 by running it alone, then among the other members in two orders:

kiln build test/shared/testsLogical
./debug/testsLogical/testsLogical --run testMyFeature
kiln test --category unit --jobs auto --shuffle 1
kiln test --category unit --jobs auto --shuffle 2

The path is that of a build with no output.dir (The build convention).

The nightly profile repeats this check for every member: it runs each one shared and alone, and fails a member whose two verdicts differ.

A member's own program is no longer built by default, and kiln builds a member only into testsLogical. On the qbs path, qbs build -p testMyFeature still builds the member's own program, for the short loop while you work on the suite.

Reference: The shared binary, and the shared property in the qbs reference.

9. Run it and read the record

Build the suite, then run it under its contract:

kiln test test/testMyFeature --report /tmp/my-feature

kiln test builds what the suite needs, testsLogical for a member, then runs it.

Read the line of the suite, not only the exit code. PASS testMyFeature: 7 passed, 0.4s says how many slots passed. A FAIL line says why: a failed case, a count that differs from expectedPassed, a crash, or every case skipped. /tmp/my-feature/tests.json holds the same verdict, each failed case with its file, line and message, and the name of the log in logs/.

Before you commit, run the suites around yours too, for example its category, with --jobs and a --shuffle seed. A suite that passes alone and fails beside others shares a path or a singleton with them.

Reference: Read the result, and the tests.json record.

With qbs

qbs builds the same suites until it retires. Build with qbs, then run what it built with kiln test --prebuilt:

qbs build -p testMyFeature        # or -p testsLogical for a member
kiln test --prebuilt default test/testMyFeature --report /tmp/my-feature
./default/testsLogical.*/testsLogical --run testMyFeature

A new suite folder needs no step on the qbs path either. If the next qbs build does not find it, see A new suite does not build.

See also

In this section