Set up a build site

A build site is one machine carrying several checkouts of this project at once — one per branch you have work in flight on — sharing a single git object store and a single compiler cache. It is the standard working arrangement here: several agent or human sessions, each in its own lane, none of them stepping on another's index.

This page is the whole setup, start to finish. It exists so that standing up a new machine is a runbook rather than an interview with whoever set up the last one.

TL;DR

1. One bare repo at shared/OctoMY.git; every lane is a git worktree.
2. Qt 6.10.0+ (§Qt).
3. One kiln for the whole site, in shared/kiln/ (§Kiln). Kiln is the default build path.
4. cache.dir = ../shared in each lane's .kiln.local for the vendored LLVM/llama/rl trees, and kiln's object cache on a disk with room (§Kiln).
5. earlyoom, and for qbs RAM and swap together of 64 GB or more (§Memory, swap and disk space).
6. Run the verification checklist — every step has a command that proves it, because most of these fail silently.
7. Only for qbs, the second path until it retires: the qbs path — qbs ≥ 3.3.1 built from source, ccache, mold and project.buildCacheDir.

---

The layout

<site-root>/                      <- ccache base_dir points HERE
├── shared/
│   ├── OctoMY.git/               <- the ONLY git object store
│   ├── kiln/                     <- the site's kiln (§Kiln)
│   └── llvm/  llama/  rl/        <- kiln's cache.dir and qbs's project.buildCacheDir, keyed by pin
├── main/                         <- one worktree per lane
├── <branch>/                     <- named for its branch, always
└── <branch>/

A lane may be a linked worktree (preferred — one object store, one fetch) or an independent clone. Older sites grew a mix of both, because each lane was originally cloned on its own. Worktrees are the target arrangement: see §Convert an existing clone into a worktree for the procedure and for what it breaks.

Know which kind you are in before reasoning about disk or fetches — git -C shared/OctoMY.git worktree list shows only the worktrees, so a lane missing from that list is a clone, not a bug. The other tell is .git itself: a file containing gitdir: means worktree, a directory means clone.

Two rules make the rest work:

• A lane directory is named for its branch, and holds only that branch. The name is how a session answers "where am I?" — see the pwd && git status habit in AGENTS.md. A lane whose directory name and branch disagree is the single most reliable way to land work on the wrong branch.
• <site-root> contains lanes and shared/, nothing else. A lane's .kiln.local names ../shared as its cache, and ccache's base_dir for the qbs path is set to the site root, which is what lets two lanes hash a file identically (§What is shared).

---

Git — bare repo plus worktrees

One clone. Every lane is a linked worktree off it, so a fetch serves all lanes and there is exactly one copy of the objects on disk.

SITE=~/Desktop/Projects/octomy/OctoMY      # your <site-root>
mkdir -p "$SITE/shared" && cd "$SITE"

git clone --bare git@gitlab.com:octomy/octomy.git shared/OctoMY.git
# A bare clone sets no remote-tracking refspec; without this, `git fetch`
# updates nothing and every lane silently goes stale.
git -C shared/OctoMY.git config remote.origin.fetch '+refs/heads/*:refs/remotes/origin/*'
git -C shared/OctoMY.git fetch origin

for b in main <other-branches>; do
    git -C shared/OctoMY.git worktree add "$SITE/$b" "$b"
done

Use the SSH remote if your key is on the GitLab account. HTTPS works for read-only and is the right fallback on a machine you will only build on, but you cannot push from it — decide this up front rather than discovering it at the first git push.

To add a lane later, for a branch that does not exist yet:

git -C shared/OctoMY.git worktree add -b <branch> "$SITE/<branch>" origin/main

Enable rerere once — with this many parallel branches the same merge conflict is resolved repeatedly, and the cache is per-repo, so all lanes share one memory of how you resolved it:

git -C shared/OctoMY.git config rerere.enabled true

Set the commit identity on the bare repo — worktrees inherit it, so this is one setting for the whole site:

git -C shared/OctoMY.git config user.name  "<your name>"
git -C shared/OctoMY.git config user.email "<your address>"

---

Convert an existing clone into a worktree

Sites that predate the bare-repo layout have lanes that are independent clones — each carrying its own object store (~1 GB per lane here) and needing its own fetch. Converting is worth doing, but there is no in-place transform: the conversion is delete-and-recreate, so the order below is the safety story.

1. Prove the lane holds nothing that exists only on this disk

Three separate questions. All must answer zero:

LANE=<lane>
git -C "$LANE" fetch origin
git -C "$LANE" status --porcelain -uall | wc -l                   # uncommitted (-uall: untracked too)
git -C "$LANE" stash list | wc -l                                 # stashes are commits nobody counts
git -C "$LANE" log --oneline HEAD --not --remotes=origin | wc -l  # reachable from NO remote ref

The third is the one to trust, and it is the only one that answers the real question. A clean git status says nothing about whether the branch was ever pushed. "Ahead of main" is a different question again — a lane can sit 50 commits ahead of main with every one of them already safe on origin/<branch>. Conversely a lane can look idle and be the only copy of 29 commits, because its branch was never pushed anywhere.

Ask a running session before converting its lane. Neither git state nor file mtimes reveal an agent mid-task; a lane that is clean this second can have uncommitted work thirty seconds later.

2. Push what is local-only — but never force

If step 1 found unpushed commits, push them before anything else. First check whether the remote branch has diverged:

git -C "$LANE" rev-list --left-right --count origin/<branch>...HEAD   # behind<TAB>ahead

A non-zero behind count means the remote carries commits this disk does not have — another machine's work. Do not force-push. Push to a backup ref instead, which is purely additive and cannot destroy anything:

git -C "$LANE" push origin "<branch>:refs/heads/backup/$(date +%Y-%m)/<branch>-local-$(date +%Y%m%d)"

Reconciling the two lineages is separate work, and it is better done later with both sides safe than under time pressure with a directory waiting to be deleted.

3. Anchor every lane's HEAD in the bare repo before deleting anything

A commit that exists only in a directory you are about to rm -rf is one command away from being unreachable. Create the branch in the bare repo first, and verify the object is actually there:

SITE=~/Desktop/Projects/octomy/OctoMY
SH="$SITE/shared/OctoMY.git"
BR=$(git -C "$LANE" rev-parse --abbrev-ref HEAD)
SHA=$(git -C "$LANE" rev-parse HEAD)

git -C "$SH" fetch origin
git -C "$SH" cat-file -e "$SHA^{commit}"   # must succeed BEFORE you delete
git -C "$SH" branch "$BR" "$SHA"

4. Delete and re-create

Re-run the step-1 checks immediately before the delete — not once at the top of a batch loop — and skip any lane that fails:

rm -rf "$SITE/$LANE"
git -C "$SH" worktree add "$SITE/$LANE" "$BR"

Converting the lane a session is currently running in works, provided the delete and the worktree add happen in a single command: the directory is re-created before anything else needs it.

---

What the conversion breaks, and how to repair it

Worktree linkage itself comes out correct — .git file, back-reference, HEAD, and git worktree list registration all fine. Everything that breaks is gitignored or untracked infra that a checkout does not recreate.

Build-dir symlinks — recreate them, do not rebuild

debug/, release/ and, for the qbs path, default/ are gitignored symlinks into the build disk when a site arranges its lanes so (§Build output). They do not survive. A lane whose .kiln.local sets output.dir needs no link: the file is gitignored too, so write it again.

This matters far more than it sounds, and in your favour: rm -rf removes a symlink, not its target. The build trees under /build/octomy-builds/<lane>/ are still there, intact, after the lane directory is deleted. Recreating the links restores the full build state — hundreds of GB on a warm lane — and turns what looks like a mandatory from-scratch rebuild into nothing at all.

BUILDDISK=/build/octomy-builds
for cfg in debug default release; do
    mkdir -p "$BUILDDISK/$LANE/$cfg"
    ln -sfn "$BUILDDISK/$LANE/$cfg" "$SITE/$LANE/$cfg"
done

Recreate the links before concluding anything about disk cost. Only a lane that genuinely built inside its own directory, rather than through a symlink, loses its build output — and that loss is real and unrecoverable, so check ls /build/octomy-builds/<lane> before deleting rather than after.

src/libs/libllvm/llvm-src

Gitignored on most branches, so absent after conversion. Absence is not automatically breakage — several long-standing lanes have never had it, and it matters only to a lane that builds LLVM. On the android branch it is a tracked symlink and comes back with the checkout.

Before recreating one by hand, read §Vendored sources: the supported mechanism is the fetch tool materialising a tree by copy from <buildCacheDir>/<pinKey>/<commit>, and linking was tried and withdrawn (I-15). A hand-made symlink to a shared tree is a different arrangement from the documented one; know which you are choosing.

*.qbs.user

Delete any that reappear: rm -f "$SITE/$LANE"/*.qbs.user.

The first qbs resolve is slow and needs --force-probe-execution

Kiln needs no step here: its graph of the checkout sees the new paths. On the qbs path, new paths mean probes must re-run, and cached probe verdicts otherwise survive (§Vendored sources). Measured at ~6 minutes — run it detached, not in a foreground call with a two-minute timeout, or it is SIGTERM'd mid-graph:

cd "$SITE/$LANE" && nohup qbs resolve profile:qt6 config:default --force-probe-execution >resolve.log 2>&1 &

Root-owned build leftovers block the delete

A containerised packaging run (build-deb/) leaves root-owned files. rm -rf fails partway through with Permission denied, having already deleted the source tree and .git — a half-deleted lane. chmod -R u+w does not help; the files are owned by root, not merely read-only.

Anchoring first (step 3) is what makes this survivable rather than a data-loss event. To finish:

sudo rm -rf "$SITE/$LANE"       # verify the path holds no .git first

Or, to avoid sudo, rename the remnant aside and create the worktree beside it — renaming needs write permission only on the parent, which you own.

Post-conversion checklist

cat "$SITE/$LANE/.git"                          # gitdir: .../shared/OctoMY.git/worktrees/<lane>
git -C "$SH" worktree list | grep "$LANE"       # registered
ls -l "$SITE/$LANE"/{default,release}           # symlinks, each -> its OWN lane dir
ccache -p | grep -E 'cache_dir|base_dir'        # base_dir = SITE ROOT (see below)
rm -f "$SITE/$LANE"/*.qbs.user

Then the §Verify the site steps, unchanged.

base_dir is the site root — the directory containing the lanes and shared/. It is not the lane, and not the site root's parent. A shared config file means any process can clobber it for every lane at once (a build-test run has done exactly this, leaving it pointing at a pytest temp directory), so check it after conversion even though conversion does not touch it. Near-zero hit rate on pre-existing files is the symptom.

---

Qt

Qt 6.10.0+ is the declared floor (integration/toolchain-requirements.json). Debian 13 ships 6.8.2, which is below it — so on Debian the Qt online installer is the only option for the Qt itself. Install the apt qt6-*-dev packages anyway for the transitive system -dev libraries:

Kiln builds with the Qt that qt.qmake names in each lane's .kiln.local. kiln check holds it to the floor, and when it is not set it names the newest Qt installation under ~/Qt or /opt/Qt.

On the qbs path, whichever you choose, the qbs profile must point at the same Qt the project resolves against. Getting this wrong does not produce a clear error; it produces hundreds of lines of Dependency 'Qt.bluetooth' not found and product '<lib>' is disabled, because the project's own probe found one Qt and qbs's module provider found another. See Troubleshooting.

---

Kiln

Kiln is the default build path. Every lane of the site builds with it, through one broker for each account.

A new site gets its first kiln from the standalone kiln or from a stage-0 kit, not from qbs: a Makefile and a bash script that kiln writes from its own build plan. The kit needs GNU make, a C++ compiler, Qt with Qt Svg and zlib's development package, and nothing of the qbs path. The procedure is in Build from source. On katana, 8 threads and a 5400 rpm disk, a kit built its first kiln in 67 minutes with GCC, LLVM included, and the machine kept 4 GB of memory free with the default job count.

Once the site has a kiln, kiln kit write <folder> writes a kit for the next site.

Where the site keeps its kiln

Keep one kiln for the whole site in <site-root>/shared/kiln/. Every lane and every session uses that kiln. A session that starts on the site reads the broker's door file to find the kiln, and when no broker runs it looks here (Start here, step 1). So this folder is the answer to "which kiln?" on the site, and nobody has to remember it.

Put a kiln there with kiln publish, which copies the kiln that runs it, with the headers its compiler reads, the headers first and the program last. Publish the first kiln of the site this way, then each newer kiln, built in a lane:

cd "$SITE/main"
kiln build --variant release src/tools/kiln
release/kiln/kiln publish "$SITE/shared/kiln"
"$SITE/shared/kiln/kiln" --version

A broker that runs keeps the program it started from. The next broker, which the next build starts after the broker ends with no work for 10 minutes, runs the new one. Don't let a lane build with a kiln of its own build folder: the broker of the account runs one kiln, and kiln check names the one to use.

A site that keeps its kiln in another folder says so in one place that every session reads, such as the site's own notes, never in this repository: a path of one machine does not belong in the documents of the project.

The vendored trees of kiln

kiln vendor writes the pinned trees (LLVM, llama.cpp, rl.tools and the Vulkan validation layer) into the checkout. Without a cache it downloads each tree that is missing or stale. LLVM took 46 s on y8 on 2026-10-13, in every new checkout. Give each lane the cache folder of the site in its .kiln.local. The path is relative to the checkout, and it is the folder that qbs uses as project.buildCacheDir (§Vendored sources):

cache.dir = ../shared

Then the first kiln vendor of the site downloads a tree once, and every other lane copies it from the cache. kiln check in a lane says when a tree is missing, and gives kiln vendor as the fix.

The object cache of kiln

kiln keeps the objects it compiled in ~/.cache/kiln/objects, which every lane of the user shares. The trim of the cache keeps 60 GB of the disk free (diskFloorGb). On a small system disk with less than 60 GB free, the trim removes every entry, and the cache serves nothing. katana had 37 GB free on 2026-10-11, and its cache kept 0 of 14 105 entries. The broker now says so once, on its standard error, and kiln status shows the same warning while the cause lasts:

kiln: warning: the cache at <folder> keeps nothing: its disk has <n> GiB free and the cache holds <n> GiB, below the disk floor of <n> GiB, so every trim empties it. ...

On a site with a build disk, put the cache there behind a link. Do it when no kiln build or test of the user runs. A broker ends after 10 minutes with no work, and kiln status shows a broker that still runs.

kiln status                                     # no broker
mkdir -p /build/kiln-cache
mv ~/.cache/kiln/objects /build/kiln-cache/objects
ln -s /build/kiln-cache/objects ~/.cache/kiln/objects

The trim measures the free space of the disk that the link points to. If the cache must stay on a small disk, set a floor below its free space:

kiln cache policy set diskFloorGb=10

Memory, swap and disk space

qbs needs much memory and, in debug, much disk. kiln needs much less of both. Compare your machine with this table before the first build. Each value was measured on a site of this page.

The 1.2 TB is from before 4f6ebb59f (2026-10-11), which gives GCC's debug links a gdb index and halves a test program. A full qbs debug tree after that fix was not measured. Do not start a full qbs debug build on a disk with less than 1 TB free.

Give a machine for qbs 64 GB or more of RAM and swap together. With 31 GB of RAM and 1 GB of swap, the first resolve on honeybadger32 froze the machine. The kernel did not stop a process. It dropped the code pages of the running programs and read them back, until the power was cut. Add a swap file:

sudo fallocate -l 64G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
echo '/swapfile none swap sw,pri=10 0 0' | sudo tee -a /etc/fstab
sudo swapon -a
cat /proc/swaps                                 # the file is listed with its size

If the site has a build disk, put the swap file there (Replace or add the build disk).

Install earlyoom. It stops a build tool before the machine stops answering, and it keeps SSH and the desktop alive. The values of honeybadger32:

sudo apt install earlyoom
sudo tee /etc/default/earlyoom <<'EOF'
EARLYOOM_ARGS="-m 5 -s 10 -r 3600 --prefer '^(qbs|cc1plus|cc1|ld|ld\.mold|mold|clang.*|lld)$' --avoid '^(sshd|systemd.*|kwin_wayland.*|plasmashell|Xwayland|sddm.*|earlyoom)$'"
EOF
sudo systemctl restart earlyoom
systemctl is-active earlyoom                    # active

-m 5 -s 10 acts when less than 5 % of the RAM and less than 10 % of the swap are free. On a desktop with another session manager, put its processes in --avoid.

Run a long build detached. Start it with setsid nohup … &, write its output to a log, and follow it with kiln progress --wait (Start here, steps 4 and 5). A build in a plain SSH command or in a tool call with a time limit ends with that command.

The test display (a development machine with a desktop)

On a machine where you work at the desktop, UI tests must not open their windows there. Each window takes the keyboard and mouse focus. Install the test display once, from the lane that stays:

integration/test-display/test-display.sh --install

This writes the systemd user unit with the path of this checkout, then enables and starts it.

A build machine with no desktop does not need it. The details, and what to do when test windows appear on the desktop again, are in Run tests.

---

Xvfb, lavapipe and pactl (GPU suites with no GPU, silent audio)

Two kinds of run start helper programs, and the site must have them:

sudo apt install xvfb mesa-vulkan-drivers pulseaudio-utils

Check them:

command -v Xvfb pactl                     # both programs are on PATH
ls /usr/share/vulkan/icd.d/lvp_icd*.json  # the lavapipe manifest
pactl info | head -3                      # pactl reaches a sound server

Without Xvfb or the lavapipe manifest, kiln test --profile gpu-sw stops with exit code 4 and names the package. Without pactl, a run with test.audio = null writes a warning and goes on, and the sound of the suites can reach the speakers. A build server that runs many test runs can load one null sink when it starts. kiln then uses that sink and leaves it loaded. See Run tests.

---

Verify the site

Every one of these fails silently if you skip it. Run them in order.

# 1. All lanes present, each on its own branch.
git -C shared/OctoMY.git worktree list

# 2. The site's kiln is there, and is the kiln you expect.
"$SITE/shared/kiln/kiln" --version

# 3. Each lane is ready to build. Run each command a fix line gives, until
#    the last line is: next: kiln build
cd "$SITE/main" && "$SITE/shared/kiln/kiln" check

# 4. The cache of the vendored trees serves: the second lane copies, and
#    downloads nothing.
grep cache.dir "$SITE/main/.kiln.local"

# 5. kiln's object cache keeps what it stores: no warning about the disk floor.
"$SITE/shared/kiln/kiln" cache policy

Step 3 also tells you when a broker of the account runs another kiln than the site's. Use the kiln it names, and publish a newer one into shared/kiln/ when the lanes need it (§Where the site keeps its kiln).

On the qbs path, run Verify the qbs path too.

---

What is shared between lanes

Build output is never shared — but it may be relocated

Two different things, and conflating them is how sites break.

Never shared. Each lane has its own build folder (Start here → The build convention). The qbs build graph (default/default.bg) stores absolute paths to every source and artifact, so a default/ tree reached by two lanes makes qbs read sources from one lane and write artifacts into another, silently. See Qbs reference → The qbs build convention; after copying a checkout, delete every copied build tree before the first build.

Relocating is supported, and is how a site with a small system disk and a large fast one is arranged. For kiln, set output.dir in each lane's .kiln.local to a folder of that lane on the build disk, for example output.dir = /build/octomy-builds/<lane>. For qbs, give each lane its own directory on the build disk and symlink to it, one-to-one:

BUILDDISK=/build/octomy-builds
for lane in <lanes>; do
    mkdir -p "$BUILDDISK/$lane/default" "$BUILDDISK/$lane/release"
    ln -sfn "$BUILDDISK/$lane/default" "$SITE/$lane/default"
    ln -sfn "$BUILDDISK/$lane/release" "$SITE/$lane/release"
done

This is safe because the path qbs records — <lane>/default — stays stable; only the storage behind it moves. It stays safe only while the mapping is exactly one lane to one target directory, named for that lane.

Audit it, because a wrong link is silent and looks like ordinary staleness — a lane building fine while its artifacts land in another lane's tree:

for d in "$SITE"/*/; do
    lane=$(basename "$d")
    for k in default release; do
        [ -L "$d$k" ] || continue
        tgt=$(readlink -f "$d$k")
        case "$tgt" in
            */"$lane"/"$k") ;;
            *) echo "MISMATCH: $lane/$k -> $tgt" ;;
        esac
    done
done

Watch free space on the build disk too — build trees are tens of GB per lane and a full disk shows up as a mid-build kill, not a compile error.

Do not hand-delete parts of a live default/ tree. Removing a product directory the graph still references fails at link with ar: <object>: No such file or directory. Delete the whole default/ and re-resolve.

The compiler cache of the qbs path is partitioned by branch

Kiln's object cache keys a compile by its command and inputs, which are checkout-relative, so lanes on different branches share every compile whose inputs match. The rest of this section is about ccache, on the qbs path.

base_dir and hash_dir=false remove the path difference between lanes, so the same file at <site>/paint/… and <site>/comms/… hashes identically. That is the part that works.

But every compile command also carries -DOC_GIT_BRANCH="<branch>", a gitStableDefine applied globally to every product, and ccache hashes the full command line. Two lanes on different branches therefore share no cache entries at all, for any object. The cache is partitioned one bucket per branch.

What cross-checkout sharing actually buys, then:

• A lane whose default/ is deleted and rebuilt — near-total hits.
• A lane re-created at the same branch, or a tree copied between machines.
• Two lanes that genuinely sit on the same branch.

It does not make eight lanes on eight branches share compile work. That is the intended trade — OC_GIT_BRANCH is how a running node reports its provenance — but it is worth knowing before you size max_size or explain a disappointing hit rate. Measurements belong in content/doc/internal/specs/build-performance/.

---

The qbs path

qbs is the second build path until it retires, and some legs of the build farm still use it. A site that builds only with kiln needs nothing of this section. The facts of the qbs path are in Qbs reference → The qbs build convention.

ccache

Full rationale, gotchas and maintenance live in ccache Setup. The site-level requirement is just this:

ccache --set-config base_dir="$SITE"      # the dir CONTAINING the lanes
ccache --set-config hash_dir=false
ccache --set-config max_size=40G        # 500G with a dedicated build disk

base_dir is the site root, never a lane. Point it at a lane and that lane alone gets cache hits, which looks like it is working right up until you check the second one. The build prints a warning naming these commands if they are missing.

That is the whole of it — do not also wire ccache into the qbs profile. The build adds ccache to the compiler wrapper itself; a profile that repeats it double-wraps. See §Qbs.

Read §What is shared before assuming this makes lanes share cache entries — it is narrower than it sounds.

If the cache and the build trees go on a dedicated disk, prepare it with Replace or add the build disk. That page also lists what a disk swap loses.

Qbs

The qbs path requires a qbs newer than any distribution ships. The floor, the versions proven on it and the ones known bad all live in integration/toolchain-requirements.json — plain JSON, readable without running qbs, deliberately. Read qbs.minimum from there; it is the only place that number is stated (internal spec build-self-sufficiency, §BSS-P-15).

Debian 13's qbs is on the knownBad list: its parallel scheduler SIGSEGVs on the android multi-arch graph.

The commands below carry a concrete version because a curl line must. If they fail against the floor, the JSON is right and this page is stale — fix this page.

project.qbs enforces the floor and its error message carries the recipe. Two build dependencies the recipe does not name, and which fail configure in a way that does not obviously point at them:

sudo apt install qt6-base-private-dev qt6-5compat-dev cmake ninja-build

• Without qt6-base-private-dev: Imported target "Qt6::CorePrivate" includes non-existent path .../QtCore/<version>.
• Without qt6-5compat-dev: Could NOT find Qt6Core5Compat.

Then:

curl -sLO https://download.qt.io/official_releases/qbs/3.3.1/qbs-src-3.3.1.tar.gz
tar xf qbs-src-3.3.1.tar.gz
cmake -S qbs-src-3.3.1 -B b -GNinja -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_PREFIX_PATH=<QtDir> -DCMAKE_INSTALL_PREFIX=/usr/local \
      -DCMAKE_INSTALL_RPATH=<QtDir>/lib -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON \
      -DWITH_TESTS=OFF
cmake --build b -j"$(nproc)" && sudo cmake --install b

Building qbs against the distro Qt is the simplest choice and needs neither rpath flag: qbs then runs on system libraries and is unaffected by what you later install or remove under ~/Qt. The two rpath flags matter only when <QtDir> is a ~/Qt tree — without them the installed binary picks up the distro Qt at runtime and dies with version 'Qt_6.10' not found.

/usr/local/bin precedes /usr/bin on a default Debian PATH, so the built qbs shadows any packaged one. Confirm with which -a qbs — both will be listed, and the first must be the one you built.

One-time profile setup:

qbs setup-toolchains --detect
qbs setup-qt <QtDir>/bin/qmake6 qt6
qbs config --list profiles | grep -oE '^profiles\.[a-zA-Z0-9_-]+' | sort -u   # find the DETECTED name
qbs config profiles.qt6.baseProfile x86_64-linux-gnu-gcc-14                   # <- that name, not "gcc"
qbs config defaultProfile qt6

baseProfile must name a profile setup-toolchains --detect created, such as x86_64-linux-gnu-gcc-14. The grep above prints the ones you actually have.

A wrong baseProfile is silently valid and fails far away. gcc is a name qbs accepts without complaint, and the resolve then dies thousands of lines later as something that looks like a project bug. Measured 2026-09-01, one machine, one wrong value, two unrelated-looking errors:

ERROR: Cyclic dependencies detected:
  First product is 'libglslang'.  Second product is 'stl-watermark'.

ERROR: Dependency 'Qt.core' not found for product 'autotest-runner'.

All three products are innocent. Neither message mentions profiles. If you see either, check baseProfile before you read any project source.

Do not set cpp.compilerWrapper. BuildOptions.qbs appends CompilerWrapper.ccache() to the low-priority prefix itself, so setting it in the profile too yields nice -n 19 ccache ccache g++ — two nested cache processes on every compile. Nothing breaks, which is why it survives unnoticed; it had been live on the main build machine long enough that nobody remembered setting it. CcacheProbe now warns about this at resolve time, so you should not have to remember this paragraph.

Linkers

kiln links with the lld that it carries, so a kiln build needs no linker on the machine. The rest of this section is about qbs builds.

mold is recommended for a qbs debug build. project.useMoldLinker is true by default, so a debug link uses -fuse-ld=mold when mold is there. lld is used only for a release build with clang, where project.enableLto is true by default. A release build with GCC links its LTO objects with bfd, which comes with binutils.

sudo apt install mold lld

The resolve probes both linkers (linkerProbe in project.qbs). If a wanted linker is not there, the resolve gives one warning, and the build links with the toolchain's default linker. The link is slower, and the build does not fail. To build without mold on purpose: qbs build project.useMoldLinker:false.

Vendored sources and project.buildCacheDir

LLVM, llama.cpp and rl.tools are vendored: pinned by commit, fetched on demand, and cached per commit so that a pin bump is a cache miss rather than a silent compile of the previous version.

qbs resolve project.buildCacheDir:"$SITE/shared"

This was BUILD_CACHE_DIR, an environment variable, until 2026-09-16 (no-environment-variables row B6). The advice then was to pass it inline on every command rather than export it, so that it stayed visible at the point it mattered and could not silently redirect every future resolve in every lane.

A project property is that advice made structural, and better in the one way inline passing was weak: it sticks in the build graph. A resolve that forgot the inline prefix used to fall back to no cache and fetch from scratch, quietly. Set it once per build directory and every later qbs build in that directory uses it; change it with another qbs resolve.

Trees are then keyed at <buildCacheDir>/<pinKey>/<commit>[+<overrides-fingerprint>] — note the directory is the pin key (llvm), not the in-tree directory name (llvm-src) — shared by every lane, and materialised into a lane by copying — never by symlink. Linking was tried and withdrawn (I-15): a builder running as root wrote a link to a container-only path into a bind-mounted worktree, where it was permanently dangling. A copy the fetch tool performs, only when asked, is the trade taken instead.

Bootstrap on a fresh site — the fetch tool is itself a product of this project, so it does not exist until something builds it. The full contract, and why the second resolve cannot be skipped, is in Qbs reference → Bootstrap:

qbs build -p llvm-source                # builds llvm-fetch, then runs it
qbs resolve --force-probe-execution     # a plain re-resolve is NOT enough

Repeat per subsystem you want: llvm-source, llama-source, rl-source. Measured here from cold: LLVM 3 m 26 s (download + unpack, 10465 files) and 0.9 s for the next lane, served from the shared cache.

Probe results live in the build graph and cache on properties rather than files, so a probe that once saw "missing" serves that verdict forever without --force-probe-execution.

Leaving project.buildCacheDir unset is a supported state: the three subsystems resolve as BUILT OUT, the build proceeds, and -DOC_NO_LLVM is defined. It is the right choice on a machine that will never need firmware compilation or local inference — those are large builds.

This used to carry a note arguing that the environment variable was not a violation of the no-environment-variables rule, on the grounds that AGENTS.md forbids the application depending on env vars and this one was read by the build at resolve time, never by a running node.

That argument was overturned on 2026-09-15. The rule now covers the build too: "qbs and JavaScript take project properties, not Environment.getEnv" (§NEV-3.3). The exemption is recorded here rather than deleted because it was reasoned, and a reader who remembers it should see that it was overturned rather than forgotten.

The pre-push build hook (optional)

integration/install-git-hooks.sh installs a pre-push hook that refuses to push a tree that does not compile. Before each push it runs, from the root of the worktree that pushes:

qbs resolve --force-probe-execution project.fileCacheMode:"never"
qbs build --keep-going

That is the plain qbs build of the qbs path, after a resolve that defeats the probe cache so new or deleted test folders are seen. The hook runs no policy guard: a guard can be red on purpose while a known violation is open, and that must not stop a push. It prints how many compile and link steps ran, up to 20 files that did not compile, and a warning when src/libs/libllvm/llvm-src arrived during the build. The full qbs output goes to octomy-pre-push-build.log in the worktree's git folder, and is deleted when the build passes.

integration/install-git-hooks.sh               # install
integration/install-git-hooks.sh --uninstall   # remove
integration/install-git-hooks.sh --repo <dir>  # install into another checkout

The installer refuses to overwrite or remove a pre-push hook it did not write.

Every lane shares the hook. Linked worktrees share one hooks folder (the installer asks git rev-parse --git-path hooks, which also honours core.hooksPath). Installing it in one lane puts the build gate on every push from every lane on the site, so it is the owner's decision, not a local one.

To push without it, use git's own escape:

git push --no-verify

There is no environment variable for this. The OCTOMY_SKIP_BUILD_HOOK variable the older hook read is gone.

The installer and the hook it writes are POSIX sh and POSIX awk, because git runs the hook before any build exists. The design is §SCP-D-3 in content/doc/internal/completed/script-ports-2026-10-15/design.md.

Verify the qbs path

These checks are for the qbs path. Kiln needs none of them: kiln check in a lane covers its part (Verify the site).

Every one of these fails silently if you skip it. Run them in order.

# 1. All lanes present, each on its own branch.
git -C shared/OctoMY.git worktree list

# 2. ccache configured for the SITE, not a lane.
ccache -p | grep -E 'base_dir|hash_dir|max_size'

# 3. The qbs you built is the one on PATH.
which -a qbs && qbs --version            # must be >= 3.3.1

# 3b. The profile is the one you think, and baseProfile names a DETECTED
#     toolchain. A wrong-but-valid value here is the single most expensive
#     mistake on this page — see §Qbs.
qbs config defaultProfile
qbs config --list | grep -E '^profiles\.<profile>\.'
#     baseProfile must be e.g. x86_64-linux-gnu-gcc-14, never plain "gcc".
#     cpp.compilerWrapper must be ABSENT.

# 4. Resolve is clean. Check the Qt path it prints against your intended Qt,
#    and the optional-module table for anything unexpectedly MISSING.
cd "$SITE/main" && qbs resolve

# 5. ccache is in the compile command — EXACTLY ONE of them.
qbs build --command-echo-mode command-line -p libutil 2>&1 | grep -m1 -o 'ccache [^ ]*g++'

# 6. The cache actually serves. Delete the build tree, rebuild, expect ~100% hits.
#    `default` can be a link to the build disk: delete its target, not the link.
T=$(readlink -f default)
ccache -z && rm -rf "$T" && mkdir -p "$T" && qbs resolve && qbs build -p libutil
ccache -s | grep -iE 'hits:|misses:'

Step 5 is the one people skip. cpp.compilerWrapper is evaluated at resolve time, so a graph resolved on a machine without ccache keeps building without it after you install ccache — until you re-resolve. "Is ccache on?" really means "was it present when this tree was last resolved?"

Step 5 also catches the double-wrap. nice -n 19 ccache ccache g++ means the profile is setting cpp.compilerWrapper on top of the one the build supplies; the grep above shows only the tail, so read the whole line, not the match. CcacheProbe warns about this at resolve time too, but only when ccache is installed — the manual check has no such precondition.

Step 3b costs a second and would have saved an hour on 2026-09-01, when a baseProfile of gcc produced a fake cyclic-dependency error naming three innocent products.

---

Troubleshooting

Each entry names a symptom, its cause and the fix.

• Symptom: kiln check gives a fix line. Cause: The lane is not ready for kiln yet. Fix: Run each command it gives until next: kiln build.
• Symptom: kiln check says the broker runs another kiln. Cause: The lane runs a kiln that is not the broker's. Fix: Use the kiln it names, the site's kiln (§Where the site keeps its kiln).
• Symptom: qbs 2.6.1 is below the minimum 3.3.1. Cause: Distro qbs on PATH. Fix: Build 3.3.1 from source (§Qbs).
• Symptom: Cyclic dependencies detected naming libglslang / stl-watermark. Cause: baseProfile names a profile that is not a detected toolchain (e.g. plain gcc). The products are innocent. Fix: qbs config profiles.<p>.baseProfile x86_64-linux-gnu-gcc-14, then re-resolve (§Qbs).
• Symptom: Dependency 'Qt.core' not found for product 'autotest-runner'. Cause: Same wrong baseProfile — a different lane surfaces it differently. Fix: As above. Check the profile before reading any project source.
• Symptom: Compile line shows nice -n 19 ccache ccache g++. Cause: Profile sets cpp.compilerWrapper and the build appends its own. Fix: qbs config --unset profiles.<p>.cpp.compilerWrapper (§Qbs).
• Symptom: Hundreds of product '<lib>' is disabled, Dependency 'Qt.bluetooth' not found. Cause: qbs profile points at a different Qt than the project resolved. Fix: qbs setup-qt <the Qt the resolve log printed>/bin/qmake6 qt6, then re-resolve.
• Symptom: Imported target "Qt6::CorePrivate" includes non-existent path. Cause: Building qbs without Qt private headers. Fix: apt install qt6-base-private-dev.
• Symptom: Could NOT find Qt6Core5Compat. Cause: Building qbs on a lean Qt. Fix: apt install qt6-5compat-dev.
• Symptom: Compile line shows ccache ccache g++. Cause: cpp.compilerWrapper set by hand. Fix: qbs config --unset profiles.<p>.cpp.compilerWrapper.
• Symptom: ccache installed, zero hits, no warning. Cause: Graph resolved before ccache existed. Fix: qbs resolve.
• Symptom: vendored sources missing after a successful fetch. Cause: Probe answer cached in the build graph. Fix: qbs resolve --force-probe-execution.
• Symptom: ar: <object>: No such file or directory. Cause: Part of a live default/ tree was deleted. Fix: T=$(readlink -f default); rm -rf "$T" && mkdir -p "$T" && qbs resolve (a default link stays).
• Symptom: git fetch updates nothing on a bare clone. Cause: No remote-tracking refspec. Fix: See §Git.
• Symptom: A lane builds fine but its artifacts never change. Cause: <lane>/default symlinked to another lane's directory. Fix: Run the audit loop in §Build output.
• Symptom: Build killed mid-way, no compile error. Cause: Build disk full. Fix: df -h the build disk; delete stale .build/ trees.
• Symptom: The machine stops answering during a qbs resolve or qbs build. Cause: Too little RAM and swap: the kernel drops program code and reads it back, and stops no process. Fix: Power off, then add swap and earlyoom (§Memory, swap and disk space).
• Symptom: kiln build takes nothing from its cache, and the cache stays empty. Cause: The cache disk has less free space than diskFloorGb (60 GB), so the trim removes every entry. Fix: Put the cache on the build disk, or lower the floor (§The object cache of kiln).
• Symptom: Cache hit rate near zero across the whole site. Cause: base_dir left pointing at a stale or unrelated path. Fix: ccache -p | grep base_dir — it must be the site root.
• Symptom: rm -rf <lane> stops with Permission denied, leaving a half-deleted lane. Cause: Root-owned files from a containerized build-deb/ run. Fix: sudo rm -rf the remnant, or rename it aside; see §Convert.
• Symptom: A freshly converted lane wants a full from-scratch rebuild. Cause: default/release symlinks were not recreated — the targets on the build disk are still there. Fix: Recreate the links (§Convert); do not rebuild until you have.
• Symptom: A converted lane's git status is clean but its work is nowhere on origin. Cause: Branch was never pushed; status does not answer that question. Fix: git log --oneline HEAD --not --remotes=origin before converting, never after.
• Symptom: qbs resolve killed after ~2 minutes on a converted lane. Cause: First resolve re-runs probes and takes ~6 min. Fix: Run it detached (§Convert); kiln needs no resolve.