Set up a build site

A build site is one machine carrying several checkouts of this project at once — one per branch you have work in flight on — sharing a single git object store and a single compiler cache. It is the standard working arrangement here: several agent or human sessions, each in its own lane, none of them stepping on another's index.

This page is the whole setup, start to finish. It exists so that standing up a new machine is a runbook rather than an interview with whoever set up the last one.

TL;DR

1. One bare repo at shared/OctoMY.git; every lane is a git worktree.
2. ccache, with base_dir pointing at the directory that contains the lanes.
3. Qt 6.10.0+, and qbs ≥ 3.3.1 — which you must build from source (§Qbs).
4. mold and lld — both required, neither probed (§Linkers).
5. project.buildCacheDir for the vendored LLVM/llama/rl trees.
6. Run the verification checklist — every step has a command that proves it, because most of these fail silently.

---

The layout

<site-root>/                      <- ccache base_dir points HERE
├── shared/
│   ├── OctoMY.git/               <- the ONLY git object store
│   └── llvm/  llama/  rl/        <- project.buildCacheDir, keyed by pin
├── main/                         <- one worktree per lane
├── <branch>/                     <- named for its branch, always
└── <branch>/

A lane may be a linked worktree (preferred — one object store, one fetch) or an independent clone. Older sites grew a mix of both, because each lane was originally cloned on its own. Worktrees are the target arrangement: see §Convert an existing clone into a worktree for the procedure and for what it breaks.

Know which kind you are in before reasoning about disk or fetches — git -C shared/OctoMY.git worktree list shows only the worktrees, so a lane missing from that list is a clone, not a bug. The other tell is .git itself: a file containing gitdir: means worktree, a directory means clone.

Two rules make the rest work:

• A lane directory is named for its branch, and holds only that branch. The name is how a session answers "where am I?" — see the pwd && git status habit in AGENTS.md. A lane whose directory name and branch disagree is the single most reliable way to land work on the wrong branch.
• <site-root> contains lanes and shared/, nothing else. ccache's base_dir is set to it, and that is what lets two lanes hash a file identically (§What is shared).

---

Git — bare repo plus worktrees

One clone. Every lane is a linked worktree off it, so a fetch serves all lanes and there is exactly one copy of the objects on disk.

SITE=~/Desktop/Projects/octomy/OctoMY      # your <site-root>
mkdir -p "$SITE/shared" && cd "$SITE"

git clone --bare git@gitlab.com:octomy/octomy.git shared/OctoMY.git
# A bare clone sets no remote-tracking refspec; without this, `git fetch`
# updates nothing and every lane silently goes stale.
git -C shared/OctoMY.git config remote.origin.fetch '+refs/heads/*:refs/remotes/origin/*'
git -C shared/OctoMY.git fetch origin

for b in main <other-branches>; do
    git -C shared/OctoMY.git worktree add "$SITE/$b" "$b"
done

Use the SSH remote if your key is on the GitLab account. HTTPS works for read-only and is the right fallback on a machine you will only build on, but you cannot push from it — decide this up front rather than discovering it at the first git push.

To add a lane later, for a branch that does not exist yet:

git -C shared/OctoMY.git worktree add -b <branch> "$SITE/<branch>" origin/main

Enable rerere once — with this many parallel branches the same merge conflict is resolved repeatedly, and the cache is per-repo, so all lanes share one memory of how you resolved it:

git -C shared/OctoMY.git config rerere.enabled true

Set the commit identity on the bare repo — worktrees inherit it, so this is one setting for the whole site:

git -C shared/OctoMY.git config user.name  "<your name>"
git -C shared/OctoMY.git config user.email "<your address>"

---

Convert an existing clone into a worktree

Sites that predate the bare-repo layout have lanes that are independent clones — each carrying its own object store (~1 GB per lane here) and needing its own fetch. Converting is worth doing, but there is no in-place transform: the conversion is delete-and-recreate, so the order below is the safety story.

1. Prove the lane holds nothing that exists only on this disk

Three separate questions. All must answer zero:

LANE=<lane>
git -C "$LANE" fetch origin
git -C "$LANE" status --porcelain -uall | wc -l                   # uncommitted (-uall: untracked too)
git -C "$LANE" stash list | wc -l                                 # stashes are commits nobody counts
git -C "$LANE" log --oneline HEAD --not --remotes=origin | wc -l  # reachable from NO remote ref

The third is the one to trust, and it is the only one that answers the real question. A clean git status says nothing about whether the branch was ever pushed. "Ahead of main" is a different question again — a lane can sit 50 commits ahead of main with every one of them already safe on origin/<branch>. Conversely a lane can look idle and be the only copy of 29 commits, because its branch was never pushed anywhere.

Ask a running session before converting its lane. Neither git state nor file mtimes reveal an agent mid-task; a lane that is clean this second can have uncommitted work thirty seconds later.

2. Push what is local-only — but never force

If step 1 found unpushed commits, push them before anything else. First check whether the remote branch has diverged:

git -C "$LANE" rev-list --left-right --count origin/<branch>...HEAD   # behind<TAB>ahead

A non-zero behind count means the remote carries commits this disk does not have — another machine's work. Do not force-push. Push to a backup ref instead, which is purely additive and cannot destroy anything:

git -C "$LANE" push origin "<branch>:refs/heads/backup/$(date +%Y-%m)/<branch>-local-$(date +%Y%m%d)"

Reconciling the two lineages is separate work, and it is better done later with both sides safe than under time pressure with a directory waiting to be deleted.

3. Anchor every lane's HEAD in the bare repo before deleting anything

A commit that exists only in a directory you are about to rm -rf is one command away from being unreachable. Create the branch in the bare repo first, and verify the object is actually there:

SITE=~/Desktop/Projects/octomy/OctoMY
SH="$SITE/shared/OctoMY.git"
BR=$(git -C "$LANE" rev-parse --abbrev-ref HEAD)
SHA=$(git -C "$LANE" rev-parse HEAD)

git -C "$SH" fetch origin
git -C "$SH" cat-file -e "$SHA^{commit}"   # must succeed BEFORE you delete
git -C "$SH" branch "$BR" "$SHA"

4. Delete and re-create

Re-run the step-1 checks immediately before the delete — not once at the top of a batch loop — and skip any lane that fails:

rm -rf "$SITE/$LANE"
git -C "$SH" worktree add "$SITE/$LANE" "$BR"

Converting the lane a session is currently running in works, provided the delete and the worktree add happen in a single command: the directory is re-created before anything else needs it.

---

What the conversion breaks, and how to repair it

Worktree linkage itself comes out correct — .git file, back-reference, HEAD, and git worktree list registration all fine. Everything that breaks is gitignored or untracked infra that a checkout does not recreate.

Build-dir symlinks — recreate them, do not rebuild

default/ and release/ are gitignored symlinks into the build disk (§Build output). They do not survive.

This matters far more than it sounds, and in your favour: rm -rf removes a symlink, not its target. The build trees under /build/octomy-builds/<lane>/ are still there, intact, after the lane directory is deleted. Recreating the links restores the full build state — hundreds of GB on a warm lane — and turns what looks like a mandatory from-scratch rebuild into nothing at all.

BUILDDISK=/build/octomy-builds
for cfg in default release; do
    mkdir -p "$BUILDDISK/$LANE/$cfg"
    ln -sfn "$BUILDDISK/$LANE/$cfg" "$SITE/$LANE/$cfg"
done

Recreate the links before concluding anything about disk cost. Only a lane that genuinely built inside its own directory, rather than through a symlink, loses its build output — and that loss is real and unrecoverable, so check ls /build/octomy-builds/<lane> before deleting rather than after.

src/libs/libllvm/llvm-src

Gitignored on most branches, so absent after conversion. Absence is not automatically breakage — several long-standing lanes have never had it, and it matters only to a lane that builds LLVM. On the android branch it is a tracked symlink and comes back with the checkout.

Before recreating one by hand, read §Vendored sources: the supported mechanism is the fetch tool materialising a tree by copy from <buildCacheDir>/<pinKey>/<commit>, and linking was tried and withdrawn (I-15). A hand-made symlink to a shared tree is a different arrangement from the documented one; know which you are choosing.

*.qbs.user

Delete any that reappear: rm -f "$SITE/$LANE"/*.qbs.user.

The first resolve is slow and needs --force-probe-execution

New paths mean probes must re-run, and cached probe verdicts otherwise survive (§Vendored sources). Measured at ~6 minutes — run it detached, not in a foreground call with a two-minute timeout, or it is SIGTERM'd mid-graph:

cd "$SITE/$LANE" && nohup qbs resolve profile:qt6101 config:default --force-probe-execution >resolve.log 2>&1 &

Root-owned build leftovers block the delete

A containerised packaging run (build-deb/) leaves root-owned files. rm -rf fails partway through with Permission denied, having already deleted the source tree and .git — a half-deleted lane. chmod -R u+w does not help; the files are owned by root, not merely read-only.

Anchoring first (step 3) is what makes this survivable rather than a data-loss event. To finish:

sudo rm -rf "$SITE/$LANE"       # verify the path holds no .git first

Or, to avoid sudo, rename the remnant aside and create the worktree beside it — renaming needs write permission only on the parent, which you own.

Post-conversion checklist

cat "$SITE/$LANE/.git"                          # gitdir: .../shared/OctoMY.git/worktrees/<lane>
git -C "$SH" worktree list | grep "$LANE"       # registered
ls -l "$SITE/$LANE"/{default,release}           # symlinks, each -> its OWN lane dir
ccache -p | grep -E 'cache_dir|base_dir'        # base_dir = SITE ROOT (see below)
rm -f "$SITE/$LANE"/*.qbs.user

Then the §Verify the site steps, unchanged.

base_dir is the site root — the directory containing the lanes and shared/. It is not the lane, and not the site root's parent. A shared config file means any process can clobber it for every lane at once (a build-test run has done exactly this, leaving it pointing at a pytest temp directory), so check it after conversion even though conversion does not touch it. Near-zero hit rate on pre-existing files is the symptom.

---

ccache

Full rationale, gotchas and maintenance live in ccache Setup. The site-level requirement is just this:

ccache --set-config base_dir="$SITE"      # the dir CONTAINING the lanes
ccache --set-config hash_dir=false
ccache --set-config max_size=40G        # 500G with a dedicated build disk

base_dir is the site root, never a lane. Point it at a lane and that lane alone gets cache hits, which looks like it is working right up until you check the second one. The build prints a warning naming these commands if they are missing.

That is the whole of it — do not also wire ccache into the qbs profile. The build adds ccache to the compiler wrapper itself; a profile that repeats it double-wraps. See §Qbs.

Read §What is shared before assuming this makes lanes share cache entries — it is narrower than it sounds.

If the cache and the build trees go on a dedicated disk, prepare it with Replace or add the build disk. That page also lists what a disk swap loses.

---

Qt

Qt 6.10.0+ is the declared floor (integration/toolchain-requirements.json). Debian 13 ships 6.8.2, which is below it — so on Debian the Qt online installer is the only option for the Qt itself. Install the apt qt6-*-dev packages anyway for the transitive system -dev libraries:

Whichever you choose, the qbs profile must point at the same Qt the project resolves against. Getting this wrong does not produce a clear error; it produces hundreds of lines of Dependency 'Qt.bluetooth' not found and product '<lib>' is disabled, because the project's own probe found one Qt and qbs's module provider found another. See Troubleshooting.

---

Qbs

The build requires a qbs newer than any distribution ships. The floor, the versions proven on it and the ones known bad all live in integration/toolchain-requirements.json — plain JSON, readable without running qbs, deliberately. Read qbs.minimum from there; it is the only place that number is stated (internal spec build-self-sufficiency, §BSS-P-15).

Debian 13's qbs is on the knownBad list: its parallel scheduler SIGSEGVs on the android multi-arch graph.

The commands below carry a concrete version because a curl line must. If they fail against the floor, the JSON is right and this page is stale — fix this page.

project.qbs enforces the floor and its error message carries the recipe. Two build dependencies the recipe does not name, and which fail configure in a way that does not obviously point at them:

sudo apt install qt6-base-private-dev qt6-5compat-dev cmake ninja-build

• Without qt6-base-private-dev: Imported target "Qt6::CorePrivate" includes non-existent path .../QtCore/<version>.
• Without qt6-5compat-dev: Could NOT find Qt6Core5Compat.

Then:

curl -sLO https://download.qt.io/official_releases/qbs/3.3.1/qbs-src-3.3.1.tar.gz
tar xf qbs-src-3.3.1.tar.gz
cmake -S qbs-src-3.3.1 -B b -GNinja -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_PREFIX_PATH=<QtDir> -DCMAKE_INSTALL_PREFIX=/usr/local \
      -DCMAKE_INSTALL_RPATH=<QtDir>/lib -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON \
      -DWITH_TESTS=OFF
cmake --build b -j"$(nproc)" && sudo cmake --install b

Building qbs against the distro Qt is the simplest choice and needs neither rpath flag: qbs then runs on system libraries and is unaffected by what you later install or remove under ~/Qt. The two rpath flags matter only when <QtDir> is a ~/Qt tree — without them the installed binary picks up the distro Qt at runtime and dies with version 'Qt_6.10' not found.

/usr/local/bin precedes /usr/bin on a default Debian PATH, so the built qbs shadows any packaged one. Confirm with which -a qbs — both will be listed, and the first must be the one you built.

One-time profile setup:

qbs setup-toolchains --detect
qbs setup-qt <QtDir>/bin/qmake6 qt6
qbs config --list profiles | grep -oE '^profiles\.[a-zA-Z0-9_-]+' | sort -u   # find the DETECTED name
qbs config profiles.qt6.baseProfile x86_64-linux-gnu-gcc-14                   # <- that name, not "gcc"
qbs config defaultProfile qt6

baseProfile must name a profile setup-toolchains --detect created, such as x86_64-linux-gnu-gcc-14. The grep above prints the ones you actually have.

A wrong baseProfile is silently valid and fails far away. gcc is a name qbs accepts without complaint, and the resolve then dies thousands of lines later as something that looks like a project bug. Measured 2026-09-01, one machine, one wrong value, two unrelated-looking errors:

ERROR: Cyclic dependencies detected:
  First product is 'libglslang'.  Second product is 'stl-watermark'.

ERROR: Dependency 'Qt.core' not found for product 'autotest-runner'.

All three products are innocent. Neither message mentions profiles. If you see either, check baseProfile before you read any project source.

Do not set cpp.compilerWrapper. BuildOptions.qbs appends CompilerWrapper.ccache() to the low-priority prefix itself, so setting it in the profile too yields nice -n 19 ccache ccache g++ — two nested cache processes on every compile. Nothing breaks, which is why it survives unnoticed; it had been live on the main build machine long enough that nobody remembered setting it. CcacheProbe now warns about this at resolve time, so you should not have to remember this paragraph.

---

Linkers

mold is required for the default (debug) build — project.useMoldLinker defaults to true, so debug links pass -fuse-ld=mold. lld is required for release, where project.enableLto defaults to true and LTO needs it.

sudo apt install mold lld

Unlike ccache, neither is probed at resolve time: a site without mold compiles the entire tree and then fails at the link step. Install both up front rather than discovering it an hour in. To build without mold on purpose: qbs build project.useMoldLinker:false.

---

Vendored sources and project.buildCacheDir

LLVM, llama.cpp and rl.tools are vendored: pinned by commit, fetched on demand, and cached per commit so that a pin bump is a cache miss rather than a silent compile of the previous version.

qbs resolve project.buildCacheDir:"$SITE/shared"

This was BUILD_CACHE_DIR, an environment variable, until 2026-09-16 (no-environment-variables row B6). The advice then was to pass it inline on every command rather than export it, so that it stayed visible at the point it mattered and could not silently redirect every future resolve in every lane.

A project property is that advice made structural, and better in the one way inline passing was weak: it sticks in the build graph. A resolve that forgot the inline prefix used to fall back to no cache and fetch from scratch, quietly. Set it once per build directory and every later qbs build in that directory uses it; change it with another qbs resolve.

Trees are then keyed at <buildCacheDir>/<pinKey>/<commit>[+<overrides-fingerprint>] — note the directory is the pin key (llvm), not the in-tree directory name (llvm-src) — shared by every lane, and materialised into a lane by copying — never by symlink. Linking was tried and withdrawn (I-15): a builder running as root wrote a link to a container-only path into a bind-mounted worktree, where it was permanently dangling. A copy the fetch tool performs, only when asked, is the trade taken instead.

Bootstrap on a fresh site — the fetch tool is itself a product of this project, so it does not exist until something builds it. The full contract, and why the second resolve cannot be skipped, is in Qbs reference → Bootstrap:

qbs build -p llvm-source                # builds llvm-fetch, then runs it
qbs resolve --force-probe-execution     # a plain re-resolve is NOT enough

Repeat per subsystem you want: llvm-source, llama-source, rl-source. Measured here from cold: LLVM 3 m 26 s (download + unpack, 10465 files) and 0.9 s for the next lane, served from the shared cache.

Probe results live in the build graph and cache on properties rather than files, so a probe that once saw "missing" serves that verdict forever without --force-probe-execution.

Leaving project.buildCacheDir unset is a supported state: the three subsystems resolve as BUILT OUT, the build proceeds, and -DOC_NO_LLVM is defined. It is the right choice on a machine that will never need firmware compilation or local inference — those are large builds.

This used to carry a note arguing that the environment variable was not a violation of the no-environment-variables rule, on the grounds that AGENTS.md forbids the application depending on env vars and this one was read by the build at resolve time, never by a running node.

That argument was overturned on 2026-09-15. The rule now covers the build too: "qbs and JavaScript take project properties, not Environment.getEnv" (§NEV-3.3). The exemption is recorded here rather than deleted because it was reasoned, and a reader who remembers it should see that it was overturned rather than forgotten.

---

The test display (a development machine with a desktop)

On a machine where you work at the desktop, UI tests must not open their windows there. Each window takes the keyboard and mouse focus. Install the test display once, from the lane that stays:

mkdir -p ~/.config/systemd/user
ln -sf "$PWD/integration/test-display/octomy-test-display.service" ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now octomy-test-display

A build machine with no desktop does not need it. The details, and what to do when test windows appear on the desktop again, are in Run tests.

---

Verify the site

Every one of these fails silently if you skip it. Run them in order.

# 1. All lanes present, each on its own branch.
git -C shared/OctoMY.git worktree list

# 2. ccache configured for the SITE, not a lane.
ccache -p | grep -E 'base_dir|hash_dir|max_size'

# 3. The qbs you built is the one on PATH.
which -a qbs && qbs --version            # must be >= 3.3.1

# 3b. The profile is the one you think, and baseProfile names a DETECTED
#     toolchain. A wrong-but-valid value here is the single most expensive
#     mistake on this page — see §Qbs.
qbs config defaultProfile
qbs config --list | grep -E '^profiles\.<profile>\.'
#     baseProfile must be e.g. x86_64-linux-gnu-gcc-14, never plain "gcc".
#     cpp.compilerWrapper must be ABSENT.

# 4. Resolve is clean. Check the Qt path it prints against your intended Qt,
#    and the optional-module table for anything unexpectedly MISSING.
cd "$SITE/main" && qbs resolve

# 5. ccache is in the compile command — EXACTLY ONE of them.
qbs build --command-echo-mode command-line -p libutil 2>&1 | grep -m1 -o 'ccache [^ ]*g++'

# 6. The cache actually serves. Delete the build tree, rebuild, expect ~100% hits.
ccache -z && rm -rf default && qbs resolve && qbs build -p libutil
ccache -s | grep -iE 'hits:|misses:'

Step 5 is the one people skip. cpp.compilerWrapper is evaluated at resolve time, so a graph resolved on a machine without ccache keeps building without it after you install ccache — until you re-resolve. "Is ccache on?" really means "was it present when this tree was last resolved?"

Step 5 also catches the double-wrap. nice -n 19 ccache ccache g++ means the profile is setting cpp.compilerWrapper on top of the one the build supplies; the grep above shows only the tail, so read the whole line, not the match. CcacheProbe warns about this at resolve time too, but only when ccache is installed — the manual check has no such precondition.

Step 3b costs a second and would have saved an hour on 2026-09-01, when a baseProfile of gcc produced a fake cyclic-dependency error naming three innocent products.

---

What is shared between lanes

Build output is never shared — but it may be relocated

Two different things, and conflating them is how sites break.

Never shared. The qbs build graph (default/default.bg) stores absolute paths to every source and artifact. A default/ tree reached by two lanes makes qbs read sources from one lane and write artifacts into another, silently. See Qbs reference → Canonical build convention; after copying a checkout, delete every copied build tree before the first build.

Relocating is supported, and is how a site with a small system disk and a large fast one is arranged. Give each lane its own directory on the build disk and symlink to it, one-to-one:

BUILDDISK=/build/octomy-builds
for lane in <lanes>; do
    mkdir -p "$BUILDDISK/$lane/default" "$BUILDDISK/$lane/release"
    ln -sfn "$BUILDDISK/$lane/default" "$SITE/$lane/default"
    ln -sfn "$BUILDDISK/$lane/release" "$SITE/$lane/release"
done

This is safe because the path qbs records — <lane>/default — stays stable; only the storage behind it moves. It stays safe only while the mapping is exactly one lane to one target directory, named for that lane.

Audit it, because a wrong link is silent and looks like ordinary staleness — a lane building fine while its artifacts land in another lane's tree:

for d in "$SITE"/*/; do
    lane=$(basename "$d")
    for k in default release; do
        [ -L "$d$k" ] || continue
        tgt=$(readlink -f "$d$k")
        case "$tgt" in
            */"$lane"/"$k") ;;
            *) echo "MISMATCH: $lane/$k -> $tgt" ;;
        esac
    done
done

Watch free space on the build disk too — build trees are tens of GB per lane and a full disk shows up as a mid-build kill, not a compile error.

Do not hand-delete parts of a live default/ tree. Removing a product directory the graph still references fails at link with ar: <object>: No such file or directory. Delete the whole default/ and re-resolve.

The compiler cache is partitioned by branch

base_dir and hash_dir=false remove the path difference between lanes, so the same file at <site>/paint/… and <site>/comms/… hashes identically. That is the part that works.

But every compile command also carries -DOC_GIT_BRANCH="<branch>", a gitStableDefine applied globally to every product, and ccache hashes the full command line. Two lanes on different branches therefore share no cache entries at all, for any object. The cache is partitioned one bucket per branch.

What cross-checkout sharing actually buys, then:

• A lane whose default/ is deleted and rebuilt — near-total hits.
• A lane re-created at the same branch, or a tree copied between machines.
• Two lanes that genuinely sit on the same branch.

It does not make eight lanes on eight branches share compile work. That is the intended trade — OC_GIT_BRANCH is how a running node reports its provenance — but it is worth knowing before you size max_size or explain a disappointing hit rate. Measurements belong in content/doc/internal/specs/build-performance/.

---

Troubleshooting