Build through the kiln queue

Kiln puts one queue between every compile, every link and every qbs run on a machine, so several people or agent sessions can build at once without the machine running out of memory.

Why a queue

A qbs resolve of this project needs 24 to 30 GB. A link needs about 1.4 GB. Two builds fit on a typical development machine and four do not, and when four are started anyway the result is a machine stuck in swap and an out-of-memory killer that takes a build graph with it.

Kiln admits work while the machine can carry it and makes work wait when it cannot. It decides by what is actually free, and it learns what each kind of job really costs.

Try it

Kiln is opt-in. Nothing changes for a build that does not ask for it.

1. Build kiln in one build directory:

   qbs build -p kiln
   

2. Point a different build directory at that binary when you resolve:

   qbs resolve project.kilnBinary:/path/to/kiln
   

   A kiln cannot wrap the build that produces it, so the binary has to exist before this resolve. A path that does not exist stops the resolve and says so.

3. Build as usual. Every compile and link now goes through the queue, and the output reads exactly as it did without it.

To run a whole qbs build as one admitted job, so that several full builds wait their turn instead of starting together:

kiln run -- qbs build

See what the machine is doing

kiln status

It shows the current ceiling and why it is there, every running job with its estimated cost, everything waiting and for how long, and the last reason work was refused. A build that looks stopped is almost always a build that is waiting, and this is where you see what it waits for.

A waiting job writes nothing to its build log. The queue is visible here and nowhere else, so a log reads the same with or without kiln.

When a build says it cannot reach the broker

kiln: no kiln broker is listening on kiln-1000 and one could not be started.

The first build on a machine starts the broker itself, so this means the broker could not start. Run it by hand to see why:

kiln serve

Kiln never falls back to running the compiler without the queue. A build that cannot be metered stops, because an unmetered build is the thing the queue exists to prevent. To build without kiln, resolve with project.kilnBinary empty.

Tune the broker

The broker's defaults suit most machines. When they do not, start it yourself with flags before the first build:

A qbs run is never stopped, only admitted when it fits, because stopping qbs damages its build graph.

A budget smaller than a qbs run itself still lets that build finish: its compiles then run one at a time, on whatever the machine has free, and the broker says once that the budget is too small for the build. A qbs of this tree holds about 32 GB, so a budget of 26 GB builds, slowly. A link that optimises at link time holds four slots and runs four optimisation jobs, so kiln status can show fewer jobs than slots in use.

Read the build description

Kiln also reads the .kiln files that will describe the build once kiln builds the tree itself. Three commands show what they say:

kiln explain src/libs/libutil          # every setting, and the file and line that set it
kiln explain src/libs/libutil deps     # one setting
kiln deps src/libs/libutil             # the libraries its includes really use
kiln graph                             # what changed in the checkout since kiln last looked
kiln keys                              # every key a .kiln file may set

kiln explain also lists the product's sources, every folder it skips and why, and the generators it needs. --variant release explains another variant. The keys, conditions and syntax are in the kiln reference.

These commands answer from a graph kiln keeps of the checkout, one file per checkout in your cache folder. The first call scans the checkout, about two seconds for this tree; after that each call costs only what changed, and a new file, a rename or a moved folder is seen with no other step. While a broker runs it keeps the graph of every checkout its jobs name current, and kiln status shows them.

See also

• Kiln reference
• Set up a build site
• ccache Setup