Code coverage

Measure test coverage with kiln and LLVM source-based coverage

Code coverage tells which lines and functions of the code the tests run. The tree uses LLVM's source-based coverage. kiln builds the instrumented tree, collects a profile from each suite, and counts the result by library. No host llvm-profdata or llvm-cov is needed: kiln reads the profiles with the LLVM it vendors, so the tools always match the compiler.

Coverage builds run on Linux x86-64 only.

Measure the coverage of a test run

Make a build folder of its own for the coverage build. The instrumented programs are larger and slower, so do not mix them with your daily build. In the checkout's .kiln.local, set:
```
output.dir = /build/octomy-builds/<lane>-cov
option.enableCoverage = on
```
If src/libs/libllvm/llvm-src/compiler-rt/lib/profile does not exist, run kiln vendor. It extracts the profile runtime's sources with the rest of the LLVM pin.
Build:
```
kiln build
```
The debug variant compiles every product with -fprofile-instr-generate -fcoverage-mapping, except the vendored libraries (libllvm, libllama, librl, libvulkanvalidation). kiln builds the profile runtime from the vendored compiler-rt into <build root>/clang-rt, and links it into each program and shared library.
Run the tests:
```
kiln test --report /build/octomy-builds/<lane>-cov/runs/cov1 --profile ci
```
Each suite writes its profile into <report>/profiles. kiln empties that folder at the start of the run.
Count the result:
```
kiln coverage --report /build/octomy-builds/<lane>-cov/runs/cov1
```
kiln merges the profiles into coverage.profdata, reads the coverage mapping of every instrumented program and library of the build folder, and prints one row per library. It writes the same table into coverage.md, and the counts of every file and library into coverage.json.

Read the table

Column	Meaning
Library	A folder under `src/libs`
Lines	The lines that hold code, in the library's source files and headers
Lines run	The lines that ran at least once
Functions	The functions that start in the library's files. A template counts once
Functions run	The functions that ran at least once

The last two rows are every library together, and every file of ours, which adds the applications, the tools and the tests.

The table leaves out:

A file under a prefix of integration/lint/vendored-paths.txt.
A file that the build generated into the build folder.
A file outside the checkout, for example a Qt or a system header.

Troubleshooting

`kiln coverage` says there is no profile

Run kiln test first, with the same --report folder. Make sure that option.enableCoverage = on is in .kiln.local of the checkout that built the programs: kiln test collects profiles only when the suites are instrumented.

A suite's lines count as not run

The profile runtime writes the profile when the process ends normally. A suite that crashed, or that kiln test stopped at its timeout, writes nothing. Read the suite's log in <report>/logs.

The command reports mismatched functions

A mismatched function has a profile from another build of the function. The programs changed after the run. Build and run again, then count again.

qbs

qbs can still build an instrumented tree with qbs build config:debug project.enableCoverage:true. Read its profiles with the llvm-profdata and llvm-cov of the same LLVM version as the compiler. kiln is the supported path.