libcompression

Overview

libcompression is the single seam for compressed and archived bytes, at src/libs/libcompression/. It is Core tier: Qt Core plus the system zlib and nothing else, so host tools — including the vendoring tools that run before the rest of the tree exists — can depend on it.

It provides five things:

Why zlib rather than a vendored inflate

Qt gives us zlib at runtime — libQt6Core.so links libz.so.1 — but does not install zlib.h, so "Qt has zlib" is true at runtime and false at compile time. What this library links is the system zlib1g-dev, which libanon has declared for a long time, Windows path included.

Writing an inflate instead would mean owning a decompressor's correctness and security surface to avoid a dependency the process already loads.

Qt framing is a passthrough, deliberately

qCompress emits Qt's own 4-byte length prefix, and those bytes are already in saved projects, replay files, RHI journals and on the wire between nodes. A wrapper that was merely equivalent rather than identical would not fail at build time and would not fail a round trip — it would fail when someone opened an old file. testCompression therefore asserts byte-identity against qCompress at every level actually used (1 for RHI journals, 6 for banter/paint, plus default) rather than trusting it.

TarReader — scope drawn from measurement

tar is not one format. Over forty years it accumulated ustar, GNU, PAX, star and Solaris variants, sparse encodings and several ways to spell a long path. A general reader is a large amount of code carrying the risk of silently mis-reading an archive it only half understands, and for a vendoring tool the alternative to an error is an incomplete source tree that compiles into a subtly wrong artifact.

So the scope was measured rather than chosen from the spec's menu. Across 47 real release tarballs, git archive run against this repository, and the two URLs the LLVM and llama.cpp pins actually fetch:

The g row is the load-bearing one. GitHub's /archive/ endpoint serves git archive output, and every such tarball opens with a pax_global_header carrying the commit sha. Reproduce it against the exact pinned inputs:

C=$(grep -m1 'LLVM_COMMIT=' src/libs/libllvm/prepare-llvm.sh | cut -d'"' -f2)
curl -sL -r 0-60000 "https://github.com/llvm/llvm-project/archive/$C.tar.gz" \
  | gunzip -c 2>/dev/null | head -c 512 | head -1
# => pax_global_header

A reader that refused PAX — as this one was originally specified to do — would have refused both of its consumers on the first 512 bytes.

Flavour matters past byte 257

Offset 345 is ustar's prefix field, but in GNU format the same bytes are atime. Reading a prefix from a GNU header would paste a timestamp's bytes onto the front of every path. The reader branches on the magic to decide, which testTarDoesNotReadGnuAtimeAsPathPrefix locks in.

Path safety

Extraction rejects absolute paths and any entry escaping the destination via ... The check compares a resolved path rather than inspecting the string, so a/../../b is caught by where it lands, not by whether it happens to contain ... An archive is untrusted input even when fetched over TLS from a known host.

This check is necessary but not sufficient — see Symlinks for the escape that no member path reveals.

Memory

extract() writes each member as it is parsed rather than materialising the whole archive first: holding every payload as well would roughly double the peak. gunzip() reserves its output from the gzip trailer's ISIZE up front, which removes the realloc chain that otherwise holds two copies of a multi-gigabyte result at once — capped by zlib's maximum ratio, since ISIZE comes from the input and a hostile stream could claim 4 GiB.

Measured on the pinned LLVM tarball — 252 MB compressed, 187,392 members — extractTarGz peaks at 2.33 GiB in 9.2 s and produces a tree byte-identical to tar -xzf, symlink targets included. Before the reserve it was 4.25 GiB.

read() does retain every payload — it is the API for inspecting a small archive, not for extracting a large one.

The file forms hold far less. TarStream is the parse every call uses: it takes the tar in slices of any size and keeps one header and one member's payload. TarReader::visitFile and extractFile feed it a file a MiB at a time through Compression::decompressFile, which tells plain tar, gzip and xz by their magic, and ArchiveDigest::treeDigestOfFile hashes members as they pass. Measured on the pinned Qt archive — 1.0 GB of .tar.xz, 5.5 GB of tar — with debug builds: the tree digest in 211 s at a 486 MiB peak, and the fetch of its 853 MB slice, digest included, in 235 s at 438 MiB. The whole-buffer path would have held some 6.6 GB.

Extraction is not transactional: a failure part-way leaves what it already wrote. Callers are expected to materialise into a scratch directory they can discard.

Usage

#include "compression/Compression.hpp"
#include "compression/TarReader.hpp"

QString err;

// gzip
QByteArray gz;
if (!Compression::gzip(payload, &gz, &err)) { /* err names the failure */ }

// Extract a fetched source tarball
if (!TarReader::extractTarGz(downloaded, scratchDir, &err)) {
    qWarning() << "vendoring failed:" << err;
}

// Inspect without writing
QList<TarEntry> entries;
if (TarReader::readTarGz(downloaded, &entries, &err)) {
    for (const TarEntry &e : entries) { /* e.path, e.type, e.size, e.data */ }
}

Every entry point returns bool and fills an error string naming the entry and offset at fault — a message naming a syscall without naming the member is not actionable against an archive of tens of thousands.

ArchiveDigest — pinning what a download must be

Hashing bytes is trivial. Choosing which bytes to hash is the part that decides whether a pin still verifies in a year, and it is the only reason this class exists rather than a bare QCryptographicHash call at each site.

A vendoring tool fetches .../archive/<commit>.tar.gz. That URL is git archive piped through gzip, and neither layer of the delivered bytes is stable:

• The gzip wrapper is a function of GitHub's server configuration. GitHub changed it in 2023 and every ecosystem pinning the compressed digest broke at once. GitHub has since stated that checksums of generated archives are not guaranteed stable.
• The tar stream is not a function of the commit either. For LLVM commit 7b26069828aa, GitHub served a tar with modes 0775 and 0664 on 2026-10-02, and the earlier tar with modes 0755 and 0644 on 2026-10-03.

So a pin records treeDigest(): the SHA-256 of a manifest of the tree. The manifest has one line per file and symbolic link, sorted as bytes:

<sha256 of the content>  <git mode> <path>

The mode is 100644, 100755 or 120000, as git stores it. A link's content is its target. The path is the member path after stripComponents. Directories have no line. Both LLVM tars give the same tree digest.

One exception, detected automatically. A file that a .gitattributes in the tree marks export-subst has the word export-subst in place of its content hash. git archive writes git describe output into such a file, and the two LLVM tars abbreviated the hash in clang/bindings/python/.git_archival.txt to 14 and 13 characters. Only a literal pattern is read. A glob, a quoted pattern, a macro or an unset form is refused, because a partial copy of git's matching that disagrees with git pins the wrong content.

llvm-fetch --digest prints the tree digest of an archive on disk, so a pin bump needs no build of the tree:

llvm-fetch --digest <archive> --strip 1                 # the digest to record
llvm-fetch --digest <archive> --strip 1 --manifest      # one line per member
llvm-fetch --digest <archive> --pin <file> --key <name> # compare with the pin

The last form takes the strip count from the pin, prints the digest, and exits 1 when the pin records another one. A missing archive or pin exits 4, and one that cannot be read or described exits 3. llvm-fetch calls treeDigest(), which is also what the fetch verifies with.

testCompression keeps a second implementation. Its test archives are byte literals, one ustar tar and one PAX tar in the form git archive writes, and not the output of TarWriter. A test helper computes the manifest of the same tree with QCryptographicHash alone, and both must give the same constant.

Three digests are available, because the choice must be visible:

Four failures, four errors. A malformed expectation, a mismatched tree, an archive that will not decompress and a tree that cannot be described without ambiguity each name themselves. A tree is ambiguous when two members have one path, when a strip removes a file, when a path has a line feed, or when a .gitattributes uses export-subst in a form the digest does not read.

Verifying costs a second decompression unless you avoid it

treeDigest() on a .tar.gz decompresses it. If you then call extractTarGz(), it decompresses again. For large archives, gunzip once and pass the .tar to both. The tree digest of a .tar and of a .tar.gz of it is the same value:

QByteArray tar;
if (!Compression::gunzip(downloaded, &tar, &err)) { /* … */ }
if (!ArchiveDigest::verify(tar, stripComponents, pinnedTreeSha256, &err)) { /* … */ }
if (!TarReader::extract(tar, destDir, options, &err)) { /* … */ }

Qt's SHA-256 is not hardware-accelerated, so treeManifest() hashes the members on a thread pool. A cap of 256 MiB bounds the payload copies that wait for a worker.

Taking a slice, not a tree

TarExtractOptions lets a caller take part of an archive: stripComponents drops leading path components as tar --strip-components does, and prefixes selects what to keep, compared after stripping.

Vendoring needs this. The LLVM tarball holds 187,392 members inside one llvm-project-<sha>/ directory and the build wants about 11,000 of them:

Prefix matching is on a path boundary, never a substring, so llvm/lib selects llvm/lib/... and never llvm/libfoo — a case the test covers explicitly because getting it wrong would silently vendor the wrong tree.

Testing

test/testCompression/ — 25 cases. The ones worth knowing about are those written to avoid passing for the wrong reason:

• Interop, not self-consistency. Archives come from the system tar(1) and from git(1), not from our own writer. A reader tested only against its own writer passes even if both agree on a format nobody else speaks.
• testTarFlavoursAgreeMemberForMember reads the same staged tree through --format=v7|gnu|ustar|posix and demands identical members — which is what proves the reader branches on flavour rather than guessing.
• testGunzipRejectsCorruptTrailer flips a byte the compressed data survives, so only the CRC can catch it.
• testTarRejectsPathTraversal uses a real malicious archive built by tar --transform, not a hand-written header.
• testTarRefusesSymlinkEscapingDestination covers the escape no member path can reveal, and was confirmed to fail with the check removed rather than assumed to be testing something.

Hand-built headers are used only where no cooperating producer will emit the case (unknown PAX keys, GNU atime bytes).

Beyond the unit tests:

• Diffed member-for-member against Python's tarfile over 47 real release tarballs, 99,342 members — zero mismatches.
• Extracted the pinned LLVM tarball in full — 171,398 files, 15,974 directories, 19 symlinks — and diffed the result against tar -xzf: identical, symlink targets and executable bits included.

Symlinks

Links are materialised after every other member has landed, and only once their target has been checked.

The check. A symlink is the one member type whose target, not its path, decides where later writes go. An archive holding escape -> ../outside followed by escape/pwned.txt writes outside the destination with every member path clean — nothing absolute, no .. — so the path check above cannot see it. Targets are therefore resolved against the link's own directory and required to stay inside the destination root. Verified by testTarRefusesSymlinkEscapingDestination, which fails if the check is removed.

Windows has no usable symlink here: QFile::link() writes a .lnk shortcut that no compiler or include resolver can follow. Links are materialised by copying the target instead, recursing for directory links. A copy is not a link — it does not track its source, and anything testing link semantics sees something different — but it yields a tree that builds, which for vendored source is the better trade.

The pinned LLVM tarball contains 19 symlinks, all relative and all inside the tree; 18 point to files and one to a directory. All are test inputs.

Related

• libcrypt — the other Core-tier wrapper over a system crypto/ compression dependency.
• Build convention: Start here → The build convention. Kiln sees new source files with no other step. On the qbs path, new source files need a resolve: Qbs reference → The qbs build convention.