Benchmark harness (make bench-lib / make bench-scenarios)
Grit compares library micro-benchmarks (Criterion) and command-level scenarios (grit-bench + hyperfine) against the system git binary on the same machine. Two Make targets run the full suites from a clean checkout after building release binaries.
make bench-lib
Runs every Criterion bench under grit-lib/benches/ (with the test-support feature where a bench needs fixtures or hooks), then writes a JSON summary:
| Output | Description |
|---|---|
target/criterion/ |
Raw Criterion results (estimates.json per benchmark) |
target/bench/criterion-summary.json |
Aggregated summary produced by grit-bench criterion-summary |
$ make bench-lib
Criterion summary JSON (criterion-summary.json)
| Field | Type | Meaning |
|---|---|---|
schema_version |
1 |
Summary schema version |
timestamp |
RFC3339 string | Capture time (UTC) |
machine |
object | CPU, RAM, kernel, rustc, cargo profile (same shape as grit-bench v1) |
tools |
object | git is n/a (criterion micro-benchmarks); grit identifies the library build |
scenarios[] |
array | One row per Criterion new/estimates.json |
scenarios[].driver |
"criterion" |
Not a git vs grit hyperfine row |
scenarios[].grit |
timing block | Mean/median/stddev in milliseconds (converted from Criterion nanoseconds) |
Each scenario id is derived from the path under target/criterion/ (for example criterion-hash_sha1-4KiB).
Patch parsing and staging
apply_blob includes parse_patch_small, which parses a valid traditional unified diff and fails if parsing fails. It measures parsing independently of repository discovery, object writes, and index updates:
$ cargo bench --profile dist -p grit-lib --bench apply_blob -- parse_patch_small
For a command-level comparison with system Git, add/patch-file@synth-10k applies a worktree patch to the index with grit add --patch-file and git apply --cached:
$ cargo build --profile dist -p grit-cli -p grit-utils
$ ./target/dist/grit-bench daily --only add/patch-file@synth-10k --output target/bench/patch-file.json
Use the same fixture, build profile, and system Git for before/after comparisons. Each timed mutation must start with the same index and worktree state; preparation runs outside the timed region.
make bench-scenarios
Prepares registry fixtures, runs every grit-bench scenario group for the chosen synthetic tier, and writes both JSON and a markdown table:
| Output | Description |
|---|---|
target/bench/scenarios-<host>.json |
Merged grit-bench report (schema_version: 1) |
target/bench/scenarios-<host>.md |
Human-readable ratio table |
# Default tier M (full hyperfine samples)
$ make bench-scenarios TIER=M
# Small fixtures + one timed iteration per scenario (CI smoke)
$ make bench-scenarios TIER=S SMOKE=1
Tier (TIER=S|M|L)
| Tier | Registry fixture | Typical use |
|---|---|---|
| S | synth-s |
CI smoke, fast local checks |
| M | synth-m |
Default local run |
| L | synth-l |
Large synthetic repo (long run) |
SMOKE=1 sets --smoke on grit-bench scenarios: warmup 0, exactly one hyperfine run per scenario, tiny network/parallel fixtures, and skips heavy groups (ODB backend, bitmaps, refs). CI uses TIER=S SMOKE=1; timings are not gated—the job fails only if a scenario errors.
Scenario report JSON (scenarios-<host>.json)
Uses the same grit-bench v1 schema as individual subcommands (schema_version, machine, tools, fixtures, scenarios). Each scenario includes git and grit timing blocks (milliseconds) and ratio (grit_median_ms / git_median_ms).
Groups included in the merged report:
| Group | Source |
|---|---|
worktree |
status / add / commit / diff on the tier fixture |
history |
log, show, rev-list, blame, cat-file batch |
rewrite-history |
grit history reword/drop/fixup/squash vs git history / git rebase |
network |
clone, fetch, push, ls-remote (file + smart HTTP) |
parallel |
threads=1 vs auto on the 100k-class fixture |
revwalk, fsck, maintenance |
porcelain vs git |
resolve-cat, ignore, local-clone, hot-paths |
CLI hot paths |
deep-leaf, dense-leaf |
smoke layout only when SMOKE=1 |
odb-backend, bitmaps, refs |
full runs only (SMOKE=0) |
grit-bench partial-clone [--files N] [--packs N] runs on its own, outside the merged report: a blob:none file:// clone of a source holding --packs packs (50 by default, one incremental repack per batch of files), the same clone with its checkout, and the reset --hard in a blobless clone, the last two lazily fetching every HEAD blob in one request, all against Git.
Equivalent CLI:
$ cargo build --release -p grit-cli -p grit-http-server -p grit-utils
$ ./target/release/grit-bench scenarios --tier S --smoke --format json --output target/bench/scenarios-smoke.json
$ ./target/release/grit-bench criterion-summary --criterion-dir target/criterion --format json --output target/bench/criterion-summary.json
See also Benchmarks for committed baselines and acceptance ratios.
grit-bench daily (factory census)
Use --min-runs 10 for performance investigations that require at least ten
samples per tool. Daily runs honor this common option, raising the usual run cap
when the requested minimum is larger. --smoke still uses one sample, and the
scenario time budget can stop expensive scenarios early; inspect each row's runs.
Daily reports reject a plan whose two commands are placeholders instead of an
operation. Reword uses a scripted editor with git history on supporting Git
versions; deepen starts from one branch boundary with tags excluded and checks
that Git and grit produce identical new shallow boundaries.
Headline scenarios compare grit and system git with the direct-spawn runner (wall time + peak RSS per run). Reports use the grit-daily-bench/1 schema (distinct from hyperfine v1).
Published daily numbers (the factory dashboard and the Performance overview) measure the dist profile, the fat-LTO build that releases ship. Build grit-bench in the same profile: it then times target/dist/grit by default and records "cargo_profile": "dist" in the report.
$ cargo build --profile dist -p grit-cli -p grit-utils
$ ./target/dist/grit-bench fixtures prepare --only synth-10k-smoke,many-refs-smoke,git-git-wt,git-git
$ ./target/dist/grit-bench daily --smoke --offline --output target/bench/daily-smoke.json
$ ./target/dist/grit-bench daily validate target/bench/daily-smoke.json
The daily census switched from release to dist on 2026-10-28. That one-time switch moves the baseline of every scenario: short commands drop by roughly 10–15% in time and 20% in peak RSS (show/head@git-git went from 3.08 to 2.70 ms and from 16.1 to 13.1 MB on the same VM), because LTO packs the code a run touches into fewer pages. Read the step at that date as a build change, not a run of wins, and compare reports only within one profile (check machine.cargo_profile).
A release build stays fine for iterating on a change (it rebuilds about three times faster), as long as the before and after binaries use the same profile. Code-layout noise is larger in release: any change to grit-lib reshuffles its 16 codegen units, which can move a 2–3 ms command by ±50 µs and its peak RSS by ±1 MB with no change in the work done. Confirm small differences on dist.
| Flag | Meaning |
|---|---|
--smoke |
Smoke fixtures, 0 warmups, 1 timed run per tool |
--only id,... |
Run a subset of scenario ids |
--area <area> |
Filter by functional area (worktree, branching, remotes, …) |
--baseline-grit <path> |
Also time a second grit binary on headline rows |
--offline |
Do not clone git.git (fixtures must already be cached) |
All 17 headline ids run in --smoke mode (including the four remote headlines on cached git-git bare and synth-10k bare remotes). Additional remote matrix rows (fetch/noop, ls-remote/all, HTTP clone/fetch/push) run on full daily reports and in grit-utils remote e2e tests. CI job bench-daily-smoke runs the smoke command above; timings are not gated.
Blobless hard reset (#2054)
Measured on 2026-10-26 with the stock system Git 2.43.0, Rust 1.99.0, release-profile grit 0.5.21, four Intel Xeon vCPUs and 16 GiB RAM on Linux/ext4. The main baseline is fa6a9fd559b2b5e0ac4779f1d2b4c115f70c24f0; the comparison is the factory/batch-reset-blobs change. Both binaries were retained and measured in the same session, with no concurrent builds or tests.
Wall times below are median milliseconds, and RSS is the maximum KiB across samples. GNU /usr/bin/time -v measured peak client RSS; Python time.monotonic() bracketed the complete command. Clone preparation, object counting, byte/symlink verification, clean git status --porcelain and git fsck --strict ran outside the timed region. Every materialization used a fresh --no-checkout client. The 200-file case has three runs per tool; Cargo and HTTP blobless cases have one.
| Blobless materialization | Git: ms / KiB | Main: ms / KiB | Fixed: ms / KiB |
|---|---|---|---|
| 200 files, file transport | 30.24 / 5,704 | 1,394.26 / 21,792 | 33.03 / 17,504 |
| 200 files, smart HTTP | 161.84 / 9,584 | 1,804.66 / 23,968 | 69.63 / 18,112 |
| Cargo, file transport | 678.57 / 45,136 | 72,193.59 / 41,644 | 757.40 / 45,344 |
| Case | Data fetches: Git / main / fixed | Total packs: Git / main / fixed | HTTP requests: Git / main / fixed |
|---|---|---|---|
| 200 files | 1 / 200 / 1 | 2 / 201 / 2 | — |
| 200 files, HTTP | 1 / 200 / 1 | 2 / 201 / 2 | 2 / 400 / 2 |
| Cargo | 1 / 2,022 / 1 | 3 / 2,023 / 2 | — |
The local HTTP server was the same debug-build grit-http-server for all three clients, with --log-headers enabled. HTTP is a single sample and includes server warmup effects; request counts are the regression gate. File-transport fetch counts come from system-Git upload-pack traces. A second reset downloads nothing, as covered by library and CLI tests.
Cargo was cloned from https://github.com/rust-lang/cargo.git and its source branch/worktree reset to b57863321f8d0a0f76fcf9ad636b6e39b024d0c0 before client creation. All three blobless clients started with 59,756 missing history blobs and identical target contents; the current source tag set differs from the original issue reproduction. The verifier compares symlink targets without following directory symlinks.
Cargo blobless checkout improves about 95×. Its peak RSS increases by about 3.6 MiB relative to main and is close to system Git. The larger single fetch replaces thousands of tiny packs; this measured memory cost is retained in the baseline rather than hidden by the speedup. The 200-file and HTTP blobless cases use less peak memory than main.
Ordinary full-clone resets check the common path separately. Cargo values use four fresh clients per tool; each clean reset follows successful full materialization.
| Full-clone reset | Git: ms / KiB | Main: ms / KiB | Fixed: ms / KiB |
|---|---|---|---|
| 200 files, materialize | 5.49 / 5,672 | 9.82 / 13,880 | 9.58 / 15,464 |
| 200 files, clean | 5.19 / 5,416 | 6.35 / 14,544 | 5.97 / 15,900 |
| Cargo, materialize | 149.66 / 46,100 | 180.57 / 56,740 | 178.71 / 56,860 |
| Cargo, clean | 14.03 / 15,300 | 17.92 / 20,384 | 17.35 / 19,976 |
Full Cargo materialization stays within the observed run-to-run range; clean Cargo resets improve slightly and use slightly less peak RSS. Tiny full-clone cases show about 1.3–1.6 MiB more peak RSS, so the results do not claim a universal memory reduction. The raw samples and environment are committed in grit-utils/baselines/blobless-reset-before.json and grit-utils/baselines/blobless-reset-after.json.
To reproduce the small input repository:
$ case_root=$(mktemp -d)
$ git init -q -b main "$case_root/source"
$ for i in $(seq 1 200); do
$ { printf 'file %s\n' "$i"; for j in $(seq 1 100); do printf 'build payload\n'; done; } > "$case_root/source/file-$i.txt"
$ done
$ git -C "$case_root/source" add .
$ git -C "$case_root/source" -c user.name=CI -c user.email=ci@example.test commit -qm inputs
$ git -C "$case_root/source" config uploadpack.allowFilter true
$ grit clone --blobs none --no-checkout "file://$case_root/source" "$case_root/client"
$ /usr/bin/time -v grit -C "$case_root/client" reset --hard HEAD
$ git -C "$case_root/client" fsck --strict
Use a fresh destination for each Git/main/fixed sample, substituting git clone --filter=blob:none --no-checkout for Git. Omit the blob filter for full materialization; run reset again on the resulting clean worktree for the clean case. For Cargo, substitute the pinned source above. Keep the source upload-pack executable as the stock system Git.
The daily remotes matrix also measures materialization separately from clone time:
$ grit-bench daily --only reset/blobless@git-git --output target/bench/blobless-reset.json
$ grit-bench daily validate target/bench/blobless-reset.json
Its prepare hook creates fresh blobless clients outside each timed iteration. Correctness checks enforce clean worktrees, strict fsck and a bounded pack count.
gc/default@synth-10k works the same way: before each timed run, its prepare hook resets synth-10k to the fixture's base commit and adds 40 empty commits as loose objects. Each prepare puts a fresh nonce in those commit messages. Without it, a prepare that runs within a second of the previous one recreates byte-identical commits that the last gc already packed, git writes nothing loose, and the timed run measures a gc with nothing to do (about 13 ms instead of about 140 ms). The prepare fails when the newest seeded commit is not stored loose.
Startup relocation comparison
Startup changes need separate release and shipping dist measurements. Keep
PIE, ASLR, LTO, codegen units and panic policy unchanged. A CLI-only RELR probe
changes the final link, without rebuilding dependencies with workspace-wide flags:
$ mkdir -p /tmp/grit-startup/{release,dist}/{stock,relr}
$ cargo build --release -p grit-cli --locked
$ cp target/release/grit /tmp/grit-startup/release/stock/grit
$ cargo rustc --release -p grit-cli --locked --bin grit -- \
-C link-arg=-Wl,-z,pack-relative-relocs \
-C link-arg=-Wl,-Map=/tmp/grit-startup/release-relr.map
$ cp target/release/grit /tmp/grit-startup/release/relr/grit
$ cargo build --profile dist -p grit-cli --locked
$ cp target/dist/grit /tmp/grit-startup/dist/stock/grit
$ cargo rustc --profile dist -p grit-cli --locked --bin grit -- \
-C link-arg=-Wl,-z,pack-relative-relocs \
-C link-arg=-Wl,-Map=/tmp/grit-startup/dist-relr.map
$ cp target/dist/grit /tmp/grit-startup/dist/relr/grit
$ cargo build -p grit-utils --bin grit-bench --example startup_compare
$ cc -O2 grit-utils/examples/startup_measure.c -o /tmp/grit-startup/measure
Stop builds and tests before timing. The focused example uses the daily runner's
measure-direct-run helper and statistics, with five warmups and 60 measured
runs per tool (minimum 30), rotating stock/candidate/system-Git order. The daily
default policy caps at 20 and is insufficient for this investigation.
$ ./target/debug/examples/startup_compare \
--stock /tmp/grit-startup/dist/stock/grit \
--candidate /tmp/grit-startup/dist/relr/grit \
--helper "$PWD/target/debug/grit-bench" \
--native-helper /tmp/grit-startup/measure \
--fixtures /tmp/grit-startup/fixtures --profile dist \
--output /tmp/grit-startup/dist.json
Repeat with the frozen release binaries and --profile release. Fixtures cover
one local branch, 20,000 refs, 20,000 tracked paths, and 20,000 commits. Every
fixture passes system git fsck --strict; each candidate command must preserve
stock stdout/stderr. Branch, status, resolve, help and a 1,000-entry log run on
all four shapes. Git's branch list is unbounded while grit's default list is
bounded; the comparison reports the actual argv and does not equate their output
volume on the many-ref fixture.
Reports retain individual wall-time samples, medians, standard deviations and
per-child peak RSS. The existing helper measures after spawn, through child
exit; the small C parent records full fork/exec/wait times, per-child RSS and page faults in the native block for startup conclusions.
Cross-check RSS with /usr/bin/time -v, using a fresh small parent per child;
a long-lived Python parent's inherited RSS can swamp short-command measurements.
Record minor/major page faults separately, and keep LD_DEBUG=statistics loader
cycles apart from end-to-end wall time. Neither loader cycles nor relocation
metadata bytes measure heap use.
Inspect readelf -hW -lW -dW -rW -VW, nm -S -n, and the linker maps. RELR
compresses relative relocation metadata; it still writes the same pointer slots.
Dependency tables, including regex Unicode tables, contribute relocations even
when their code executes lazily. Preserve Unicode matching, filtering, schemas
and tracing when considering source changes.
The release workflow packages the original glibc 2.34 x86_64 GNU asset before
performing this CLI-only RELR link for a separate -gnu-relr asset. The latter
requires glibc 2.36+ and remains a manual download. Stable/nightly installers
always use the original GNU asset on glibc systems: the final-tip review found
a 1–2% startup regression, despite earlier gains. Before enabling automatic
selection, benchmark stock and RELR from the exact release source again with
builds/tests stopped; require a demonstrated benefit rather than inferring one
from compressed metadata or loader support. This preserves the existing loader
minimum and leaves other targets unchanged. Run both ELF/runtime checks and the
pinned 2.34/2.36 container smokes in .github/workflows/release.yml; see
TESTING.md.
The initial measured dist gain is 5–8% for tiny commands. The earlier Xeon
shipping follow-up measured 4–5% for branch/status/resolve, with help within
its sample spread. Both reports retain variance and independent RSS; layout
changes mean metadata compression is not a general memory reduction. The
shipping JSON also records 30 loader-only observations and ten GNU-time peak
RSS samples per tiny command/tool.
The final-tip follow-up in startup-relocations-review-{release,dist}.json
records exact 3fcfb24e9 CLI/library source, frozen binary hashes and the actual
AMD EPYC 7763 host. Tiny dist commands regress 0.88–2.07%; release changes are
near zero. The native block includes paired-bootstrap 95% intervals from 5,000
resamples of the 60 round indices (reset RNG seed 4626 per case), taking
median(candidate) - median(stock). Loader-only and GNU-time cross-checks use
the runner's isolated environment, recorded in isolated_env_overrides.
Hot text layout
Short commands' peak RSS is mostly the binary's own code: the kernel maps file-backed text
in 64 KB fault-around windows, so a function run once makes its whole window resident, and
the library functions an operation runs are otherwise scattered over the whole .text.
grit-lib's hot-text-layout feature lays the library's hot code path out contiguously in
every binary that links it; grit enables it and lays out its own hot code the same way
(see Benchmarks and
Benchmarks).
grit-lib/hot-text.ldlists the functions of grit-lib, the Rust runtime and grit-lib's own dependencies that the library's common operations run. Anything that names another crate, such as a CLI type or clap, is left out.grit-lib/build.rspublishes the fragment as a native library. LLD reads it as an implicit linker script and places the listed functions in one.text.grit_hotoutput section ahead of.text.grit-cli/hot-text.ldlists everything else the same workloads run in thegritbinary: the CLI's own code, clap, and library generics the CLI instantiates.grit-cli/build.rspasses it to the final link ofgritonly, which places those functions in.text.grit_cli_hot, also ahead of.text. Every command runs argument parsing and CLI start-up first, so this half matters as much as the library's for short commands.- GNU ld rejects
SECTIONSin implicit scripts, so both fragments only act onx86_64-unknown-linux-gnu, where rustc links with LLD by default. SetGRIT_HOT_TEXT_LAYOUT=0to turn them off, for example when linking with another linker. - Patterns glob input-section names with crate hashes wildcarded, so version bumps keep
them matching. A function that is renamed or no longer runs simply stays in
.text. - Every function matches with and without a local-copy suffix. ThinLTO in the
releaseprofile renames a function it shares between codegen units toname.llvm.<n>, while the single-unitdistbuild keeps most names bare, so a list that pinned either form would miss much of the other build. - The instantiating crate at the end of a generic's symbol is wildcarded as well. LLVM
merges identical instantiations from different crates into one copy, and which crate's
name the survivor keeps changes between builds: the stripped
distbinary can name<String as Display>::fmtafterserde_jsonwhere the profiled unstripped one named it aftergrit. About one pattern in eight names an instantiating crate; with it pinned,grit branchon the shipped build still faulted in 12 windows of plain.text, against 5 with the wildcard. - Every alias of an executed function is listed. LLVM merges functions with identical code
(
RawVec<T>::grow_onefor every same-sizedT) into one body and keeps the other names as aliases at the same address. The body's input section is named after one of them, while callgrind reports whichever name it picks for the address, so listing only that name left 15 such bodies in.texton areleasebuild ofgrit status. - Impl disambiguators (
Ms<n>_) are kept, and they depend on grit-lib's features. Buildinggrittogether withgrit-utils, as the daily census does, turns onserde-models, whose extra impls renumber them, while shipped binaries buildgrit-clialone. The generator profiles both builds of both profiles, so the lists match either one (see Benchmarks). - The
hot_text_layouttest fails if either section is missing, smaller than its floor (256 KiB for the library, 128 KiB for the CLI, the sign that most patterns stopped matching), or not ahead of.text.
Regenerate both lists after large library or CLI changes, from an unstripped dist build
and a release build:
$ CARGO_PROFILE_DIST_STRIP=false CARGO_TARGET_DIR=target/hot-text \
cargo build --profile dist -p grit-cli
$ CARGO_TARGET_DIR=target/hot-text cargo build --release -p grit-cli
$ python3 scripts/gen_hot_text_layout.py \
--grit target/hot-text/dist/grit --grit target/hot-text/release/grit
The script needs valgrind. It builds a 10,000-file clone (packed objects and refs, tags,
an upstream) and runs the library's common operations through each binary, so inlining
matches what is linked: status, diff, log (one page and a streamed --json listing), show,
revision walks, blame, refs, branch listing and edits, config, add, commit, stash, and
switch twice: to a branch about 300 files away (the parallel checkout) and to one 50 files
away, below checkout.thresholdForParallelism, which writes files on the serial path every
small switch takes. Each workload starts right after a reset, so its first step also takes the
index-refresh path. Callgrind records every function each step runs; the script merges what
the builds ran and splits it into the library's functions and the rest. Each list keeps the
functions that the same set of workloads runs together, and orders those groups in reflected
binary (Gray code) order over the workloads, status first: neighbouring groups differ in as
few workloads as possible, so each command's code forms a few long runs. Sorting by how many
workloads share a function, as the lists did before, cut a command's code into one slice per
sharing count. The order of WORKLOADS matters: the first two workloads' code is one run
each, and the further down the list a workload is, the more runs its code splits into, so
the commands whose memory matters most come first. The CLI's list keeps the older sort by
sharing count and uses the Gray order only between equally shared functions: every command
runs the CLI's start-up and argument parsing, so that shared core stays dense at the front,
and the full Gray order there cost config and ref listings a window each.
Both orders are then refined by a search: it moves whole groups and keeps a move when the
64 KB windows all workloads touch, counted with each build's code sizes and averaged over
four section start offsets, do not rise in total and no workload's own count rises above
what the starting order gave it. The search is seeded, so a regeneration from the same
builds is reproducible; --search-iterations 0 keeps the plain Gray order. Both builds
matter: the shipped dist binary inlines more under fat LTO, while the release build that
development and local benchmarking use keeps functions such as clap's command construction
separate, and those are cold in a list taken from dist alone.
To see where a run's resident pages are, stop the process at exit_group under gdb and
read /proc/<pid>/smaps (per mapping) or /proc/<pid>/pagemap (per page, to map back to
symbols with nm).