grit
CLILibrary
HEAD → docs/library/storage-backends
Library guide / Writing storage backendsMarkdown

Writing storage backends

Implement object and ref stores, open a repository on them, test with conformance suites, and know what still needs a .git directory on disk.

Embedders replace grit’s default files layout with their own object store, ref store, or both. This page is the end-to-end guide: traits to implement, how to wire RepositoryOptions, optional capabilities, performance hooks, and how to prove correctness with the shared conformance suites. For lookup order, alternates, pack ingest, and benchmark tables on custom primaries, see Object stores.

What to implement

Object storage (read path)

ObjectStore is the minimum surface:

Method Contract
read / read_info / open_stream Return Ok(None) when the id is absent; propagate I/O or corruption errors (do not map outages to “missing”).
contains Must match what read would return for the same id.
hash_algo Must match extensions.objectformat / repository init.
for_each_object List every stored id; release internal locks before calling the callback (callbacks often call back into the store).
lookup_prefix / contains_many Override when a full scan per call is too expensive (see Performance).

Optional streaming helpers: ObjectStore::read_into (default copies from read), ObjectStore::with_stream on the repository Odb handle for large blobs.

Object storage (write path)

WritableObjectStore: idempotent write, plus flush / rollback when you batch transactions. Pack fetch, push, bundle install, and commits all end in flush on the primary.

Pruning (optional)

PrunableObjectStore: implement remove, as_prunable, and optionally object_mtime for grace-period prune. Without it, gc, prune, and Odb::gc return RepositoryError::UnsupportedObjectStore with operation prune — never touch objects/ on disk silently.

Ref storage

RefStore: read_raw, resolve, iteration (for_each_ref with the empty-prefix rule from RefStore), reflog read/write, and transactions via prepare → commit / abort.

You do not reimplement Git’s ref rules: use refs::store::backend helpers (check_transaction, apply_transaction, RefBatchState, SingleWriterGate for in-process writers). Report backend failures as RefStoreError::Backend.

Opening and initializing

Inject stores when opening

RepositoryOptions::with_object_store is the only supported way to set a custom primary object store. Add read layers, an attached commit-graph, or alternates with RepositoryOptions::with_odb_builder (OdbBuilder).

RepositoryOptions::with_ref_store replaces loose/packed/reftable refs under refs/ and the on-disk HEAD file. Pair it with with_initial_branch when HEAD should start on an unborn branch.

use std::sync::Arc;

use grit_lib::config::environment::RepositoryOptions;
use grit_lib::objects::odb::store::MemoryStore;
use grit_lib::objects::HashAlgo;
use grit_lib::refs::store::MemoryRefStore;
use grit_lib::repo::{InitOptions, Repository};

let dir = tempfile::tempdir()?;
let (repo, _) = Repository::init(
    dir.path(),
    &InitOptions::new()
        .with_initial_branch("main")
        .with_repository_options(
            RepositoryOptions::empty()
                .with_object_store(Arc::new(MemoryStore::new(HashAlgo::Sha1)))
                .with_ref_store(Arc::new(MemoryRefStore::new()))
                .with_work_tree(dir.path()),
        ),
)?;

What Repository::init writes

Repository::init with InitOptions::with_repository_options creates only what your stores do not own:

Injected On disk under the git directory
Object store only Everything except objects/
Ref store only Everything except HEAD and refs/
Both config only (plus the work tree beside it when not bare)

Object format comes from the store’s hash_algo; do not fight it with InitOptions::object_format unless you mean to error with RepositoryError::ObjectFormatMismatch.

Capability table

PrunableObjectStore

API On a prunable custom primary
Maintenance::prune_loose_objects Deletes unreachable objects past gc.pruneExpire
Maintenance::gc Pack-refs + reflog expiry + prune (skips on-disk repack / commit-graph)
Odb::gc Deletes all unreachable objects (no grace)

FilesOnlyOperation (typed refusal)

Operations that need the on-disk objects/ layout return RepositoryError::UnsupportedObjectStore with a FilesOnlyOperation variant — match on the enum, not strings. Grit’s fallbacks on a custom primary:

Operation Fallback on custom primary
Repack / .pack install Pack stream ingested object-by-object (ingest_pack_into, ingest_received_pack)
Commit-graph file Build in memory with Maintenance::build_commit_graph; attach bytes via OdbBuilder::commit_graph
Promisor / partial clone ODB Not supported (no silent filesystem promisor)
Prune / gc object deletion Only with PrunableObjectStore; otherwise typed prune refusal

The generated variant table lives on Object stores (| Name | Variant | Description | | --- | --- | --- | | gc | [Gc](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.Gc) | Garbage collection without a repository: [Odb::gc](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/struct.Odb.html#method.gc) on an Odbbuilt standalone, with no git directory to find reachability roots in. | |prune | [Prune](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.Prune) | Deleting unreachable objects from a custom primary that cannot delete ([WritableObjectStore::as_prunable](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/trait.WritableObjectStore.html#method.as_prunable) is None): [Maintenance::prune_loose_objects](https://docs.rs/grit-lib/latest/grit_lib/maintenance/struct.Maintenance.html#method.prune_loose_objects), [refs::gc::prune_loose_unreachable](https://docs.rs/grit-lib/latest/grit_lib/refs/gc/fn.prune_loose_unreachable.html), [Odb::gc](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/struct.Odb.html#method.gc), [Maintenance::gc](https://docs.rs/grit-lib/latest/grit_lib/maintenance/struct.Maintenance.html#method.gc). Implement [PrunableObjectStore](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/store/trait.PrunableObjectStore.html) to support them. | | repack | [Repack](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.Repack) | Repacking into objects/pack/: [Maintenance::repack](https://docs.rs/grit-lib/latest/grit_lib/maintenance/struct.Maintenance.html#method.repack) and [Maintenance::repack_all](https://docs.rs/grit-lib/latest/grit_lib/maintenance/struct.Maintenance.html#method.repack_all). | | commit_graph_write | [CommitGraphWrite](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.CommitGraphWrite) | Writing objects/info/commit-graph: [Odb::write_commit_graph](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/struct.Odb.html#method.write_commit_graph) and [Maintenance::write_commit_graph](https://docs.rs/grit-lib/latest/grit_lib/maintenance/struct.Maintenance.html#method.write_commit_graph). A custom store builds its graph in memory with [Maintenance::build_commit_graph](https://docs.rs/grit-lib/latest/grit_lib/maintenance/struct.Maintenance.html#method.build_commit_graph) instead. | | midx_write | [MidxWrite](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.MidxWrite) | Writing objects/pack/multi-pack-index: [Maintenance::write_midx](https://docs.rs/grit-lib/latest/grit_lib/maintenance/struct.Maintenance.html#method.write_midx). | | pack_store | [PackStore](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.PackStore) | The files backend’s pack and MIDX read cache: [Odb::pack_store](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/struct.Odb.html#method.pack_store). | | pack_install | [PackInstall](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.PackInstall) | Installing a pack file and its .idxunderobjects/pack/: [index_pack::install_pack_bytes](https://docs.rs/grit-lib/latest/grit_lib/objects/index_pack/fn.install_pack_bytes.html) and [index_pack::install_pack_path](https://docs.rs/grit-lib/latest/grit_lib/objects/index_pack/fn.install_pack_path.html). Use [index_pack::ingest_received_pack](https://docs.rs/grit-lib/latest/grit_lib/objects/index_pack/fn.ingest_received_pack.html) to write a pack into any primary. | | largest_pack_index | [LargestPackIndex](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.LargestPackIndex) | Finding the largest pack index in objects/pack/: [Odb::largest_pack_index_path](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/struct.Odb.html#method.largest_pack_index_path). | | loose_write | [LooseWrite](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.LooseWrite) | Writing loose object files into objects/: [Odb::write_local](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/struct.Odb.html#method.write_local), [Odb::write_raw_local](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/struct.Odb.html#method.write_raw_local) and [Odb::write_loose_materialize](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/struct.Odb.html#method.write_loose_materialize). [Odb::write](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/struct.Odb.html#method.write) works with any primary. | | files_source | [FilesSource](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.FilesSource) | The files-backed primary source itself: [Odb::primary](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/struct.Odb.html#method.primary). Use [Odb::primary_writable](https://docs.rs/grit-lib/latest/grit_lib/objects/odb/struct.Odb.html#method.primary_writable) for the primary of any kind. | | offload | [Offload](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.Offload) | Moving blobs to a large-object promisor: [Maintenance::offload_blobs](https://docs.rs/grit-lib/latest/grit_lib/maintenance/struct.Maintenance.html#method.offload_blobs), for the main repository and for the promisor repository. | | promisor_rewrite | [PromisorRewrite](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.PromisorRewrite) | Rewriting packs to leave out promised objects, as offload and dropping promised blobs do. | | partial_fetch | [PartialFetch](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.PartialFetch) | Fetching or cloning with an object filter (a partial clone) into a custom primary. | | receive_quarantine | [ReceiveQuarantine](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.ReceiveQuarantine) | The receive-pack quarantine directory under objects/. A custom primary receives pushes through an in-memory overlay instead. | | drop_large_blobs | [DropLargeBlobs](https://docs.rs/grit-lib/latest/grit_lib/repo/error/enum.FilesOnlyOperation.html#variant.DropLargeBlobs) | Dropping promised blobs from promisor packs: [Maintenance::drop_promised_blobs](https://docs.rs/grit-lib/latest/grit_lib/maintenance/struct.Maintenance.html#method.drop_promised_blobs). | ).

Performance checklist

Hook Why
read_into Rev-list, merge-base, and change-id scans reuse one buffer per commit instead of allocating every read.
contains_many Write-tree and fetch/push negotiation ask about many ids at once; default is one contains per id.
lookup_prefix Short-id and RevWalk::abbreviate; default scans all objects. Use ObjectIdPrefix::lowest / highest for sorted stores.
Attached commit-graph Walks skip commit reads; see Object stores — Commit-graphs for a custom store and the benchmark table under <!-- grit:benchmark-tables --> on Benchmarks for rev-list / merge-base rows.

Testing

Three workspace conformance crates exercise backends before you wire a full Repository:

Suite Crate module What it checks
Object read/write grit_test_support::odb_conformance run_read_suite, run_write_suite on ObjectStore / WritableObjectStore
Ref store grit_test_support::refstore_conformance run_suite on Arc<dyn RefStore>
Repository grit_test_support::repository_conformance BackendFixture + run_repository_suite, run_parity for typed RepositorySnapshot parity

Example pattern (object store):

use grit_test_support::odb_conformance::{run_read_suite, run_write_suite};

#[test]
fn my_store_passes_odb_conformance() {
    run_read_suite(|| seed_my_store(&FIXTURE_OBJECTS));
    run_write_suite(|| MyStore::new(HashAlgo::Sha1));
}

End-to-end SQLite + in-memory refs (parity against the files backend):

After changing integration tests under grit-examples/tests/, run python3 scripts/sync_integration_tests.py.

What still needs a .git directory on disk

When both stores are injected, Repository::open does not require a populated git directory; config can come from Environment::push_config_override only. Even then, several subsystems still read or write paths under the git directory (or require a work tree):

Area Without on-disk git layout
Objects, refs, config (injected) Fully served by your stores + environment
Log, rev-parse, merge trees, notes, bundles to Write Work through stores
Fetch (local) Works; pack spool uses temp files; FETCH_HEAD not written
Index, staging, COMMIT_EDITMSG, merge/rebase state Need git-dir files
Work-tree commands (status, checkout, stash) Need index + work tree
Linked worktrees, rerere, sparse-checkout Need their admin/state files
Hooks expecting another git to read refs/objects Not supported for purely injected stores
gc / repack / on-disk commit-graph Typed UnsupportedObjectStore unless prunable / in-memory graph (see above)

The same matrix appears in Object stores — No git directory at all. Injecting only the object store leaves refs and reflogs on disk as usual.

  • Object stores — Odb layering, alternates, pack ingest, benchmarks
  • Refs — References handle, reflog, branch tracking
  • Embedding — Environment, diagnostics, subprocess injection