Skip to content

Add incremental Rust compilation via Bazel-managed tree artifacts#3925

Open
rejuvenile wants to merge 1 commit intobazelbuild:mainfrom
rejuvenile:feat-incremental-compilation
Open

Add incremental Rust compilation via Bazel-managed tree artifacts#3925
rejuvenile wants to merge 1 commit intobazelbuild:mainfrom
rejuvenile:feat-incremental-compilation

Conversation

@rejuvenile
Copy link
Copy Markdown

Summary

  • Adds --@rules_rust//rust/settings:incremental=true build setting to enable incremental Rust compilation in Bazel
  • Incremental state is stored as Bazel-managed tree artifacts, compatible with sandboxing and remote execution
  • Process wrapper seeds the incremental cache from previous builds using hardlinks (with EXDEV fallback for cross-device scenarios)
  • Configurable size threshold (incremental_seed_threshold_mb, default 20) skips seeding small crates where rustc's dep-graph loading overhead exceeds benefit
  • unused_inputs_list ensures seed changes don't invalidate action cache keys

Design

Rustc's incremental compilation stores state in a persistent directory. This PR makes that state a Bazel-managed tree artifact so:

  1. The path is identical on local and remote workers (no absolute-path divergence)
  2. Bazel distributes the cache via CAS for consistent state across dynamic execution
  3. Sandboxing and remote execution work normally

Only workspace crates get incremental compilation (external deps are excluded via ctx.label.workspace_root gate). Exec-config targets (proc macros, build scripts) are also excluded. Both main and metadata actions use -Cincremental for matching SVH but with separate tree artifacts to prevent corruption from --rustc-quit-on-rmeta.

The process wrapper handles cache seeding:

  • --copy-seed: hardlinks a seed tree artifact into the output directory
  • --seed-prev-dir: reads directly from the previous build's output (for the RustcIncrSeed action)
  • --seed-min-mb: configurable threshold (default 20 MiB) to skip small caches
  • --write-unused-inputs: populates the unused inputs list so Bazel excludes the seed from cache keys
  • --exit-early: exits after pre-exec operations without spawning a child process
  • Parallel hardlinking for trees with >256 files

Test plan

  • Build with --@rules_rust//rust/settings:incremental=true and verify incremental compilation works
  • Verify cache sharing across builds (second build should be faster)
  • Verify external deps are not affected by incremental compilation
  • Verify builds work without the flag set (default behavior unchanged)

…acts

Rustc's incremental compilation cache is stored as Bazel-managed tree
artifacts so the path is identical on local and remote workers. The
process wrapper seeds the cache from previous builds using hardlinks
(with EXDEV fallback), with a configurable size threshold to skip
small crates where loading overhead exceeds benefit.

Key changes:
- rustc.bzl: Declare tree artifact outputs for incremental state,
  gate on workspace crates, separate main/metadata artifacts
- rust.bzl/settings.bzl: Add incremental and seed_threshold_mb settings
- process_wrapper: Add --copy-seed, --seed-prev-dir, --seed-min-mb,
  --write-unused-inputs, --exit-early flags for cache management
- Parallel hardlinking for trees with >256 files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant