Implement multiple tiers caching with fallback and backfilling

[Felixoid]

This PR addresses multiple requests to have tiered caching.

Closes #30
Closes #1020
Closes #2493
Closes #2566

What is added:

• multilevel configuration for the caching chain
• config parameter for cache writing tolerance: ignore errors, fail on write to L0, fail on a write to any level
• added stats for each caching level
• documentation
• tests
• integration tests

[codecov-commenter]

Codecov Report

❌ Patch coverage is 89.60422% with 197 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.18%. Comparing base (f5edaf1) to head (a24dc69).

Files with missing lines	Patch %	Lines
src/cache/multilevel.rs	87.54%	66 Missing ⚠️
src/cache/multilevel_test.rs	93.99%	63 Missing ⚠️
src/config.rs	79.91%	46 Missing ⚠️
src/cache/cache.rs	70.73%	12 Missing ⚠️
src/cache/disk.rs	81.25%	6 Missing ⚠️
src/cache/readonly.rs	0.00%	3 Missing ⚠️
src/server.rs	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2581      +/-   ##
==========================================
+ Coverage   73.38%   74.18%   +0.80%     
==========================================
  Files          68       70       +2     
  Lines       37340    39208    +1868     
==========================================
+ Hits        27402    29087    +1685     
- Misses       9938    10121     +183     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Felixoid]

I've done quite a few features today, like control for write error handling, and added server stats for multilevel caching.

[sylvestre]

this is a huge patch, can you split a bit more? thanks

[Felixoid]

Probably I could take something out of cache.rs, but the rest is actually multilevel.rs, and 2/3 of 2k LoC are tests.

[Felixoid]

Rebased to the current main.

This PR now contains the multi-level cache feature exclusively. Most of the changes are the tests and documentation, which are around 70%.

[AJIOB]

Hi @drahnr and @sylvestre ,

Is it OK for you? Can we merge these changes?

[sunnyguan]

Thanks a lot for the work here, just curious if SCCACHE_BASEDIRS is handled properly when using the multi-tier setup?

[sylvestre]

i am not ignoring you. just higher priorities lately, sorry

[Felixoid]

Thanks a lot for the work here, just curious if SCCACHE_BASEDIRS is handled properly when using the multi-tier setup?

I don't have any specific tests for it, but I don't see how it could break.

I'll try to add it too, just in case

[Felixoid]

@sunnyguan thanks a lot for the spotlight, I indeed forgot to implement the basedirs treat for the MultiLevelStorage 🤦

[Felixoid]

I believe the cmake-modules failure is unrelated; the fix of test is in #2649

The dedicated issue for cmake 4.0 failure #2650

[Felixoid]

The merged #2649 caused a conflict. Resolved it by rebasing