Add an experimental opt in tar based implementation of the local cache#1065
Add an experimental opt in tar based implementation of the local cache#1065christiango wants to merge 9 commits intomainfrom
Conversation
| const rm = promisify(fs.rm); | ||
| const readdir = promisify(fs.readdir); | ||
| const stat = promisify(fs.stat); |
There was a problem hiding this comment.
This is outdated--you can import these directly from fs/promises now (that's usually also preferable to fs.promises since it's easier to mock if needed)
There was a problem hiding this comment.
Good catch, I missed that
| // Read all output files in parallel, then build and write the tar | ||
| // in one shot. This is dramatically faster than tar.create() which | ||
| // processes each file sequentially through Minipass streams. | ||
| const fileEntries: TarFileEntry[] = await Promise.all( |
There was a problem hiding this comment.
I'm not sure firing off an unbounded number of parallel read or write requests is a good idea? (same in other places)
There was a problem hiding this comment.
So locally in our repo it's been faster than what we currently do with the local cache provider. Do we have some examples of limits in other cache implementations I can apply here?
| } | ||
|
|
||
| interface TarFileEntry { | ||
| path: string; |
There was a problem hiding this comment.
Can you add a comment about whether this is relative or absolute?
There was a problem hiding this comment.
Adding jsdoc comments to all interfaces
| } | ||
|
|
||
| /** | ||
| * Build a tar buffer from in-memory file entries. |
There was a problem hiding this comment.
I assume Claude wrote this but is there some documentation for this format?
| ? undefined | ||
| const useLocalTarCache = cacheOptions?.useLocalTarCache === true; | ||
|
|
||
| const localCacheProvider = |
There was a problem hiding this comment.
Especially now that we have backfill in the same repo, I think it would be better if the tar caching option and implementation could be added to the backfill side.
2 participants
We're seeing that for operations that are fast (such as transpiling with SWC) lage caching overhead is extremely high from copying the files over to the cache. This implements a local cache that uses tar for storing the cache. Here is the performance delta for this implementation from an internal repo
And for reference here is the overhead before and after compared to lage --no-cache on that same repo