feat: add ReadHeader, ReadMany, and ReadManyHeaders#38
Open
ajroetker wants to merge 3 commits intotidwall:masterfrom
Open
feat: add ReadHeader, ReadMany, and ReadManyHeaders#38ajroetker wants to merge 3 commits intotidwall:masterfrom
ajroetker wants to merge 3 commits intotidwall:masterfrom
Conversation
ReadHeader returns only the first n bytes of an entry's data, enabling efficient scanning of fixed-size metadata headers without copying full payloads. This supports use cases like WAL-backed queues that embed retry counts or item counts in entry headers. ReadHeader always copies the result (even with NoCopy enabled) because the segment buffer may be evicted by the LRU cache, making a slice into it unsafe for the caller to hold.
Add batch read methods that reuse loaded segments across contiguous index ranges, avoiding redundant segment lookups. Extract readEntry and readEntryHeader helpers from Read and ReadHeader. Refactor Read to use readEntry. - ReadMany(start, count) returns [][]byte for a range of entries - ReadManyHeaders(start, count, n) returns first n bytes of each entry - Segment reuse: only calls loadSegment when crossing segment boundaries
ajroetker
changed the title
Author
|
also conveniently aligns with #17 I think |
- Fix integer overflow in end-index calculation for ReadMany/
ReadManyHeaders by using overflow-safe arithmetic
- Fix NoCopy safety: ReadMany always copies entry data since segment
eviction can invalidate slices from earlier iterations
- Fix ReadHeader returning []byte{} instead of nil for n<=0, and move
n<=0 check after corrupt/closed guards so closed logs still error
- Extract shared readMany helper to deduplicate ReadMany/ReadManyHeaders
- Add tests: max-count overflow, closed-n-zero, JSON format headers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New methods for efficient partial and batch reads from the WAL.
ReadHeader(index, n)— read the firstnbytes of an entry without copying the full payload. Useful for inspecting embedded metadata headers (retry counts, item counts, etc.) without deserializing the full entry.ReadMany(start, count)— batch read a contiguous range of entries, returning[][]byte. Reuses loaded segments across the range instead of callingloadSegmentper entry.ReadManyHeaders(start, count, n)— batch read the firstnbytes of each entry in a range. Combines the efficiency ofReadHeaderwith the segment reuse ofReadMany.Design
ReadHeaderalways copies the result even withNoCopy— the segment buffer can be evicted by the LRU cache between calls.ReadManyandReadManyHeaderstrack the current segment and only callloadSegmentwhen crossing segment boundaries.readEntryandreadEntryHeaderare extracted fromRead/ReadHeaderand shared by the batch methods.Readis refactored to usereadEntry.RLock, consistent with existing read paths.[FirstIndex, LastIndex]— out-of-bounds requests return partial results rather than errors.