Add packfile library design doc

[tamirms]

Summary

• Adds a design document for a compact, immutable file format that provides O(1) random access to ordinal-indexed data (events, bitmaps, ledgers) with minimal I/O
• This is a building block for a full history implementation of RPC (v2)
• The doc covers the file format, read/write paths, integrity model, and API reference
• The goal is to align on this design before proceeding with implementation — please flag any concerns or blockers so we can address them during the design stage

Note: This PR targets main for now but will be retargeted to the full history RPC / v2 branch once we align on that before merging.

Further reading

• Reference implementation: packfile, intpack
• Benchmarks — packfile vs RocksDB sstable across write throughput, sequential/random reads, compression efficiency, and bitmap indexing
• Format comparison — technical evaluation of packfile vs RocksDB sstable across performance, code complexity, dependencies, and suitability for remote storage

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a design document proposing a “packfile” immutable, ordinal-indexed file format and Go library API intended as a building block for Stellar RPC v2 full-history storage (compact storage with O(1) random access and minimal I/O).

Changes:

• Introduces the packfile/intpack concepts, goals/non-goals, and usage examples.
• Specifies the on-disk layout (records, index, trailer) plus integrity/content-hash model.
• Documents proposed Go APIs for writer/reader, concurrency behavior, and error surface.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.