feat(network): Batches and parallelizes block downloads#10442
feat(network): Batches and parallelizes block downloads#10442
Conversation
… for FindBlocks/FindHeaders requests
…mprove performance
…e verified and download batches of blocks from multiple peers in parallel
…chain tip height during checkpoint sync
…ing downloads Move download_batch from a free function to an instance method on Downloads, add pending_download_hashes tracking to prevent duplicate downloads, and handle duplicate block verification gracefully instead of panicking. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR improves Zebra's initial block sync performance by batching and parallelizing block downloads. Instead of requesting blocks one-at-a-time sequentially, the syncer now groups blocks into adaptively-sized batches, sends multiple batches concurrently, and overlaps download progress with tip discovery. This reduces per-request overhead, maximizes network utilization, and eliminates pipeline stalls. The PR also improves peer routing during sync: multi-block requests target peers at or above the current chain height, and tip discovery requests spread fanout across distinct peers for better diversity.
Changes:
- Implement batched block downloads with per-block retry logic and dynamic batch sizing based on observed block sizes
- Run multiple batches concurrently using
SelectAllto interleave block arrivals and prevent waiting for entire batches - Extend tips proactively while downloads are in-flight, and drain completed downloads during tip discovery, eliminating pipeline stalls
- Add peer routing logic to prefer peers at/above chain tip height for multi-block requests and distinct peers for tip-discovery fanout
- Update integration tests to verify batched request behavior and concurrent verification
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| zebrad/src/components/sync/downloads.rs | Implements batched download logic with dynamic batch sizing, retry mechanism, and block size tracking for adaptive batching |
| zebrad/src/components/sync.rs | Integrates batched downloads, adds drain_ready_downloads() to pipeline during tip discovery, and refactors tip extension to run concurrently |
| zebra-network/src/peer_set/set.rs | Adds peer routing for multi-block requests (targeting tall peers) and tip-discovery requests (targeting distinct peers), with recent-peer tracking |
| zebra-network/src/peer/load_tracked_client.rs | Adds remote_height() method to expose peer's reported chain height for peer selection during sync |
| zebrad/src/components/sync/tests/vectors.rs | Updates test expectations to verify batched request behavior instead of sequential per-block requests |
…r improved readability
…mic batch size calculation
| // Report block size for adaptive batch sizing. | ||
| { | ||
| use zebra_chain::serialization::ZcashSerialize; | ||
| let _ = block_size_sender.send(block.zcash_serialized_size()); |
There was a problem hiding this comment.
Record the number of every hundredth block. Start at a random number in 0..99, i.e. compute and send the serialized size if block.height + random_num % 100 == 0
Motivation
We want to improve Zebra's initial sync performance.
Solution
See commit messages for more information
Tests
Running full Testnet sync locally.
Follow-up Work
AI Disclosure
PR Checklist