Version1 Rebased - Coroutine Migration#167
Closed
fpelliccioni wants to merge 14 commits intomasterfrom
Closed
Conversation
Add infrastructure for C++20 coroutine support detection and testing: - Add compiler_constraints.hpp with compile-time requirements: - C++23 standard required - Coroutine support (__cpp_impl_coroutine) - std::expected support (__cpp_lib_expected) - Concepts support (__cpp_concepts) - Compiler version checks (GCC 13+, Clang 17+, MSVC 19.36+) - Add coroutine_config.hpp with feature flag auto-detection - KTH_USE_COROUTINES: main feature flag - Version detection for Boost.Asio and standalone Asio - Helper macros for conditional compilation - Add comprehensive smoke tests for coroutine support - Asio version detection tests - C++20 and coroutine support detection tests - Basic coroutine functionality tests (when enabled) - Timer, error handling, racing (||), and parallel (&&) operators - Exclude network integration tests from CI (require real TCP connections) This is Phase 0 of the coroutine migration plan, establishing the foundation for future networking code modernization.
Phase 1 of the coroutine migration. Adds base infrastructure for modern async programming with C++23 coroutines and standalone Asio 1.36.0. - `threadpool.hpp`: Rewritten using `asio::thread_pool` - `async_channel.hpp`: Type aliases for async communication channels - `async_channel<T>`: NOT thread-safe, use with strand - `concurrent_channel<T>`: Thread-safe for multi-threaded scenarios - `awaitable_helpers.hpp`: Coroutine utilities (with_timeout, etc.) - Added `asio_standalone` option to conanfile.py (default: True) - Added `KTH_ASIO_STANDALONE` CMake option and macro - Updated to standalone Asio 1.36.0 - Changed `.service()` → `.get_executor()` across codebase - Simplified `sequencer`, `work` classes to use modern executor API - Added comprehensive tests for threadpool stop/join behavior - Added tests for async_channel with strand serialization - Added tests for concurrent_channel multi-threaded scenarios - Added stress test with 20 producers × 10 items `asio::experimental::channel` is NOT thread-safe for concurrent access. Documentation: "Distinct objects: Safe. Shared objects: Unsafe." Use `concurrent_channel` for multi-producer/consumer scenarios.
Add coroutine-based peer_session class that replaces proxy/channel: - peer_session: modern async peer connection handler using C++20 coroutines - Concurrent read/write/timer loops using asio::awaitable operators - Message framing with Bitcoin protocol validation (magic, checksum, payload size) - Inactivity and expiration timers for connection lifecycle Add connection helper functions (Phase 3): - async_connect(): DNS resolution + TCP connect with timeout - async_accept(): accept incoming connections - async_listen(): create listening acceptor on port Includes comprehensive unit tests (31 test cases): - Construction, properties, lifecycle management - Message send/receive with protocol validation - Timer behavior (inactivity, expiration) - Bidirectional communication - Connection helpers (listen, accept, connect, timeout)
#144) Add peer_manager class that replaces the legacy pending<connector>/ pending<channel> collections with a single unified collection. Key features: - All operations are coroutines (asio::awaitable) - Uses asio::strand for serialization (no mutex needed) - add/remove/find operations with nonce or authority lookup - broadcast template for sending messages to all peers - for_each handler for iterating over active peers - Capacity limiting with max_connections parameter Includes comprehensive unit tests for all functionality.
Implement protocol logic as free functions using C++20 coroutines: - wait_for_message<T>() and wait_for_any_message() helpers - perform_handshake() for version/verack exchange - run_ping_pong() for keepalive loop - request_addresses() and send_addresses() for addr protocol These replace the complex protocol class hierarchy with simple, composable coroutine functions that operate on peer_session.
Add p2p_node class that integrates all coroutine-based networking components into a unified P2P networking interface. Components: - p2p_node: Main networking class with lifecycle management - Uses async_connect/async_accept/async_listen from peer_session.hpp - Uses peer_manager for connection tracking - Uses protocols_coro for handshake, ping/pong, address requests - Supports seeding, inbound/outbound connections, broadcasting Features: - Coroutine-based API (asio::awaitable) - Automatic seeding when host pool is low - Configurable inbound/outbound connections - Top block tracking for version negotiation - Thread pool based execution Note: p2p_node will be renamed to p2p after legacy code removal in Phase 7 cleanup.
- Complete Phase 7 of the coroutine migration with a fully functional P2P network layer - Delete legacy node sessions and protocols - Comment out legacy network code (to be deleted after full validation) - Add blockchain protocol declarations and implementations in protocols_coro - Add headers-first sync infrastructure (sync_session, header_organizer, validate_header) - Add TUI mining dashboard and display modes - All legacy session classes (session_inbound, session_outbound, session_manual, session_block_sync, session_header_sync) - All legacy protocol classes (protocol_block_in/out, protocol_header_sync, protocol_transaction_in/out, protocol_double_spend_proof_in/out) - Legacy sessions (session.cpp, session_*.cpp) - Legacy protocols (protocol_*.cpp) - Legacy core (p2p.cpp, proxy.cpp, channel.cpp, acceptor.cpp, connector.cpp, message_subscriber.cpp) - Legacy test (test/p2p.cpp) - Blockchain protocol declarations in protocols_coro.hpp (request_headers, request_block, send_inventory, etc.) - Full implementations in protocols_coro.cpp - sync_session: coordinates headers-first sync with peers - header_organizer: caches and validates headers during IBD with memory-aware batching - validate_header: header validation (PoW, checkpoints, chain continuity) - broadcaster template for async channel subscriptions - system_memory utilities for cross-platform memory estimation - block_chain::organize_headers_batch() for batch header storage - TUI mining dashboard with sync statistics - Display mode support (headless, tui, etc.) - coroutine-migration-status.md documenting current state and TODOs - [x] Compilation passes - [x] Unit tests pass - [x] Verified on real BCH mainnet: - Seeding works (obtains 1000+ addresses) - Outbound connections work - Version/verack handshake completes - Ping/pong keepalive works - Message reception works (inv, addr, getheaders, etc.) - Headers sync starts (receives 2000 headers per batch)
- Get block hashes from header_index instead of database - Headers are stored in memory during IBD, not persisted to DB yet - Enables proper Phase 1 -> Phase 2 transition after header sync completes
See issue #157 for full implementation details.
… download (#162) * refactor(executor): modernize executor with internal io_context and graceful shutdown - Refactor executor to own its io_context and run it in a dedicated thread - Replace three atomic bools with single state enum (stopped/starting/running/stopping) - Move user_agent configuration from executor to full_node - Add stopped() checks to all peer_manager async methods to prevent strand access after pool shutdown (fixes segfault on Ctrl-C) - Fix signal handling in wait_for_stop_signal() with proper work_guard - Update C API to use new executor interface - Improve TUI dashboard with network statistics display - Inline KTH_* message macros directly where used * feat(network): implement structured concurrency with task_group and peer_supervisor Replace detached coroutines with structured concurrency patterns: - Add task_group utility for managing parallel coroutines with automatic cleanup - Implement peer_supervisor as a nursery that manages peer lifecycles - Use concurrent_channel for peer events instead of fire-and-forget spawns - Add supervisor_ready_ atomic for deterministic synchronization - Close response channels (headers_responses_, block_responses_) on peer stop to properly propagate shutdown to waiting sync operations - Modernize executor with internal io_context and graceful shutdown - Fix thread_ceiling to use all cores when threads=0 Key changes: - p2p_node::run() now uses task_group instead of co_spawn with detached - peer_supervisor tracks all peer tasks and waits for completion on shutdown - Clean shutdown: all coroutines complete orderly, "Good bye!" prints reliably - Manual peer connections wait for supervisor_ready_ before connecting * feat(node): add parallel block download with lock-free coordinator v2 Implement parallel block download system with two coordinator versions: V1 (strand-based): - Uses asio::strand for serialization instead of mutex - Avoids coroutine blocking issues from std::mutex V2 (lock-free): - Atomic CAS operations for chunk claiming - Slot-based system: 800 slots per round (8 peers × 100 multiplier) - 3 states: FREE (0), IN_PROGRESS (1), COMPLETED (2) - Peer watcher detects and adds new peers dynamically - Round advances when all slots complete Features: - Parallel downloads from multiple peers simultaneously - In-order block validation pipeline - Timeout detection for stalled downloads - Dynamic peer addition during sync - Progress logging with ETA, peer count, in-flight chunks Files added: - block_download_coordinator.hpp/cpp (V1) - block_download_coordinator_v2.hpp/cpp (V2) - parallel_sync.hpp/cpp (V1) - parallel_sync_v2.hpp/cpp (V2) # Conflicts: # src/blockchain/include/kth/blockchain/pools/header_organizer.hpp # src/blockchain/src/pools/header_organizer.cpp # src/network/include/kth/network/peer_session.hpp # src/network/src/banlist.cpp # src/network/src/p2p_node.cpp # src/network/src/peer_manager.cpp # src/network/src/peer_session.cpp # src/node-exe/src/main.cpp # src/node/include/kth/node.hpp # src/node/include/kth/node/sync_session.hpp # src/node/src/executor/executor.cpp # src/node/src/full_node.cpp
…raceful shutdown - Refactor executor to own its io_context and run it in a dedicated thread - Replace three atomic bools with single state enum (stopped/starting/running/stopping) - Move user_agent configuration from executor to full_node - Add stopped() checks to all peer_manager async methods to prevent strand access after pool shutdown (fixes segfault on Ctrl-C) - Fix signal handling in wait_for_stop_signal() with proper work_guard - Update C API to use new executor interface - Improve TUI dashboard with network statistics display - Inline KTH_* message macros directly where used
…eer_supervisor Replace detached coroutines with structured concurrency patterns: - Add task_group utility for managing parallel coroutines with automatic cleanup - Implement peer_supervisor as a nursery that manages peer lifecycles - Use concurrent_channel for peer events instead of fire-and-forget spawns - Add supervisor_ready_ atomic for deterministic synchronization - Close response channels (headers_responses_, block_responses_) on peer stop to properly propagate shutdown to waiting sync operations - Modernize executor with internal io_context and graceful shutdown - Fix thread_ceiling to use all cores when threads=0 Key changes: - p2p_node::run() now uses task_group instead of co_spawn with detached - peer_supervisor tracks all peer tasks and waits for completion on shutdown - Clean shutdown: all coroutines complete orderly, "Good bye!" prints reliably - Manual peer connections wait for supervisor_ready_ before connecting
Implement parallel block download system with two coordinator versions: V1 (strand-based): - Uses asio::strand for serialization instead of mutex - Avoids coroutine blocking issues from std::mutex V2 (lock-free): - Atomic CAS operations for chunk claiming - Slot-based system: 800 slots per round (8 peers × 100 multiplier) - 3 states: FREE (0), IN_PROGRESS (1), COMPLETED (2) - Peer watcher detects and adds new peers dynamically - Round advances when all slots complete Features: - Parallel downloads from multiple peers simultaneously - In-order block validation pipeline - Timeout detection for stalled downloads - Dynamic peer addition during sync - Progress logging with ETA, peer count, in-flight chunks Files added: - block_download_coordinator.hpp/cpp (V1) - block_download_coordinator_v2.hpp/cpp (V2) - parallel_sync.hpp/cpp (V1) - parallel_sync_v2.hpp/cpp (V2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Same changes as version1 -> master PR (#151), rebased on top of master with style-cleanup
Changes