Conversation
- Add MessagePipeline<T> interface for end-to-end (byte[] <-> T) processing - Refactor MessageSink to a single-argument Consumer<T> for processed objects - Update KPipeConsumer, CompositeMessageSink, and all sinks/tests to use new MessageSink<T> signature - Simplify registry and pipeline builder APIs for typed pipelines and sinks - Enhance AvroFormat with default schema support and implement deserialize - Remove legacy processJson/processAvro helpers in favor of pipeline-based processing - Update tests and integration tests to use new typed sink and pipeline APIs - Improve error handling and logging consistency in sinks
…ocessing logic - Add getSink() method to MessagePipeline interface for terminal sink retrieval - Update KPipeConsumer to invoke pipeline sink if present, with fallback to consumer sink - Adjust TypedPipelineBuilder to implement getSink() - Improve documentation for inScopedCaches methods and registry sink wrapping - Enhance AvroFormat deserialization with error handling - Update App to use withPipeline instead of withProcessor
…or typed pipelines - Replace format-specific pipeline builders with unified pipeline(MessageFormat) API in MessageProcessorRegistry - Update App to use the new typed pipeline builder for protobuf processing - Simplify consumer creation by removing explicit sink registry wiring - Improve documentation and usage examples for new pipeline API in registry and processor classes
…ion with additional types - Refactor MessageProcessorRegistry to streamline operator and sink wrapping, consolidating metrics logic and improving type safety - Update getOperator and wrapSink to always return a wrapped function, ensuring consistent behavior - Add predefined registry keys and registrations for JSON map and Avro generic record sinks in MessageSinkRegistry - Improve TypedPipelineBuilder to log a warning when a sink key is missing in the registry - Remove unused startTime field from App
…ine benchmark pipelines - Replace byte[] sink keys with typed RegistryKey variants for JSON and Avro sinks - Remove redundant sink keys and registrations from MessageSinkRegistry - Update benchmarks and App to use new typed pipeline and sink APIs - Refactor manual processing in benchmarks to use MessageFormat for serialization/deserialization - Minor cleanup in TypedPipelineBuilder and test assertions
…ator/sink addition - Remove redundant sink key factory from RegistryKey and clarify type signatures for json/avro helpers - Simplify TypedPipelineBuilder operator addition to support varargs and direct registry lookup - Refactor sink addition in TypedPipelineBuilder to allow multiple sinks and remove unnecessary logging - Update usage in App and benchmarks to use new registry key helpers and streamlined pipeline builder API - Improve documentation for clarity and consistency
…tors for sink registration and retrieval
…chmarks and pipeline builder
…into helper methods in KPipeConsumer
…itialization in KPipeConsumer
…age processors and consumer
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.