A Clojure library designed to map functions over kafka topics.
The kafka-adapter function is the main function, it takes one function and an options map. It returns a channel, any activity (writing or closing) on it stops the adapter.
The adapter dispatches to workers, manages dynamic rebalancing.
The following options are available:
:configis a kafka config map (strings to strings) shared by consumer and producer.:consumer-configand:producer-configare kafka config maps specific to consumers and producers.:input-topicsa collection of topic names to consume.:topic-aliasesa map from aliases to topic names; this is used to route "messages" emitted byfto actual topics. There are two reserved aliases::outand:state. Both are mandatory for stateful mode.:edn,:edn-inand:edn-outall default to true and control if the adpater should automatically read and print to edn.:tracesa channel to which maps of the form{:from deps :to [topic partition offset]}are sent.:deps-fna function from an outgoing value to a collection of dependencies ids.
Messages are read from the topics listed under :input-topics.
When a message arrives, its value (parsed as edn by default) is passed to f which returns a collection of pairs [alias v] where alias is a topic alias used to lookup the actual topic in the map specified under :topic-aliases. This functionality can be used for routing as messages can be dispatched to serveral outputs.
When the :traces option is set to a channel, this channel receives values like {:from deps :to [topic partition offset]} where deps is a collection of source messages ids (for Kafka messages they are identified by a triple [topic partition offset]).
If :traces is set but not :deps-fn then the incoming message is considered to be the only dependency of the outgoing message.
Copyright © 2016 HCA
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.