Competing Consumers	Messaging Patterns  Previous   Next
	Messaging Patterns » Messaging Endpoints » Competing Consumers

An application is using Messaging. However, it cannot process messages as fast as they’re being added to the channel.

How can a messaging client process multiple messages concurrently?

Create multiple Competing Consumers on a single channel so that the consumers can process multiple messages concurrently.

Competing Consumers are multiple consumers that are all created to receive messages from a single Point-to-Point Channel. When the channel delivers a message, any of the consumers could potentially receive it. The messaging system's implementation determines which consumer actually receives the message, but in effect the consumers compete with each other to be the receiver. Once a consumer receives a message, it can delegate to the rest of its application to help process the message. (This solution only works with Point-to-Point Channels; multiple consumers on a Publish-Subscribe Channel just create more copies of each message.)

Example: Apache KafkaNEW

Apache Kafka is a distributed streaming platform that allows clients to consume messages in both Publish-Subscribe Channel and Competing Consumers semantics. Kafka is optimized for high throughout and horizontal scalability and therefore tries to avoid the overhead that can be inherent in coordinating across multiple Competing Consumers. So while the client interface allows multiple consumers to consume messages off a topic in a competing fashion, guaranteeing that each message is only consumed by one of the consumers, the implementation actually relies on multiple partitions implemented as s:

In the example above, a Kafka topic consists of three partitions divided across two consumers. A Message Router distributes the messages across the three Point-to-Point Channels. This router is generally a that works off the message key. This allows the sender to control the distribution of messages, e.g. to assure related messages go onto the same partition. This can be necessary as in-order delivery across partitions is not guaranteed.

This design implies that the decision which consumer receives the message is made a priori, meaning consumers don't actually compete but rather read from independent channels. This limits the ability to load balance between consumers, e.g. if one channel runs empty, the corresponding consumer won't be able to aid other consumers whose channels still have messages. Also, as the number of partitions is set independently, not all consumers will service the same number of partitions, as illustrated in the example above.

However, as Kafka partitions are relatively inexpensive, there are usually lots of them, meaning that each consumer is in reality likely to read of many partitions and therefore channels. This highlights another trade-off: there can't be more competing consumers in a consumer group than there are partitions on that topic.

Related patterns: Event-Driven Consumer, Message, Message Channel, Message Dispatcher, Message Router, Messaging, Point-to-Point Channel, Polling Consumer, Publish-Subscribe Channel, Transactional Client

Find the full description of this pattern in: Enterprise Integration Patterns Gregor Hohpe and Bobby Woolf ISBN 0321200683 650 pages Addison-Wesley	From Enterprise Integration to Enterprise Transformation: My new book describes how architects can play a critical role in IT transformation by applying their technical, communication, and organizational skills with 37 episodes from large-scale enterprise IT. DRM-free eBook on Leanpub.com Print book on Amazon.com
Parts of this page are made available under the Creative Commons Attribution license. You can reuse the pattern icon, the pattern name, the problem and solution statements (in bold), and the sketch under this license. Other portions of the text, such as text chapters or the full pattern text, are protected by copyright.

Messaging Patterns » Messaging Endpoints » Competing Consumers

Previous   Next