Apache Kafka cluster architecture


May 26, 2021 16:00 Apache Kafka



Take a look at the illustration below. /b10> It shows Kafka's cluster diagram.

Apache Kafka cluster architecture

The following table describes each component shown in the figure above.

S.No Components and instructions
1

Broker (Agent)

Kafka clusters typically consist of multiple agents to maintain load balancing. /b10> Kafka agents are stateless, so they use ZooKeeper to maintain their cluster state. /b11>A Kafka proxy instance can handle hundreds of thousands of reads and writes per second, and each Broker can process TB messages without performance impact. /b12> Kafka broker leadership elections can be completed by ZooKeeper.

2

ZooKeeper

ZooKeeper is used to manage and coordinate Kafka agents. /b10> The ZooKeeper service is primarily used to notify producers and consumers of any new agents in the Kafka system or agent failures in the Kafka system. /b11> According to Zookeeper, notifications of the existence or failure of agents are received, and then producers and consumers take decisions and begin coordinating their tasks with certain other agents.

3

Producers (producers)

Producers push data to brokers. /b10> When a new agent starts, all producers search for it and automatically send messages to the new agent. /b11> Kafka producers do not wait for confirmation from the agent and send messages as quickly as the agent can handle.

4

Consumers (Consumers)

Because the Kafka agent is stateless, this means that consumers must maintain how much messages have been consumed by using partition offsets. /b10> If the consumer confirms a specific message offset, it means that the consumer has consumed all previous messages. /b11> The consumer makes an asynchronous pull request to the agent to have a byte buffer ready to consume. /b12> Consumers can simply regress or jump to any point in the partition by providing an offset value. /b13> The consumer offset value is notified by ZooKeeper.