Apache Storm cluster architecture
May 26, 2021 Apache Storm
The main highlight of Apache Storm is that it is a fault-facing, fast, no "single point failure" (SPOF) distributed application. ns.
Let's look at how the Apache Storm cluster designed and its internal architecture. The following diagram describes the cluster design.
Apache Storm has two types of nodes, Nimbus (primary node) and Supervisor (working node). N imbus is a core component of Apache Storm. N imbus's main job is to run the Storm topology. 20> t then assigns tasks to available supervisors.
Supervisor will have one or more work processes. isor delegates tasks to the work process. T he work process will generate as many executors as necessary and run the task. ms.
| Component | Describe |
|---|---|
| Nimbus (primary node) | Nimbus is the primary node of the Storm cluster. A ll other nodes in the cluster are called working nodes. The primary node is responsible for distributing data across all work nodes, assigning tasks to the work nodes, and monitoring failures. |
| Supervisor (working node) | The node that follows the instructions is called Supervisors. Supervisor has multiple work processes that manage the process to complete tasks assigned by nimbus. |
| Worker process (work process) | The work process performs tasks related to a specific topology. I nstead of running tasks themselves, the work process creates executors and asks them to perform specific tasks. The work process will have more than one executor. |
| Executor (Executor) | An executor is just a single thread generated by a worker process. The executor runs one or more tasks, but only for specific spouts or bolts. |
| Task (Task) | The task performs actual data processing. So, it's a spout or bolt. |
| ZooKeeper Framework (ZooKeeper Frame) |
Apache's ZooKeeper is a service that uses clusters (node groups) to coordinate between using clusters (node groups) and maintaining shared data with powerful synchronization technologies. de. ZooKeeper's help supervisor interacts with nimbus. I t is responsible for maintaining the state of nimbus, supervisor. |
Storm is stateless. Even if stateless has its drawbacks, it actually helps Storm process real-time data in the best possible and fastest way.
Storm is not completely stateless. I t stores its status in Apache ZooKeeper. re. ypically, a service monitoring tool like monit monitors Nimbus and restarts it in the event of any failure.
Apache Storm also has an advanced topology called Trident Topology, which has state maintenance and also provides an advanced API such as Pig. We'll discuss all of these features in the next chapter.