Spark Streaming monitoring applications

Spark Streaming monitors the app

In addition to Spark's monitoring capabilities, Spark Streaming adds some proprietary features. W hen StreamingContext is applied, the Spark web UI displays the Streaming which displays statistics for running receivers (whether the receivers are alive, the number of records received, receiver errors, etc.) and the completed batch (batch time, queue wait). This can be used to monitor the processing of flow applications.

Processing Time and Processing Time in Scheduling Delay are two metrics that are important. T he first metric represents the time the batch data was processed, and the second metric represents the wait time for the current batch in the queue after the previous batch has finished. I f the batch time lasts longer than the batch interval or the queue wait time continues to increase, this indicates that the system cannot process the data at the rate at which the batch data is generated, and the entire process lags behind. In this case, consider reducing batch time.

The processing of the Spark Streaming program can also be monitored through the StreamingListener interface, which allows you to get the receiver status and processing time. Note that this interface is a developer API and it is likely to provide more information in the future.