May 17, 2021 Spark Programming guide
Similar to Spark, Spark Streaming can also take advantage of the maven repository. To write your own Spark Streaming program, you need to introduce the following dependencies into your SBT or Maven project
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>1.2</version>
</dependency>
In order to get data from Kafka, Flume, and Kinesis, sources that are not available in Spark's core API, we need
spark-streaming-xyz_2.10
to the dependency.
For example, some generic components are shown in the following table:
Source | Artifact |
---|---|
Kafka | spark-streaming-kafka_2.10 |
Flume | spark-streaming-flume_2.10 |
Kinesis | spark-streaming-kinesis-asl_2.10 |
spark-streaming-twitter_2.10 | |
ZeroMQ | spark-streaming-zeromq_2.10 |
Mqtt | spark-streaming-mqtt_2.10 |
For the latest list, visit Apache repository