blog/BigData-Notes
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update Spark_Streaming整合Kafka.md

Browse Source
This commit is contained in:
heibaiying 2019-05-26 19:07:10 +08:00 committed by GitHub
parent 4dd53cfbe1
commit 5ccbdbf89e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 4 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
notes/Spark_Streaming整合Kafka.md
View File

@ -26,7 +26,7 @@ Spark针对Kafka的不同版本，提供了两套整合方案：`spark-streaming
| Direct DStream | Yes | Yes |
| SSL / TLS Support | No | Yes |
| Offset Commit API(偏移量提交) | No | Yes |
| Dynamic Topic Subscription(动态主题订阅) | No | Yes |
| Dynamic Topic Subscription<br/>(动态主题订阅) | No | Yes |
本文使用的Kafka版本为`kafka_2.12-2.2.0`，故采用第二种方式进行整合。
@ -48,7 +48,8 @@ Spark针对Kafka的不同版本，提供了两套整合方案：`spark-streaming
<artifactId>spark-streaming-kafka-0-10_${scala.version}</artifactId>
<version>2.4.3</version>
</dependency>
<!--本地测试时如果出现异常:Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 10582 则是因为paranamer版本问题，可以添加下面的依赖包进行解决-->
<!--本地测试时如果出现异常:Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 10582
则是因为paranamer版本问题，可以添加下面的依赖包进行解决-->
<dependency>
<groupId>com.thoughtworks.paranamer</groupId>
<artifactId>paranamer</artifactId>
@ -320,4 +321,4 @@ bin/kafka-console-producer.sh --broker-list hadoop001:9092 --topic spark-streami
## 参考资料
1. https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
1. https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html