Getting spark to read kafka data



Setup kafka

bin/zookeeper-server-start.sh config/zookeeper.properties


bin/kafka-server-start.sh config/server.properties


Create a topic


bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor
1 --partitions 1 --topic test



To write something to your topic

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test



Working with spark

You can get the script from here.

To submit spark python code


spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.1
kafka.py localhost subscribe test

Comments

Popular posts from this blog

The specified initialization vector (IV) does not match the block size for this algorithm