Using flume to create stream for spark streaming.

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Using flume to create stream for spark streaming.

Ravi Hemnani

I am using the following flume flow,

Flume agent 1 consisting of Rabbitmq-> source, files-> channet, avro-> sink sending data to a slave node of spark cluster.
Flume agent 2, slave node of spark cluster, consisting of avro-> source, files-> channel, now for the sink i tried avro, hdfs, file_roll as sink but i am not able to read the DStream from any of these where for avro sink type, i am giving sink address as the same slave node and some other port and i am asking the spark streaming program to listen to slave node and the port of the sink defined in the conf of slave node. Thus spark streaming is giving me no result.

I am running the program as java -jar <jar>  on master of the cluster.

What should be the sink type that should be used on the slave node?
I have stuck on this since two weeks now and i am confused how to approach this.

Any help?