Read Avro Data using Spark Streaming

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Read Avro Data using Spark Streaming

Divya Narayan
Hi,

I produced avro data to kafka topic using schema registry and now I want to use spark streaming to read that data and do some computation in real time. Can some one please give a sample code for doing that . I couldn't find any working code online. I am using spark version 2.2.0 and spark-streaming-kafka-0-10_2.11.

Thanks
Divya
Reply | Threaded
Open this post in threaded view
|

Re: Read Avro Data using Spark Streaming

chandan prakash
In case you are using old Spark Streaming and processing Avro data from kafka, this might be useful:
https://www.linkedin.com/pulse/avro-data-processing-spark-streaming-kafka-chandan-prakash/

Regards,
Chandan

On Sat, Nov 3, 2018 at 9:04 AM Divya Narayan <[hidden email]> wrote:
Hi,

I produced avro data to kafka topic using schema registry and now I want to use spark streaming to read that data and do some computation in real time. Can some one please give a sample code for doing that . I couldn't find any working code online. I am using spark version 2.2.0 and spark-streaming-kafka-0-10_2.11.

Thanks
Divya


--
Chandan Prakash

Reply | Threaded
Open this post in threaded view
|

Re: Read Avro Data using Spark Streaming

smikesh
In reply to this post by Divya Narayan
Hi, 

you can use this project in order to read Avro using Spark Structured Streaming: 

Spark 2.4 has also built in support for Avro, so you can use from_avro function in Spark 2.4. 

Best,
Michael


On Sat, Nov 3, 2018 at 4:34 AM Divya Narayan <[hidden email]> wrote:
Hi,

I produced avro data to kafka topic using schema registry and now I want to use spark streaming to read that data and do some computation in real time. Can some one please give a sample code for doing that . I couldn't find any working code online. I am using spark version 2.2.0 and spark-streaming-kafka-0-10_2.11.

Thanks
Divya