Spark streaming with multiple Kafka topics

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark streaming with multiple Kafka topics

lalitha bandaru
Hi Team,

I have a spark streaming application configured to consume events from 2 Kafka topics. But when I run the application locally, the messages are consumed from either of these topics only and not both. If the first event is published to say topic2 and second message to topic1 then only the messages from topic2 are consumed.
But when I deployed the same code to an EMR cluster then the messages from both topics are being consumed.
Can someone please help what configuration could be missing when run locally? Thanks much in advance!

Thanks,
Lalitha
--
Thanks,
Lalitha B
Mobile - +1 8046290486