[Structured Streaming] [Kafka] How to repartition the data and distribute the processing among worker nodes

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[Structured Streaming] [Kafka] How to repartition the data and distribute the processing among worker nodes

karthikjay
Any help appreciated. please find the question in the link:

https://stackoverflow.com/questions/49951022/spark-structured-streaming-with-kafka-how-to-repartition-the-data-and-distribu




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Structured Streaming] [Kafka] How to repartition the data and distribute the processing among worker nodes

Bowden, Chris
The primary role of a sink is storing output tuples. Consider groupByKey and map/flatMapGroupsWithState instead.

-Chris
From: karthikjay <[hidden email]>
Sent: Friday, April 20, 2018 4:49:49 PM
To: [hidden email]
Subject: [Structured Streaming] [Kafka] How to repartition the data and distribute the processing among worker nodes
 
Any help appreciated. please find the question in the link:

https://stackoverflow.com/questions/49951022/spark-structured-streaming-with-kafka-how-to-repartition-the-data-and-distribu




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]