How to write dataframe to kafka topic in spark streaming application using pyspark other than collect?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to write dataframe to kafka topic in spark streaming application using pyspark other than collect?

umargeek
This post was updated on .
Can anyone provide me code snippet/ steps to write a data frame to Kafka topic in a spark streaming application using pyspark with spark 2.2 and Kafka 0.8 using Direct Stream Approach other than below method of using collect after converting my DF to RDD?

for row in RDD.collect():

    producer.send(topicname, str(row).encode('utf-8'))

producer.flush()

Thanks,
Umar



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Reply | Threaded
Open this post in threaded view
|

Re: How to write dataframe to kafka topic in spark streaming application using pyspark other than collect?

umargeek
Hi Team,

Can someone please advise me on the above post since because of this I have
written data file to HDFS location.
So as of now am just passing the filename into Kafka topic and not utilizing
Kafka potential at the best looking forward to suggestions.

Thanks,
Umar



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]