help needed in perforance improvement of spark structured streaming

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

help needed in perforance improvement of spark structured streaming

amit kumar singh
Hi Community,

I have a use case where i need to call stored procedure through structured streaming.

I am able to send kafka message and call stored procedure ,

but since foreach sink keeps on executing stored procedure per message 

i want to combine all the messages in single dtaframe and then call  stored procedure at once

is it possible to do 


current code

select('value cast "string",'topic)
.select('topic,concat_ws(",", 'value cast "string") as 'value1)
.groupBy('topic cast "string").count()
.coalesce(1)
.as[String]
.writeStream
.trigger(ProcessingTime("60 seconds"))
.option("checkpointLocation", checkpointUrl)
.foreach(new SimpleSqlServerSink(jdbcUrl, connectionProperties))



thanks
rohit