How to bulk insert using spark streaming job

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to bulk insert using spark streaming job

amit kumar singh

How to bulk insert using spark streaming job 


Sent from my iPhone
Reply | Threaded
Open this post in threaded view
|

Re: How to bulk insert using spark streaming job

ayan guha
by writing code, I suppose :) Jokes apart, I think you need to articulate the problem with more details for others to help. 

Do you mean you want to batch up data in memory and then write as a chunk? Where do want to insert? Etc etc...

On Fri, Apr 20, 2018 at 1:08 PM, amit kumar singh <[hidden email]> wrote:

How to bulk insert using spark streaming job 


Sent from my iPhone



--
Best Regards,
Ayan Guha
Reply | Threaded
Open this post in threaded view
|

Re: How to bulk insert using spark streaming job

scorpio
In reply to this post by amit kumar singh
You need to insert per partition per batch. Normally database drivers meant
for spark have bulk update feature built in. They take a RDD and do a bulk
insert per partition.
In case db driver you are using doesn't provide this feature, you can
aggregate records per partition and then send out to db by writing your own
code.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]