How to Create one DB connection per executor and close it after the job is done?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

How to Create one DB connection per executor and close it after the job is done?

kant kodali
Hi All,

I understand creating a connection forEachPartition but I am wondering can I create one DB connection per executor and close it after the job is done? any sample code would help. you can imagine I am running a simple batch processing application.

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: How to Create one DB connection per executor and close it after the job is done?

patrickmcgloin
You could use an object in Scala, of which only one instance will be created on each JVM / Executor. E.g.

object MyDatabseSingleton {
    var dbConn = ???
}

On Sat, 28 Jul 2018, 08:34 kant kodali, <[hidden email]> wrote:
Hi All,

I understand creating a connection forEachPartition but I am wondering can I create one DB connection per executor and close it after the job is done? any sample code would help. you can imagine I am running a simple batch processing application.

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: How to Create one DB connection per executor and close it after the job is done?

kant kodali
Hi Patrick,

This object must be serializable right? I wonder if I will access to this object in my driver(since it is getting created on the executor side) so I can close when I am done with my batch?

Thanks!

On Mon, Jul 30, 2018 at 7:37 AM, Patrick McGloin <[hidden email]> wrote:
You could use an object in Scala, of which only one instance will be created on each JVM / Executor. E.g.

object MyDatabseSingleton {
    var dbConn = ???
}

On Sat, 28 Jul 2018, 08:34 kant kodali, <[hidden email]> wrote:
Hi All,

I understand creating a connection forEachPartition but I am wondering can I create one DB connection per executor and close it after the job is done? any sample code would help. you can imagine I am running a simple batch processing application.

Thanks!

Reply | Threaded
Open this post in threaded view
|

Re: How to Create one DB connection per executor and close it after the job is done?

Vadim Semenov-2
object MyDatabseSingleton {
    @transient
    lazy val dbConn = DB.connect(…)

`transient` marks the variable to be excluded from serialization

and `lazy` would open connection only when it's needed and also makes
sure that the val is thread-safe

http://fdahms.com/2015/10/14/scala-and-the-transient-lazy-val-pattern/
http://code-o-matic.blogspot.com/2009/05/double-checked-locking-idiom-sweet-in.html
On Mon, Jul 30, 2018 at 1:32 PM kant kodali <[hidden email]> wrote:

>
> Hi Patrick,
>
> This object must be serializable right? I wonder if I will access to this object in my driver(since it is getting created on the executor side) so I can close when I am done with my batch?
>
> Thanks!
>
> On Mon, Jul 30, 2018 at 7:37 AM, Patrick McGloin <[hidden email]> wrote:
>>
>> You could use an object in Scala, of which only one instance will be created on each JVM / Executor. E.g.
>>
>> object MyDatabseSingleton {
>>     var dbConn = ???
>> }
>>
>> On Sat, 28 Jul 2018, 08:34 kant kodali, <[hidden email]> wrote:
>>>
>>> Hi All,
>>>
>>> I understand creating a connection forEachPartition but I am wondering can I create one DB connection per executor and close it after the job is done? any sample code would help. you can imagine I am running a simple batch processing application.
>>>
>>> Thanks!
>
>


--
Sent from my iPhone

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]