Broadcast variable size limit?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Broadcast variable size limit?

klrmowse
is it currently still ~2GB (Integer.MAX_VALUE) ??

or am i misinformed, since that's what google-search and scouring this
mailing list seem to say... ?



Thanks



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Broadcast variable size limit?

Jörn Franke
I think if you need more then you should anyway think about something different than broadcast variable ...

> On 5. Aug 2018, at 16:51, klrmowse <[hidden email]> wrote:
>
> is it currently still ~2GB (Integer.MAX_VALUE) ??
>
> or am i misinformed, since that's what google-search and scouring this
> mailing list seem to say... ?
>
>
>
> Thanks
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Broadcast variable size limit?

klrmowse
i don't need more, per se... i just need to watch the size of the variable;
then, if it's within the size limit, go ahead and broadcast it; if not, then
i won't broadcast...

so, that would be a yes then? (2 GB, or which is it exactly?)



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Broadcast variable size limit?

Vadim Semenov-2
That’s the max size of a byte array in Java, limited by the length which is defined as integer, and in most JVMS arrays can’t hold more than Int.MaxValue - 8 elements. Other way to overcome this is to create multiple broadcast variables 

On Sunday, August 5, 2018, klrmowse <[hidden email]> wrote:
i don't need more, per se... i just need to watch the size of the variable;
then, if it's within the size limit, go ahead and broadcast it; if not, then
i won't broadcast...

so, that would be a yes then? (2 GB, or which is it exactly?)



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Sent from my iPhone