Problems with broadcast large datastructure

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Problems with broadcast large datastructure – Spark repeatedly fails broadcast a large object on a cluster of 25 machines for me. I get log messages like this: [spark-akka.actor.default...
If your object size > 10MB you may need to change spark.akka.frameSize. What is your spark, spark.akka.timeOut ? did you change spark.a...
I have occurred the same problem with you . I have a node of 20 machines, and I just run the broadcast example, what I do is just change the dat...
On Mon, Jan 13, 2014 at 4:17 AM, lihu <lihu723@...> wrote: > I have occurred the same problem with you . > I have a node of 20 mac...
In my opinion, the spark system is for big data, then 400M seem not big . I read slides about the broadcast, in my understanding, the executor ...
broadcast is supposed to send data from the driver to the executors and not the other direction. can you share the code snippet you are using to ...
Yes, I just using the code snippet from the broadcast example, and using the spark-shell run this code. I thought the broadcast is driver send t...
Size calculation is correct, but broadcast happens from the driver to the workers. btw, your code is broadcasting 400MB 30 times, which are no...
Oh, I misleading by the following log info, that I thought the broadcast variable is send back to driver. then the sending result to driver has ...
400MB isn't really that big. Broadcast is expected to work with several GB of data and in even larger clusters (100s of machines). if you are ...
What's the size of your large object to be broadcast? On Tue, Jan 7, 2014 at 8:55 AM, Sebastian Schelter <ssc@...> wrote: > Spark...
Loading...