How Kryo serializer allocates buffer in Spark

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
nir
Reply | Threaded
Open this post in threaded view
|

How Kryo serializer allocates buffer in Spark

nir
I am getting following error in spark task. Default max value is 64mb! Document says it should be large enough to store largest object in my application. I don't think I have any object thhhat is bigger then 64mb. SO what these values (spark.kryoserializer.buffer, spark.kryoserializer.buffer.max) means?

Is that a buffer per executor or buffer per executor per core ? I have 6 cores per executors so do all 6 are writing to this common buffer?   in that case I have 16mb buffer per core. Please explain. Thanks!


Job aborted due to stage failure: Task 3 in stage 4.0 failed 10 times, most recent failure: Lost task 3.9 in stage 4.0 (TID 16, iadprd01mpr005.mgmt-a.xactly.iad.dc.local): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 19. To avoid this, increase spark.kryoserializer.buffer.max value.
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:300)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:313)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)