java.lang.OutOfMemoryError: Java heap space - Spark driver.

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

java.lang.OutOfMemoryError: Java heap space - Spark driver.

Guillermo Ortiz Fernández
I got this error from spark driver, it seems that I should increase the memory in the driver although it's 5g (and 4 cores) right now. It seems weird to me because I'm not using Kryo or broadcast in this process but in the log there are references to Kryo and broadcast.
How could I figure out the reason of this outOfMemory? Is it normal that there are references to Kryo and broadcasting when I'm not using it?

05:11:19.110 [streaming-job-executor-0] WARN  c.datastax.driver.core.CodecRegistry - Ignoring codec DateRangeCodec ['org.apache.cassandra.db.marshal.DateRangeType' <-> com.datastax.driver.dse.search.DateRange] because it collides with previously registered codec DateRangeCodec ['org.apache.cassandra.db.marshal.DateRangeType' <-> com.datastax.driver.dse.search.DateRange]
05:11:26.806 [dag-scheduler-event-loop] WARN  org.apache.spark.util.Utils - Suppressing exception in finally: Java heap space
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_162]
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_162]
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadcast.scala:231) ~[spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadcast.scala:231) ~[spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.util.io.ChunkedByteBufferOutputStream.allocateNewChunkIfNeeded(ChunkedByteBufferOutputStream.scala:87) ~[spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.util.io.ChunkedByteBufferOutputStream.write(ChunkedByteBufferOutputStream.scala:75) ~[spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205) ~[lz4-1.3.0.jar:na]
    at net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158) ~[lz4-1.3.0.jar:na]
    at com.esotericsoftware.kryo.io.Output.flush(Output.java:181) ~[kryo-3.0.3.jar:na]
    at com.esotericsoftware.kryo.io.Output.close(Output.java:191) ~[kryo-3.0.3.jar:na]
    at org.apache.spark.serializer.KryoSerializationStream.close(KryoSerializer.scala:209) ~[spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$blockifyObject$1.apply$mcV$sp(TorrentBroadcast.scala:238) ~[spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1319) ~[spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:237) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:107) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:86) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:56) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1387) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1012) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:933) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:936) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:935) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at scala.collection.immutable.List.foreach(List.scala:392) [scala-library-2.11.11.jar:na]
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:935) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:873) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1630) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1622) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1611) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) [spark-core_2.11-2.0.2.15.jar:2.0.2.15]
05:40:53.535 [dse-app-client-thread-pool-0] WARN  c.datastax.driver.core.CodecRegistry - Ignoring codec DateRangeCodec ['org.apache.cassandra.db.marshal.DateRangeType' <-> com.datastax.driver.dse.search.DateRange] because it collides with previously registered codec DateRangeCodec ['org.apache.cassandra.db.marshal.DateRangeType' <-> com.datastax.driver.dse.search.DateRange]
05:41:10.332 [dse-app-client-thread-pool-0] WARN  c.datastax.driver.core.CodecRegistry - Ignoring codec DateRangeCodec ['org.apache.cassandra.db.marshal.DateRangeType' <-> com.datastax.driver.dse.search.DateRange] because it collides with previously registered codec DateRangeCodec ['org.apache.cassandra.db.marshal.DateRangeType' <-> com.datastax.driver.dse.search.DateRange]
05:41:18.