sparksql 2.4.0 java.lang.NoClassDefFoundError: com/esotericsoftware/minlog/Log

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

sparksql 2.4.0 java.lang.NoClassDefFoundError: com/esotericsoftware/minlog/Log

Ivan Petrov
Hi there!
I'm seeing this exception in Spark Driver log.
Executor log stays empty. No exceptions, nothing.
8 tasks out of 402 failed with this exception.
What is the right way to debug it? 
Thank you.

java.lang.NoClassDefFoundError: com/esotericsoftware/minlog/Log
at com.esotericsoftware.kryo.serializers.FieldSerializerConfig.<init>(FieldSerializerConfig.java:47)
at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:154)
at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:161)
at com.twitter.chill.KryoBase.<init>(KryoBase.scala:34)
at com.twitter.chill.EmptyScalaKryoInstantiator.newKryo(ScalaKryoInstantiator.scala:36)
at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:97)
at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:324)
at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:309)
at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:218)
at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:305)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$apply$2.apply(TorrentBroadcast.scala:235)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:211)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1371)
at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:207)
at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:66)
at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:66)
at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:96)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:89)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Reply | Threaded
Open this post in threaded view
|

Re: sparksql 2.4.0 java.lang.NoClassDefFoundError: com/esotericsoftware/minlog/Log

Ivan Petrov
spark/jars -> minlog-1.3.0.jar
I see that jar is there. What do I do wrong?

чт, 9 июл. 2020 г. в 20:43, Ivan Petrov <[hidden email]>:
Hi there!
I'm seeing this exception in Spark Driver log.
Executor log stays empty. No exceptions, nothing.
8 tasks out of 402 failed with this exception.
What is the right way to debug it? 
Thank you.

java.lang.NoClassDefFoundError: com/esotericsoftware/minlog/Log
at com.esotericsoftware.kryo.serializers.FieldSerializerConfig.<init>(FieldSerializerConfig.java:47)
at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:154)
at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:161)
at com.twitter.chill.KryoBase.<init>(KryoBase.scala:34)
at com.twitter.chill.EmptyScalaKryoInstantiator.newKryo(ScalaKryoInstantiator.scala:36)
at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:97)
at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:324)
at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:309)
at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:218)
at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:305)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$apply$2.apply(TorrentBroadcast.scala:235)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:211)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1371)
at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:207)
at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:66)
at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:66)
at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:96)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:89)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)