Spark 2.1.1 Error:java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Spark 2.1.1 Error:java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;

zzcclp
This post has NOT been accepted by the mailing list yet.
Hi guys:
  I am using spark 2.1.1 to test on CDH 5.7.1, when i run on yarn with following command, error 'NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;' appears sometimes:

  command:
  su cloudera-scm -s "/bin/sh" -c "/opt/spark2/bin/spark-shell --master yarn --deploy-mode client --files /opt/spark2/conf/log4j_all.properties#log4j.properties --driver-memory 8g --num-executors 2 --executor-memory 8g --executor-cores 5 --driver-library-path :/opt/cloudera/parcels/CDH/lib/hadoop/lib/native --driver-class-path /opt/spark2/libs/mysql-connector-java-5.1.36.jar --jars /opt/spark2/libs/mysql-connector-java-5.1.36.jar "

  error messages:
  2017-07-17 17:15:25,255 - WARN - org.apache.spark.network.server.TransportChannelHandler.exceptionCaught(TransportChannelHandler.java:78) - rpc-client-1-1 -Exception in connection from /ip:60099
java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;
        at org.apache.spark.rpc.netty.NettyRpcHandler.channelActive(NettyRpcEnv.scala:614)
        at org.apache.spark.network.server.TransportRequestHandler.channelActive(TransportRequestHandler.java:87)
        at org.apache.spark.network.server.TransportChannelHandler.channelActive(TransportChannelHandler.java:88)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
        at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
        at org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelActive(IdleStateHandler.java:251)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
        at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
        at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
        at org.spark_project.io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1282)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
        at org.spark_project.io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:887)
        at org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:262)
        at org.spark_project.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:292)
        at org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:640)
        at org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
        at org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
        at org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
        at org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
        at org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:745)
2017-07-17 17:15:25,256 - DEBUG - org.spark_project.io.netty.util.internal.logging.Slf4JLogger.debug(Slf4JLogger.java:81) - rpc-client-1-1 -An exception java.lang.NoSuchMethodError: org.apache.spark.network.client.TransportClient.getChannel()Lio/netty/channel/Channel;
        at org.apache.spark.rpc.netty.NettyRpcHandler.exceptionCaught(NettyRpcEnv.scala:595)
        at org.apache.spark.network.server.TransportRequestHandler.exceptionCaught(TransportRequestHandler.java:82)
        at org.apache.spark.network.server.TransportChannelHandler.exceptionCaught(TransportChannelHandler.java:80)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:291)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:854)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:221)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
        at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
        at org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelActive(IdleStateHandler.java:251)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:205)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:198)
        at org.spark_project.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
        at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:219)

   my conf:
   spark.serializer                      org.apache.spark.serializer.KryoSerializer
spark.rdd.compress                    true

spark.default.parallelism               10

spark.rpc.netty.dispatcher.numThreads  3

spark.shuffle.manager                 tungsten-sort

spark.memory.useLegacyMode            false
spark.memory.fraction                 0.5
spark.memory.storageFraction          0.3
 
spark.memory.offHeap.enabled          false
spark.memory.offHeap.size             0

spark.yarn.executor.memoryOverhead    1024
spark.yarn.am.memoryOverhead          1024

spark.yarn.maxAppAttempts             6

spark.shuffle.file.buffer             32k

spark.locality.wait                     0s

spark.scheduler.minRegisteredResourcesRatio         1.0
spark.scheduler.maxRegisteredResourcesWaitingTime 60s

  Does anyone encounters this problems? Please help me, Thanks.

 
Loading...