Help Required - Unable to run spark-submit on YARN client mode

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Help Required - Unable to run spark-submit on YARN client mode

Debabrata Ghosh
Hi Everyone,
I have been trying to run spark-shell in YARN client mode, but am getting lot of ClosedChannelException errors, however the program works fine on local mode.  I am using spark 2.2.0 build for Hadoop 2.7.3.  If you are familiar with this error, please can you help with the possible resolution.

Any help would be greatly appreciated!

Here is the error message:

18/05/08 00:01:18 ERROR TransportClient: Failed to send RPC 7905321254854295784 to /9.30.94.43:60220: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
18/05/08 00:01:18 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(5,0,Map(),Set()) to AM was unsuccessful
java.io.IOException: Failed to send RPC 7905321254854295784 to /9.30.94.43:60220: java.nio.channels.ClosedChannelException
        at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
        at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
        at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException

Cheers,

Debu
Reply | Threaded
Open this post in threaded view
|

Re: Help Required - Unable to run spark-submit on YARN client mode

Deepak Sharma
Can you try increasing the partition for the base RDD/dataframe that you are working on?


On Tue, May 8, 2018 at 5:05 PM, Debabrata Ghosh <[hidden email]> wrote:
Hi Everyone,
I have been trying to run spark-shell in YARN client mode, but am getting lot of ClosedChannelException errors, however the program works fine on local mode.  I am using spark 2.2.0 build for Hadoop 2.7.3.  If you are familiar with this error, please can you help with the possible resolution.

Any help would be greatly appreciated!

Here is the error message:

18/05/08 00:01:18 ERROR TransportClient: Failed to send RPC 7905321254854295784 to /9.30.94.43:60220: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
18/05/08 00:01:18 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(5,0,Map(),Set()) to AM was unsuccessful
java.io.IOException: Failed to send RPC 7905321254854295784 to /9.30.94.43:60220: java.nio.channels.ClosedChannelException
        at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
        at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
        at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException

Cheers,

Debu



--