issue Running Spark Job on Yarn Cluster

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

issue Running Spark Job on Yarn Cluster

sachin Singh
This post has NOT been accepted by the mailing list yet.
Hi,
I want to run my spark Job in Hadoop yarn Cluster mode,
I am using below command -
spark-submit --master yarn-cluster --driver-memory 1g --executor-memory 1g
--executor-cores 1 --class com.dc.analysis.jobs.AggregationJob
sparkanalitic.jar param1 param2 param3
I am getting error as under, kindly suggest whats going wrong ,is command is
proper or not ,thanks in advance,

 diagnostics: Application application_1424284032717_0066 failed 2 times due to AM Container for appattempt_1424284032717_0066_000002 exited with  exitCode: 15 due to: Exception from container-launch.
Container id: container_1424284032717_0066_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 15
.Failing this attempt.. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: root.hdfs
         start time: 1424699723648
         final status: FAILED
         tracking URL: http://myhostname:8088/cluster/app/application_1424284032717_0066
         user: hdfs
2015-02-23 19:26:04 DEBUG Client - stopping client from cache: org.apache.hadoop.ipc.Client@4085f1ac
2015-02-23 19:26:04 DEBUG Utils - Shutdown hook called
2015-02-23 19:26:05 DEBUG Utils - Shutdown hook called
Reply | Threaded
Open this post in threaded view
|

Re: issue Running Spark Job on Yarn Cluster

sachin Singh
This post has NOT been accepted by the mailing list yet.
I am using CDH5.3.1
Reply | Threaded
Open this post in threaded view
|

Re: issue Running Spark Job on Yarn Cluster

avilevi3
This post has NOT been accepted by the mailing list yet.
you should fetch the complete logs for the application using 'yarn logs' command, like so:

yarn logs -applicationId [the application's id]

and look for the real error info
Reply | Threaded
Open this post in threaded view
|

Re: issue Running Spark Job on Yarn Cluster

poolis
This post has NOT been accepted by the mailing list yet.
In reply to this post by sachin Singh
Did you resolve this issue?
Reply | Threaded
Open this post in threaded view
|

Re: issue Running Spark Job on Yarn Cluster

oubrik
This post has NOT been accepted by the mailing list yet.
Hi
try --master yarn-client

Regards
Reply | Threaded
Open this post in threaded view
|

Re: issue Running Spark Job on Yarn Cluster

MooseSpark
This post has NOT been accepted by the mailing list yet.
In reply to this post by sachin Singh
Please check logs in your hadoop yarn cluster, there you would get precise error or exception.
Reply | Threaded
Open this post in threaded view
|

Re: issue Running Spark Job on Yarn Cluster

Mandar Vaidya
This post has NOT been accepted by the mailing list yet.
Hi,
I am facing same issue.I am using HDP 2.3.0 on which running spark 1.3.1.
I try to set the HADOOP_CONF_DIR but getting same error.

Error :

15/08/19 16:40:53 INFO Client:
         client token: N/A
         diagnostics: Application application_1439464132837_0015 failed 2 times due to AM Container for appattempt_1439464132837_0015_000002 exited with  exitCode: 15
For more detailed output, check application tracking page:http://node1.xxxx.com:8088/cluster/app/application_1439464132837_0015Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e04_1439464132837_0015_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
        at org.apache.hadoop.util.Shell.run(Shell.java:456)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 15
Failing this attempt. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1439982582802
         final status: FAILED
         tracking URL: http://node1.xxxx.com:8088/cluster/app/application_1439464132837_0015
         user: hdfs
Error: application failed with exception
org.apache.spark.SparkException: Application finished with failed status
        at org.apache.spark.deploy.yarn.Client.run(Client.scala:622)
        at org.apache.spark.deploy.yarn.Client$.main(Client.scala:647)
        at org.apache.spark.deploy.yarn.Client.main(Client.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:577)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:174)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:197)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

kindly suggest some input.

Regards,
Mandar Vaidya.
Reply | Threaded
Open this post in threaded view
|

Re: issue Running Spark Job on Yarn Cluster

stark_summer
This post has NOT been accepted by the mailing list yet.
In reply to this post by sachin Singh
Please  look at more about  hadoop logs, such as  yarn logs -applicationId xxx
attach more logs to this topic
Reply | Threaded
Open this post in threaded view
|

Re: issue Running Spark Job on Yarn Cluster

sivasonai
Come across such issue in our project and got it resolved by clearing the space under hdfs directory - "/user/spark". Please check if you have enough space/privileges for this hdfs directory - "/user/spark"

Sent from the Apache Spark User List mailing list archive at Nabble.com.