How to run spark shell using YARN

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

How to run spark shell using YARN

kant kodali
Hi All,

I am trying to use YARN for the very first time. I believe I configured all the resource manager and name node fine. And then I run the below command

./spark-shell --master yarn --deploy-mode client

I get the below output and it hangs there forever (I had been waiting over 10 minutes)

18/03/12 23:36:32 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.

Any idea?

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

Marcelo Vanzin
That's not an error, just a warning. The docs [1] have more info about
the config options mentioned in that message.

[1] http://spark.apache.org/docs/latest/running-on-yarn.html

On Mon, Mar 12, 2018 at 4:42 PM, kant kodali <[hidden email]> wrote:

> Hi All,
>
> I am trying to use YARN for the very first time. I believe I configured all
> the resource manager and name node fine. And then I run the below command
>
> ./spark-shell --master yarn --deploy-mode client
>
> I get the below output and it hangs there forever (I had been waiting over
> 10 minutes)
>
> 18/03/12 23:36:32 WARN Client: Neither spark.yarn.jars nor
> spark.yarn.archive is set, falling back to uploading libraries under
> SPARK_HOME.
>
> Any idea?
>
> Thanks!



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

kant kodali
Hi,

I read that doc several times now. I am stuck with the below error message when I run ./spark-shell --master yarn --deploy-mode client.

I have my HADOOP_CONF_DIR set to /usr/local/hadoop-2.7.3/etc/hadoop and SPARK_HOME set to /usr/local/spark on all 3 machines (1 node for Resource Manager and NameNode, 2 Nodes for Node Manager and DataNodes). 

Any idea?



18/03/13 00:19:13 INFO LineBufferedStream: stdout: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/centos/.sparkStaging/application_1520898664848_0003/__spark_libs__2434167523839846774.zip could only be replicated to 0 nodes instead of minReplication (=1).  There are 2 datanode(s) running and no node(s) are excluded in this operation.


18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at java.security.AccessController.doPrivileged(Native Method)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at javax.security.auth.Subject.doAs(Subject.java:422)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
18/03/13


Thanks! 


On Mon, Mar 12, 2018 at 4:46 PM, Marcelo Vanzin <[hidden email]> wrote:
That's not an error, just a warning. The docs [1] have more info about
the config options mentioned in that message.

[1] http://spark.apache.org/docs/latest/running-on-yarn.html

On Mon, Mar 12, 2018 at 4:42 PM, kant kodali <[hidden email]> wrote:
> Hi All,
>
> I am trying to use YARN for the very first time. I believe I configured all
> the resource manager and name node fine. And then I run the below command
>
> ./spark-shell --master yarn --deploy-mode client
>
> I get the below output and it hangs there forever (I had been waiting over
> 10 minutes)
>
> 18/03/12 23:36:32 WARN Client: Neither spark.yarn.jars nor
> spark.yarn.archive is set, falling back to uploading libraries under
> SPARK_HOME.
>
> Any idea?
>
> Thanks!



--
Marcelo

Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

Marcelo Vanzin
Looks like you either have a misconfigured HDFS service, or you're
using the wrong configuration on the client.

BTW, as I said in the previous response, the message you saw initially
is *not* an error. If you're just trying things out, you don't need to
do anything and Spark should still work.

On Mon, Mar 12, 2018 at 6:13 PM, kant kodali <[hidden email]> wrote:

> Hi,
>
> I read that doc several times now. I am stuck with the below error message
> when I run ./spark-shell --master yarn --deploy-mode client.
>
> I have my HADOOP_CONF_DIR set to /usr/local/hadoop-2.7.3/etc/hadoop and
> SPARK_HOME set to /usr/local/spark on all 3 machines (1 node for Resource
> Manager and NameNode, 2 Nodes for Node Manager and DataNodes).
>
> Any idea?
>
>
>
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /user/centos/.sparkStaging/application_1520898664848_0003/__spark_libs__2434167523839846774.zip
> could only be replicated to 0 nodes instead of minReplication (=1).  There
> are 2 datanode(s) running and no node(s) are excluded in this operation.
>
>
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> java.security.AccessController.doPrivileged(Native Method)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> javax.security.auth.Subject.doAs(Subject.java:422)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> 18/03/13
>
>
> Thanks!
>
>
> On Mon, Mar 12, 2018 at 4:46 PM, Marcelo Vanzin <[hidden email]> wrote:
>>
>> That's not an error, just a warning. The docs [1] have more info about
>> the config options mentioned in that message.
>>
>> [1] http://spark.apache.org/docs/latest/running-on-yarn.html
>>
>> On Mon, Mar 12, 2018 at 4:42 PM, kant kodali <[hidden email]> wrote:
>> > Hi All,
>> >
>> > I am trying to use YARN for the very first time. I believe I configured
>> > all
>> > the resource manager and name node fine. And then I run the below
>> > command
>> >
>> > ./spark-shell --master yarn --deploy-mode client
>> >
>> > I get the below output and it hangs there forever (I had been waiting
>> > over
>> > 10 minutes)
>> >
>> > 18/03/12 23:36:32 WARN Client: Neither spark.yarn.jars nor
>> > spark.yarn.archive is set, falling back to uploading libraries under
>> > SPARK_HOME.
>> >
>> > Any idea?
>> >
>> > Thanks!
>>
>>
>>
>> --
>> Marcelo
>
>



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

vermanurag
In reply to this post by kant kodali
This does not look like Spark error. Looks like yarn has not been able to
allocate resources for spark driver. If you check resource manager UI you
are likely to see this as spark application waiting for resources. Try
reducing the driver node memory and/ or other bottlenecks based on what you
see in the resource manager UI.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

kant kodali
I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website and these are the only three files I changed Do I need to set or change anything in mapred-site.xml (As of now I have not touched mapred-site.xml)? 

when I do yarn -node -list -all I can see both node manager and resource managers are running fine.

But when I run spark-shell --master yarn --deploy-mode client

 
it just keeps looping forever and never stops with the following messages

18/03/14 07:07:47 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:48 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:49 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:50 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:51 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:52 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

when I go to RM UI I see this 

ACCEPTED: waiting for AM container to be allocated, launched and register with RM.




On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <[hidden email]> wrote:
This does not look like Spark error. Looks like yarn has not been able to
allocate resources for spark driver. If you check resource manager UI you
are likely to see this as spark application waiting for resources. Try
reducing the driver node memory and/ or other bottlenecks based on what you
see in the resource manager UI.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

kant kodali
any idea?

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:
I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website and these are the only three files I changed Do I need to set or change anything in mapred-site.xml (As of now I have not touched mapred-site.xml)? 

when I do yarn -node -list -all I can see both node manager and resource managers are running fine.

But when I run spark-shell --master yarn --deploy-mode client

 
it just keeps looping forever and never stops with the following messages

18/03/14 07:07:47 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:48 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:49 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:50 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:51 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:52 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

when I go to RM UI I see this 

ACCEPTED: waiting for AM container to be allocated, launched and register with RM.




On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <[hidden email]> wrote:
This does not look like Spark error. Looks like yarn has not been able to
allocate resources for spark driver. If you check resource manager UI you
are likely to see this as spark application waiting for resources. Try
reducing the driver node memory and/ or other bottlenecks based on what you
see in the resource manager UI.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

kant kodali
I am using spark 2.3.0 and hadoop 2.7.3. 

Also I have done the following and restarted all. But I still see ACCEPTED: waiting for AM container to be allocated, launched and register with RM. And i am unable to spawn spark-shell.

editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the following property value from 0.1 to something higher. I changed to 0.5 (50%)

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.5</value>
    <description>
        Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications.
    </description>
</property>

You may have to allocate more memory to YARN by editing yarn-site.xml by updating the following property:

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>8192</value>
</property>



On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:
any idea?

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:
I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website and these are the only three files I changed Do I need to set or change anything in mapred-site.xml (As of now I have not touched mapred-site.xml)? 

when I do yarn -node -list -all I can see both node manager and resource managers are running fine.

But when I run spark-shell --master yarn --deploy-mode client

 
it just keeps looping forever and never stops with the following messages

18/03/14 07:07:47 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:48 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:49 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:50 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:51 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:52 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

when I go to RM UI I see this 

ACCEPTED: waiting for AM container to be allocated, launched and register with RM.




On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <[hidden email]> wrote:
This does not look like Spark error. Looks like yarn has not been able to
allocate resources for spark driver. If you check resource manager UI you
are likely to see this as spark application waiting for resources. Try
reducing the driver node memory and/ or other bottlenecks based on what you
see in the resource manager UI.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]




Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

femibyte
Make sure you have enough memory allocated for Spark workers, try specifying executor memory as follows:

--executor-memory <memory>

to spark-submit.

On Wed, Mar 14, 2018 at 3:25 AM, kant kodali <[hidden email]> wrote:
I am using spark 2.3.0 and hadoop 2.7.3. 

Also I have done the following and restarted all. But I still see ACCEPTED: waiting for AM container to be allocated, launched and register with RM. And i am unable to spawn spark-shell.

editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the following property value from 0.1 to something higher. I changed to 0.5 (50%)

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.5</value>
    <description>
        Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications.
    </description>
</property>

You may have to allocate more memory to YARN by editing yarn-site.xml by updating the following property:

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>8192</value>
</property>



On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:
any idea?

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:
I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website and these are the only three files I changed Do I need to set or change anything in mapred-site.xml (As of now I have not touched mapred-site.xml)? 

when I do yarn -node -list -all I can see both node manager and resource managers are running fine.

But when I run spark-shell --master yarn --deploy-mode client

 
it just keeps looping forever and never stops with the following messages

18/03/14 07:07:47 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:48 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:49 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:50 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:51 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:52 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

when I go to RM UI I see this 

ACCEPTED: waiting for AM container to be allocated, launched and register with RM.




On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <[hidden email]> wrote:
This does not look like Spark error. Looks like yarn has not been able to
allocate resources for spark driver. If you check resource manager UI you
are likely to see this as spark application waiting for resources. Try
reducing the driver node memory and/ or other bottlenecks based on what you
see in the resource manager UI.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]







--
"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.
Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

kant kodali
Tried this

 ./spark-shell --master yarn --deploy-mode client --executor-memory 4g


Same issue. Keeps going forever..


18/03/14 09:31:25 INFO Client: 

client token: N/A

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: default

start time: 1521019884656

final status: UNDEFINED

tracking URL: http://ip-172-31-0-54:8088/proxy/application_1521014458020_0004/

user: centos


18/03/14 09:30:08 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:09 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:10 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:11 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:12 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:13 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:14 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:15 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)


On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony <[hidden email]> wrote:
Make sure you have enough memory allocated for Spark workers, try specifying executor memory as follows:

--executor-memory <memory>

to spark-submit.

On Wed, Mar 14, 2018 at 3:25 AM, kant kodali <[hidden email]> wrote:
I am using spark 2.3.0 and hadoop 2.7.3. 

Also I have done the following and restarted all. But I still see ACCEPTED: waiting for AM container to be allocated, launched and register with RM. And i am unable to spawn spark-shell.

editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the following property value from 0.1 to something higher. I changed to 0.5 (50%)

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.5</value>
    <description>
        Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications.
    </description>
</property>

You may have to allocate more memory to YARN by editing yarn-site.xml by updating the following property:

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>8192</value>
</property>



On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:
any idea?

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:
I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website and these are the only three files I changed Do I need to set or change anything in mapred-site.xml (As of now I have not touched mapred-site.xml)? 

when I do yarn -node -list -all I can see both node manager and resource managers are running fine.

But when I run spark-shell --master yarn --deploy-mode client

 
it just keeps looping forever and never stops with the following messages

18/03/14 07:07:47 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:48 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:49 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:50 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:51 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:52 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

when I go to RM UI I see this 

ACCEPTED: waiting for AM container to be allocated, launched and register with RM.




On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <[hidden email]> wrote:
This does not look like Spark error. Looks like yarn has not been able to
allocate resources for spark driver. If you check resource manager UI you
are likely to see this as spark application waiting for resources. Try
reducing the driver node memory and/ or other bottlenecks based on what you
see in the resource manager UI.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]







--
"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.

Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

femibyte
What's the hardware configuration of the box you're running on i.e. how much memory does it have ?

Femi

On Wed, Mar 14, 2018 at 5:32 AM, kant kodali <[hidden email]> wrote:
Tried this

 ./spark-shell --master yarn --deploy-mode client --executor-memory 4g


Same issue. Keeps going forever..


18/03/14 09:31:25 INFO Client: 

client token: N/A

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: default

start time: 1521019884656

final status: UNDEFINED

tracking URL: http://ip-172-31-0-54:8088/proxy/application_1521014458020_0004/

user: centos


18/03/14 09:30:08 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:09 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:10 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:11 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:12 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:13 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:14 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:15 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)


On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony <[hidden email]> wrote:
Make sure you have enough memory allocated for Spark workers, try specifying executor memory as follows:

--executor-memory <memory>

to spark-submit.

On Wed, Mar 14, 2018 at 3:25 AM, kant kodali <[hidden email]> wrote:
I am using spark 2.3.0 and hadoop 2.7.3. 

Also I have done the following and restarted all. But I still see ACCEPTED: waiting for AM container to be allocated, launched and register with RM. And i am unable to spawn spark-shell.

editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the following property value from 0.1 to something higher. I changed to 0.5 (50%)

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.5</value>
    <description>
        Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications.
    </description>
</property>

You may have to allocate more memory to YARN by editing yarn-site.xml by updating the following property:

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>8192</value>
</property>



On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:
any idea?

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:
I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website and these are the only three files I changed Do I need to set or change anything in mapred-site.xml (As of now I have not touched mapred-site.xml)? 

when I do yarn -node -list -all I can see both node manager and resource managers are running fine.

But when I run spark-shell --master yarn --deploy-mode client

 
it just keeps looping forever and never stops with the following messages

18/03/14 07:07:47 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:48 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:49 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:50 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:51 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:52 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

when I go to RM UI I see this 

ACCEPTED: waiting for AM container to be allocated, launched and register with RM.




On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <[hidden email]> wrote:
This does not look like Spark error. Looks like yarn has not been able to
allocate resources for spark driver. If you check resource manager UI you
are likely to see this as spark application waiting for resources. Try
reducing the driver node memory and/ or other bottlenecks based on what you
see in the resource manager UI.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]







--
"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.




--
"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.
Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

kant kodali
16GB RAM.  AWS m4.xlarge. It's a three node cluster and I only have YARN and  HDFS running. Resources are barely used however I believe there is something in my config that is preventing YARN to see that I have good amount of resources I think (thats my guess I never worked with YARN before). My mapred-site.xml is empty. Do I even need this? if so, what should I set it to?

On Wed, Mar 14, 2018 at 2:46 AM, Femi Anthony <[hidden email]> wrote:
What's the hardware configuration of the box you're running on i.e. how much memory does it have ?

Femi

On Wed, Mar 14, 2018 at 5:32 AM, kant kodali <[hidden email]> wrote:
Tried this

 ./spark-shell --master yarn --deploy-mode client --executor-memory 4g


Same issue. Keeps going forever..


18/03/14 09:31:25 INFO Client: 

client token: N/A

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: default

start time: 1521019884656

final status: UNDEFINED

tracking URL: http://ip-172-31-0-54:8088/proxy/application_1521014458020_0004/

user: centos


18/03/14 09:30:08 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:09 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:10 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:11 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:12 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:13 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:14 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:15 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)


On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony <[hidden email]> wrote:
Make sure you have enough memory allocated for Spark workers, try specifying executor memory as follows:

--executor-memory <memory>

to spark-submit.

On Wed, Mar 14, 2018 at 3:25 AM, kant kodali <[hidden email]> wrote:
I am using spark 2.3.0 and hadoop 2.7.3. 

Also I have done the following and restarted all. But I still see ACCEPTED: waiting for AM container to be allocated, launched and register with RM. And i am unable to spawn spark-shell.

editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the following property value from 0.1 to something higher. I changed to 0.5 (50%)

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.5</value>
    <description>
        Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications.
    </description>
</property>

You may have to allocate more memory to YARN by editing yarn-site.xml by updating the following property:

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>8192</value>
</property>



On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:
any idea?

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:
I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website and these are the only three files I changed Do I need to set or change anything in mapred-site.xml (As of now I have not touched mapred-site.xml)? 

when I do yarn -node -list -all I can see both node manager and resource managers are running fine.

But when I run spark-shell --master yarn --deploy-mode client

 
it just keeps looping forever and never stops with the following messages

18/03/14 07:07:47 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:48 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:49 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:50 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:51 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:52 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

when I go to RM UI I see this 

ACCEPTED: waiting for AM container to be allocated, launched and register with RM.




On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <[hidden email]> wrote:
This does not look like Spark error. Looks like yarn has not been able to
allocate resources for spark driver. If you check resource manager UI you
are likely to see this as spark application waiting for resources. Try
reducing the driver node memory and/ or other bottlenecks based on what you
see in the resource manager UI.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]







--
"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.




--
"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.

Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

Anthony, Olufemi

After you updated your yarn-site.xml  file, did you restart the YARN resource manager ?

 

https://aws.amazon.com/premiumsupport/knowledge-center/restart-service-emr/

 

Femi

 

From: kant kodali <[hidden email]>
Date: Wednesday, March 14, 2018 at 6:16 AM
To: Femi Anthony <[hidden email]>
Cc: vermanurag <[hidden email]>, "user @spark" <[hidden email]>
Subject: Re: How to run spark shell using YARN

 

 

On Wed, Mar 14, 2018 at 2:46 AM, Femi Anthony <[hidden email]> wrote:

What's the hardware configuration of the box you're running on i.e. how much memory does it have ?

 

Femi

 

On Wed, Mar 14, 2018 at 5:32 AM, kant kodali <[hidden email]> wrote:

Tried this

 

 ./spark-shell --master yarn --deploy-mode client --executor-memory 4g



Same issue. Keeps going forever..



18/03/14 09:31:25 INFO Client: 

client token: N/A

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: default

start time: 1521019884656

final status: UNDEFINED

tracking URL: http://ip-172-31-0-54:8088/proxy/application_1521014458020_0004/

user: centos



18/03/14 09:30:08 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:09 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:10 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:11 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:12 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:13 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:14 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:15 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

 

On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony <[hidden email]> wrote:

Make sure you have enough memory allocated for Spark workers, try specifying executor memory as follows:

--executor-memory <memory>

to spark-submit.

 

On Wed, Mar 14, 2018 at 3:25 AM, kant kodali <[hidden email]> wrote:

I am using spark 2.3.0 and hadoop 2.7.3. 

 

Also I have done the following and restarted all. But I still see ACCEPTED: waiting for AM container to be allocated, launched and register with RM. And i am unable to spawn spark-shell.

 

editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the following property value from 0.1 to something higher. I changed to 0.5 (50%)

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.5</value>
    <description>
        Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications.
    </description>
</property>

You may have to allocate more memory to YARN by editing yarn-site.xml by updating the following property:

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>8192</value>
</property>

 

 

 

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:

any idea?

 

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:

I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website and these are the only three files I changed Do I need to set or change anything in mapred-site.xml (As of now I have not touched mapred-site.xml)? 

 

when I do yarn -node -list -all I can see both node manager and resource managers are running fine.

 

But when I run spark-shell --master yarn --deploy-mode client

 

 

it just keeps looping forever and never stops with the following messages

 

18/03/14 07:07:47 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:48 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:49 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:50 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:51 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:52 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

 

when I go to RM UI I see this 

 

ACCEPTED: waiting for AM container to be allocated, launched and register with RM.

 

 

 

 

On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <[hidden email]> wrote:

This does not look like Spark error. Looks like yarn has not been able to
allocate resources for spark driver. If you check resource manager UI you
are likely to see this as spark application waiting for resources. Try
reducing the driver node memory and/ or other bottlenecks based on what you
see in the resource manager UI.



--
Sent from:
http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail:
[hidden email]

 

 

 



 

--

"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.

 



 

--

"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.

 



The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

kant kodali
Hi, 

I am not using emr. And yes I restarted several times.

On Wed, Mar 14, 2018 at 6:35 AM, Anthony, Olufemi <[hidden email]> wrote:

After you updated your yarn-site.xml  file, did you restart the YARN resource manager ?

 

https://aws.amazon.com/premiumsupport/knowledge-center/restart-service-emr/

 

Femi

 

From: kant kodali <[hidden email]>
Date: Wednesday, March 14, 2018 at 6:16 AM
To: Femi Anthony <[hidden email]>
Cc: vermanurag <[hidden email]>, "user @spark" <[hidden email]>
Subject: Re: How to run spark shell using YARN

 

 

On Wed, Mar 14, 2018 at 2:46 AM, Femi Anthony <[hidden email]> wrote:

What's the hardware configuration of the box you're running on i.e. how much memory does it have ?

 

Femi

 

On Wed, Mar 14, 2018 at 5:32 AM, kant kodali <[hidden email]> wrote:

Tried this

 

 ./spark-shell --master yarn --deploy-mode client --executor-memory 4g



Same issue. Keeps going forever..



18/03/14 09:31:25 INFO Client: 

client token: N/A

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: default

start time: 1521019884656

final status: UNDEFINED

tracking URL: http://ip-172-31-0-54:8088/proxy/application_1521014458020_0004/

user: centos



18/03/14 09:30:08 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:09 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:10 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:11 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:12 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:13 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:14 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:15 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

 

On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony <[hidden email]> wrote:

Make sure you have enough memory allocated for Spark workers, try specifying executor memory as follows:

--executor-memory <memory>

to spark-submit.

 

On Wed, Mar 14, 2018 at 3:25 AM, kant kodali <[hidden email]> wrote:

I am using spark 2.3.0 and hadoop 2.7.3. 

 

Also I have done the following and restarted all. But I still see ACCEPTED: waiting for AM container to be allocated, launched and register with RM. And i am unable to spawn spark-shell.

 

editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the following property value from 0.1 to something higher. I changed to 0.5 (50%)

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.5</value>
    <description>
        Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications.
    </description>
</property>

You may have to allocate more memory to YARN by editing yarn-site.xml by updating the following property:

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>8192</value>
</property>

 

 

 

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:

any idea?

 

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:

I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website and these are the only three files I changed Do I need to set or change anything in mapred-site.xml (As of now I have not touched mapred-site.xml)? 

 

when I do yarn -node -list -all I can see both node manager and resource managers are running fine.

 

But when I run spark-shell --master yarn --deploy-mode client

 

 

it just keeps looping forever and never stops with the following messages

 

18/03/14 07:07:47 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:48 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:49 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:50 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:51 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:52 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

 

when I go to RM UI I see this 

 

ACCEPTED: waiting for AM container to be allocated, launched and register with RM.

 

 

 

 

On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <[hidden email]> wrote:

This does not look like Spark error. Looks like yarn has not been able to
allocate resources for spark driver. If you check resource manager UI you
are likely to see this as spark application waiting for resources. Try
reducing the driver node memory and/ or other bottlenecks based on what you
see in the resource manager UI.



--
Sent from:
http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail:
[hidden email]

 

 

 



 

--

"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.

 



 

--

"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.

 



The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.


Reply | Threaded
Open this post in threaded view
|

Re: How to run spark shell using YARN

kant kodali
Do I need to set SPARK_DIST_CLASSPATH or SPARK_CLASSPATH ? The latest version of spark (2.3) only has SPARK_CLASSPATH.

On Wed, Mar 14, 2018 at 11:37 AM, kant kodali <[hidden email]> wrote:
Hi, 

I am not using emr. And yes I restarted several times.

On Wed, Mar 14, 2018 at 6:35 AM, Anthony, Olufemi <[hidden email]> wrote:

After you updated your yarn-site.xml  file, did you restart the YARN resource manager ?

 

https://aws.amazon.com/premiumsupport/knowledge-center/restart-service-emr/

 

Femi

 

From: kant kodali <[hidden email]>
Date: Wednesday, March 14, 2018 at 6:16 AM
To: Femi Anthony <[hidden email]>
Cc: vermanurag <[hidden email]>, "user @spark" <[hidden email]>
Subject: Re: How to run spark shell using YARN

 

 

On Wed, Mar 14, 2018 at 2:46 AM, Femi Anthony <[hidden email]> wrote:

What's the hardware configuration of the box you're running on i.e. how much memory does it have ?

 

Femi

 

On Wed, Mar 14, 2018 at 5:32 AM, kant kodali <[hidden email]> wrote:

Tried this

 

 ./spark-shell --master yarn --deploy-mode client --executor-memory 4g



Same issue. Keeps going forever..



18/03/14 09:31:25 INFO Client: 

client token: N/A

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: default

start time: 1521019884656

final status: UNDEFINED

tracking URL: http://ip-172-31-0-54:8088/proxy/application_1521014458020_0004/

user: centos



18/03/14 09:30:08 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:09 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:10 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:11 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:12 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:13 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:14 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:15 INFO Client: Application report for application_1521014458020_0003 (state: ACCEPTED)

 

On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony <[hidden email]> wrote:

Make sure you have enough memory allocated for Spark workers, try specifying executor memory as follows:

--executor-memory <memory>

to spark-submit.

 

On Wed, Mar 14, 2018 at 3:25 AM, kant kodali <[hidden email]> wrote:

I am using spark 2.3.0 and hadoop 2.7.3. 

 

Also I have done the following and restarted all. But I still see ACCEPTED: waiting for AM container to be allocated, launched and register with RM. And i am unable to spawn spark-shell.

 

editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the following property value from 0.1 to something higher. I changed to 0.5 (50%)

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.5</value>
    <description>
        Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications.
    </description>
</property>

You may have to allocate more memory to YARN by editing yarn-site.xml by updating the following property:

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>8192</value>
</property>

 

 

 

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:

any idea?

 

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <[hidden email]> wrote:

I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website and these are the only three files I changed Do I need to set or change anything in mapred-site.xml (As of now I have not touched mapred-site.xml)? 

 

when I do yarn -node -list -all I can see both node manager and resource managers are running fine.

 

But when I run spark-shell --master yarn --deploy-mode client

 

 

it just keeps looping forever and never stops with the following messages

 

18/03/14 07:07:47 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:48 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:49 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:50 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:51 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

18/03/14 07:07:52 INFO Client: Application report for application_1521011212550_0001 (state: ACCEPTED)

 

when I go to RM UI I see this 

 

ACCEPTED: waiting for AM container to be allocated, launched and register with RM.

 

 

 

 

On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <[hidden email]> wrote:

This does not look like Spark error. Looks like yarn has not been able to
allocate resources for spark driver. If you check resource manager UI you
are likely to see this as spark application waiting for resources. Try
reducing the driver node memory and/ or other bottlenecks based on what you
see in the resource manager UI.



--
Sent from:
http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail:
[hidden email]

 

 

 



 

--

"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.

 



 

--

"Great spirits have always encountered violent opposition from mediocre minds." - Albert Einstein.

 



The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.