How to set task number in a container

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How to set task number in a container

hequn8128
When i increase my input data size, the executor will be failed and lost. see below:

14/03/11 20:44:18 INFO AppClient$ClientActor: Executor updated: app-20140311204343-0008/8 is now FAILED (Command exited with code 134)
14/03/11 20:44:18 INFO SparkDeploySchedulerBackend: Executor app-20140311204343-0008/8 removed: Command exited with code 134
14/03/11 20:44:18 INFO SparkDeploySchedulerBackend: Executor 8 disconnected, so removing it
14/03/11 20:44:18 ERROR TaskSchedulerImpl: Lost executor 8 on Salve10.Hadoop: Unknown executor exit code (134) (died from signal 6?)
14/03/11 20:44:18 INFO TaskSetManager: Re-queueing tasks for 8 from TaskSet 1.0
14/03/11 20:44:18 WARN TaskSetManager: Lost TID 8 (task 1.0:22)
14/03/11 20:44:18 WARN TaskSetManager: Lost TID 49 (task 1.0:49)
14/03/11 20:44:18 WARN TaskSetManager: Lost TID 31 (task 1.0:40)
14/03/11 20:44:18 WARN TaskSetManager: Lost TID 1 (task 1.0:10)
14/03/11 20:44:18 WARN TaskSetManager: Lost TID 15 (task 1.0:48)
14/03/11 20:44:18 INFO AppClient$ClientActor: Executor added: app-20140311204343-0008/11 on worker-20140311172513-Salve10.Hadoop-56435 (Salve10.Hadoop:56435) with 8 cores
14/03/11 20:44:18 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140311204343-0008/11 on hostPort Salve10.Hadoop:56435 with 8 cores, 4.0 GB RAM
14/03/11 20:44:18 INFO AppClient$ClientActor: Executor updated: app-20140311204343-0008/11 is now RUNNING

It seems that too many tasks are running in a container and the total task memory exceed the "spark.executor.memory"

so how to set the task number in a executor?
thank you