too many "Lost executor" shark on yarn

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

too many "Lost executor" shark on yarn

martinxu
This post has NOT been accepted by the mailing list yet.
shark 0.9.1
shark 0.9.1
hadoop cdh 5.0.1
centOS 64

two issues:
1, executor  number is away less than 4, no matter  how to set .

2, almost 1/3 task failed when I run sql.
 it show " CANNOT FIND ADDRESS" on spark ui.
 and many errors in hive.log.





2014-06-12 09:19:18,249 ERROR cluster.YarnClientClusterScheduler (Logging.scala:logError(66)) - Lost executor 2 on host49
: remote Akka client disassociated
2014-06-12 09:19:18,279 ERROR remote.EndpointWriter (Slf4jLogger.scala:apply$mcV$sp(65)) - AssociationError [akka.tcp://s
park@host50:4629] -> [akka.tcp://spark@host49:53281]: Error [Association failed with [akka.tcp://spark@host49:53281]] [
2014-06-12 09:19:18,279 ERROR remote.EndpointWriter (Slf4jLogger.scala:apply$mcV$sp(65)) - AssociationError [akka.tcp://s
park@host50:4629] -> [akka.tcp://sparkExecutor@host49:27326]: Error [Association failed with [akka.tcp://sparkExecutor@ho
st49:27326]] [
2014-06-12 09:19:18,283 ERROR remote.EndpointWriter (Slf4jLogger.scala:apply$mcV$sp(65)) - AssociationError [akka.tcp://s
park@host50:4629] -> [akka.tcp://spark@host49:53281]: Error [Association failed with [akka.tcp://spark@host49:53281]] [
2014-06-12 09:19:18,284 ERROR remote.EndpointWriter (Slf4jLogger.scala:apply$mcV$sp(65)) - AssociationError [akka.tcp://s
park@host50:4629] -> [akka.tcp://sparkExecutor@host49:27326]: Error [Association failed with [akka.tcp://sparkExecutor@ho
st49:27326]] [
2014-06-12 09:19:18,287 ERROR remote.EndpointWriter (Slf4jLogger.scala:apply$mcV$sp(65)) - AssociationError [akka.tcp://s
park@host50:4629] -> [akka.tcp://spark@host49:53281]: Error [Association failed with [akka.tcp://spark@host49:53281]] [
2014-06-12 09:19:18,289 ERROR remote.EndpointWriter (Slf4jLogger.scala:apply$mcV$sp(65)) - AssociationError [akka.tcp://s
park@host50:4629] -> [akka.tcp://sparkExecutor@host49:27326]: Error [Association failed with [akka.tcp://sparkExecutor@ho
st49:27326]] [
2014-06-12 09:19:18,431 ERROR cluster.YarnClientClusterScheduler (Logging.scala:logError(66)) - Lost executor 1 on host50
: remote Akka client disassociated