GC overhead limit exceeded

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

GC overhead limit exceeded

AlexModestov
This post has NOT been accepted by the mailing list yet.
I get the error in the apache spark...

"spark.driver.memory 60g
spark.python.worker.memory 60g
spark.master local[*]"

The amount of data is about 5Gb, but spark says that "GC overhead limit exceeded". I guess that my conf-file gives enought resources.

"16/05/16 15:13:02 WARN NettyRpcEndpointRef: Error sending message [message = Heartbeat(driver,[Lscala.Tuple2;@87576f9,BlockManagerId(driver, localhost, 59407))] in 1 attempts
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval
        at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)
        at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)
        at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:449)
        at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:470)
        at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470)
        at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470)
        at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1765)
        at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:470)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds]
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
        at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
        at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
        at scala.concurrent.Await$.result(package.scala:107)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
        ... 14 more
16/05/16 15:13:02 WARN NettyRpcEnv: Ignored message: HeartbeatResponse(false)
05-16 15:13:26.398 127.0.0.1:54321       2059   #e Thread WARN: Swapping!  GC CALLBACK, (K/V:29.74 GB + POJO:16.74 GB + FREE:11.03 GB == MEM_MAX:57.50 GB), desiredKV=7.19 GB OOM!
05-16 15:13:44.528 127.0.0.1:54321       2059   #e Thread WARN: Swapping!  GC CALLBACK, (K/V:29.74 GB + POJO:16.86 GB + FREE:10.90 GB == MEM_MAX:57.50 GB), desiredKV=7.19 GB OOM!
05-16 15:13:56.847 127.0.0.1:54321       2059   #e Thread WARN: Swapping!  GC CALLBACK, (K/V:29.74 GB + POJO:16.88 GB + FREE:10.88 GB == MEM_MAX:57.50 GB), desiredKV=7.19 GB OOM!
05-16 15:14:10.215 127.0.0.1:54321       2059   #e Thread WARN: Swapping!  GC CALLBACK, (K/V:29.74 GB + POJO:16.90 GB + FREE:10.86 GB == MEM_MAX:57.50 GB), desiredKV=7.19 GB OOM!
05-16 15:14:33.622 127.0.0.1:54321       2059   #e Thread WARN: Swapping!  GC CALLBACK, (K/V:29.74 GB + POJO:16.91 GB + FREE:10.85 GB == MEM_MAX:57.50 GB), desiredKV=7.19 GB OOM!
05-16 15:14:47.075 127.0.0.1:54321       2059   #e Thread WARN: Swapping!  GC CALLBACK, (K/V:29.74 GB + POJO:16.93 GB + FREE:10.84 GB == MEM_MAX:57.50 GB), desiredKV=7.19 GB OOM!
05-16 15:15:10.555 127.0.0.1:54321       2059   #e Thread WARN: Swapping!  GC CALLBACK, (K/V:29.74 GB + POJO:16.92 GB + FREE:10.84 GB == MEM_MAX:57.50 GB), desiredKV=7.19 GB OOM!
05-16 15:15:25.520 127.0.0.1:54321       2059   #e Thread WARN: Swapping!  GC CALLBACK, (K/V:29.74 GB + POJO:16.93 GB + FREE:10.84 GB == MEM_MAX:57.50 GB), desiredKV=7.19 GB OOM!
05-16 15:15:39.087 127.0.0.1:54321       2059   #e Thread WARN: Swapping!  GC CALLBACK, (K/V:29.74 GB + POJO:16.93 GB + FREE:10.84 GB == MEM_MAX:57.50 GB), desiredKV=7.19 GB OOM!
Exception in thread "HashSessionScavenger-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.concurrent.ConcurrentHashMap$ValuesView.iterator(ConcurrentHashMap.java:4683)
        at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:314)
        at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:285)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
16/05/16 15:22:26 ERROR Executor: Exception in task 0.0 in stage 10.0 (TID 107)
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.Double.valueOf(Double.java:519)
        at scala.runtime.BoxesRunTime.boxToDouble(BoxesRunTime.java:84)
        at org.apache.spark.sql.catalyst.expressions.MutableRow.setDouble(rows.scala:176)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown Source)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:30)
        at org.spark-project.guava.collect.Ordering.leastOf(Ordering.java:665)
        at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
        at org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$29.apply(RDD.scala:1391)
        at org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$29.apply(RDD.scala:1388)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
05-16 15:22:26.947 127.0.0.1:54321       2059   #e Thread WARN: Unblock allocations; cache below desired, but also OOM: GC CALLBACK, (K/V:29.74 GB + POJO:16.93 GB + FREE:10.83 GB == MEM_MAX:57.50 GB), desiredKV=38.52 GB OOM!
05-16 15:22:26.948 127.0.0.1:54321       2059   #e Thread WARN: Swapping!  GC CALLBACK, (K/V:29.74 GB + POJO:14.94 GB + FREE:12.83 GB == MEM_MAX:57.50 GB), desiredKV=8.65 GB OOM!
16/05/16 15:22:26 WARN HeartbeatReceiver: Removing executor driver with no recent heartbeats: 144662 ms exceeds timeout 120000 ms
16/05/16 15:22:26 ERROR ActorSystemImpl: exception on LARS’ timer thread
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at akka.dispatch.AbstractNodeQueue.<init>(AbstractNodeQueue.java:22)
        at akka.actor.LightArrayRevolverScheduler$TaskQueue.<init>(Scheduler.scala:443)
        at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:409)
        at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
        at java.lang.Thread.run(Thread.java:745)
16/05/16 15:22:26 INFO ActorSystemImpl: starting new LARS thread
16/05/16 15:22:26 ERROR TaskSchedulerImpl: Lost executor driver on localhost: Executor heartbeat timed out after 144662 ms
16/05/16 15:22:26 WARN TaskSetManager: Lost task 3.0 in stage 10.0 (TID 110, localhost): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 144662 ms
16/05/16 15:22:26 ERROR TaskSetManager: Task 3 in stage 10.0 failed 1 times; aborting job
16/05/16 15:22:26 WARN TaskSetManager: Lost task 6.0 in stage 10.0 (TID 113, localhost): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 144662 ms
16/05/16 15:22:26 WARN TaskSetManager: Lost task 0.0 in stage 10.0 (TID 107, localhost): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 144662 ms
16/05/16 15:22:26 WARN TaskSetManager: Lost task 2.0 in stage 10.0 (TID 109, localhost): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 144662 ms
16/05/16 15:22:26 WARN TaskSetManager: Lost task 5.0 in stage 10.0 (TID 112, localhost): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 144662 ms
16/05/16 15:22:26 WARN TaskSetManager: Lost task 7.0 in stage 10.0 (TID 114, localhost): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 144662 ms
16/05/16 15:22:26 WARN TaskSetManager: Lost task 1.0 in stage 10.0 (TID 108, localhost): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 144662 ms
16/05/16 15:22:26 WARN TaskSetManager: Lost task 4.0 in stage 10.0 (TID 111, localhost): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 144662 ms
16/05/16 15:22:26 INFO TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool
16/05/16 15:22:26 ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkDriverActorSystem-scheduler-1] shutting down ActorSystem [sparkDriverActorSystem]
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at akka.dispatch.AbstractNodeQueue.<init>(AbstractNodeQueue.java:22)
        at akka.actor.LightArrayRevolverScheduler$TaskQueue.<init>(Scheduler.scala:443)
        at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:409)
        at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
        at java.lang.Thread.run(Thread.java:745)
16/05/16 15:22:27 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/05/16 15:22:27 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/05/16 15:22:27 WARN NettyRpcEnv: Ignored message: true
16/05/16 15:22:27 WARN NettyRpcEnv: Ignored message: true
16/05/16 15:22:27 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-14,5,main]
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.Double.valueOf(Double.java:519)
        at scala.runtime.BoxesRunTime.boxToDouble(BoxesRunTime.java:84)
        at org.apache.spark.sql.catalyst.expressions.MutableRow.setDouble(rows.scala:176)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown Source)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:30)
        at org.spark-project.guava.collect.Ordering.leastOf(Ordering.java:665)
        at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37)
        at org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$29.apply(RDD.scala:1391)
        at org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$29.apply(RDD.scala:1388)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
16/05/16 15:22:27 INFO TaskSchedulerImpl: Cancelling stage 10
16/05/16 15:22:27 WARN SparkContext: Killing executors is only supported in coarse-grained mode
16/05/16 15:22:27 INFO DAGScheduler: ResultStage 10 (head at <ipython-input-13-f753ebdb6b0f>:13) failed in 667.824 s
16/05/16 15:22:27 INFO Executor: Told to re-register on heartbeat
16/05/16 15:22:27 INFO BlockManager: BlockManager re-registering with master
16/05/16 15:22:27 INFO BlockManagerMaster: Trying to register BlockManager
16/05/16 15:22:27 INFO BlockManagerMaster: Registered BlockManager
16/05/16 15:22:27 INFO BlockManager: Reporting 8 blocks to the master.
16/05/16 15:22:27 INFO DAGScheduler: Executor lost: driver (epoch 2)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerMasterEndpoint: Trying to remove executor driver from BlockManagerMaster.
16/05/16 15:22:27 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(driver, localhost, 59407)
16/05/16 15:22:27 INFO DAGScheduler: Job 8 failed: head at <ipython-input-13-f753ebdb6b0f>:13, took 667.845630 s
16/05/16 15:22:27 ERROR BlockManager: Failed to report broadcast_15_piece0 to master; giving up.
16/05/16 15:22:27 INFO BlockManagerMaster: Removed driver successfully in removeExecutor
16/05/16 15:22:27 INFO DAGScheduler: Host added was in lost list earlier: localhost
16/05/16 15:22:27 INFO SparkContext: Invoking stop() from shutdown hook
16/05/16 15:22:27 INFO Executor: Told to re-register on heartbeat
16/05/16 15:22:27 INFO BlockManager: BlockManager re-registering with master
16/05/16 15:22:27 INFO BlockManagerMaster: Trying to register BlockManager
16/05/16 15:22:27 INFO BlockManagerMasterEndpoint: Registering block manager localhost:59407 with 51.5 GB RAM, BlockManagerId(driver, localhost, 59407)
16/05/16 15:22:27 INFO BlockManagerMaster: Registered BlockManager
16/05/16 15:22:27 INFO BlockManager: Reporting 8 blocks to the master.
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:59407 (size: 8.2 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO Executor: Told to re-register on heartbeat
16/05/16 15:22:27 INFO BlockManager: BlockManager re-registering with master
16/05/16 15:22:27 INFO BlockManagerMaster: Trying to register BlockManager
16/05/16 15:22:27 INFO BlockManagerMaster: Registered BlockManager
16/05/16 15:22:27 INFO BlockManager: Reporting 8 blocks to the master.
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:59407 (size: 8.2 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO Executor: Told to re-register on heartbeat
16/05/16 15:22:27 INFO BlockManager: BlockManager re-registering with master
16/05/16 15:22:27 INFO BlockManagerMaster: Trying to register BlockManager
16/05/16 15:22:27 INFO BlockManagerMaster: Registered BlockManager
16/05/16 15:22:27 INFO BlockManager: Reporting 8 blocks to the master.
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:59407 (size: 8.2 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO Executor: Told to re-register on heartbeat
16/05/16 15:22:27 INFO BlockManager: BlockManager re-registering with master
16/05/16 15:22:27 INFO BlockManagerMaster: Trying to register BlockManager
16/05/16 15:22:27 INFO BlockManagerMaster: Registered BlockManager
16/05/16 15:22:27 INFO BlockManager: Reporting 8 blocks to the master.
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:59407 (size: 8.2 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO Executor: Told to re-register on heartbeat
16/05/16 15:22:27 INFO BlockManager: BlockManager re-registering with master
16/05/16 15:22:27 INFO BlockManagerMaster: Trying to register BlockManager
16/05/16 15:22:27 INFO BlockManagerMaster: Registered BlockManager
16/05/16 15:22:27 INFO BlockManager: Reporting 8 blocks to the master.
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO SparkUI: Stopped Spark web UI at http://192.168.107.30:4040
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:59407 (size: 8.2 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:27 INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:59407 (size: 19.3 KB, free: 51.5 GB)
16/05/16 15:22:56 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
05-16 15:22:56.111 127.0.0.1:54321       2059   #e Thread WARN: Swapping!  GC CALLBACK, (K/V:29.74 GB + POJO:15.20 GB + FREE:12.56 GB == MEM_MAX:57.50 GB), desiredKV=8.12 GB OOM!
16/05/16 15:22:56 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
16/05/16 15:22:56 WARN NettyRpcEndpointRef: Error sending message [message = Heartbeat(driver,[Lscala.Tuple2;@797268e9,BlockManagerId(driver, localhost, 59407))] in 1 attempts
org.apache.spark.SparkException: Could not find HeartbeatReceiver or it has been stopped.
        at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:161)
        at org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:126)
        at org.apache.spark.rpc.netty.NettyRpcEnv.ask(NettyRpcEnv.scala:227)
        at org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:511)
        at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100)
        at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:449)
        at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:470)
        at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470)
        at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470)
        at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1765)
        at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:470)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)"