Stopping StreamingContext

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Stopping StreamingContext

Sidney Feiner

Hey,

I have a Spark Streaming application processing some events.

Sometimes, I want to stop the application if a get a specific event. I collect the executor's results in the driver and based on those results, I kill the StreamingContext using StreamingContext.stop(stopSparkContext=true).

When I do that, I can see in the logs that the app is shutting down, closing receivers etc.

But when I go to the master's web UI I can still see the app under "Running Applications". But if I click it, it says the endpoint doesn't exist.

When I check the open processes on the machine, I can see that the job's process is still running.

Am I closing the application wrong?

 

Those are the logs once I call the stop() method:

 

2018-03-28 11:59:04 INFO  KafkaProducer:615 - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.

2018-03-28 11:59:04 INFO  ReceiverTracker:54 - Sent stop signal to all 1 receivers

2018-03-28 11:59:05 INFO  BlockManagerInfo:54 - Added input-0-1522238344750 in memory on i-va-spark1:59059 (size: 1632.0 B, free: 579.2 MB)

2018-03-28 11:59:05 ERROR ReceiverTracker:70 - Deregistered receiver for stream 0: Stopped by driver

2018-03-28 11:59:05 INFO  BlockManagerInfo:54 - Added input-0-1522238345000 in memory on i-va-spark1:59059 (size: 272.0 B, free: 579.2 MB)

2018-03-28 11:59:05 INFO  TaskSetManager:54 - Finished task 0.0 in stage 2.0 (TID 70) in 30213 ms on i-va-spark1 (executor 0) (1/1)

2018-03-28 11:59:05 INFO  TaskSchedulerImpl:54 - Removed TaskSet 2.0, whose tasks have all completed, from pool

2018-03-28 11:59:05 INFO  DAGScheduler:54 - ResultStage 2 (start at UserLocationHistoryJob.scala:38) finished in 30.213 s

2018-03-28 11:59:05 INFO  ReceiverTracker:54 - All of the receivers have deregistered successfully

2018-03-28 11:59:05 INFO  ReceiverTracker:54 - ReceiverTracker stopped

2018-03-28 11:59:05 INFO  JobGenerator:54 - Stopping JobGenerator immediately

2018-03-28 11:59:05 INFO  RecurringTimer:54 - Stopped timer for JobGenerator after time 1522238340000

2018-03-28 11:59:05 INFO  JobGenerator:54 - Stopped JobGenerator

2018-03-28 11:59:07 INFO  JobScheduler:54 - Stopped JobScheduler

2018-03-28 11:59:07 INFO  StreamingContext:54 - StreamingContext stopped successfully

2018-03-28 11:59:07 INFO  BlockManagerInfo:54 - Removed broadcast_5_piece0 on 10.0.0.243:41976 in memory (size: 2.4 KB, free: 488.4 MB)

2018-03-28 11:59:07 INFO  BlockManagerInfo:54 - Removed broadcast_5_piece0 on i-va-spark1:59059 in memory (size: 2.4 KB, free: 579.2 MB)

2018-03-28 11:59:07 INFO  BlockManagerInfo:54 - Removed broadcast_4_piece0 on 10.0.0.243:41976 in memory (size: 23.9 KB, free: 488.4 MB)

2018-03-28 11:59:07 INFO  BlockManagerInfo:54 - Removed broadcast_4_piece0 on i-va-spark1:59059 in memory (size: 23.9 KB, free: 579.2 MB)

2018-03-28 11:59:37 WARN  QueuedThreadPool:178 - SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop Thread[SparkUI-171-selector-ServerConnectorManager@478b3e9/2,5,main]

2018-03-28 11:59:37 WARN  QueuedThreadPool:178 - SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop Thread[SparkUI-172-selector-ServerConnectorManager@478b3e9/3,5,main]

2018-03-28 11:59:37 WARN  QueuedThreadPool:178 - SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop Thread[SparkUI-169-selector-ServerConnectorManager@478b3e9/0,5,main]

2018-03-28 11:59:37 WARN  QueuedThreadPool:178 - SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop Thread[SparkUI-170-selector-ServerConnectorManager@478b3e9/1,5,main]

2018-03-28 13:22:01 INFO  DiskBlockManager:54 - Shutdown hook called

2018-03-28 13:22:01 INFO  ShutdownHookManager:54 - Shutdown hook called

2018-03-28 13:22:01 INFO  ShutdownHookManager:54 - Deleting directory /data/spark/scratch/spark-69a3a8a6-5504-4153-a4c1-059676861581

2018-03-28 13:22:01 INFO  ShutdownHookManager:54 - Deleting directory /data/spark/scratch/spark-69a3a8a6-5504-4153-a4c1-059676861581/userFiles-8a970eec-da41-442b-9ccf-1621b9e9e045

 

 

 

Sidney Feiner / SW Developer

M: +972.528197720 / Skype: sidney.feiner.startapp

 

emailsignature

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Stopping StreamingContext

Dhaval Modi
+1

Regards,
Dhaval Modi
[hidden email]

On 29 March 2018 at 19:57, Sidney Feiner <[hidden email]> wrote:

Hey,

I have a Spark Streaming application processing some events.

Sometimes, I want to stop the application if a get a specific event. I collect the executor's results in the driver and based on those results, I kill the StreamingContext using StreamingContext.stop(stopSparkContext=true).

When I do that, I can see in the logs that the app is shutting down, closing receivers etc.

But when I go to the master's web UI I can still see the app under "Running Applications". But if I click it, it says the endpoint doesn't exist.

When I check the open processes on the machine, I can see that the job's process is still running.

Am I closing the application wrong?

 

Those are the logs once I call the stop() method:

 

2018-03-28 11:59:04 INFO  KafkaProducer:615 - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.

2018-03-28 11:59:04 INFO  ReceiverTracker:54 - Sent stop signal to all 1 receivers

2018-03-28 11:59:05 INFO  BlockManagerInfo:54 - Added input-0-1522238344750 in memory on i-va-spark1:59059 (size: 1632.0 B, free: 579.2 MB)

2018-03-28 11:59:05 ERROR ReceiverTracker:70 - Deregistered receiver for stream 0: Stopped by driver

2018-03-28 11:59:05 INFO  BlockManagerInfo:54 - Added input-0-1522238345000 in memory on i-va-spark1:59059 (size: 272.0 B, free: 579.2 MB)

2018-03-28 11:59:05 INFO  TaskSetManager:54 - Finished task 0.0 in stage 2.0 (TID 70) in 30213 ms on i-va-spark1 (executor 0) (1/1)

2018-03-28 11:59:05 INFO  TaskSchedulerImpl:54 - Removed TaskSet 2.0, whose tasks have all completed, from pool

2018-03-28 11:59:05 INFO  DAGScheduler:54 - ResultStage 2 (start at UserLocationHistoryJob.scala:38) finished in 30.213 s

2018-03-28 11:59:05 INFO  ReceiverTracker:54 - All of the receivers have deregistered successfully

2018-03-28 11:59:05 INFO  ReceiverTracker:54 - ReceiverTracker stopped

2018-03-28 11:59:05 INFO  JobGenerator:54 - Stopping JobGenerator immediately

2018-03-28 11:59:05 INFO  RecurringTimer:54 - Stopped timer for JobGenerator after time 1522238340000

2018-03-28 11:59:05 INFO  JobGenerator:54 - Stopped JobGenerator

2018-03-28 11:59:07 INFO  JobScheduler:54 - Stopped JobScheduler

2018-03-28 11:59:07 INFO  StreamingContext:54 - StreamingContext stopped successfully

2018-03-28 11:59:07 INFO  BlockManagerInfo:54 - Removed broadcast_5_piece0 on 10.0.0.243:41976 in memory (size: 2.4 KB, free: 488.4 MB)

2018-03-28 11:59:07 INFO  BlockManagerInfo:54 - Removed broadcast_5_piece0 on i-va-spark1:59059 in memory (size: 2.4 KB, free: 579.2 MB)

2018-03-28 11:59:07 INFO  BlockManagerInfo:54 - Removed broadcast_4_piece0 on 10.0.0.243:41976 in memory (size: 23.9 KB, free: 488.4 MB)

2018-03-28 11:59:07 INFO  BlockManagerInfo:54 - Removed broadcast_4_piece0 on i-va-spark1:59059 in memory (size: 23.9 KB, free: 579.2 MB)

2018-03-28 11:59:37 WARN  QueuedThreadPool:178 - SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop Thread[SparkUI-171-selector-ServerConnectorManager@478b3e9/2,5,main]

2018-03-28 11:59:37 WARN  QueuedThreadPool:178 - SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop Thread[SparkUI-172-selector-ServerConnectorManager@478b3e9/3,5,main]

2018-03-28 11:59:37 WARN  QueuedThreadPool:178 - SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop Thread[SparkUI-169-selector-ServerConnectorManager@478b3e9/0,5,main]

2018-03-28 11:59:37 WARN  QueuedThreadPool:178 - SparkUI{STOPPING,8<=9<=200,i=0,q=4} Couldn't stop Thread[SparkUI-170-selector-ServerConnectorManager@478b3e9/1,5,main]

2018-03-28 13:22:01 INFO  DiskBlockManager:54 - Shutdown hook called

2018-03-28 13:22:01 INFO  ShutdownHookManager:54 - Shutdown hook called

2018-03-28 13:22:01 INFO  ShutdownHookManager:54 - Deleting directory /data/spark/scratch/spark-69a3a8a6-5504-4153-a4c1-059676861581

2018-03-28 13:22:01 INFO  ShutdownHookManager:54 - Deleting directory /data/spark/scratch/spark-69a3a8a6-5504-4153-a4c1-059676861581/userFiles-8a970eec-da41-442b-9ccf-1621b9e9e045

 

 

 

Sidney Feiner / SW Developer

M: +972.528197720 / Skype: sidney.feiner.startapp

 

emailsignature