Stopping StreamingContext does not kill receiver

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Stopping StreamingContext does not kill receiver

Nick Chammas

From the interactive shell I’ve created a StreamingContext.

I call ssc.start() and take a look at http://master_url:4040/streaming/ and see that I have an active Twitter receiver. Then I call ssc.stop(stopSparkContext = false, stopGracefully = true) and wait a bit, but the receiver seems to stay active.

Is this expected? I’m running 1.0.1 on EC2.

Nick

Reply | Threaded
Open this post in threaded view
|

Re: Stopping StreamingContext does not kill receiver

Nick Chammas

To add a potentially relevant piece of information, around when I stop the StreamingContext, I get the following warning:

14/07/12 22:16:18 WARN ReceiverTracker: All of the receivers have not deregistered, Map(0 -> ReceiverInfo(0,TwitterReceiver-0,Actor[akka.tcp://spark@url-here:49776/user/Receiver-0-1405203122224#-273455949],true,url-here-too,,))

Nick



On Sat, Jul 12, 2014 at 6:03 PM, Nick Chammas <[hidden email]> wrote:

From the interactive shell I’ve created a StreamingContext.

I call ssc.start() and take a look at http://master_url:4040/streaming/ and see that I have an active Twitter receiver. Then I call ssc.stop(stopSparkContext = false, stopGracefully = true) and wait a bit, but the receiver seems to stay active.

Is this expected? I’m running 1.0.1 on EC2.

Nick



View this message in context: Stopping StreamingContext does not kill receiver
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Stopping StreamingContext does not kill receiver

Tathagata Das
Yes, thats a bug i just discovered. Race condition in the Twitter Receiver, will fix asap. 

TD


On Sat, Jul 12, 2014 at 3:21 PM, Nicholas Chammas <[hidden email]> wrote:

To add a potentially relevant piece of information, around when I stop the StreamingContext, I get the following warning:

14/07/12 22:16:18 WARN ReceiverTracker: All of the receivers have not deregistered, Map(0 -> ReceiverInfo(0,TwitterReceiver-0,Actor[akka.tcp://spark@url-here:49776/user/Receiver-0-1405203122224#-273455949],true,url-here-too,,))

Nick



On Sat, Jul 12, 2014 at 6:03 PM, Nick Chammas <[hidden email]> wrote:

From the interactive shell I’ve created a StreamingContext.

I call ssc.start() and take a look at http://master_url:4040/streaming/ and see that I have an active Twitter receiver. Then I call ssc.stop(stopSparkContext = false, stopGracefully = true) and wait a bit, but the receiver seems to stay active.

Is this expected? I’m running 1.0.1 on EC2.

Nick



View this message in context: Stopping StreamingContext does not kill receiver
Sent from the Apache Spark User List mailing list archive at Nabble.com.


Reply | Threaded
Open this post in threaded view
|

Re: Stopping StreamingContext does not kill receiver

Nick Chammas
Okie doke. Thanks for filing the JIRA.


On Sat, Jul 12, 2014 at 6:45 PM, Tathagata Das <[hidden email]> wrote:
Yes, thats a bug i just discovered. Race condition in the Twitter Receiver, will fix asap. 

TD


On Sat, Jul 12, 2014 at 3:21 PM, Nicholas Chammas <[hidden email]> wrote:

To add a potentially relevant piece of information, around when I stop the StreamingContext, I get the following warning:

14/07/12 22:16:18 WARN ReceiverTracker: All of the receivers have not deregistered, Map(0 -> ReceiverInfo(0,TwitterReceiver-0,Actor[akka.tcp://spark@url-here:49776/user/Receiver-0-1405203122224#-273455949],true,url-here-too,,))

Nick



On Sat, Jul 12, 2014 at 6:03 PM, Nick Chammas <[hidden email]> wrote:

From the interactive shell I’ve created a StreamingContext.

I call ssc.start() and take a look at http://master_url:4040/streaming/ and see that I have an active Twitter receiver. Then I call ssc.stop(stopSparkContext = false, stopGracefully = true) and wait a bit, but the receiver seems to stay active.

Is this expected? I’m running 1.0.1 on EC2.

Nick



View this message in context: Stopping StreamingContext does not kill receiver
Sent from the Apache Spark User List mailing list archive at Nabble.com.



Reply | Threaded
Open this post in threaded view
|

Re: Stopping StreamingContext does not kill receiver

lbustelo
I'm running on spark 1.0.0 and I see a similar problem when using the socketTextStream receiver. The ReceiverTracker task sticks around after a ssc.stop(false).
Reply | Threaded
Open this post in threaded view
|

Re: Stopping StreamingContext does not kill receiver

Tathagata Das
Can you give the stack trace?

This was the fix for the twitter stream. 


You could try doing the same.

TD



On Wed, Aug 6, 2014 at 2:41 PM, lbustelo <[hidden email]> wrote:
I'm running on spark 1.0.0 and I see a similar problem when using the
socketTextStream receiver. The ReceiverTracker task sticks around after a
ssc.stop(false).



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11587.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Stopping StreamingContext does not kill receiver

lbustelo
Sorry about the screenshot… but that is what I have handy right now. You can see that we get a WARN and it ultimately say that it stopped successfully. When looking that the application in Spark UI, it still shows the ReceiverTracker task running.

It is easy to recreate. On the spark repl we are running a modified version of https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala.

Then do a ssc.stop(false).

Reply | Threaded
Open this post in threaded view
|

Re: Stopping StreamingContext does not kill receiver

Tathagata Das
Okay let me give it a shot.


On Wed, Aug 6, 2014 at 3:57 PM, lbustelo <[hidden email]> wrote:
Sorry about the screenshot… but that is what I have handy right now. You can
see that we get a WARN and it ultimately say that it stopped successfully.
When looking that the application in Spark UI, it still shows the
ReceiverTracker task running.

It is easy to recreate. On the spark repl we are running a modified version
of
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala.

Then do a ssc.stop(false).

<http://apache-spark-user-list.1001560.n3.nabble.com/file/n11595/Screen_Shot_2014-08-06_at_4.png>



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11595.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Stopping StreamingContext does not kill receiver

Tathagata Das
I narrowed down the error. Unfortunately this is not quick fix. I have opened a JIRA for this.


On Wed, Aug 6, 2014 at 3:59 PM, Tathagata Das <[hidden email]> wrote:
Okay let me give it a shot.


On Wed, Aug 6, 2014 at 3:57 PM, lbustelo <[hidden email]> wrote:
Sorry about the screenshot… but that is what I have handy right now. You can
see that we get a WARN and it ultimately say that it stopped successfully.
When looking that the application in Spark UI, it still shows the
ReceiverTracker task running.

It is easy to recreate. On the spark repl we are running a modified version
of
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala.

Then do a ssc.stop(false).

<http://apache-spark-user-list.1001560.n3.nabble.com/file/n11595/Screen_Shot_2014-08-06_at_4.png>



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Stopping-StreamingContext-does-not-kill-receiver-tp9522p11595.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]