Spark stuck at removing broadcast variable

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark stuck at removing broadcast variable

SparkUser6
I am running a simple Spark structured streaming application that is pulling data from a Kafka Topic. I have a Kafka Topic with nearly 1000 partitions. I am running this app on 6 node EMR cluster with 4 cores and 16GB RAM. I observed that Spark is trying to pull data from all 1024 Kafka partition and after running successful for few iteration it is stuck with following exception:

20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 101
20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 66
20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 77
20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 78

20/04/18 00:51:41 INFO BlockManagerInfo: Removed broadcast_2_piece0 on  in memory (size: 4.5 KB, free: 2.7 GB)
20/04/18 00:51:41 INFO BlockManagerInfo: Removed broadcast_2_piece0 on ip- in memory (size: 4.5 KB, free: 2.7 GB)
20/04/18 00:51:41 INFO BlockManagerInfo: Removed broadcast_2_piece0 on ip- in memory (size: 4.5 KB, free: 2.7 GB)
Then Sparks show RUNNING but it is NOT Processing any data.

Reply | Threaded
Open this post in threaded view
|

Re: Spark stuck at removing broadcast variable

srowen
I don't think that means it's stuck on removing something; it was
removed. Not sure what it is waiting on - more data perhaps?

On Sat, Apr 18, 2020 at 2:22 PM Alchemist <[hidden email]> wrote:

>
> I am running a simple Spark structured streaming application that is pulling data from a Kafka Topic. I have a Kafka Topic with nearly 1000 partitions. I am running this app on 6 node EMR cluster with 4 cores and 16GB RAM. I observed that Spark is trying to pull data from all 1024 Kafka partition and after running successful for few iteration it is stuck with following exception:
>
> 20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 101
> 20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 66
> 20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 77
> 20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 78
>
> 20/04/18 00:51:41 INFO BlockManagerInfo: Removed broadcast_2_piece0 on  in memory (size: 4.5 KB, free: 2.7 GB)
> 20/04/18 00:51:41 INFO BlockManagerInfo: Removed broadcast_2_piece0 on ip- in memory (size: 4.5 KB, free: 2.7 GB)
> 20/04/18 00:51:41 INFO BlockManagerInfo: Removed broadcast_2_piece0 on ip- in memory (size: 4.5 KB, free: 2.7 GB)
> Then Sparks show RUNNING but it is NOT Processing any data.
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark stuck at removing broadcast variable

Waleed Fateem
This might be obvious but just checking anyways, did you confirm whether or not all of the messages have already been consumed by Spark? If that's the case then I wouldn't expect much to happen unless new data comes into your Kafka topic. 

If you're a hundred percent sure that there's still plenty more data to be consumed by Spark and that didn't happen, then I would suggest generating Java thread dumps (use Java's jstack command) from your driver's process.

On Sat, Apr 18, 2020 at 2:43 PM Sean Owen <[hidden email]> wrote:
I don't think that means it's stuck on removing something; it was
removed. Not sure what it is waiting on - more data perhaps?

On Sat, Apr 18, 2020 at 2:22 PM Alchemist <[hidden email]> wrote:
>
> I am running a simple Spark structured streaming application that is pulling data from a Kafka Topic. I have a Kafka Topic with nearly 1000 partitions. I am running this app on 6 node EMR cluster with 4 cores and 16GB RAM. I observed that Spark is trying to pull data from all 1024 Kafka partition and after running successful for few iteration it is stuck with following exception:
>
> 20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 101
> 20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 66
> 20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 77
> 20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 78
>
> 20/04/18 00:51:41 INFO BlockManagerInfo: Removed broadcast_2_piece0 on  in memory (size: 4.5 KB, free: 2.7 GB)
> 20/04/18 00:51:41 INFO BlockManagerInfo: Removed broadcast_2_piece0 on ip- in memory (size: 4.5 KB, free: 2.7 GB)
> 20/04/18 00:51:41 INFO BlockManagerInfo: Removed broadcast_2_piece0 on ip- in memory (size: 4.5 KB, free: 2.7 GB)
> Then Sparks show RUNNING but it is NOT Processing any data.
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]