Partitions cached by updatStateByKey does not seem to be getting evicted forever

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
SRK
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Partitions cached by updatStateByKey does not seem to be getting evicted forever

SRK
This post has NOT been accepted by the mailing list yet.
Hi,

We use updateStateByKey in our Spark streaming application. The partitions cached by updateStateByKey does not seem to be getting evicted. It was getting evicted fine with spark.cleaner.ttl in 1.5.1. I am facing issues with partitions not getting evicted with Stateful Streaming after Spark 2.1. upgrade.

Any idea as to why this happens?


Thanks!
Loading...