Partitions cached by updatStateByKey does not seem to be getting evicted forever
This post has NOT been accepted by the mailing list yet.
We use updateStateByKey in our Spark streaming application. The partitions cached by updateStateByKey does not seem to be getting evicted. It was getting evicted fine with spark.cleaner.ttl in 1.5.1. I am facing issues with partitions not getting evicted with Stateful Streaming after Spark 2.1. upgrade.