Spark Streaming RDD Cleanup too slow

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark Streaming RDD Cleanup too slow

Prashant Sharma-2
I have a Spark Streaming job which takes too long to delete temp RDD's. I collect about 4MM telemetry metrics per minute and do minor aggregations in the Streaming Job. 

I am using Amazon R4 instances.  The Driver RPC call although Async,i believe, is slow getting the handle for future object  at "askAsync call.  Here  is the Spark code which does the cleanup - https://github.com/apache/spark/blob/5264164a67df498b73facae207eda12ee133be7d/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala#L125

Any chance anyone else encountered similar issue with their Streaming jobs? About 20% of our time (~60 secs) is spent in cleaning the temp RDDs.
best,
Prashant