Spark Stremaing - Dstreams - Removing RDD

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark Stremaing - Dstreams - Removing RDD

forece85
We are using Spark Streaming (Dstreams) with Kinesis batch interval as 10sec.
For every random batch, processing time is taking very long. While checking
logs, we found below log lines when ever we are getting spike in processing
time:

<http://apache-spark-user-list.1001560.n3.nabble.com/file/t10925/Untitled.png>

Processing Time:

<http://apache-spark-user-list.1001560.n3.nabble.com/file/t10925/Untitled2.png>

We dont have any code to manually remove RDDs. How to get rid of these?

Thanks in Advance.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]