Issues when trying to recover a textFileStream from checkpoint in Spark streaming

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
SRK
Reply | Threaded
Open this post in threaded view
|

Issues when trying to recover a textFileStream from checkpoint in Spark streaming

SRK
This post has NOT been accepted by the mailing list yet.
Hi,

I am facing issues while trying to recover a textFileStream from checkpoint. Basically it is trying to load the files from the begining  of the job start whereas I am deleting the files after processing them. I have the following configs set so was thinking that it should not look for files beyond 2 minutes when trying to recover from checkpoint. Any suggestions on this would be of great help.

      sparkConf.set("spark.streaming.minRememberDuration","120s")
      sparkConf.set("spark.streaming.fileStream.minRememberDuration","120s")

Thanks,
Swetha