Issues when trying to recover a textFileStream from checkpoint in Spark streaming
This post has NOT been accepted by the mailing list yet.
I am facing issues while trying to recover a textFileStream from checkpoint. Basically it is trying to load the files from the begining of the job start whereas I am deleting the files after processing them. I have the following configs set so was thinking that it should not look for files beyond 2 minutes when trying to recover from checkpoint. Any suggestions on this would be of great help.