spark strucured csv file stream not detecting new files
I am running a unit test designed to stream a folder where I am manually copying csv files. The files do not always get picked up. They only get detected when the job starts with the files already in the folder.
I even tried using the option of fileNameOnly newly included in 2.2.0. Have I missed something in the documentation. This problem does not seem to occur in DStreams examples