This post has NOT been accepted by the mailing list yet.
When writing a dataframe, a _SUCCESS file is created to mark that the entire dataframe is written. However, the existence of this _SUCCESS does not seem to be validated by default on reads. This would allow in some cases for partially written dataframes to be read back. Is this behavior configurable? Is lack of validation intentional?
Here is an example from spark 2.1.0 shell. I would expect the read step to fail because I've manually removed the _SUCCESS file: