Getting Corrupt Records while loading data into dataframe from csv file

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Getting Corrupt Records while loading data into dataframe from csv file

Shuporno Choudhury
Hi all,

I have a manually created schema using which I am loading data from multiple csv files to a dataframe.
Now, if there are certain records that fail the provided schema, is there a way to get those rejected records and continue with the process of loading data into the dataframe?
As of now, it seems the options that I have the are the 3 modes (PERMISSIVE, DROPMALFORMED and FAILFAST), none of which seem to fulfill the objective.


--
--Thanks,
Shuporno Choudhury