Spark 3 datasource v2: Can't extract user provided schema Dataframewriter save operation

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark 3 datasource v2: Can't extract user provided schema Dataframewriter save operation

Rahul Kumar
Dear friends, I'm implementing datasource v2 for a custom NoSql database. I'm facing following issue a) It seems while doing save operation, there is no way to access user specified schema on dataframe. There is an existing unresolved ongoing conversation here. b)I have also noticed that it is not possible to extract write modes etc(save, append, overwrite, error_if_exists ..) from the dataframewriter. I believe, I can force usesrs to supply these things through options I'm still curious if there is cleaner way to access this information. Please shed some light. Thank you very much, my next release is dependent on it lol Best, Rahul

Sent from the Apache Spark User List mailing list archive at Nabble.com.