Let’s say that I have a spark dataframe as 3 columns:
id, name, age.
When I save it into HDFS/S3, it saves as:
(where I have used “partitionBy(id, name)”)
If I want not to include “id=” and “name=” in
directory structures, what should I do
Therefore I want my final output to be: