Dataframe from partitioned parquet table missing partition columns from schema

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Dataframe from partitioned parquet table missing partition columns from schema

Nirav Patel
I created a hive table with parquet storage using sparkSql. Now in hive cli when I do describe and Select I can see partition columns in both as regular columns as well as partition column. However if I try to do same in sparkSql (Dataframe) I don't see partition columns.

I need to do projection on partition column in spark. How do I do that now?



What's New with Xactly

        
Reply | Threaded
Open this post in threaded view
|

Re: Dataframe from partitioned parquet table missing partition columns from schema

Nirav Patel
Just found out that I need following option while reading:



On Tue, Jul 17, 2018 at 3:48 PM, Nirav Patel <[hidden email]> wrote:
I created a hive table with parquet storage using sparkSql. Now in hive cli when I do describe and Select I can see partition columns in both as regular columns as well as partition column. However if I try to do same in sparkSql (Dataframe) I don't see partition columns.

I need to do projection on partition column in spark. How do I do that now?




What's New with Xactly