Creating hive table through df.write.mode("overwrite").saveAsTable("DB.TABLE")

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Creating hive table through df.write.mode("overwrite").saveAsTable("DB.TABLE")

Mich Talebzadeh

Hi,

In Spark I specifically specify the format of the table to be created

 sqltext = """
     CREATE TABLE test.randomDataPy(
       ID INT
     , CLUSTERED INT
     , SCATTERED INT
     , RANDOMISED INT
     , RANDOM_STRING VARCHAR(50)
     , SMALL_VC VARCHAR(50)
     , PADDING  VARCHAR(4000)
    )
    STORED AS PARQUET
    """
  sqlContext.sql(sqltext)

However, I can store it through

df.write.mode("overwrite").saveAsTable("test.ABCD")

The table created through the above is stored as parquet.

 desc test.abcd
+----------------+------------+----------+
|    col_name    | data_type  | comment  |
+----------------+------------+----------+
| id             | bigint     |          |
| clustered      | double     |          |
| scattered      | double     |          |
| randomised     | double     |          |
| random_string  | string     |          |
| small_vc       | string     |          |
| padding        | string     |          |
+----------------+------------+----------+


Is this the default behaviour expected?

Thanks

Mich



LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.