Hive External Table Partiton Data Type.

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Hive External Table Partiton Data Type.

SB M
Hi All,
   Am trying to create a dynamic partition with external table on hive metastore using spark sql.

when am trying to create a partition column data type as bigint, partition is not working even i tried with repair table. data is not shown when i ran sample query select * from table.


but if i tried to create a dynamic partition with string as data type for partition its working fine. partition are working as expected.


is there something am doing it wrong ??


Regards,
Sree

Reply | Threaded
Open this post in threaded view
|

Re: Hive External Table Partiton Data Type.

Harsh
Please share the spark version you are using .

On Fri, 13 Dec, 2019, 4:02 PM SB M, <[hidden email]> wrote:
Hi All,
   Am trying to create a dynamic partition with external table on hive metastore using spark sql.

when am trying to create a partition column data type as bigint, partition is not working even i tried with repair table. data is not shown when i ran sample query select * from table.


but if i tried to create a dynamic partition with string as data type for partition its working fine. partition are working as expected.


is there something am doing it wrong ??


Regards,
Sree

Reply | Threaded
Open this post in threaded view
|

Re: Hive External Table Partiton Data Type.

SB M
spark version 2.1.0

Regards,
Sbm

On Mon, 16 Dec, 2019, 10:04 HARSH TAKKAR, <[hidden email]> wrote:
Please share the spark version you are using .

On Fri, 13 Dec, 2019, 4:02 PM SB M, <[hidden email]> wrote:
Hi All,
   Am trying to create a dynamic partition with external table on hive metastore using spark sql.

when am trying to create a partition column data type as bigint, partition is not working even i tried with repair table. data is not shown when i ran sample query select * from table.


but if i tried to create a dynamic partition with string as data type for partition its working fine. partition are working as expected.


is there something am doing it wrong ??


Regards,
Sree

Reply | Threaded
Open this post in threaded view
|

Re: Hive External Table Partiton Data Type.

Harsh

Hi, 


I am able to make it in spark 2.3.0, If you can change the version to spark 2.3 it will be good , otherwise let me know , i 'll check on the Spark version 2.1.0. 

Following is the code for spark 2.3.0.

scala> var seq = Seq((10L,"Hello"),(10L,"Hi"))
seq: Seq[(Long, String)] = List((10,Hello), (10,Hi))

scala> seq.toDF("a","b")
res0: org.apache.spark.sql.DataFrame = [a: bigint, b: string]

scala> res0.show()
+---+-----+
|  a|    b|
+---+-----+
| 10|Hello|
| 10|   Hi|
+---+-----+

scala> res0.printSchema
root
 |-- a: long (nullable = false)
 |-- b: string (nullable = true)


scala> res0.write.option("path","/tmp/longpartition1").mode("overwrite").partitionBy("a").saveAsTable("default.longPartition")


hive> select * from longpartition
    > ;
OK
Hello 10
Hi 10
Time taken: 0.356 seconds, Fetched: 2 row(s)
hive> describe longpartition;
OK
b                   string
a                   bigint

# Partition Information
# col_name             data_type           comment

a                   bigint

On Mon, Dec 16, 2019 at 11:05 AM SB M <[hidden email]> wrote:
spark version 2.1.0

Regards,
Sbm

On Mon, 16 Dec, 2019, 10:04 HARSH TAKKAR, <[hidden email]> wrote:
Please share the spark version you are using .

On Fri, 13 Dec, 2019, 4:02 PM SB M, <[hidden email]> wrote:
Hi All,
   Am trying to create a dynamic partition with external table on hive metastore using spark sql.

when am trying to create a partition column data type as bigint, partition is not working even i tried with repair table. data is not shown when i ran sample query select * from table.


but if i tried to create a dynamic partition with string as data type for partition its working fine. partition are working as expected.


is there something am doing it wrong ??


Regards,
Sree