Support for storing date time fields as TIMESTAMP_MILLIS(INT64)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Support for storing date time fields as TIMESTAMP_MILLIS(INT64)

Rahul Raj
Hi,

I believe spark writes datetime fields as INT96. What are the implications of https://issues.apache.org/jira/browse/SPARK-10364(Support Parquet logical type TIMESTAMP_MILLIS) which is part of 2.2.0?

I am having issues while reading spark generated parquet dates using Apache Drill (Drill supports INT64 default). Will the change allow me to store datetime fields as INT64?

Regards,
Rahul


*** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom it is addressed. If you are not the named addressee then you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and delete this e-mail from your system.***