Timestamp changing while writing

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Timestamp changing while writing

sk skk
Hello,

I am using createDataframe and passing java row rdd and schema . But it is changing the time value when I write that data frame to a parquet file.

Can any one help .

Thank you,
Sudhir
Reply | Threaded
Open this post in threaded view
|

Re: Timestamp changing while writing

Bryan Cutler
Spark internally stores timestamps as UTC values, so cearteDataFrame will covert from local time zone to UTC. I think there was a Jira to correct parquet output. Are the values you are seeing offset from your local time zone?

On Jan 11, 2018 4:49 PM, "sk skk" <[hidden email]> wrote:
Hello,

I am using createDataframe and passing java row rdd and schema . But it is changing the time value when I write that data frame to a parquet file.

Can any one help .

Thank you,
Sudhir