How do I generate current UTC timestamp in raw spark sql?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How do I generate current UTC timestamp in raw spark sql?

kant kodali
Hi All,

How do I generate current UTC timestamp using spark sql?

When I do curent_timestamp() it is giving me local time.

to_utc_timestamp(current_time(), ????) takes timezone in the second parameter and I see no udf that can give me current timezone.

when I do 

spark.conf.set('spark.sql.session.timeZone', 'UTC')
current_timestamp()

But this wont work for me because I dont have access to spark conf object .

Any other solutions using raw sql alone and default udfs would be helpful.

Thanks!

Reply | Threaded
Open this post in threaded view
|

Re: How do I generate current UTC timestamp in raw spark sql?

Nikita Goyal
Hi,

One way is to write own UDF and use UTC zone inside it.

Something like  :  

import org.joda.time.{DateTime,DateTimeZone}
import java.sql.Timestamp
val getCurrentTimestampUTC = udf(() => {
new Timestamp(new DateTime(new Date()).withZone(DateTimeZone.UTC).getMillis)
})
Note : I've not tested the code.  

Regards,
Nikita

On Tue, Aug 28, 2018 at 2:34 PM kant kodali <[hidden email]> wrote:
Hi All,

How do I generate current UTC timestamp using spark sql?

When I do curent_timestamp() it is giving me local time.

to_utc_timestamp(current_time(), ????) takes timezone in the second parameter and I see no udf that can give me current timezone.

when I do 

spark.conf.set('spark.sql.session.timeZone', 'UTC')
current_timestamp()

But this wont work for me because I dont have access to spark conf object .

Any other solutions using raw sql alone and default udfs would be helpful.

Thanks!