Spark SQL Dataset and BigDecimal

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark SQL Dataset and BigDecimal

Ivan Petrov
Hi, I'm using Spark Scala Dataset API to write spark sql jobs.
I've noticed that Spark dataset accepts scala BigDecimal as the value but it always returns java.math.BigDecimal when you read it back.

Is it by design?
Should I use java.math.BigDecimal everywhere instead?
Is there any performance penalty for  using scala BigDecimal? it's more convenient from an API point of view than java.math.BigDecimal.
Reply | Threaded
Open this post in threaded view
|

Re: Spark SQL Dataset and BigDecimal

Takeshi Yamamuro
Yea, I think that's because it's needed for interoperability between scala/java.
If it returns a scala decimal, java code cannot handle it.

If you want a scala decimal, you need to convert it by yourself.

Bests,
Takeshi

On Wed, Feb 17, 2021 at 9:48 PM Ivan Petrov <[hidden email]> wrote:
Hi, I'm using Spark Scala Dataset API to write spark sql jobs.
I've noticed that Spark dataset accepts scala BigDecimal as the value but it always returns java.math.BigDecimal when you read it back.

Is it by design?
Should I use java.math.BigDecimal everywhere instead?
Is there any performance penalty for  using scala BigDecimal? it's more convenient from an API point of view than java.math.BigDecimal.


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Spark SQL Dataset and BigDecimal

Ivan Petrov
I'm fine with both. So does it make sense to use java.math.BigDecimal everywhere to avoid perf penalty for value conversion? 
scala BigMath looks like a wrapper around java.math.BigDecimal though...

чт, 18 февр. 2021 г. в 00:33, Takeshi Yamamuro <[hidden email]>:
Yea, I think that's because it's needed for interoperability between scala/java.
If it returns a scala decimal, java code cannot handle it.

If you want a scala decimal, you need to convert it by yourself.

Bests,
Takeshi

On Wed, Feb 17, 2021 at 9:48 PM Ivan Petrov <[hidden email]> wrote:
Hi, I'm using Spark Scala Dataset API to write spark sql jobs.
I've noticed that Spark dataset accepts scala BigDecimal as the value but it always returns java.math.BigDecimal when you read it back.

Is it by design?
Should I use java.math.BigDecimal everywhere instead?
Is there any performance penalty for  using scala BigDecimal? it's more convenient from an API point of view than java.math.BigDecimal.


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Spark SQL Dataset and BigDecimal

Khalid Mammadov
As Scala book says Value types are mapped/transformed to java primitive types. So when you use Integer for example it will compile to int. So Integer is a syntactic sugar and makes it more readable in Scala code than plain int and plus Scala adds extra perks through implicits etc. I think the same goes for this BigDecimal case. So I personally would go with Scala types and compiler should do the rest for you.

Cheers,
Khalid Mammadov

On 18 Feb 2021, at 09:39, Ivan Petrov <[hidden email]> wrote:


I'm fine with both. So does it make sense to use java.math.BigDecimal everywhere to avoid perf penalty for value conversion? 
scala BigMath looks like a wrapper around java.math.BigDecimal though...

чт, 18 февр. 2021 г. в 00:33, Takeshi Yamamuro <[hidden email]>:
Yea, I think that's because it's needed for interoperability between scala/java.
If it returns a scala decimal, java code cannot handle it.

If you want a scala decimal, you need to convert it by yourself.

Bests,
Takeshi

On Wed, Feb 17, 2021 at 9:48 PM Ivan Petrov <[hidden email]> wrote:
Hi, I'm using Spark Scala Dataset API to write spark sql jobs.
I've noticed that Spark dataset accepts scala BigDecimal as the value but it always returns java.math.BigDecimal when you read it back.

Is it by design?
Should I use java.math.BigDecimal everywhere instead?
Is there any performance penalty for  using scala BigDecimal? it's more convenient from an API point of view than java.math.BigDecimal.


--
---
Takeshi Yamamuro