Issue With mod function in Spark SQL

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Issue With mod function in Spark SQL

Tzahi File
I have in my spark sql query a calculated field that gets the value if field1 % 3. 

I'm using this field as a partition so I expected to get 3 partitions in the mentioned case, and I do get. The issue happened with even numbers (instead of 3 - 4,2 ... ). 
When I tried to use even numbers, for example 4 I got only 2 partitions - 1 and 3. 
Field 1 datatype is bigint. 

Do you have any suggestions?


-- 
thanks,
Tzahi 
Reply | Threaded
Open this post in threaded view
|

Re: Issue With mod function in Spark SQL

Russell Spitzer
Is there a chance your data is all even or all odd?

On Tue, Dec 17, 2019 at 11:01 AM Tzahi File <[hidden email]> wrote:
I have in my spark sql query a calculated field that gets the value if field1 % 3. 

I'm using this field as a partition so I expected to get 3 partitions in the mentioned case, and I do get. The issue happened with even numbers (instead of 3 - 4,2 ... ). 
When I tried to use even numbers, for example 4 I got only 2 partitions - 1 and 3. 
Field 1 datatype is bigint. 

Do you have any suggestions?


-- 
thanks,
Tzahi 
Reply | Threaded
Open this post in threaded view
|

Re: Issue With mod function in Spark SQL

Tzahi File
no.. there're 100M records both even and odd

On Tue, Dec 17, 2019 at 8:13 PM Russell Spitzer <[hidden email]> wrote:
Is there a chance your data is all even or all odd?

On Tue, Dec 17, 2019 at 11:01 AM Tzahi File <[hidden email]> wrote:
I have in my spark sql query a calculated field that gets the value if field1 % 3. 

I'm using this field as a partition so I expected to get 3 partitions in the mentioned case, and I do get. The issue happened with even numbers (instead of 3 - 4,2 ... ). 
When I tried to use even numbers, for example 4 I got only 2 partitions - 1 and 3. 
Field 1 datatype is bigint. 

Do you have any suggestions?


-- 
thanks,
Tzahi 


--
Tzahi File
Data Engineer
ironSource
mobile <a href="tel:+972-546864835" style="color:rgb(3,0,85)" target="_blank">+972-546864835
ironSource HQ - 121 Derech Menachem Begin st. Tel Aviv
ironsrc.com
linkedintwitterfacebookgoogleplus
This email (including any attachments) is for the sole use of the intended recipient and may contain confidential information which may be protected by legal privilege. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication and/or its content is strictly prohibited. If you are not the intended recipient, please immediately notify us by reply email or by telephone, delete this email and destroy any copies. Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: Issue With mod function in Spark SQL

Enrico Minack
I think some example code would help to understand what you are doing.

Am 18.12.19 um 08:12 schrieb Tzahi File:
no.. there're 100M records both even and odd

On Tue, Dec 17, 2019 at 8:13 PM Russell Spitzer <[hidden email]> wrote:
Is there a chance your data is all even or all odd?

On Tue, Dec 17, 2019 at 11:01 AM Tzahi File <[hidden email]> wrote:
I have in my spark sql query a calculated field that gets the value if field1 % 3. 

I'm using this field as a partition so I expected to get 3 partitions in the mentioned case, and I do get. The issue happened with even numbers (instead of 3 - 4,2 ... ). 
When I tried to use even numbers, for example 4 I got only 2 partitions - 1 and 3. 
Field 1 datatype is bigint. 

Do you have any suggestions?


-- 
thanks,
Tzahi 


--
Tzahi File
Data Engineer
ironSource
mobile <a href="tel:+972-546864835" style="color:rgb(3,0,85)" target="_blank" moz-do-not-send="true">+972-546864835
ironSource HQ - 121 Derech Menachem Begin st. Tel Aviv
ironsrc.com
linkedintwitterfacebookgoogleplus
This email (including any attachments) is for the sole use of the intended recipient and may contain confidential information which may be protected by legal privilege. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication and/or its content is strictly prohibited. If you are not the intended recipient, please immediately notify us by reply email or by telephone, delete this email and destroy any copies. Thank you.