Passing an array of more than 22 elements in a UDF

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Passing an array of more than 22 elements in a UDF

Aakash Basu-2
Hi,

I am using Spark 2.2 using Java, can anyone please suggest me how to take more than 22 parameters in an UDF? I mean, if I want to pass all the parameters as an array of integers?

Thanks,
Aakash.
Reply | Threaded
Open this post in threaded view
|

Re: Passing an array of more than 22 elements in a UDF

ayan guha
Hi I think you are in correct track. You can stuff all your param in a suitable data structure like array or dict and pass this structure as a single param in your udf.

On Fri, 22 Dec 2017 at 2:55 pm, Aakash Basu <[hidden email]> wrote:
Hi,

I am using Spark 2.2 using Java, can anyone please suggest me how to take more than 22 parameters in an UDF? I mean, if I want to pass all the parameters as an array of integers?

Thanks,
Aakash.
--
Best Regards,
Ayan Guha
Reply | Threaded
Open this post in threaded view
|

Re: Passing an array of more than 22 elements in a UDF

Felix Cheung
Or use it with Scala 2.11?


From: ayan guha <[hidden email]>
Sent: Friday, December 22, 2017 3:15:14 AM
To: Aakash Basu
Cc: user
Subject: Re: Passing an array of more than 22 elements in a UDF
 
Hi I think you are in correct track. You can stuff all your param in a suitable data structure like array or dict and pass this structure as a single param in your udf.

On Fri, 22 Dec 2017 at 2:55 pm, Aakash Basu <[hidden email]> wrote:
Hi,

I am using Spark 2.2 using Java, can anyone please suggest me how to take more than 22 parameters in an UDF? I mean, if I want to pass all the parameters as an array of integers?

Thanks,
Aakash.
--
Best Regards,
Ayan Guha
Reply | Threaded
Open this post in threaded view
|

Re: Passing an array of more than 22 elements in a UDF

Aakash Basu-2
What's the privilege of using that specific version for this? Please throw some light onto it.

On Mon, Dec 25, 2017 at 6:51 AM, Felix Cheung <[hidden email]> wrote:
Or use it with Scala 2.11?


From: ayan guha <[hidden email]>
Sent: Friday, December 22, 2017 3:15:14 AM
To: Aakash Basu
Cc: user
Subject: Re: Passing an array of more than 22 elements in a UDF
 
Hi I think you are in correct track. You can stuff all your param in a suitable data structure like array or dict and pass this structure as a single param in your udf.

On Fri, 22 Dec 2017 at 2:55 pm, Aakash Basu <[hidden email]> wrote:
Hi,

I am using Spark 2.2 using Java, can anyone please suggest me how to take more than 22 parameters in an UDF? I mean, if I want to pass all the parameters as an array of integers?

Thanks,
Aakash.
--
Best Regards,
Ayan Guha

Reply | Threaded
Open this post in threaded view
|

Re: Passing an array of more than 22 elements in a UDF

Felix Cheung
Generally the 22 limitation is from Scala 2.10.

In Scala 2.11, the issue with case class is fixed, but with that said I’m not sure if with UDF in Java other limitation might apply.

_____________________________
From: Aakash Basu <[hidden email]>
Sent: Monday, December 25, 2017 9:13 PM
Subject: Re: Passing an array of more than 22 elements in a UDF
To: Felix Cheung <[hidden email]>
Cc: ayan guha <[hidden email]>, user <[hidden email]>


What's the privilege of using that specific version for this? Please throw some light onto it.

On Mon, Dec 25, 2017 at 6:51 AM, Felix Cheung <[hidden email]> wrote:
Or use it with Scala 2.11?


From: ayan guha <[hidden email]>
Sent: Friday, December 22, 2017 3:15:14 AM
To: Aakash Basu
Cc: user
Subject: Re: Passing an array of more than 22 elements in a UDF
 
Hi I think you are in correct track. You can stuff all your param in a suitable data structure like array or dict and pass this structure as a single param in your udf.

On Fri, 22 Dec 2017 at 2:55 pm, Aakash Basu <[hidden email]> wrote:
Hi,

I am using Spark 2.2 using Java, can anyone please suggest me how to take more than 22 parameters in an UDF? I mean, if I want to pass all the parameters as an array of integers?

Thanks,
Aakash.
--
Best Regards,
Ayan Guha



Reply | Threaded
Open this post in threaded view
|

Re: Passing an array of more than 22 elements in a UDF

Vadim Semenov

On Tue, Dec 26, 2017 at 2:19 PM, Felix Cheung <[hidden email]> wrote:
Generally the 22 limitation is from Scala 2.10.

In Scala 2.11, the issue with case class is fixed, but with that said I’m not sure if with UDF in Java other limitation might apply.

_____________________________
From: Aakash Basu <[hidden email]>
Sent: Monday, December 25, 2017 9:13 PM
Subject: Re: Passing an array of more than 22 elements in a UDF
To: Felix Cheung <[hidden email]>
Cc: ayan guha <[hidden email]>, user <[hidden email]>



What's the privilege of using that specific version for this? Please throw some light onto it.

On Mon, Dec 25, 2017 at 6:51 AM, Felix Cheung <[hidden email]> wrote:
Or use it with Scala 2.11?


From: ayan guha <[hidden email]>
Sent: Friday, December 22, 2017 3:15:14 AM
To: Aakash Basu
Cc: user
Subject: Re: Passing an array of more than 22 elements in a UDF
 
Hi I think you are in correct track. You can stuff all your param in a suitable data structure like array or dict and pass this structure as a single param in your udf.

On Fri, 22 Dec 2017 at 2:55 pm, Aakash Basu <[hidden email]> wrote:
Hi,

I am using Spark 2.2 using Java, can anyone please suggest me how to take more than 22 parameters in an UDF? I mean, if I want to pass all the parameters as an array of integers?

Thanks,
Aakash.
--
Best Regards,
Ayan Guha