Pyspark access to scala/java libraries

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Pyspark access to scala/java libraries

Mohit Jaggi
Folks,
I am writing some Scala/Java code and want it to be usable from pyspark.

For example:
class MyStuff(addend: Int)  {
        def myMapFunction(x: Int) = x + addend
}

I want to call it from pyspark as:

df = ...
mystuff = sc._jvm.MyStuff(5)
df[‘x’].map(lambda x: mystuff.myMapFunction(x))

How can I do this?

Mohit.



---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Pyspark access to scala/java libraries

Mohit Jaggi
Trying again…anyone know how to make this work?

> On Jul 9, 2018, at 3:45 PM, Mohit Jaggi <[hidden email]> wrote:
>
> Folks,
> I am writing some Scala/Java code and want it to be usable from pyspark.
>
> For example:
> class MyStuff(addend: Int)  {
> def myMapFunction(x: Int) = x + addend
> }
>
> I want to call it from pyspark as:
>
> df = ...
> mystuff = sc._jvm.MyStuff(5)
> df[‘x’].map(lambda x: mystuff.myMapFunction(x))
>
> How can I do this?
>
> Mohit.
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Pyspark access to scala/java libraries

Holden Karau-2
If you want to see some examples in a library shows a way to do it - https://github.com/sparklingpandas/sparklingml and high performance spark also talks about it.

On Sun, Jul 15, 2018, 11:57 AM <[hidden email]> wrote:
Check https://stackoverflow.com/questions/31684842/calling-java-scala-function-from-a-task

​Sent with ProtonMail Secure Email.​

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On July 15, 2018 8:01 AM, Mohit Jaggi <[hidden email]> wrote:

> Trying again…anyone know how to make this work?
>
> > On Jul 9, 2018, at 3:45 PM, Mohit Jaggi [hidden email] wrote:
> >
> > Folks,
> >
> > I am writing some Scala/Java code and want it to be usable from pyspark.
> >
> > For example:
> >
> > class MyStuff(addend: Int) {
> >
> > def myMapFunction(x: Int) = x + addend
> >
> > }
> >
> > I want to call it from pyspark as:
> >
> > df = ...
> >
> > mystuff = sc._jvm.MyStuff(5)
> >
> > df[‘x’].map(lambda x: mystuff.myMapFunction(x))
> >
> > How can I do this?
> >
> > Mohit.
>
> --
>
> To unsubscribe e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Pyspark access to scala/java libraries

Mohit Jaggi
Thanks 0xF0F0F0 and Ashutosh for the pointers. 

Holden,
I am trying to look into sparklingml...what am I looking for? Also which chapter/page of your book should I look at?

Mohit.

On Sun, Jul 15, 2018 at 3:02 AM Holden Karau <[hidden email]> wrote:
If you want to see some examples in a library shows a way to do it - https://github.com/sparklingpandas/sparklingml and high performance spark also talks about it.

On Sun, Jul 15, 2018, 11:57 AM <[hidden email]> wrote:
Check https://stackoverflow.com/questions/31684842/calling-java-scala-function-from-a-task

​Sent with ProtonMail Secure Email.​

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On July 15, 2018 8:01 AM, Mohit Jaggi <[hidden email]> wrote:

> Trying again…anyone know how to make this work?
>
> > On Jul 9, 2018, at 3:45 PM, Mohit Jaggi [hidden email] wrote:
> >
> > Folks,
> >
> > I am writing some Scala/Java code and want it to be usable from pyspark.
> >
> > For example:
> >
> > class MyStuff(addend: Int) {
> >
> > def myMapFunction(x: Int) = x + addend
> >
> > }
> >
> > I want to call it from pyspark as:
> >
> > df = ...
> >
> > mystuff = sc._jvm.MyStuff(5)
> >
> > df[‘x’].map(lambda x: mystuff.myMapFunction(x))
> >
> > How can I do this?
> >
> > Mohit.
>
> --
>
> To unsubscribe e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Pyspark access to scala/java libraries

Harsh
Hi

You can access your java packages using following in pySpark 

obj = sc._jvm.yourPackage.className()


Kind Regards
Harsh Takkar

On Wed, Jul 18, 2018 at 4:00 AM Mohit Jaggi <[hidden email]> wrote:
Thanks 0xF0F0F0 and Ashutosh for the pointers. 

Holden,
I am trying to look into sparklingml...what am I looking for? Also which chapter/page of your book should I look at?

Mohit.

On Sun, Jul 15, 2018 at 3:02 AM Holden Karau <[hidden email]> wrote:
If you want to see some examples in a library shows a way to do it - https://github.com/sparklingpandas/sparklingml and high performance spark also talks about it.

On Sun, Jul 15, 2018, 11:57 AM <[hidden email]> wrote:
Check https://stackoverflow.com/questions/31684842/calling-java-scala-function-from-a-task

​Sent with ProtonMail Secure Email.​

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On July 15, 2018 8:01 AM, Mohit Jaggi <[hidden email]> wrote:

> Trying again…anyone know how to make this work?
>
> > On Jul 9, 2018, at 3:45 PM, Mohit Jaggi [hidden email] wrote:
> >
> > Folks,
> >
> > I am writing some Scala/Java code and want it to be usable from pyspark.
> >
> > For example:
> >
> > class MyStuff(addend: Int) {
> >
> > def myMapFunction(x: Int) = x + addend
> >
> > }
> >
> > I want to call it from pyspark as:
> >
> > df = ...
> >
> > mystuff = sc._jvm.MyStuff(5)
> >
> > df[‘x’].map(lambda x: mystuff.myMapFunction(x))
> >
> > How can I do this?
> >
> > Mohit.
>
> --
>
> To unsubscribe e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]