Using Custom Scala Spark ML Estimator in PySpark

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Using Custom Scala Spark ML Estimator in PySpark

Harsh
Hi , 

I have created a custom Estimator in scala, which i can use successfully by creating a pipeline model in Java and scala, But when i try to load the pipeline model saved using scala api in pyspark, i am getting an error saying module not found.

I have included my custom model jar in the class pass using "spark.jars"

Can you please help, if i am missing something. 

Kind Regards
Harsh Takkar
Reply | Threaded
Open this post in threaded view
|

Re: Using Custom Scala Spark ML Estimator in PySpark

Mich Talebzadeh
Hi,

Specifically is this a run time or compilation error.

I gather by class path you mean something like below

spark-submit --master yarn --deploy-mode client --driver-class-path <full_path_to_custom_jar>  --jars ......

HTH





LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 



On Tue, 16 Feb 2021 at 05:23, HARSH TAKKAR <[hidden email]> wrote:
Hi , 

I have created a custom Estimator in scala, which i can use successfully by creating a pipeline model in Java and scala, But when i try to load the pipeline model saved using scala api in pyspark, i am getting an error saying module not found.

I have included my custom model jar in the class pass using "spark.jars"

Can you please help, if i am missing something. 

Kind Regards
Harsh Takkar
Reply | Threaded
Open this post in threaded view
|

Re: Using Custom Scala Spark ML Estimator in PySpark

srowen
In reply to this post by Harsh
You won't be able to use it in python if it is implemented in Java - needs a python wrapper too. 

On Mon, Feb 15, 2021, 11:29 PM HARSH TAKKAR <[hidden email]> wrote:
Hi , 

I have created a custom Estimator in scala, which i can use successfully by creating a pipeline model in Java and scala, But when i try to load the pipeline model saved using scala api in pyspark, i am getting an error saying module not found.

I have included my custom model jar in the class pass using "spark.jars"

Can you please help, if i am missing something. 

Kind Regards
Harsh Takkar
Reply | Threaded
Open this post in threaded view
|

Re: Using Custom Scala Spark ML Estimator in PySpark

Harsh
Hello Sean,

Thanks for the advice, can you please point me to an example where i can find a custom wrapper for python.


Kind Regards
Harsh Takkar 

On Tue, 16 Feb, 2021, 8:25 pm Sean Owen, <[hidden email]> wrote:
You won't be able to use it in python if it is implemented in Java - needs a python wrapper too. 

On Mon, Feb 15, 2021, 11:29 PM HARSH TAKKAR <[hidden email]> wrote:
Hi , 

I have created a custom Estimator in scala, which i can use successfully by creating a pipeline model in Java and scala, But when i try to load the pipeline model saved using scala api in pyspark, i am getting an error saying module not found.

I have included my custom model jar in the class pass using "spark.jars"

Can you please help, if i am missing something. 

Kind Regards
Harsh Takkar