Pickling Keras models for use in UDFs

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Pickling Keras models for use in UDFs

erp12
I would like to create a Spark UDF which returns the a prediction made with a
trained Keras model. Keras models are not typically pickle-able, however I
have used the monkey patch approach to making Keras models pickle-able, as
described here: http://zachmoshe.com/2017/04/03/pickling-keras-models.html

This allows for models to be sent from the PySpark driver to the workers,
however the worker python processes do not have the monkey patched Model
class, and thus cannot properly un-pickle the models. To fix this issue, I
know I must call the monkey patching function (make_keras_picklable()) once
on each worker, however I have been unable to figure out how to do this.

I am curious to hear if anyone has a fix for this issue, or would like to
offer an alternative way to make predictions with a Keras model within a
Spark UDF.

Here is a Stack Overflow question with more details:
https://stackoverflow.com/questions/50007126/pickling-monkey-patched-keras-model-for-use-in-pyspark

Thank you!



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Pickling Keras models for use in UDFs

Khaled Zaouk
Why don't you try to encapsulate your keras model within a wrapper class (an estimator let's say), and you implement inside this wrapper class the two functions: __getstate__ and __setstate__

On Thu, May 3, 2018 at 5:27 PM erp12 <[hidden email]> wrote:
I would like to create a Spark UDF which returns the a prediction made with a
trained Keras model. Keras models are not typically pickle-able, however I
have used the monkey patch approach to making Keras models pickle-able, as
described here: http://zachmoshe.com/2017/04/03/pickling-keras-models.html

This allows for models to be sent from the PySpark driver to the workers,
however the worker python processes do not have the monkey patched Model
class, and thus cannot properly un-pickle the models. To fix this issue, I
know I must call the monkey patching function (make_keras_picklable()) once
on each worker, however I have been unable to figure out how to do this.

I am curious to hear if anyone has a fix for this issue, or would like to
offer an alternative way to make predictions with a Keras model within a
Spark UDF.

Here is a Stack Overflow question with more details:
https://stackoverflow.com/questions/50007126/pickling-monkey-patched-keras-model-for-use-in-pyspark

Thank you!



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Pickling Keras models for use in UDFs

erp12
Great idea! That works perfectly, thank you!



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]