Issue in parallelization of CNN model using spark

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Issue in parallelization of CNN model using spark

Mukhtaj Khan
Dear Spark User

I am trying to parallelize the CNN (convolutional neural network) model using spark. I have developed the model using python and Keras library. The model works fine on a single machine but when we try on multiple machines, the execution time remains the same as sequential.
Could you please tell me that there is any built-in library for CNN to parallelize in spark framework. Moreover, MLLIB does not have any support for CNN.
Best regards
Mukhtaj

  


Reply | Threaded
Open this post in threaded view
|

Re: Issue in parallelization of CNN model using spark

Juan Martín Guillén
Hi Mukhtaj,

Parallelization on Spark is abstracted on the DataFrame.

You can run anything locally on the driver but to make it run in parallel on the cluster you'll need to use the DataFrame abstraction.


Regards,
Juan Martín.


El lunes, 13 de julio de 2020 08:59:35 ART, Mukhtaj Khan <[hidden email]> escribió:


Dear Spark User

I am trying to parallelize the CNN (convolutional neural network) model using spark. I have developed the model using python and Keras library. The model works fine on a single machine but when we try on multiple machines, the execution time remains the same as sequential.
Could you please tell me that there is any built-in library for CNN to parallelize in spark framework. Moreover, MLLIB does not have any support for CNN.
Best regards
Mukhtaj

  


Reply | Threaded
Open this post in threaded view
|

Re: Issue in parallelization of CNN model using spark

srowen
In reply to this post by Mukhtaj Khan
There is a multilayer perceptron implementation in Spark ML, but
that's not what you're looking for.
To parallelize model training developed using standard libraries like
Keras, use Horovod from Uber.
https://horovod.readthedocs.io/en/stable/spark_include.html

On Mon, Jul 13, 2020 at 6:59 AM Mukhtaj Khan <[hidden email]> wrote:

>
> Dear Spark User
>
> I am trying to parallelize the CNN (convolutional neural network) model using spark. I have developed the model using python and Keras library. The model works fine on a single machine but when we try on multiple machines, the execution time remains the same as sequential.
> Could you please tell me that there is any built-in library for CNN to parallelize in spark framework. Moreover, MLLIB does not have any support for CNN.
> Best regards
> Mukhtaj
>
>
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Issue in parallelization of CNN model using spark

Anwar AliKhan
This is very useful for me leading on from week4 of the Andrew Ng course.


On Mon, 13 Jul 2020, 15:18 Sean Owen, <[hidden email]> wrote:
There is a multilayer perceptron implementation in Spark ML, but
that's not what you're looking for.
To parallelize model training developed using standard libraries like
Keras, use Horovod from Uber.
https://horovod.readthedocs.io/en/stable/spark_include.html

On Mon, Jul 13, 2020 at 6:59 AM Mukhtaj Khan <[hidden email]> wrote:
>
> Dear Spark User
>
> I am trying to parallelize the CNN (convolutional neural network) model using spark. I have developed the model using python and Keras library. The model works fine on a single machine but when we try on multiple machines, the execution time remains the same as sequential.
> Could you please tell me that there is any built-in library for CNN to parallelize in spark framework. Moreover, MLLIB does not have any support for CNN.
> Best regards
> Mukhtaj
>
>
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Issue in parallelization of CNN model using spark

Anwar AliKhan
In reply to this post by srowen
 link to a free book  which may be useful.

Hands-On Machine Learning with Scikit-Learn, Keras, and Tensorflow Concepts, Tools, and Techniques to Build Intelligent Systems by Aurélien Géron






 13 Jul 2020, 15:18 Sean Owen, <[hidden email]> wrote:
There is a multilayer perceptron implementation in Spark ML, but
that's not what you're looking for.
To parallelize model training developed using standard libraries like
Keras, use Horovod from Uber.
https://horovod.readthedocs.io/en/stable/spark_include.html

On Mon, Jul 13, 2020 at 6:59 AM Mukhtaj Khan <[hidden email]> wrote:
>
> Dear Spark User
>
> I am trying to parallelize the CNN (convolutional neural network) model using spark. I have developed the model using python and Keras library. The model works fine on a single machine but when we try on multiple machines, the execution time remains the same as sequential.
> Could you please tell me that there is any built-in library for CNN to parallelize in spark framework. Moreover, MLLIB does not have any support for CNN.
> Best regards
> Mukhtaj
>
>
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Issue in parallelization of CNN model using spark

Patrick McCarthy-2
Please don't advocate for piracy, this book is not freely available. 

I own it and it's wonderful, Mr. Géron deserves to benefit from it.

On Mon, Jul 13, 2020 at 9:59 PM Anwar AliKhan <[hidden email]> wrote:
 link to a free book  which may be useful.

Hands-On Machine Learning with Scikit-Learn, Keras, and Tensorflow Concepts, Tools, and Techniques to Build Intelligent Systems by Aurélien Géron






 13 Jul 2020, 15:18 Sean Owen, <[hidden email]> wrote:
There is a multilayer perceptron implementation in Spark ML, but
that's not what you're looking for.
To parallelize model training developed using standard libraries like
Keras, use Horovod from Uber.
https://horovod.readthedocs.io/en/stable/spark_include.html

On Mon, Jul 13, 2020 at 6:59 AM Mukhtaj Khan <[hidden email]> wrote:
>
> Dear Spark User
>
> I am trying to parallelize the CNN (convolutional neural network) model using spark. I have developed the model using python and Keras library. The model works fine on a single machine but when we try on multiple machines, the execution time remains the same as sequential.
> Could you please tell me that there is any built-in library for CNN to parallelize in spark framework. Moreover, MLLIB does not have any support for CNN.
> Best regards
> Mukhtaj
>
>
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--

Patrick McCarthy 

Senior Data Scientist, Machine Learning Engineering

Dstillery

470 Park Ave South, 17th Floor, NYC 10016

Reply | Threaded
Open this post in threaded view
|

Re: Issue in parallelization of CNN model using spark

Anwar AliKhan
Please note It is freely available because it is an early unedited raw edition. 
It is not 100% complete , it is not entirely same as yours.
So it is not piracy.
I agree it is a good book.







On Tue, 14 Jul 2020, 14:30 Patrick McCarthy, <[hidden email]> wrote:
Please don't advocate for piracy, this book is not freely available. 

I own it and it's wonderful, Mr. Géron deserves to benefit from it.

On Mon, Jul 13, 2020 at 9:59 PM Anwar AliKhan <[hidden email]> wrote:
 link to a free book  which may be useful.

Hands-On Machine Learning with Scikit-Learn, Keras, and Tensorflow Concepts, Tools, and Techniques to Build Intelligent Systems by Aurélien Géron






 13 Jul 2020, 15:18 Sean Owen, <[hidden email]> wrote:
There is a multilayer perceptron implementation in Spark ML, but
that's not what you're looking for.
To parallelize model training developed using standard libraries like
Keras, use Horovod from Uber.
https://horovod.readthedocs.io/en/stable/spark_include.html

On Mon, Jul 13, 2020 at 6:59 AM Mukhtaj Khan <[hidden email]> wrote:
>
> Dear Spark User
>
> I am trying to parallelize the CNN (convolutional neural network) model using spark. I have developed the model using python and Keras library. The model works fine on a single machine but when we try on multiple machines, the execution time remains the same as sequential.
> Could you please tell me that there is any built-in library for CNN to parallelize in spark framework. Moreover, MLLIB does not have any support for CNN.
> Best regards
> Mukhtaj
>
>
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--

Patrick McCarthy 

Senior Data Scientist, Machine Learning Engineering

Dstillery

470 Park Ave South, 17th Floor, NYC 10016

Reply | Threaded
Open this post in threaded view
|

Re: Issue in parallelization of CNN model using spark

srowen
It is still copyrighted material, no matter its state of editing. Yes,
you should not be sharing this on the internet.

On Tue, Jul 14, 2020 at 9:46 AM Anwar AliKhan <[hidden email]> wrote:
>
> Please note It is freely available because it is an early unedited raw edition.
> It is not 100% complete , it is not entirely same as yours.
> So it is not piracy.
> I agree it is a good book.
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Issue in parallelization of CNN model using spark

Anwar AliKhan
Ok, thanks. 
You can buy it here 


This book is like an accompaniment to the Andrew Ng course on coursera.
It uses exact same mathematical notations , examples etc. so it is a smooth transition from that courses.
  



On Tue, 14 Jul 2020, 15:52 Sean Owen, <[hidden email]> wrote:
It is still copyrighted material, no matter its state of editing. Yes,
you should not be sharing this on the internet.

On Tue, Jul 14, 2020 at 9:46 AM Anwar AliKhan <[hidden email]> wrote:
>
> Please note It is freely available because it is an early unedited raw edition.
> It is not 100% complete , it is not entirely same as yours.
> So it is not piracy.
> I agree it is a good book.
>
Reply | Threaded
Open this post in threaded view
|

Re: Issue in parallelization of CNN model using spark

Mukhtaj Khan
In reply to this post by Anwar AliKhan
Dear All
Thanks all of you for your reply. 
I am trying to parallelize the CNN model using Keras2DML library, however, I am getting the error message: NO Module Named Systemml.mllearn. Can any body guide me how to install systemml using ubuntu

best regards
 

On Tue, Jul 14, 2020 at 4:34 AM Anwar AliKhan <[hidden email]> wrote:
This is very useful for me leading on from week4 of the Andrew Ng course.


On Mon, 13 Jul 2020, 15:18 Sean Owen, <[hidden email]> wrote:
There is a multilayer perceptron implementation in Spark ML, but
that's not what you're looking for.
To parallelize model training developed using standard libraries like
Keras, use Horovod from Uber.
https://horovod.readthedocs.io/en/stable/spark_include.html

On Mon, Jul 13, 2020 at 6:59 AM Mukhtaj Khan <[hidden email]> wrote:
>
> Dear Spark User
>
> I am trying to parallelize the CNN (convolutional neural network) model using spark. I have developed the model using python and Keras library. The model works fine on a single machine but when we try on multiple machines, the execution time remains the same as sequential.
> Could you please tell me that there is any built-in library for CNN to parallelize in spark framework. Moreover, MLLIB does not have any support for CNN.
> Best regards
> Mukhtaj
>
>
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]