Query about Spark

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Query about Spark

Ankur Das

Good Evening Sir/Madam,
Hope you are doing well, I am experimenting on some ML techniques where I need to test it on a distributed environment.
For example a particular algorithm I want to run it on different nodes at the same time and collect the results at the end in one single node or the parent node.

So, I would like to know if it is possible or a good choice to use spark for this.

Hope to hear from you soon, Stay safe and healthy
Thanking you in advance.
--
Regards,
Ankur J Das
Research Scholar @ Tezpur University
Tezpur, Assam
Reply | Threaded
Open this post in threaded view
|

Re: Query about Spark

☼ R Nair (रविशंकर नायर)
Question is not clear..use accumulators, if I took it correctly.

Best, Ravion

On Sun, Sep 6, 2020, 9:41 AM Ankur Das <[hidden email]> wrote:

Good Evening Sir/Madam,
Hope you are doing well, I am experimenting on some ML techniques where I need to test it on a distributed environment.
For example a particular algorithm I want to run it on different nodes at the same time and collect the results at the end in one single node or the parent node.

So, I would like to know if it is possible or a good choice to use spark for this.

Hope to hear from you soon, Stay safe and healthy
Thanking you in advance.
--
Regards,
Ankur J Das
Research Scholar @ Tezpur University
Tezpur, Assam
Reply | Threaded
Open this post in threaded view
|

Re: Query about Spark

☼ R Nair (रविशंकर नायर)
Or use MLFlow's PySpark UDF. First create a mlflow.pyfunc.

Best, Ravion

On Sun, Sep 6, 2020, 9:43 AM ☼ R Nair <[hidden email]> wrote:
Question is not clear..use accumulators, if I took it correctly.

Best, Ravion

On Sun, Sep 6, 2020, 9:41 AM Ankur Das <[hidden email]> wrote:

Good Evening Sir/Madam,
Hope you are doing well, I am experimenting on some ML techniques where I need to test it on a distributed environment.
For example a particular algorithm I want to run it on different nodes at the same time and collect the results at the end in one single node or the parent node.

So, I would like to know if it is possible or a good choice to use spark for this.

Hope to hear from you soon, Stay safe and healthy
Thanking you in advance.
--
Regards,
Ankur J Das
Research Scholar @ Tezpur University
Tezpur, Assam
Reply | Threaded
Open this post in threaded view
|

Re: Query about Spark

Ankur Das
Thanks, I'll check it out. 

On Sun, Sep 6, 2020 at 7:15 PM ☼ R Nair <[hidden email]> wrote:
Or use MLFlow's PySpark UDF. First create a mlflow.pyfunc.

Best, Ravion

On Sun, Sep 6, 2020, 9:43 AM ☼ R Nair <[hidden email]> wrote:
Question is not clear..use accumulators, if I took it correctly.

Best, Ravion

On Sun, Sep 6, 2020, 9:41 AM Ankur Das <[hidden email]> wrote:

Good Evening Sir/Madam,
Hope you are doing well, I am experimenting on some ML techniques where I need to test it on a distributed environment.
For example a particular algorithm I want to run it on different nodes at the same time and collect the results at the end in one single node or the parent node.

So, I would like to know if it is possible or a good choice to use spark for this.

Hope to hear from you soon, Stay safe and healthy
Thanking you in advance.
--
Regards,
Ankur J Das
Research Scholar @ Tezpur University
Tezpur, Assam


--
Regards,
Ankur J Das
Research Scholar @ Tezpur University
Tezpur, Assam
Reply | Threaded
Open this post in threaded view
|

Re: Query about Spark

Enrico Minack
In reply to this post by Ankur Das
You could use Horovod to distribute your ML algorithm on a cluster, while Horovod also supports Spark clusters.

Enrico


Am 06.09.20 um 15:30 schrieb Ankur Das:

Good Evening Sir/Madam,
Hope you are doing well, I am experimenting on some ML techniques where I need to test it on a distributed environment.
For example a particular algorithm I want to run it on different nodes at the same time and collect the results at the end in one single node or the parent node.

So, I would like to know if it is possible or a good choice to use spark for this.

Hope to hear from you soon, Stay safe and healthy
Thanking you in advance.
--
Regards,
Ankur J Das
Research Scholar @ Tezpur University
Tezpur, Assam


Reply | Threaded
Open this post in threaded view
|

Re: Query about Spark

☼ R Nair (रविशंकर नायर)

On Mon, Sep 7, 2020, 8:29 AM Enrico Minack <[hidden email]> wrote:
You could use Horovod to distribute your ML algorithm on a cluster, while Horovod also supports Spark clusters.

Enrico


Am 06.09.20 um 15:30 schrieb Ankur Das:

Good Evening Sir/Madam,
Hope you are doing well, I am experimenting on some ML techniques where I need to test it on a distributed environment.
For example a particular algorithm I want to run it on different nodes at the same time and collect the results at the end in one single node or the parent node.

So, I would like to know if it is possible or a good choice to use spark for this.

Hope to hear from you soon, Stay safe and healthy
Thanking you in advance.
--
Regards,
Ankur J Das
Research Scholar @ Tezpur University
Tezpur, Assam


Reply | Threaded
Open this post in threaded view
|

Re: Query about Spark

Ankur Das
Thanks a lot. 

On Mon, Sep 7, 2020 at 8:36 PM ☼ R Nair <[hidden email]> wrote:

On Mon, Sep 7, 2020, 8:29 AM Enrico Minack <[hidden email]> wrote:
You could use Horovod to distribute your ML algorithm on a cluster, while Horovod also supports Spark clusters.

Enrico


Am 06.09.20 um 15:30 schrieb Ankur Das:

Good Evening Sir/Madam,
Hope you are doing well, I am experimenting on some ML techniques where I need to test it on a distributed environment.
For example a particular algorithm I want to run it on different nodes at the same time and collect the results at the end in one single node or the parent node.

So, I would like to know if it is possible or a good choice to use spark for this.

Hope to hear from you soon, Stay safe and healthy
Thanking you in advance.
--
Regards,
Ankur J Das
Research Scholar @ Tezpur University
Tezpur, Assam




--
Regards,
Ankur J Das
Research Scholar @ Tezpur University
Tezpur, Assam