Spark as computing engine vs spark cluster

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark as computing engine vs spark cluster

Santosh74
Is spark compute engine only or it's also cluster which comes with set of
hardware /nodes  ? What exactly is spark clusterr?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark as computing engine vs spark cluster

Jeff Evans
Spark is a computation engine that runs on a set of distributed nodes.  You must "bring your own" hardware, although of course there are hosted solutions available.

On Sat, Oct 10, 2020 at 9:24 AM Santosh74 <[hidden email]> wrote:
Is spark compute engine only or it's also cluster which comes with set of
hardware /nodes  ? What exactly is spark clusterr?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark as computing engine vs spark cluster

Mich Talebzadeh
In reply to this post by Santosh74
Hi Santosh,

Generally speaking, there are two ways of making a process faster:

  1. Do more intelligent work by creating indexes, cubes etc thus reducing the processing time
  2. Throw hardware and memory at it using something like Spark multi-cluster with fully managed cloud service like Google Dataproc
So the framework is a computational engine (Spark) and the physical realisation is achieved by creating a Spark cluster (multi nodes/VM hosts) that work in tandem and provide parallel processing. I suggest that you look at Spark docs  

HTH



LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 



On Sat, 10 Oct 2020 at 15:24, Santosh74 <[hidden email]> wrote:
Is spark compute engine only or it's also cluster which comes with set of
hardware /nodes  ? What exactly is spark clusterr?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark as computing engine vs spark cluster

Kushagra Deep

Hi Santosh,

 

Spark is a distributed computation engine . You may ask why distributed ? The answer is when things are distributed, memory and cores can be increased to process parallely on scale . Since it is difficult to scale things vertically we scale horizontally.

 

Thanks And Regards

Kushagra Deep

 

From: Mich Talebzadeh <[hidden email]>
Date: Monday, 12 October 2020 at 11:23 PM
To: Santosh74 <[hidden email]>
Cc: "user @spark" <[hidden email]>
Subject: Re: Spark as computing engine vs spark cluster

 

Hi Santosh,

 

Generally speaking, there are two ways of making a process faster:

 

1.       Do more intelligent work by creating indexes, cubes etc thus reducing the processing time

2.       Throw hardware and memory at it using something like Spark multi-cluster with fully managed cloud service like Google Dataproc

So the framework is a computational engine (Spark) and the physical realisation is achieved by creating a Spark cluster (multi nodes/VM hosts) that work in tandem and provide parallel processing. I suggest that you look at Spark docs  

 

HTH

 

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

 

 

Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 

 

 

On Sat, 10 Oct 2020 at 15:24, Santosh74 <[hidden email]> wrote:

Is spark compute engine only or it's also cluster which comes with set of
hardware /nodes  ? What exactly is spark clusterr?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]