Spark in Docker container?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Spark in Docker container?

lola
This post has NOT been accepted by the mailing list yet.
Hello :)

I am a student and I want to try using Spark for analysing big data. So, I have a data set that I will use it for the analysis, and I have MacBook Pro.

I want to create a cluster with multi node and applying frequent patern algorithme on the data set I have.I want to know the steps to do this.

 So, should I install Docker and create 3 or 4 containers in my MacBook and install Spark then execute the algorithme in these containers?

Thank you in advance 😊
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark in Docker container?

yncxcw
This post has NOT been accepted by the mailing list yet.
hi, 

I did not get there is a need to use docker in your data analysis work.
If it is simple for data analysis work, you just need to install a spark on your mac and load in the data from your local disk for you analysis purpose.

If you are to use Docker to simulate a multi-nodes cluster, first you may need to install Docker and then create 3-4 containers. Treating each container as a VM node and install Java, Spark on each node. In a distributed environment, you may also have to install HDFS as backend data storage. 


Wei







On Sat, Jul 8, 2017 at 7:44 AM, lola [via Apache Spark User List] <[hidden email]> wrote:
Hello :)

I am a student and I want to try using Spark for analysing big data. So, I have a data set that I will use it for the analysis, and I have MacBook Pro.

I want to create a cluster with multi node and applying frequent patern algorithme on the data set I have.I want to know the steps to do this.

 So, should I install Docker and create 3 or 4 containers in my MacBook and install Spark then execute the algorithme in these containers?

Thank you in advance 😊


If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-in-Docker-container-tp28833.html
To start a new topic under Apache Spark User List, email [hidden email]
To unsubscribe from Apache Spark User List, click here.
NAML

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark in Docker container?

lola
This post has NOT been accepted by the mailing list yet.
Thank you for replying :-)
Yes I want to apply the algorithme on a multi node cluster, just for exploring how it works,
I will try to do this :)
Loading...