How/where to set the hostname used by the spark workers?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How/where to set the hostname used by the spark workers?

maasg
Hi,

I'm trying to run Spark in Docker, using the amplab docker scripts (which I've been modifying  to support 0.9.0)

I'm trying to use Docker's own link facility instead of the provided DNS service to have master-worker communication, using plain IP addresses.

Right now, the master is working fine, but the workers are picking up the hostname when they build the remote actor address:

INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkWorker@devworker:54621]

Where 'devworker' is the name given to  the docker container and non-routable from other containers. 

For the master, by setting  `SPARK_MASTER_IP` on the `spark_env.sh`, it's working fine: 
INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@172.17.0.41:7077]

Yet, there's no SPARK_WORKER_IP option there.

How could I instruct the spark worker to use a given ip address in a similar fashion?

Thanks,

Gerard.
Reply | Threaded
Open this post in threaded view
|

Re: How/where to set the hostname used by the spark workers?

ssimanta
try setting worker/slaves ip on $SPARK_HOME/conf/slaves 


On Tue, Feb 11, 2014 at 1:14 PM, Gerard Maas <[hidden email]> wrote:
Hi,

I'm trying to run Spark in Docker, using the amplab docker scripts (which I've been modifying  to support 0.9.0)

I'm trying to use Docker's own link facility instead of the provided DNS service to have master-worker communication, using plain IP addresses.

Right now, the master is working fine, but the workers are picking up the hostname when they build the remote actor address:

INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkWorker@devworker:54621]

Where 'devworker' is the name given to  the docker container and non-routable from other containers. 

For the master, by setting  `SPARK_MASTER_IP` on the `spark_env.sh`, it's working fine: 
INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@172.17.0.41:7077]

Yet, there's no SPARK_WORKER_IP option there.

How could I instruct the spark worker to use a given ip address in a similar fashion?

Thanks,

Gerard.

Reply | Threaded
Open this post in threaded view
|

Re: How/where to set the hostname used by the spark workers?

maasg
Hi,

Thanks.  For the record, I found a solution by passing the ip address as parameter:

spark-class org.apache.spark.deploy.worker.Worker $MASTER -i $1 

-kr, Gerard.



On Tue, Feb 11, 2014 at 7:20 PM, Soumya Simanta <[hidden email]> wrote:
try setting worker/slaves ip on $SPARK_HOME/conf/slaves 


On Tue, Feb 11, 2014 at 1:14 PM, Gerard Maas <[hidden email]> wrote:
Hi,

I'm trying to run Spark in Docker, using the amplab docker scripts (which I've been modifying  to support 0.9.0)

I'm trying to use Docker's own link facility instead of the provided DNS service to have master-worker communication, using plain IP addresses.

Right now, the master is working fine, but the workers are picking up the hostname when they build the remote actor address:

INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkWorker@devworker:54621]

Where 'devworker' is the name given to  the docker container and non-routable from other containers. 

For the master, by setting  `SPARK_MASTER_IP` on the `spark_env.sh`, it's working fine: 
INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@172.17.0.41:7077]

Yet, there's no SPARK_WORKER_IP option there.

How could I instruct the spark worker to use a given ip address in a similar fashion?

Thanks,

Gerard.