Quantcast

How to deploy a Spark client outside of the cdh cluster

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
min
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to deploy a Spark client outside of the cdh cluster

min
This post has NOT been accepted by the mailing list yet.
hi,i deploy a spark client outside of cluster, and i modify the spark-env.sh, add the following lines:
export HADOOP_HOME=/home/bigdata/deploy/hadoop-2.5.0-cdh5.3.8
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_CLASSPATH=$HADOOP_HOME/share/hadoop/common/lib/*

, then i run spark streaming application on this node, but i find some errors:(10.20.216.136 is not a node in cluster, is a client node)
17/04/13 18:22:50 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.20.216.136:53779, retrying ...
17/04/13 18:22:50 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.20.216.136:53779, retrying ...
17/04/13 18:22:51 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.20.216.136:53779, retrying ...
17/04/13 18:22:51 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.20.216.136:53779, retrying ...
17/04/13 18:22:51 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.20.216.136:53779, retrying ...
17/04/13 18:22:51 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.20.216.136:53779, retrying ...
17/04/13 18:22:52 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.20.216.136:53779, retrying ...
17/04/13 18:22:53 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.20.216.136:53779, retrying ...
17/04/13 18:22:54 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.20.216.136:53779, retrying ...
17/04/13 18:22:55 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10.20.216.136:53779, retrying ...
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to deploy a Spark client outside of the cdh cluster

Adopod
This post has NOT been accepted by the mailing list yet.
You can try to add "--conf spark.driver.host=$your_ip_address" behind the command
min
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to deploy a Spark client outside of the cdh cluster

min
This post has NOT been accepted by the mailing list yet.
it's not work
Loading...