Need help with Application Detail UI link

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Need help with Application Detail UI link


I am running Standalone spark server in EC2. My cluster has 1 master and 16
worker nodes. I have a jenkins server that calls spark-submit command like

/mnt/services/spark/bin/spark-submit --master spark://
--deploy-mode cluster --conf spark.driver.maxResultSize=1g   
--executor-cores 4     --total-executor-cores 4     --executor-memory 5g   
--driver-memory 1g     --class com.test.HelloSpark /opt/helloworld.jar   

The application executes ok but one problem i am facing is when i go to
Spark Web UI and go to Application: page and then click on `Application
Detail UI`, it doesn't work

What i noticed is when i call the spark-submit from jenkins server the
Application Details UI link points to jenkinsurl:4040 . If i execute the
spark-submit from master node then it points to masternode:4040 and if i
execute it from worker node 1 it takes me to workernode1:4040, instead of
pointing to driver:4040

Please note that  i am setting SPARK_PUBLIC_DNS= parameter with value of ec2
public hostname in on each of these machines.

I was wondering how does spark figure out value of Application Details UI, i
noticed that in SparkUI.scala it gets
  protected val publicHostName =

So it should have got the value from of machine where driver is
running but seems like it always points to value of SPARK_PUBLIC_DNS of the
machine from which i invoked