The application executes ok but one problem i am facing is when i go to Spark Web UI and go to Application: page and then click on `Application Detail UI`, it doesn't work
What i noticed is when i call the spark-submit from jenkins server the Application Details UI link points to jenkinsurl:4040 . If i execute the spark-submit from master node then it points to masternode:4040 and if i execute it from worker node 1 it takes me to workernode1:4040, instead of pointing to driver:4040
Please note that i am setting SPARK_PUBLIC_DNS= parameter with value of ec2 public hostname in spark-env.sh on each of these machines.
I was wondering how does spark figure out value of Application Details UI, i noticed that in SparkUI.scala it gets protected val publicHostName = Option(conf.getenv("SPARK_PUBLIC_DNS")).getOrElse(localHostName)
So it should have got the value from spark-env.sh of machine where driver is running but seems like it always points to value of SPARK_PUBLIC_DNS of the machine from which i invoked spark-submit.sh