Accessing UI for spark running as kubernetics container on standby name node
We are launching all spark jobs as kubernetics(k8es) containers inside a k8es cluster. We also create a service on each job and do port forwarding for the spark UI (container's 4040 is mapped to SvcPort 31123).
The same set of nodes is also hosting a Yarn cluster.
Inside container, we do spark-submit to Yarn in client mode.
Now, the spark container gets launched on any of the name nodes - master or standby. There is a VIP assigned to the active name node.
When spark container gets launched on active name node, spark UI and all its tabs are easily accessible from anywhere using VIP:SvcPort
However, when spark container gets launched on standby name node, spark UI is NOT accessible and eventually request fails with 500 error (as the VIP:SvcPort gets redirected to ActiveNamenode:8080 which is not accessible as driver is now running on standby name node.
I have tried to reason and try out multiple service configurations but nothing seems to work this scenario.
May anyone pls suggest proper k8es service config for this scenario. We do not want to restrict the spark containers to active name node as let k8es address the job load within k8es cluster.