Given a session/context, we can get the UI web URL like this:
This gives me something like http://node-name.cluster-name:4040. If
opening this from outside the cluster (ex: my laptop), this redirects
via HTTP 302 to something like
For discussion purposes, call the latter one the "final web URL".
Critically, this final URL is active even after the application
terminates. The original uiWebUrl
(http://node-name.cluster-name:4040) is not available after the
application terminates, so one has to have captured the redirect in
time, if they want to provide a persistent link to that history server
UI entry (ex: for debugging purposes).
Is there a way, other than using some HTTP client, to detect what this
final URL will be directly from the SparkContext?