Spark UI stdout/stderr links point to executors internal address

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark UI stdout/stderr links point to executors internal address

Jhon Anderson Cardenas Diaz

Environment:

AWS EMR, yarn cluster.

 

Description:

On Spark ui, in Environment and Executors tabs, the links of stdout and stderr point to the internal address of the executors. This would imply to expose the executors so that links can be accessed. Shouldn't those links be pointed to master then handled internally serving the master as a proxy for these files instead of exposing the internal machines?

 

I have tried setting SPARK_PUBLIC_DNS and SPARK_LOCAL_IP variables so they contain the master ip address. I also tried with this properties: spark.yarn.appMasterEnv.SPARK_LOCAL_IP and spark.yarn.appMasterEnv.SPARK_PUBLIC_DNS but it does not seem to work.


Any suggestion?