Spark master IP on Kubernetes

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Spark master IP on Kubernetes

ffarozan
This post has NOT been accepted by the mailing list yet.
We are deploying Spark on k8s cluster. We are facing one issue with respect to Spark master IP from a worker perspective. The Spark master is exposed as a service @ 10.3.0.175:7077.

Spark worker registers with the master, but saves the pod IP, instead of the service IP.

Following are related logs.
17/03/18 03:33:15 SPARK_WORKER INFO Utils: Successfully started service 'WorkerUI' on port 8081.
17/03/18 03:33:15 SPARK_WORKER INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://10.2.58.40:8081
17/03/18 03:33:15 SPARK_WORKER INFO Worker: Connecting to master 10.3.0.175:7077...
17/03/18 03:33:15 SPARK_WORKER INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1302ede4{/metrics/json,null,AVAILABLE}
17/03/18 03:33:15 SPARK_WORKER INFO TransportClientFactory: Successfully created connection to /10.3.0.175:7077 after 89 ms (0 ms spent in bootstraps)
17/03/18 03:33:15 SPARK_WORKER INFO Worker: Successfully registered with master spark://10.2.58.68:7077


As can be seen, what gets registered is the pod IP - spark://10.2.58.68:7077.

The issue due to this behavior is that when spark master pod dies or restarts, the spark worker does not act on the disconnected message, since it checks if the disconnected IP (here, we get service IP - 10.3.0.175) matches with the locally stored IP (in this case, it is 10.2.58.68).

Any suggestions how I can override this behavior, without changing spark code?

If this can be achieved only by changing spark behavior, please share ideas how to go about.
Loading...