Hi,
I am not using Airflow but I assume your application is deployed in cluster
mode and in this case the class you are looking for is
"org.apache.spark.deploy.k8s.submit.Client" [1].
If we are talking about the first "spark-submit" used to start the
application and not "spark-submit --status" then it contains loop where the
application status is logged. This loop stops when the
"LoggingPodStatusWatcher" reports the app is completed [2] or when
"spark.kubernetes.submission.waitAppCompletion" [3] is false.
And you are right the monitoring (POD state watching) is done via REST
(HTTPS) and should be detected by
"io.fabric8.kubernetes.client.Watcher.onClose()" method so by the kubernetes
client.
I hope this helps. Some further questions if you need some more help:
1. What is the Spark version you are running?
2. Does it contain SPARK-24266 [4]?
3. If yes can you reproduce the issue without airflow and do you have the
logs about the issue?
Best regards,
Attila
[1]
https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L88-L103[2]
https://github.com/apache/spark/blob/8604db28b87b387bbdb3761df85fae292cd402a1/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L162-L166[3]
https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala#L112-L114[4]
https://issues.apache.org/jira/browse/SPARK-24266--
Sent from:
http://apache-spark-user-list.1001560.n3.nabble.com/---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org