spark on k8s driver pod exception

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

spark on k8s driver pod exception

040840219



when run the code in k8s ,  driver pod throw AnalysisException , but  the spark-submit log still  running , then how to get the exception and stop pods ?

val spark = SparkSession.builder().getOrCreate()
    import spark.implicits._
    val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key", "id%10 as value")
      .groupBy("key").agg(count("value1").as("cnt"))
    df.show()
spark.stop()

bin/spark-submit \
--master k8s://https://localhost:9443 \
--deploy-mode cluster \
--name wordcount \
--class k8s.WordCount \
--conf spark.kubernetes.container.image=rspark:v3.1.1 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \
/data/spark-example-1.0.0.jar
Reply | Threaded
Open this post in threaded view
|

Re: spark on k8s driver pod exception

Attila Zsolt Piros

For getting the logs please read Accessing Logs part of the Running Spark on Kubernetes page.

For stopping and generic management of the spark application please read the Spark Application Management, where you find the example:

$ spark-submit --kill spark:spark-pi* --master  k8s://https://192.168.2.8:8443


On Thu, Mar 11, 2021 at 1:07 PM yxl040840219 <[hidden email]> wrote:



when run the code in k8s ,  driver pod throw AnalysisException , but  the spark-submit log still  running , then how to get the exception and stop pods ?

val spark = SparkSession.builder().getOrCreate()
    import spark.implicits._
    val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key", "id%10 as value")
      .groupBy("key").agg(count("value1").as("cnt"))
    df.show()
spark.stop()

bin/spark-submit \
--master k8s://https://localhost:9443 \
--deploy-mode cluster \
--name wordcount \
--class k8s.WordCount \
--conf spark.kubernetes.container.image=rspark:v3.1.1 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \
/data/spark-example-1.0.0.jar
Reply | Threaded
Open this post in threaded view
|

Re: spark on k8s driver pod exception

Attila Zsolt Piros
but  the spark-submit log still  running

Set the "spark.kubernetes.submission.waitAppCompletion" config to false to change that. As the doc says:

"spark.kubernetes.submission.waitAppCompletion" : In cluster mode, whether to wait for the application to finish before exiting the launcher process. When changed to false, the launcher has a "fire-and-forget" behavior when launching the Spark job.

On Thu, Mar 11, 2021 at 10:05 PM Attila Zsolt Piros <[hidden email]> wrote:

For getting the logs please read Accessing Logs part of the Running Spark on Kubernetes page.

For stopping and generic management of the spark application please read the Spark Application Management, where you find the example:

$ spark-submit --kill spark:spark-pi* --master  k8s://https://192.168.2.8:8443


On Thu, Mar 11, 2021 at 1:07 PM yxl040840219 <[hidden email]> wrote:



when run the code in k8s ,  driver pod throw AnalysisException , but  the spark-submit log still  running , then how to get the exception and stop pods ?

val spark = SparkSession.builder().getOrCreate()
    import spark.implicits._
    val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key", "id%10 as value")
      .groupBy("key").agg(count("value1").as("cnt"))
    df.show()
spark.stop()

bin/spark-submit \
--master k8s://https://localhost:9443 \
--deploy-mode cluster \
--name wordcount \
--class k8s.WordCount \
--conf spark.kubernetes.container.image=rspark:v3.1.1 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \
/data/spark-example-1.0.0.jar
Reply | Threaded
Open this post in threaded view
|

Re: spark on k8s driver pod exception

040840219

when driver pod throws exception ,  driver pod still running   ?

kubectl logs  wordcount-e3141c7834d3dd68-driver

21/03/15 07:40:19 DEBUG Analyzer$ResolveReferences: Resolving 'value1 to 'value1
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`value1`' given input columns: [key, value];
'Aggregate [key#6], [key#6, count('value1) AS cnt#14]
+- Project [(id#4 % 5) AS key#6, (id#4 % 10) AS value#7]
   +- Project [value#1 AS id#4]
      +- LocalRelation [value#1]

at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:155)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:152)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:341)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73)

kubectl get pods wordcount-e3141c7834d3dd68-driver

NAME                                READY   STATUS    RESTARTS   AGE
wordcount-e3141c7834d3dd68-driver   1/1     Running   0          2m58s

On 03/12/2021 05:42[hidden email] wrote:
but  the spark-submit log still  running

Set the "spark.kubernetes.submission.waitAppCompletion" config to false to change that. As the doc says:

"spark.kubernetes.submission.waitAppCompletion" : In cluster mode, whether to wait for the application to finish before exiting the launcher process. When changed to false, the launcher has a "fire-and-forget" behavior when launching the Spark job.

On Thu, Mar 11, 2021 at 10:05 PM Attila Zsolt Piros <[hidden email]> wrote:

For getting the logs please read Accessing Logs part of the Running Spark on Kubernetes page.

For stopping and generic management of the spark application please read the Spark Application Management, where you find the example:

$ spark-submit --kill spark:spark-pi* --master  k8s://https://192.168.2.8:8443


On Thu, Mar 11, 2021 at 1:07 PM yxl040840219 <[hidden email]> wrote:



when run the code in k8s ,  driver pod throw AnalysisException , but  the spark-submit log still  running , then how to get the exception and stop pods ?

val spark = SparkSession.builder().getOrCreate()
    import spark.implicits._
    val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key", "id%10 as value")
      .groupBy("key").agg(count("value1").as("cnt"))
    df.show()
spark.stop()

bin/spark-submit \
--master k8s://https://localhost:9443 \
--deploy-mode cluster \
--name wordcount \
--class k8s.WordCount \
--conf spark.kubernetes.container.image=rspark:v3.1.1 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \
/data/spark-example-1.0.0.jar
Reply | Threaded
Open this post in threaded view
|

Re: spark on k8s driver pod exception

Attila Zsolt Piros
Sure, that is expected, see the "How it works" section in "Running Spark on Kubernetes" page, quote: 

When the application completes, the executor pods terminate and are cleaned up, but the driver pod persists logs and remains in “completed” state in the Kubernetes API until it’s eventually garbage collected or manually cleaned up.
 

On Mon, Mar 15, 2021 at 8:45 AM 040840219 <[hidden email]> wrote:

when driver pod throws exception ,  driver pod still running   ?

kubectl logs  wordcount-e3141c7834d3dd68-driver

21/03/15 07:40:19 DEBUG Analyzer$ResolveReferences: Resolving 'value1 to 'value1
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`value1`' given input columns: [key, value];
'Aggregate [key#6], [key#6, count('value1) AS cnt#14]
+- Project [(id#4 % 5) AS key#6, (id#4 % 10) AS value#7]
   +- Project [value#1 AS id#4]
      +- LocalRelation [value#1]

at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:155)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:152)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:341)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73)

kubectl get pods wordcount-e3141c7834d3dd68-driver

NAME                                READY   STATUS    RESTARTS   AGE
wordcount-e3141c7834d3dd68-driver   1/1     Running   0          2m58s

On 03/12/2021 05:42[hidden email] wrote:
but  the spark-submit log still  running

Set the "spark.kubernetes.submission.waitAppCompletion" config to false to change that. As the doc says:

"spark.kubernetes.submission.waitAppCompletion" : In cluster mode, whether to wait for the application to finish before exiting the launcher process. When changed to false, the launcher has a "fire-and-forget" behavior when launching the Spark job.

On Thu, Mar 11, 2021 at 10:05 PM Attila Zsolt Piros <[hidden email]> wrote:

For getting the logs please read Accessing Logs part of the Running Spark on Kubernetes page.

For stopping and generic management of the spark application please read the Spark Application Management, where you find the example:

$ spark-submit --kill spark:spark-pi* --master  k8s://https://192.168.2.8:8443


On Thu, Mar 11, 2021 at 1:07 PM yxl040840219 <[hidden email]> wrote:



when run the code in k8s ,  driver pod throw AnalysisException , but  the spark-submit log still  running , then how to get the exception and stop pods ?

val spark = SparkSession.builder().getOrCreate()
    import spark.implicits._
    val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key", "id%10 as value")
      .groupBy("key").agg(count("value1").as("cnt"))
    df.show()
spark.stop()

bin/spark-submit \
--master k8s://https://localhost:9443 \
--deploy-mode cluster \
--name wordcount \
--class k8s.WordCount \
--conf spark.kubernetes.container.image=rspark:v3.1.1 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \
/data/spark-example-1.0.0.jar