Do spark-submit overwrite the Spark session created manually?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Do spark-submit overwrite the Spark session created manually?

Yeikel

Hi Community ,

 

When we submit a job using ‘spark-submit’ passing options like the ‘master url’ what should be the content of the main class?

 

For example , if I create the session myself :

 

    val spark = SparkSession.builder.

      master("local[*]")

      .appName("Console")

      .config("spark.app.id", "spark-es")

      .getOrCreate()

 

Will spark-submit overwrite parameters like the master url , name , etc? 

 

Thank you

Reply | Threaded
Open this post in threaded view
|

Re: Do spark-submit overwrite the Spark session created manually?

Neo Chien
Hi 

According to official spec as below, I think the SparkSession.builder gets the highest priority to the configuration and it means the ‘spark-submit’ passing options would be ignored. Please correct me if I am wrong, many thanks.

Properties set directly on the SparkConf take highest precedence, then flags passed to spark-submit or spark-shell, then options in the spark-defaults.conf file.


Hope this helps,
Neo


<[hidden email]> 於 2019年1月1日 週二 上午6:04寫道:

Hi Community ,

 

When we submit a job using ‘spark-submit’ passing options like the ‘master url’ what should be the content of the main class?

 

For example , if I create the session myself :

 

    val spark = SparkSession.builder.

      master("local[*]")

      .appName("Console")

      .config("spark.app.id", "spark-es")

      .getOrCreate()

 

Will spark-submit overwrite parameters like the master url , name , etc? 

 

Thank you




Mailtrack Sender notified by
Mailtrack 19/01/01 下午3:43:08