Understanding deploy mode config

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Understanding deploy mode config

Nikhil Goyal
Hi all,

In a pyspark application is the python process the driver or spark will start a new driver process? If it is the same as driver then how does specifying "spark.submit.deployMode" as "cluster" in  spark conf would come in use.

conf = SparkConf()
            .setMaster("yarn")
            .set("spark.submit.deployMode", "cluster")
sc = SparkContext(conf)

Is the spark context being created on application master or on the machine where this python process is being run?

Thanks
Nikhil