Bug Report: Spark Config Defaults not Loading with python code/spark-submit
Here is an example pyspark program which illustrates this problem. If run using spark-submit, the default configurations for Spark do not seem to be loaded when a new SparkConf class is instantiated (contrary to what the loadDefaults=True keyword arg implies). When using the interactive shell for pyspark, the default configurations are loaded as expected.
I also tested this behaviour with a simple Scala program, but the Scala code behaved correctly.
#!/usr/bin/python2.7 from pyspark.conf import SparkConf from pyspark import SparkContext
# this prints nothing print 'SPARK DEFAULTS' conf = SparkConf() print conf.toDebugString()
# this prints configuration options print 'SPARK DEFAULTS' spark_context = SparkContext() conf = spark_context.getConf() print conf.toDebugString()
This bug does not seem to exist in Spark 1.6.x I have reproduced it in Spark 2.1.1 and Spark 2.2.0