Bug Report: Spark Config Defaults not Loading with python code/spark-submit

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Bug Report: Spark Config Defaults not Loading with python code/spark-submit

Nathan McLean
Here is an example pyspark program which illustrates this problem. If run using spark-submit, the default configurations for Spark do not seem to be loaded when a new SparkConf class is instantiated (contrary to what the loadDefaults=True keyword arg implies). When using the interactive shell for pyspark, the default configurations are loaded as expected.

I also tested this behaviour with a simple Scala program, but the Scala code behaved correctly.


#!/usr/bin/python2.7
from pyspark.conf import SparkConf
from pyspark import SparkContext


# this prints nothing
print 'SPARK DEFAULTS'
conf = SparkConf()
print conf.toDebugString()

# this prints configuration options
print 'SPARK DEFAULTS'
spark_context = SparkContext()
conf = spark_context.getConf()
print conf.toDebugString()


This bug does not seem to exist in Spark 1.6.x
I have reproduced it in Spark 2.1.1 and Spark 2.2.0