[How To] Using Spark Session in internal called classes

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[How To] Using Spark Session in internal called classes

Aakash Basu-2
Hi,

I have created my own Model Tuner class which I want to use to tune models and return a Model object if the user expects. This Model Tuner is in a file which I would ideally import into another file and call the class and use it.

Outer file {from where I'd be calling the Model Tuner): I am using SparkSession to start my Spark job and then read a file, do some basic operations to prepare it as a Dense Vector and then send it into the Model to tune.

While I send it into the model, I get this error:

AttributeError: Cannot load _jvm from SparkContext. Is SparkContext initialized?


The full traceback:

Traceback (most recent call last):
  File "/home/aakashbasu/PycharmProjects/bb_lite/2.0 backend/model_tuning/aakash_test.py", line 9, in <module>
    from model_tuning.model_tuning_newer import ModelTuner
  File "/home/aakashbasu/PycharmProjects/bb_lite/2.0 backend/model_tuning/model_tuning.py", line 13, in <module>
    labelCol="label", predictionCol="prediction", metricName="accuracy")
  File "/usr/local/lib/python3.5/dist-packages/pyspark/__init__.py", line 105, in wrapper
    return func(self, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/pyspark/ml/evaluation.py", line 298, in __init__
    "org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator", self.uid)
  File "/usr/local/lib/python3.5/dist-packages/pyspark/ml/wrapper.py", line 59, in _new_java_obj
    java_obj = _jvm()
  File "/usr/local/lib/python3.5/dist-packages/pyspark/ml/util.py", line 44, in _jvm
    raise AttributeError("Cannot load _jvm from SparkContext. Is SparkContext initialized?")
AttributeError: Cannot load _jvm from SparkContext. Is SparkContext initialized?



How to ensure that the inner file also uses the same Spark Context which is initiated by the outer file using SparkSession builder?

Any help?

Thanks,
Aakash.