[ML] [How-to]: How to unload the loaded W2V model in Pyspark?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[ML] [How-to]: How to unload the loaded W2V model in Pyspark?

Zhefu PENG
Hi all,

I'm using pyspark and Spark-ml to train and use Word2Vect model, here is the logic of my program:

model = Word2VecModel.load("save path")

result_list = model.findSynonymsArray(target, top_N)

Then I use the graphframe and result_list to create graph and do some computing. However the program failed due to the out of memory error: xxx is running beyond physical memory limits. As a result, I want to delete the word2vec model to free memory, since I don't need to use it after getting the result_list.

I tried using del function in Python, and 
spark.sparkContext._gateway.detach(model._java_obj)

Is there anyway to unload or delete the loaded w2v model in Spark or Pyspark? 

Really appreciate for any reply and help.

Best,
Zhefu