Multiply Matrix to it's transpose get undesired output

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Multiply Matrix to it's transpose get undesired output

Soheil Pourbafrani
Hi,

I want to compute the cosine similarities of vectors using apache spark. In a simple example, I created a vector from each document using built-in tf-idf. Here is the code:
hashingTF = HashingTF(inputCol="tokenized", outputCol="tf")
tf = hashingTF.transform(df)

idf = IDF(inputCol="tf", outputCol="feature").fit(tf)
tfidf = idf.transform(tf)

normalizer = Normalizer(inputCol="feature", outputCol="norm")
data = normalizer.transform(tfidf)
mat = IndexedRowMatrix(
data.select("id", "norm")\
.rdd.map(lambda row: IndexedRow(row.id, row.norm.toArray()))).toBlockMatrix()
dot = mat.multiply(mat.transpose())
In the output, I expect it generates a matrix with Matrix diagonal of 1 (because each vector's similarity to itself is one) and its Matrix diagonal is one, too (as desired).
[[1. 0.]
 [0. 1.]]

The problem is when I want to weight words in the vector space to something other than typical TF-IDF. So I compute the vector space and create a vector for each document that the index of document's words has new weights and other than has weights zero.
for example the following vector is for document id 0.
(0, [9.0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3.3010299956639813, 3.3010299956639813, 0, 3.3010299956639813, 0, 3.3010299956639813])

The problem is when I try to compute cosine similarity of the matrix it didn't produce the correct answer because the similarity of a document to itself is not 1:
mat = IndexedRowMatrix(
final_vectors.map(lambda row: IndexedRow(row[0], row[1]))).toBlockMatrix()
dot = mat.multiply(mat.transpose())
the output for the same dataset is :
[[124.58719613  81.        ]
 [ 81.         407.90397097]]

while with Spark TF-IDF approach it was :
[[1. 0.]
 [0. 1.]]

Where is wrong in my approach?