how to multiply large matrix on spark

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

how to multiply large matrix on spark
This post has NOT been accepted by the mailing list yet.
   i do a algorithm that use spark to multiply two large matrix.

   i use the function org.apache.spark.mllib.linalg.distributed.BlockMatrix.multiply,but i found the function use other function "simulateMultiply" that have the code "val leftMatrix=blockInfo.keys.collection()".
the collection function will return all data to driver so OOM is appear.
   i think maybe there is another way to do this.
   can you help me !