Computation time increasing every super-step

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Computation time increasing every super-step

alelulli
Hi All,

I'm facing a performance degradation running an iterative algorithm built using Spark 0.9 and GraphX.
I'm usingĀ org.apache.spark.graphx.Pregel to run the iterative algorithm.

My graph hasĀ 2395 vertex 7462 edges.

Every super step the computation time increase significantly. The steps 1-5 are executed in the order of seconds instead steps > 10 are executed in the order of tens of minutes and always increasing.

In every step each vertex executes always the same actions and sends a message to all of its neighbor. The graph doesn't change topology during execution.

I tried also to perform a checkpoint of vertices, edges and triplets at the end of each step but i'm encountering the same issue.

Could you please help me solve this issue?
Please let me know if i'm missing something or you need additional details.

Thanks
Alessandro