shuffleMapTask problem

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

shuffleMapTask problem

Mina
This post has NOT been accepted by the mailing list yet.
Spark stops as shown in the picture and taking so long to perform when I do count action:



55/58 is shuffled and stopped. I have about 10million items.
How can i speed it up?
Reply | Threaded
Open this post in threaded view
|

Re: shuffleMapTask problem

Mina
This post has NOT been accepted by the mailing list yet.
Is it because of memory problem? It took a few seconds to reach 55/58 but it is spending too long to process the rest of tasks.