How does shuffle operation work in Spark?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How does shuffle operation work in Spark?

Joe-2
Hello,
I'm looking for a detailed description of the shuffle operation in
Spark, something that would explain what are the criteria for assigning
blocks to nodes, how many go where, what happens when there are memory
constraints, etc.
If anyone knows of such a document I'd appreciate a link (or a detailed
answer).
Thanks a lot,

Joe


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]