question about barrier execution mode in Spark 2.4.0

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

question about barrier execution mode in Spark 2.4.0

Joe-2
Hello,
I was reading Spark 2.4.0 release docs and I'd like to find out more
about barrier execution mode.
In particular I'd like to know what happens when number of partitions
exceeds number of nodes (which I think is allowed, Spark tuning doc
mentions that)?
Does Spark guarantee that all tasks process all partitions
simultaneously? If not then how does barrier mode handle partitions that
are waiting to be processed?
If there are partitions waiting to be processed then I don't think it's
possible to send all data from given stage to a DL process, even when
using barrier mode?
Thanks a lot,

Joe


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: question about barrier execution mode in Spark 2.4.0

Xiangrui Meng


On Mon, Nov 12, 2018 at 7:33 AM Joe <[hidden email]> wrote:
Hello,
I was reading Spark 2.4.0 release docs and I'd like to find out more
about barrier execution mode.
In particular I'd like to know what happens when number of partitions
exceeds number of nodes (which I think is allowed, Spark tuning doc
mentions that)?

The barrier execution mode is different. It needs to run tasks for all partitions together. So when the number of partitions is greater than number of nodes, it will wait until more nodes are available and print warning messages.
 
Does Spark guarantee that all tasks process all partitions
simultaneously?

They will start all together. We provide a barrier() method in the task scope to help simple coordination among tasks.
 
If not then how does barrier mode handle partitions that
are waiting to be processed?
If there are partitions waiting to be processed then I don't think it's
possible to send all data from given stage to a DL process, even when
using barrier mode?
Thanks a lot,

Joe


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]