Curious case of Spark SQL 2.3 - number of stages different for the same query ever?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Curious case of Spark SQL 2.3 - number of stages different for the same query ever?

Jacek Laskowski
Hi,

I've got a case where the same structured query (it's union) gives 1 stage for a run and 5 stages for another. I could not find any pattern yet (and it's hard to reproduce it due to the volume and the application), but I'm pretty certain that it's *never* possible that Spark 2.3 could come up with 1 vs 5 stages for the very same query plan (even if I changed number of executors or number of cores or anything execution-related).

So my question is, is this possible that Spark SQL could give 1-stage execution plan and 5-stage execution plan for the very same query?

(I am not saying that I'm 100% sure that the query is indeed the same since I'm working on a reproducible test case and only when I got it I'll really be).

Sorry for the vague description, but I've got nothing more to share yet.