Curious case of Spark SQL 2.3 - number of stages different for the same query ever?
I've got a case where the same structured query (it's union) gives 1 stage for a run and 5 stages for another. I could not find any pattern yet (and it's hard to reproduce it due to the volume and the application), but I'm pretty certain that it's *never* possible that Spark 2.3 could come up with 1 vs 5 stages for the very same query plan (even if I changed number of executors or number of cores or anything execution-related).
So my question is, is this possible that Spark SQL could give 1-stage execution plan and 5-stage execution plan for the very same query?
(I am not saying that I'm 100% sure that the query is indeed the same since I'm working on a reproducible test case and only when I got it I'll really be).
Sorry for the vague description, but I've got nothing more to share yet.