I am running spark in cluster mode ( on K8 ). On running it on Word Count
example, the number of executors assigned are different across stages. Our
number of assigned executors is 20. While stage 1 gets all 20 of them
alloted, stage 2 gets only < 10 executors. Is there any particular reason
for the change in executor count across stages?