If you are using the same code to run on Yarn, I believe it’s still using the local mode as it overwrites the master url set by CLI. You can check the “executors” tab in the Spark UI to set how many executors are running, and verify if it matches your config.
On Tue, Nov 27, 2018 at 6:17 AM Abhijeet Kumar <[hidden email]> wrote:
Yes, it did
Thanks for the solution. I solved it locally, but I’m worried how I can do this when I’m using yarn because that same 15 Sec is taking on the yarn too :)