Local mode vs client mode with one executor

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Local mode vs client mode with one executor

Guillermo Ortiz
I have many spark processes, some of them are pretty simple and they don't have to process almost messages but they were developed with the same archeotype and they use spark.

Some of them are executed with many executors but a few ones don't make sense to process with more than 2-4 cores in only one executor. The most important reason is that the quantity of messages is so low,, that it's not worth it.

The point here it's,, any disventage if I run this few spark processes in local[2..4] instance of cluster/client mode with one executor (4 cores) and one driver?. I have read that it's a testing mode in most of cases and I use for my tests too ;)
Besides, it seems that it goes faster running in local mode in those cases.