Spark configuration params

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
Report Content as Inappropriate

Spark configuration params

This post has NOT been accepted by the mailing list yet.
Hi Team,

I have a job running with 8 executors and 512 MB memory each.This job does 4 count actions that retrieves the data from the hbase tables and aggregates the data.

As part of playing with configurations and figure out the best ideal configuration for this job, I changed the configuration to 4 executors and  two cores each and 2 GB memory each but there was degradation in performance.
In my thoughts, I provided the same no.of cores for this job also increased the memory and expecting an increase in performance/ same amount of performance but the case was completely different. May I know what internals made difference and caused the performance issue.