This post has NOT been accepted by the mailing list yet.
I have a job running with 8 executors and 512 MB memory each.This job does 4 count actions that retrieves the data from the hbase tables and aggregates the data.
As part of playing with configurations and figure out the best ideal configuration for this job, I changed the configuration to 4 executors and two cores each and 2 GB memory each but there was degradation in performance.
In my thoughts, I provided the same no.of cores for this job also increased the memory and expecting an increase in performance/ same amount of performance but the case was completely different. May I know what internals made difference and caused the performance issue.