There are three machines in the spark cluster, each with 56 cpu cores and 360GB of memory allocated. But what is strange is that
when I increase the number of cpu cores, the overall execution time of
Query#5 is reduced that can be seen from the history server web UI, but
the average time of tasks (510 in total) responsible for reading data
increases significantly. When cpu cores=32, the average task time is 7s When cpu cores=48, the average task time is 17s When cpu cores=96, the average task time is 20s Entering
the task log analysis, it is found that reading the same data file, the
task time consumption increases with the number of cpu cores. What is the reason for this?
when I kept the total executors unchanged and still increased the cpu
cores, I found that the result was the same. Of course, the
executor-memory and driver-memory are enough.
Can you explain the reason for this?
【Excuse me, the same email I sent through [hidden email] on Friday, and it shows that the sending to [hidden email] is successful, why can’t I see the email I sent at http://apache-spark-developers-list.1001551.n3.nabble.com?】