interpretation of Spark logs about remote fetch latency
Hello Spark users,
I have an inquiry while analyzing a sample Spark task. The task has remote fetches (shuffle) from few blocks. However, the remote fetch time does not really make sense to me. Can someone please help to interpret this?
The logs came from Spark REST API. The task ID 33 needs four blocks, and it has to fetch three blocks from remote machines. In the "shuffleReadMetrics" section, however, it marks as the "fetchWaitTime" as 0 while it really fetches about 2.4GB from remote machines. While in task ID 34 below, it needs to fetch 4 blocks with total size of around 3GB, it shows the fetchWaitTime is about 2.4 seconds, and only this makes sense.