How to troubleshoot MetadataFetchFailedException: Missing an output location for shuffle 0

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to troubleshoot MetadataFetchFailedException: Missing an output location for shuffle 0

Warren Zhu
Hi All,

I have seen this exception many times in my production environment for long running batch job. Is there some stigmatization of all root causes of this exception? Below is my analysis:

1. This happens when executor try to fetch MapStatus of some shuffle.
2. Each executor maintains a local cache of all map statuses. When can't find in local cache, executor will try to fetch latest from driver which acting as MapOutputTrackerMaster.
3. Driver's map statuses only be clear when epoch got updated.
4. Epoch got updated when new executor got restarted. This might be caused by executor lost. I have double confirmed this if one container(executor) is kill by Yarn for exceeding memory limits, then this exception will happen.

So I have 3 questions:
1. Is my analysis correct?
2. Is there some other clues or causes could result in this exception?
3. How to fix this exception?

Thanks,
Warren
Reply | Threaded
Open this post in threaded view
|

Re: How to troubleshoot MetadataFetchFailedException: Missing an output location for shuffle 0

Alessandro Solimando
Hi Warren,
it's often an exception stemming from an OOM at the executor level.

If you are caching data make sure you spill to disk, if needed.

You could also try to increase off-heap memory to alleviate the issue.

Of course also handing more memory to the executor helps.

Best regards,
Alessandro

Il Lun 16 Dic 2019, 21:01 Warren Zhu <[hidden email]> ha scritto:
Hi All,

I have seen this exception many times in my production environment for long running batch job. Is there some stigmatization of all root causes of this exception? Below is my analysis:

1. This happens when executor try to fetch MapStatus of some shuffle.
2. Each executor maintains a local cache of all map statuses. When can't find in local cache, executor will try to fetch latest from driver which acting as MapOutputTrackerMaster.
3. Driver's map statuses only be clear when epoch got updated.
4. Epoch got updated when new executor got restarted. This might be caused by executor lost. I have double confirmed this if one container(executor) is kill by Yarn for exceeding memory limits, then this exception will happen.

So I have 3 questions:
1. Is my analysis correct?
2. Is there some other clues or causes could result in this exception?
3. How to fix this exception?

Thanks,
Warren