[YarnShuffleService] Consistent OOMs when enabling Spark transport encryption
I have been experimenting with Spark 2.4.4 transport encryption and have
encountered an issue with a couple of our jobs: they consistently make the
YarnShuffleService die with OOM errors. It looks like the memory is full of
/io.netty.channel.ChannelOutboundBuffer$Entry/ objects each containing
- Has anyone experienced anything like this as well?
- Is there anyone with knowledge of the YarnShuffleService, and/or Netty
and/or Spark encryption that would be able to lend a hand in debugging this