We are trying to enable encryption between spark-shuffle and local FileSystem. We wanted to clarify our understanding on this. Currently we're working on Spark 2.4
According to our understanding of Spark supporting Local Storage Encryption, that is, "Enabling local disk I/O encryption", it looks like the following properties:-
needs to be enabled only in spark and not in spark-shuffle's configuration properties.
So, on performing spark-submit using external shuffle service, only when we set ioEncryption properties enabled in the configuration used for spark-submit, we can see ioEncryption related messages
in the driver log. But when we use ioEncryption properties enabled only in spark-shuffle's configuration we do not see any ioEncryption related messages in shuffle logs.