Local Storage Encryption - Spark ioEncryption

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Local Storage Encryption - Spark ioEncryption

Sinha, Breeta (Nokia - IN/Bangalore)

Hi All,

 

We are trying to enable encryption between spark-shuffle and local FileSystem. We wanted to clarify our understanding on this. Currently we're working on Spark 2.4

 

According to our understanding of Spark supporting Local Storage Encryption, that is, "Enabling local disk I/O encryption", it looks like the following properties:-

spark.io.encryption.enabled

spark.io.encryption.keySizeBits

spark.io.encryption.keygen.algorithm

spark.io.encryption.commons.config.*

needs to be enabled only in spark and not in spark-shuffle's configuration properties.

 

So, on performing spark-submit using external shuffle service, only when we set ioEncryption properties enabled in the configuration used for spark-submit, we can see ioEncryption related messages in the driver log. But when we use ioEncryption properties enabled only in spark-shuffle's configuration we do not see any ioEncryption related messages in shuffle logs.

 

We have followed the below links:-

https://spark.apache.org/docs/latest/security.html

https://dzone.com/articles/securing-apache-spark-shuffle-using-apache-commons

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/configuring-spark/content/configuring_spark_for_wire_encryption.html

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-SerializerManager.html

 

Can you please clarify on this if our understanding that ioEncryption related properties needs to be enabled only in spark is correct?

 

Thanks.

Breeta