Unable to save an RDd on S3 with SSE-KMS encryption

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Unable to save an RDd on S3 with SSE-KMS encryption

Vikash Pareek
This post was updated on .
I am trying to save an rdd on S3 with server side encryption using KMS key
(SSE-KMS), But I am getting the following exception:

Exception in thread "main"
com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS
Service: Amazon S3, AWS Request ID: 695E32175EBA568A, AWS Error Code:
InvalidArgument, AWS Error Message: The encryption method specified is not
supported, S3 Extended Request ID:
Pi+HFLg0WsAWtkdI2S/xViOcRPMCi7zdHiaO5n1f7tiwpJe2z0lPY1C2Cr53PnnUCj3358Gx3AQ=


Following is the piece of my test code to write an rdd on S3 by using
SSE-KMS for encryption:

val sparkConf = new SparkConf().
  setMaster("local[*]").
  setAppName("aws-encryption")
val sc = new SparkContext(sparkConf)

sc.hadoopConfiguration.set("fs.s3a.access.key", AWS_ACCESS_KEY)
sc.hadoopConfiguration.set("fs.s3a.secret.key", AWS_SECRET_KEY)
sc.hadoopConfiguration.setBoolean("fs.s3a.sse.enabled", true)
sc.hadoopConfiguration.set("fs.s3a.server-side-encryption-algorithm",
"SSE-KMS")
sc.hadoopConfiguration.set("fs.s3a.sse.kms.keyId", KMS_ID)

val s3a = new org.apache.hadoop.fs.s3a.S3AFileSystem
val s3aName = s3a.getClass.getName
sc.hadoopConfiguration.set("fs.s3a.impl", s3aName)

val rdd = sc.parallelize(Seq("one", "two", "three", "four"))
println("rdd is: " + rdd.collect())
rdd.saveAsTextFile(s"s3a://$bucket/$objKey")


Although, I am able to write rdd on s3 with AES256 encryption but failing
with SSE-KMS.
Does spark/hadoop have a different value for KMS key encryption instead of
"SSE-KMS" or it doesn't support SSE-KMS encryption on AWS S3?

I found in the official document of hadoop that it only supports AES256 as
of now.

<property>
  <name>fs.s3n.server-side-encryption-algorithm</name>
  <value></value>
  <description>Specify a server-side encryption algorithm for S3.
  The default is NULL, and the only other currently allowable value is
AES256.
  </description>
</property>


Can anyone please suggest what I am missing here or doing wrong?

My environment details as follow:
spark: 1.6.1
hadoop: 2.6.0
aws-java-sdk: 1.7.4

Thank you in advance.




-----

__Vikash Pareek
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


__Vikash Pareek