[Spark Core]: S3a with Openstack swift object storage not using credentials provided in sparkConf

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[Spark Core]: S3a with Openstack swift object storage not using credentials provided in sparkConf

Marius

Hey,

i am currently using Spark 2.2.0 for Hadoop 2.7.x in in a Standalone cluster for testing. I want to Access some files to share them one the nodes on the cluster using addFiles. As local directories are not supported for this i want to use s3 to do the job.

In contrast to nearly everything i have found on the internet i am using a self hosted openstack cluster using swift as object storage. Accessing swift directly would be fine, too, but all tutorials i have found seem to use keystone v2, whilst our deployment uses the v3 version.

I added the following jars:

aws-java-sdk-1.7.4.jar

hadoop-aws-2.7.3.jar

as jars and to the classpath of each executor and driver.

When i try to access an s3 bucket the following exception occurs: "Unable to load AWS credentials from any provider in the chain"

This is my config:

conf.set("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
conf.set("fs.s3a.endpoint", "https://foo/swift/v1")
conf.set("fs.s3a.access.key", System.getenv("s3Access"))
conf.set("fs.s3a.secret.key", System.getenv("s3Secret"))

From my understanding the s3 handler is not using the provided credentials.

Has anyone an idea how to fix this?


Cheers and thanks in Advance

Marius