AWS credentials needed while trying to read a model from S3 in Spark

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

AWS credentials needed while trying to read a model from S3 in Spark

Mina Aslani
Hi,

I am trying to load a ML model from AWS S3 in my spark app running in a docker container, however I need to pass the AWS credentials. 
My questions is, why do I need to pass the credentials in the path? 
And what is the workaround?

Best regards,
Mina
Reply | Threaded
Open this post in threaded view
|

Re: AWS credentials needed while trying to read a model from S3 in Spark

srinath
You could use IAM roles in AWS to access the data in S3 without credentials.
See this link and this link for an example.

Regards,
Srinath.


On Thu, May 10, 2018 at 7:04 AM, Mina Aslani <[hidden email]> wrote:
Hi,

I am trying to load a ML model from AWS S3 in my spark app running in a docker container, however I need to pass the AWS credentials. 
My questions is, why do I need to pass the credentials in the path? 
And what is the workaround?

Best regards,
Mina