S3a Committer

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

S3a Committer

aldu29
Hi,

I have some issues at the moment with S3 API of Openstack Swift (S3a).
This one is eventually consistent and it causes lots of issues with my distributed jobs in Spark.
Is the S3A committer able to fix that ? Or an "S3guard like" implementation is the only way ?

David
Reply | Threaded
Open this post in threaded view
|

Re: S3a Committer

Henoc
S3 is strongly consistent now

Regards,
Henoc

On Tue, Feb 2, 2021, 10:27 PM David Morin <[hidden email]> wrote:
Hi,

I have some issues at the moment with S3 API of Openstack Swift (S3a).
This one is eventually consistent and it causes lots of issues with my distributed jobs in Spark.
Is the S3A committer able to fix that ? Or an "S3guard like" implementation is the only way ?

David
Reply | Threaded
Open this post in threaded view
|

Re: S3a Committer

aldu29
Yes, that's true but this is not (yet) the case of the Openstack Swift S3 API

Le mar. 2 févr. 2021 à 21:41, Henoc <[hidden email]> a écrit :
S3 is strongly consistent now

Regards,
Henoc

On Tue, Feb 2, 2021, 10:27 PM David Morin <[hidden email]> wrote:
Hi,

I have some issues at the moment with S3 API of Openstack Swift (S3a).
This one is eventually consistent and it causes lots of issues with my distributed jobs in Spark.
Is the S3A committer able to fix that ? Or an "S3guard like" implementation is the only way ?

David
Reply | Threaded
Open this post in threaded view
|

RE: S3a Committer

Boris Litvak
In reply to this post by aldu29

This December AWS announced https://aws.amazon.com/s3/consistency/, are you sure this is your problem?

I think all these s3guard like wrappers are irrelevant right now. Please correct me if I am wrong.

 

From: David Morin <[hidden email]>
Sent: Tuesday, 2 February 2021 22:26
To: [hidden email]
Subject: S3a Committer

 

Hi,

 

I have some issues at the moment with S3 API of Openstack Swift (S3a).

This one is eventually consistent and it causes lots of issues with my distributed jobs in Spark.

Is the S3A committer able to fix that ? Or an "S3guard like" implementation is the only way ?

 

David

Reply | Threaded
Open this post in threaded view
|

Re: S3a Committer

YoungKun Min
Hi, 

I have almost the same problem with Ceph RGW, and currently do research about Apache Iceberg and Databricks Delta(opensource version). 
I think these libraries can address the problem.


2021년 2월 3일 (수) 오후 3:44, Boris Litvak <[hidden email]>님이 작성:

This December AWS announced https://aws.amazon.com/s3/consistency/, are you sure this is your problem?

I think all these s3guard like wrappers are irrelevant right now. Please correct me if I am wrong.

 

From: David Morin <[hidden email]>
Sent: Tuesday, 2 February 2021 22:26
To: [hidden email]
Subject: S3a Committer

 

Hi,

 

I have some issues at the moment with S3 API of Openstack Swift (S3a).

This one is eventually consistent and it causes lots of issues with my distributed jobs in Spark.

Is the S3A committer able to fix that ? Or an "S3guard like" implementation is the only way ?

 

David

Reply | Threaded
Open this post in threaded view
|

Re: S3a Committer

Gourav Sengupta
Why s3a? 

Regards,
Gourav Sengupta

On Wed, Feb 3, 2021 at 7:35 AM YoungKun Min <[hidden email]> wrote:
Hi, 

I have almost the same problem with Ceph RGW, and currently do research about Apache Iceberg and Databricks Delta(opensource version). 
I think these libraries can address the problem.


2021년 2월 3일 (수) 오후 3:44, Boris Litvak <[hidden email]>님이 작성:

This December AWS announced https://aws.amazon.com/s3/consistency/, are you sure this is your problem?

I think all these s3guard like wrappers are irrelevant right now. Please correct me if I am wrong.

 

From: David Morin <[hidden email]>
Sent: Tuesday, 2 February 2021 22:26
To: [hidden email]
Subject: S3a Committer

 

Hi,

 

I have some issues at the moment with S3 API of Openstack Swift (S3a).

This one is eventually consistent and it causes lots of issues with my distributed jobs in Spark.

Is the S3A committer able to fix that ? Or an "S3guard like" implementation is the only way ?

 

David