Writing to Google Cloud Storage with v2 algorithm safe?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Writing to Google Cloud Storage with v2 algorithm safe?

Jacek Laskowski
Hi,

I've just posted a question on StackOverflow [1] about the safety of the v2 algorithm while writing out to Google Cloud Storage. I think I'm missing some fundamentals on how cloud object stores work (GCS in particular) and hence the question.

Is this all about File.rename and how many HTTP calls are there under the covers? How to know it for GCS?

Thank you for any help you can provide. Merci beaucoup mes amis :)

Reply | Threaded
Open this post in threaded view
|

Re: Writing to Google Cloud Storage with v2 algorithm safe?

Mich Talebzadeh
Hi Jacek,

Can you please clarify your question?

with regard to your point:

"... I think I'm missing some fundamentals on how cloud object stores work (GCS in particular) and hence the question."

The end result is the safe storage of data in object storage in GCP right?

HTH




   view my Linkedin profile

 

Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 



On Sat, 3 Apr 2021 at 17:13, Jacek Laskowski <[hidden email]> wrote:
Hi,

I've just posted a question on StackOverflow [1] about the safety of the v2 algorithm while writing out to Google Cloud Storage. I think I'm missing some fundamentals on how cloud object stores work (GCS in particular) and hence the question.

Is this all about File.rename and how many HTTP calls are there under the covers? How to know it for GCS?

Thank you for any help you can provide. Merci beaucoup mes amis :)

Reply | Threaded
Open this post in threaded view
|

Re: Writing to Google Cloud Storage with v2 algorithm safe?

Jacek Laskowski
Hi,


> For object stores whose consistency model means that rename-based commits are safe use the FileOutputCommitter v2 algorithm for performance; v1 for safety.

These are "safe" and "safety" meanings.

On Sat, Apr 3, 2021 at 7:49 PM Mich Talebzadeh <[hidden email]> wrote:
Hi Jacek,

Can you please clarify your question?

with regard to your point:

"... I think I'm missing some fundamentals on how cloud object stores work (GCS in particular) and hence the question."

The end result is the safe storage of data in object storage in GCP right?

HTH




   view my Linkedin profile

 

Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 



On Sat, 3 Apr 2021 at 17:13, Jacek Laskowski <[hidden email]> wrote:
Hi,

I've just posted a question on StackOverflow [1] about the safety of the v2 algorithm while writing out to Google Cloud Storage. I think I'm missing some fundamentals on how cloud object stores work (GCS in particular) and hence the question.

Is this all about File.rename and how many HTTP calls are there under the covers? How to know it for GCS?

Thank you for any help you can provide. Merci beaucoup mes amis :)

Reply | Threaded
Open this post in threaded view
|

Re: Writing to Google Cloud Storage with v2 algorithm safe?

vaquar khan
In reply to this post by Jacek Laskowski
Hi Jecek ,

I have answered , hope you find it useful.

Regards,
Viquar khan

On Sat, Apr 3, 2021 at 11:19 AM Jacek Laskowski <[hidden email]> wrote:
Hi,

I've just posted a question on StackOverflow [1] about the safety of the v2 algorithm while writing out to Google Cloud Storage. I think I'm missing some fundamentals on how cloud object stores work (GCS in particular) and hence the question.

Is this all about File.rename and how many HTTP calls are there under the covers? How to know it for GCS?

Thank you for any help you can provide. Merci beaucoup mes amis :)



--
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago
Reply | Threaded
Open this post in threaded view
|

Re: Writing to Google Cloud Storage with v2 algorithm safe?

Jacek Laskowski
Hi Vaquar,

Thanks a lot! Accepted as the answer (yet there was the other answer that was very helpful too). Tons of reading ahead to understand it more.

That once again makes me feel that Hadoop MapReduce experience would help a great deal (and I've got none).

On Sun, Apr 4, 2021 at 7:28 AM vaquar khan <[hidden email]> wrote:
Hi Jecek ,

I have answered , hope you find it useful.

Regards,
Viquar khan

On Sat, Apr 3, 2021 at 11:19 AM Jacek Laskowski <[hidden email]> wrote:
Hi,

I've just posted a question on StackOverflow [1] about the safety of the v2 algorithm while writing out to Google Cloud Storage. I think I'm missing some fundamentals on how cloud object stores work (GCS in particular) and hence the question.

Is this all about File.rename and how many HTTP calls are there under the covers? How to know it for GCS?

Thank you for any help you can provide. Merci beaucoup mes amis :)



--
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago