saving RDD to disk - fault tolerance

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

saving RDD to disk - fault tolerance

amoc

Hi

When saving an RDD to disk, if the spark node doing the saving crashes, is the entire file rewritten to disk or can spark figure out where it left off upon RDD recomputation?

 

-Adrian

 

Reply | Threaded
Open this post in threaded view
|

Re: saving RDD to disk - fault tolerance

Mayur Rustagi
Since disk here is HDFS & HDFS only supports appending of files, i bet that RDD is rewritten. Never tried this though. 




On Thu, Feb 20, 2014 at 10:41 AM, Adrian Mocanu <[hidden email]> wrote:

Hi

When saving an RDD to disk, if the spark node doing the saving crashes, is the entire file rewritten to disk or can spark figure out where it left off upon RDD recomputation?

 

-Adrian