saving RDD to disk - fault tolerance

[amoc]

Hi

When saving an RDD to disk, if the spark node doing the saving crashes, is the entire file rewritten to disk or can spark figure out where it left off upon RDD recomputation?

 

-Adrian

 

[Mayur Rustagi]

Since disk here is HDFS & HDFS only supports appending of files, i bet that RDD is rewritten. Never tried this though. 

Mayur Rustagi
Ph: [phone]

http://www.sigmoidanalytics.com

https://twitter.com/mayur_rustagi

On Thu, Feb 20, 2014 at 10:41 AM, Adrian Mocanu <[hidden email]> wrote:

Hi

When saving an RDD to disk, if the spark node doing the saving crashes, is the entire file rewritten to disk or can spark figure out where it left off upon RDD recomputation?

 

-Adrian