RDD values and defensive copying

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RDD values and defensive copying

Allen Chang
Hi all,

I had a question: if I have an RDD containing mutable values, and I run a function over the RDD which mucks with the mutable values, what happens? What happens in the case of a cogroup? e.g.:

  inputRdd.cogroup(inputRdd2).flatMapValues(functionThatModifiesValues())

Will this result in undefined behavior? Is it a best practice then to make sure functionThatModifiesValues() performs a defensive copy if necessary?

Thanks,
Allen
Loading...