aws spot instances termination notification for checkpointing/caching

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

aws spot instances termination notification for checkpointing/caching

polya
This post has NOT been accepted by the mailing list yet.
aws sends a notification 2 mins before the (spot) instance is terminated. Is there a way in Spark/Yarn to checkpoint the data on HDFS from the worker node(that is going down) so that the work that is already done (in a long running batch process) need not be recomputed.
Loading...