Does mapWithState need checkpointing to be specified in Spark Streaming?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
SRK
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Does mapWithState need checkpointing to be specified in Spark Streaming?

SRK
This post has NOT been accepted by the mailing list yet.
Hi,

Do we need to specify checkpointing for mapWithState just like we do for updateStateByKey?

Thanks,
Swetha
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does mapWithState need checkpointing to be specified in Spark Streaming?

Yuval.Itzchakov
This post has NOT been accepted by the mailing list yet.
Yes, you do.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does mapWithState need checkpointing to be specified in Spark Streaming?

dori waldman
This post has NOT been accepted by the mailing list yet.
In reply to this post by SRK
when you are working with state spark use checkpoint.  (save metadata about your state in order to recover from restart and continue with latest state)
Loading...