How to update structured streaming apps gracefully

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

How to update structured streaming apps gracefully

Yuta Morisawa
Hi

Now I'm trying to update my structured streaming application.
But I have no idea how to update it gracefully.

Should I stop it, replace a jar file then restart it?
In my understanding, in that case, all the state will be recovered if I
use checkpoints.
Is this correct?

Thank you,


--


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How to update structured streaming apps gracefully

pmatpadi
Using checkpointing for graceful updates is my understanding as well, based on the writeup in https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing, and some prototyping. Have you faced any missed events?

On Mon, Dec 17, 2018 at 6:56 PM Yuta Morisawa <[hidden email]> wrote:
Hi

Now I'm trying to update my structured streaming application.
But I have no idea how to update it gracefully.

Should I stop it, replace a jar file then restart it?
In my understanding, in that case, all the state will be recovered if I
use checkpoints.
Is this correct?

Thank you,


--


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How to update structured streaming apps gracefully

vincent gromakowski
Checkpointing is only used for failure recovery not for app upgrades. You need to manually code the unload/load and save it to a persistent store

Le mar. 18 déc. 2018 à 17:29, Priya Matpadi <[hidden email]> a écrit :
Using checkpointing for graceful updates is my understanding as well, based on the writeup in https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing, and some prototyping. Have you faced any missed events?

On Mon, Dec 17, 2018 at 6:56 PM Yuta Morisawa <[hidden email]> wrote:
Hi

Now I'm trying to update my structured streaming application.
But I have no idea how to update it gracefully.

Should I stop it, replace a jar file then restart it?
In my understanding, in that case, all the state will be recovered if I
use checkpoints.
Is this correct?

Thank you,


--


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How to update structured streaming apps gracefully

pmatpadi
Changes in streaming query that allow or disallow recovery from checkpoint is clearly provided in https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovery-semantics-after-changes-in-a-streaming-query.

On Tue, Dec 18, 2018 at 9:45 AM vincent gromakowski <[hidden email]> wrote:
Checkpointing is only used for failure recovery not for app upgrades. You need to manually code the unload/load and save it to a persistent store

Le mar. 18 déc. 2018 à 17:29, Priya Matpadi <[hidden email]> a écrit :
Using checkpointing for graceful updates is my understanding as well, based on the writeup in https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing, and some prototyping. Have you faced any missed events?

On Mon, Dec 17, 2018 at 6:56 PM Yuta Morisawa <[hidden email]> wrote:
Hi

Now I'm trying to update my structured streaming application.
But I have no idea how to update it gracefully.

Should I stop it, replace a jar file then restart it?
In my understanding, in that case, all the state will be recovered if I
use checkpoints.
Is this correct?

Thank you,


--


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How to update structured streaming apps gracefully

vincent gromakowski
I totally missed this new feature. Thanks for the pointer

Le mar. 18 déc. 2018 à 21:18, Priya Matpadi <[hidden email]> a écrit :
Changes in streaming query that allow or disallow recovery from checkpoint is clearly provided in https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovery-semantics-after-changes-in-a-streaming-query.

On Tue, Dec 18, 2018 at 9:45 AM vincent gromakowski <[hidden email]> wrote:
Checkpointing is only used for failure recovery not for app upgrades. You need to manually code the unload/load and save it to a persistent store

Le mar. 18 déc. 2018 à 17:29, Priya Matpadi <[hidden email]> a écrit :
Using checkpointing for graceful updates is my understanding as well, based on the writeup in https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing, and some prototyping. Have you faced any missed events?

On Mon, Dec 17, 2018 at 6:56 PM Yuta Morisawa <[hidden email]> wrote:
Hi

Now I'm trying to update my structured streaming application.
But I have no idea how to update it gracefully.

Should I stop it, replace a jar file then restart it?
In my understanding, in that case, all the state will be recovered if I
use checkpoints.
Is this correct?

Thank you,


--


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How to update structured streaming apps gracefully

Yuta Morisawa
Hi Priya and  Vincent

Thank you for your reply!
It looks the new feature is implemented only in the latest version.
But I'm using Spark 2.3.0 so, in my understanding, I need to stop and
reload apps.

Thanks

On 2018/12/19 9:09, vincent gromakowski wrote:

> I totally missed this new feature. Thanks for the pointer
>
> Le mar. 18 déc. 2018 à 21:18, Priya Matpadi <[hidden email]
> <mailto:[hidden email]>> a écrit :
>
>     Changes in streaming query that allow or disallow recovery from
>     checkpoint is clearly provided in
>     https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovery-semantics-after-changes-in-a-streaming-query.
>
>     On Tue, Dec 18, 2018 at 9:45 AM vincent gromakowski
>     <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>         Checkpointing is only used for failure recovery not for app
>         upgrades. You need to manually code the unload/load and save it
>         to a persistent store
>
>         Le mar. 18 déc. 2018 à 17:29, Priya Matpadi <[hidden email]
>         <mailto:[hidden email]>> a écrit :
>
>             Using checkpointing for graceful updates is my understanding
>             as well, based on the writeup in
>             https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing,
>             and some prototyping. Have you faced any missed events?
>
>             On Mon, Dec 17, 2018 at 6:56 PM Yuta Morisawa
>             <[hidden email]
>             <mailto:[hidden email]>> wrote:
>
>                 Hi
>
>                 Now I'm trying to update my structured streaming
>                 application.
>                 But I have no idea how to update it gracefully.
>
>                 Should I stop it, replace a jar file then restart it?
>                 In my understanding, in that case, all the state will be
>                 recovered if I
>                 use checkpoints.
>                 Is this correct?
>
>                 Thank you,
>
>
>                 --
>
>
>                 ---------------------------------------------------------------------
>                 To unsubscribe e-mail: [hidden email]
>                 <mailto:[hidden email]>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]