[Structured Streaming] Reading Checkpoint data

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Structured Streaming] Reading Checkpoint data

subramgr
Hi,

I read somewhere that with Structured Streaming all the checkpoint data is
more readable (Json) like. Is there any documentation on how to read the
checkpoint data.

If I do `hadoop fs -ls` on the `state` directory I get some encoded data.

Thanks
Girish



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Structured Streaming] Reading Checkpoint data

Tathagata Das
Only the stream metadata (e.g., streamid, offsets) are stored as json. The stream state data is stored in an internal binary format.

On Mon, Jul 9, 2018 at 4:07 PM, subramgr <[hidden email]> wrote:
Hi,

I read somewhere that with Structured Streaming all the checkpoint data is
more readable (Json) like. Is there any documentation on how to read the
checkpoint data.

If I do `hadoop fs -ls` on the `state` directory I get some encoded data.

Thanks
Girish



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: [Structured Streaming] Reading Checkpoint data

subramgr
thanks



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]