Kafka 010 Spark 2.2.0 Streaming / Custom checkpoint strategy

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Kafka 010 Spark 2.2.0 Streaming / Custom checkpoint strategy

Anand Chandrashekar
Greetings!

I would like to accomplish a custom kafka checkpoint strategy (instead of hdfs, i would like to use redis). is there a strategy I can use to change this behavior; any advise will help. Thanks!

Regards,
Anand.
Reply | Threaded
Open this post in threaded view
|

Re: Kafka 010 Spark 2.2.0 Streaming / Custom checkpoint strategy

Jörn Franke
HDFS can be r placed by other filesystem plugins (eg ignitefs, s3, etc) so the easiest is to write a file system plugin. This is not a plug-in for Spark but part of the Hadoop functionality used by Spark.

> On 13. Oct 2017, at 17:41, Anand Chandrashekar <[hidden email]> wrote:
>
> Greetings!
>
> I would like to accomplish a custom kafka checkpoint strategy (instead of hdfs, i would like to use redis). is there a strategy I can use to change this behavior; any advise will help. Thanks!
>
> Regards,
> Anand.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]