how is fault tolerance achieved in spark

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

how is fault tolerance achieved in spark

amoc

Hi all,

I am curious how fault tolerance is achieved in spark. Well, more like what do I need to do to make sure my aggregations which comes from streams are fault tolerant and saved into cassandra. I will have nodes die and would not like to count “tuples” multiple times.

 

For example, in trident you have to implement different interfaces. Is there a similar process for spark?

 

Thanks

-Adrian

 

Reply | Threaded
Open this post in threaded view
|

RE: how is fault tolerance achieved in spark

amoc

Anyone willing to link some resource on how to achieve fault tolerance?

 

From: Adrian Mocanu [mailto:[hidden email]]
Sent: February-10-14 1:44 PM
To: [hidden email]
Subject: how is fault tolerance achieved in spark

 

Hi all,

I am curious how fault tolerance is achieved in spark. Well, more like what do I need to do to make sure my aggregations which comes from streams are fault tolerant and saved into cassandra. I will have nodes die and would not like to count “tuples” multiple times.

 

For example, in trident you have to implement different interfaces. Is there a similar process for spark?

 

Thanks

-Adrian

 

Reply | Threaded
Open this post in threaded view
|

Re: how is fault tolerance achieved in spark

Andrew Ash
Here's the original paper on how the framework achieves fault tolerance.  You shouldn't have to do anything special as a user of the framework.

https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf


On Tue, Feb 11, 2014 at 12:21 PM, Adrian Mocanu <[hidden email]> wrote:

Anyone willing to link some resource on how to achieve fault tolerance?

 

From: Adrian Mocanu [mailto:[hidden email]]
Sent: February-10-14 1:44 PM
To: [hidden email]
Subject: how is fault tolerance achieved in spark

 

Hi all,

I am curious how fault tolerance is achieved in spark. Well, more like what do I need to do to make sure my aggregations which comes from streams are fault tolerant and saved into cassandra. I will have nodes die and would not like to count “tuples” multiple times.

 

For example, in trident you have to implement different interfaces. Is there a similar process for spark?

 

Thanks

-Adrian

 


Reply | Threaded
Open this post in threaded view
|

Re: how is fault tolerance achieved in spark

Matei Zaharia
Administrator
Spark Streaming also provides fault tolerance by default as long as your input is in a reliable data source. Take a look at this paper: http://www.cs.berkeley.edu/~matei/papers/2013/sosp_spark_streaming.pdf or the fault tolerance section of the usage guide: http://spark.incubator.apache.org/docs/latest/streaming-programming-guide.html.

Matei

On Feb 11, 2014, at 12:27 PM, Andrew Ash <[hidden email]> wrote:

Here's the original paper on how the framework achieves fault tolerance.  You shouldn't have to do anything special as a user of the framework.

https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf


On Tue, Feb 11, 2014 at 12:21 PM, Adrian Mocanu <[hidden email]> wrote:

Anyone willing to link some resource on how to achieve fault tolerance?

 

From: Adrian Mocanu [mailto:[hidden email]]
Sent: February-10-14 1:44 PM
To: [hidden email]
Subject: how is fault tolerance achieved in spark

 

Hi all,

I am curious how fault tolerance is achieved in spark. Well, more like what do I need to do to make sure my aggregations which comes from streams are fault tolerant and saved into cassandra. I will have nodes die and would not like to count “tuples” multiple times.

 

For example, in trident you have to implement different interfaces. Is there a similar process for spark?

 

Thanks

-Adrian

 



Reply | Threaded
Open this post in threaded view
|

RE: how is fault tolerance achieved in spark

amoc

Thanks Andrew and Matei.

 

My input is from a kafka spout which can fail any time so it’s not a reliable input source. The problem is duplicate tuples coming through and I’d like those to be discarded.

 

I’ll take a look at those links provided by both you and Andrew.

-A

From: Matei Zaharia [mailto:[hidden email]]
Sent: February-11-14 4:02 PM
To: [hidden email]
Subject: Re: how is fault tolerance achieved in spark

 

Spark Streaming also provides fault tolerance by default as long as your input is in a reliable data source. Take a look at this paper: http://www.cs.berkeley.edu/~matei/papers/2013/sosp_spark_streaming.pdf or the fault tolerance section of the usage guide: http://spark.incubator.apache.org/docs/latest/streaming-programming-guide.html.

 

Matei

 

On Feb 11, 2014, at 12:27 PM, Andrew Ash <[hidden email]> wrote:



Here's the original paper on how the framework achieves fault tolerance.  You shouldn't have to do anything special as a user of the framework.

 

https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf

 

On Tue, Feb 11, 2014 at 12:21 PM, Adrian Mocanu <[hidden email]> wrote:

Anyone willing to link some resource on how to achieve fault tolerance?

 

From: Adrian Mocanu [mailto:[hidden email]]
Sent: February-10-14 1:44 PM
To:
[hidden email]
Subject: how is fault tolerance achieved in spark

 

Hi all,

I am curious how fault tolerance is achieved in spark. Well, more like what do I need to do to make sure my aggregations which comes from streams are fault tolerant and saved into cassandra. I will have nodes die and would not like to count “tuples” multiple times.

 

For example, in trident you have to implement different interfaces. Is there a similar process for spark?

 

Thanks

-Adrian

 

 

 

Reply | Threaded
Open this post in threaded view
|

Re: how is fault tolerance achieved in spark

Pankaj Mittal
Hi Adrian,
You must know all the transformations on RDD are having 'exactly once semantic' but counting multiple times is still possible because all the output operations (like foreachRDD) are not having 'exactly once semantic' instead they have 'at least once semantic'. Have a look at this link - https://spark.incubator.apache.org/docs/0.9.0/streaming-programming-guide.html#failure-of-a-worker-node

So your application or Cassandra model needs to have logic to make sure over counting doesn't happen while storing data into Cassandra. We are achieving this by storing RDD id into our data model so that we don't over count even if failed partitions are recomputed. Let me know if it helps.

Thanks
Pankaj


On Wed, Feb 12, 2014 at 2:50 AM, Adrian Mocanu <[hidden email]> wrote:

Thanks Andrew and Matei.

 

My input is from a kafka spout which can fail any time so it’s not a reliable input source. The problem is duplicate tuples coming through and I’d like those to be discarded.

 

I’ll take a look at those links provided by both you and Andrew.

-A

From: Matei Zaharia [mailto:[hidden email]]
Sent: February-11-14 4:02 PM
To: [hidden email]
Subject: Re: how is fault tolerance achieved in spark

 

Spark Streaming also provides fault tolerance by default as long as your input is in a reliable data source. Take a look at this paper: http://www.cs.berkeley.edu/~matei/papers/2013/sosp_spark_streaming.pdf or the fault tolerance section of the usage guide: http://spark.incubator.apache.org/docs/latest/streaming-programming-guide.html.

 

Matei

 

On Feb 11, 2014, at 12:27 PM, Andrew Ash <[hidden email]> wrote:



Here's the original paper on how the framework achieves fault tolerance.  You shouldn't have to do anything special as a user of the framework.

 

https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf

 

On Tue, Feb 11, 2014 at 12:21 PM, Adrian Mocanu <[hidden email]> wrote:

Anyone willing to link some resource on how to achieve fault tolerance?

 

From: Adrian Mocanu [mailto:[hidden email]]
Sent: February-10-14 1:44 PM
To:
[hidden email]
Subject: how is fault tolerance achieved in spark

 

Hi all,

I am curious how fault tolerance is achieved in spark. Well, more like what do I need to do to make sure my aggregations which comes from streams are fault tolerant and saved into cassandra. I will have nodes die and would not like to count “tuples” multiple times.

 

For example, in trident you have to implement different interfaces. Is there a similar process for spark?

 

Thanks

-Adrian