how spark dstream handles congestion?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

how spark dstream handles congestion?

Dong Mo
Dear list,

I was wondering how Spark handles congestion when the upstream is generating dstreams faster than downstream workers can handle?

Thanks
-Mo
Reply | Threaded
Open this post in threaded view
|

Re: how spark dstream handles congestion?

Evgeniy Shishkin

On 31 Mar 2014, at 21:05, Dong Mo <[hidden email]> wrote:

> Dear list,
>
> I was wondering how Spark handles congestion when the upstream is generating dstreams faster than downstream workers can handle?

It will eventually OOM.

Reply | Threaded
Open this post in threaded view
|

Re: how spark dstream handles congestion?

Dong Mo
Thanks
-Mo


2014-03-31 13:16 GMT-05:00 Evgeny Shishkin <[hidden email]>:

On 31 Mar 2014, at 21:05, Dong Mo <[hidden email]> wrote:

> Dear list,
>
> I was wondering how Spark handles congestion when the upstream is generating dstreams faster than downstream workers can handle?

It will eventually OOM.