kinesis throughput problems

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

kinesis throughput problems

Jeremy Kelley
We have a largeish kinesis stream with about 25k events per second and each record is around 142k.  I have tried multiple cluster sizes, multiple batch sizes, multiple parameters...  I am doing minimal transformations on the data.  Whatever happens I can sustain consuming 25k with minimal effort and cluster load for about 5-10 minutes and then always always the stream shapes down and hovers around 5k EPS.  

I can give MANY more details but I was curious if anyone had seen similar behavior.

Thanks,
Jeremy


-- 
Jeremy Kelley | Technical Director, Data
[hidden email] | Carbon Black Threat Engineering



smime.p7s (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: kinesis throughput problems

Gourav Sengupta
Hi Jeremy,

just out of curiosity - you do know that this is a SPARK user group?


Regards,
Gourav 

On Thu, Dec 14, 2017 at 7:03 PM, Jeremy Kelley <[hidden email]> wrote:
We have a largeish kinesis stream with about 25k events per second and each record is around 142k.  I have tried multiple cluster sizes, multiple batch sizes, multiple parameters...  I am doing minimal transformations on the data.  Whatever happens I can sustain consuming 25k with minimal effort and cluster load for about 5-10 minutes and then always always the stream shapes down and hovers around 5k EPS.  

I can give MANY more details but I was curious if anyone had seen similar behavior.

Thanks,
Jeremy


-- 
Jeremy Kelley | Technical Director, Data
[hidden email] | Carbon Black Threat Engineering



Reply | Threaded
Open this post in threaded view
|

Re: kinesis throughput problems

Jeremy Kelley
Gourav,  Yes, sorry.   Apparently I failed to mention I'm having these problems with Spark consuming  from a kinesis stream.  Been putting in late nights to figure this out and it's affecting my brain.  :^)

-jeremy

-- 
Jeremy Kelley | Technical Director, Data
[hidden email] | Carbon Black Threat Engineering


On Dec 15, 2017, at 9:12 AM, Gourav Sengupta <[hidden email]> wrote:

Hi Jeremy,

just out of curiosity - you do know that this is a SPARK user group?


Regards,
Gourav 

On Thu, Dec 14, 2017 at 7:03 PM, Jeremy Kelley <[hidden email]> wrote:
We have a largeish kinesis stream with about 25k events per second and each record is around 142k.  I have tried multiple cluster sizes, multiple batch sizes, multiple parameters...  I am doing minimal transformations on the data.  Whatever happens I can sustain consuming 25k with minimal effort and cluster load for about 5-10 minutes and then always always the stream shapes down and hovers around 5k EPS.  

I can give MANY more details but I was curious if anyone had seen similar behavior.

Thanks,
Jeremy


-- 
Jeremy Kelley | Technical Director, Data
[hidden email] | Carbon Black Threat Engineering





smime.p7s (9K) Download Attachment