Number of Spark streams in Yarn cluster

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Number of Spark streams in Yarn cluster

tnegi
Hi,

I am trying to get a sense of number of streams we can process in parallel on a Spark streaming cluster(Hadoop Yarn).
Is there any benchmark for the same?
We need a large number of streams(original + transformed) to be processed in parallel.
The number is approximately around= 30,0000.



thanks
Negi