Parallelism in custom Receiver

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Parallelism in custom Receiver

I custom a receiver that can process data from an external source. And I read the doc saying

    A DStream is associated with a single receiver. For attaining read parallelism multiple receivers i.e. multiple DStreams need to be created. A receiver is run within an executor. It occupies one core. Ensure that there are enough cores for processing after receiver slots are booked i.e. spark.cores.max should take the receiver slots into account. The receivers are allocated to executors in a round robin fashion.

So I should be able to launch multiple receiver. But my question is how to increase parallelism of Receiver? I do not see any parameter can be tuned according to doc -

   val sc = new SparkConf().setMaster("local[*]").setAppName("MyAppName")
    val ssc = new StreamingContext(sc, Seconds(1))
    val stream = ssc.receiverStream(new MyReceiver())
    Try(ssc.awaitTermination) match {
      case Success(_) => println("Finish streaming ....")
      case Failure(ex) => println(s"exception : $ex")

Right now I use local, but I would like to learn both clustered mode and local mode strategy in launching multiple receiver for parallelism. Appreciate any suggestions!