Can spark-streaming work with spark-on-yarn mode?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Can spark-streaming work with spark-on-yarn mode?

林武康
hi all,
I am a very newbie of apache spark, recently I have tried spark on yarn, it works for batch process. Now we want to try streaming process using spark-streaming, and still, use yarn for resource scheduler as we want to manager all the resource of the cluster used for computing tasks in a unified way.  Can this works?
any suggestions are welcome!


--------------------------------------------------------------------------------

Best Regards!
Reply | Threaded
Open this post in threaded view
|

Re: Can spark-streaming work with spark-on-yarn mode?

Tathagata Das
Yes, Spark and Spark Streaming programs can be deployed on YARN. Here is the documentation.

TD


On Thu, Feb 20, 2014 at 11:16 PM, 林武康 <[hidden email]> wrote:
hi all,
I am a very newbie of apache spark, recently I have tried spark on yarn, it works for batch process. Now we want to try streaming process using spark-streaming, and still, use yarn for resource scheduler as we want to manager all the resource of the cluster used for computing tasks in a unified way.  Can this works?
any suggestions are welcome!


--------------------------------------------------------------------------------

Best Regards!