Monitoring Spark Jobs

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Monitoring Spark Jobs

SamyaMaiti
This post has NOT been accepted by the mailing list yet.
Hi All,

I have a Spark SQL application to fetch data from Hive, on top I have a akka layer to run multiple Queries in parallel.

Please suggest a mechanism, so as to figure out the number of spark jobs running in the cluster at a given instance of time.

I need to do the above as, I see the average response time increasing with increase in number of requests, in-spite of increasing the number of cores in the cluster. I suspect there is a bottleneck somewhere else.

Regards,
Sam
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring Spark Jobs

Himanshu Mehra
This post has NOT been accepted by the mailing list yet.
Hi Sam,

You might want to have a look at spark UI which runs by default at localhost://8080. You can also configure Apache Ganglia to monitor over your cluster resources.

Thank you
Regards
Himanshu Mehra
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring Spark Jobs

SamyaMaiti
This post has NOT been accepted by the mailing list yet.
Hi Himanshu,

Using Spark UI wont help, because  I want to figure out the total jobs running at a given instance & my jobs runs in milliseconds. Not sure how much Ganglia will help in this scenario, will try out & let you know.

Regards,
Sam

On Wed, Jun 10, 2015 at 1:39 PM, Himanshu Mehra [via Apache Spark User List] <[hidden email]> wrote:
Hi Sam,

You might want to have a look at spark UI which runs by default at localhost://8080. You can also configure Apache Ganglia to monitor over your cluster resources.

Thank you
Regards
Himanshu Mehra


If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Monitoring-Spark-Jobs-tp23193p23243.html
To unsubscribe from Monitoring Spark Jobs, click here.
NAML