tracking resource usage for spark-shell commands

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

tracking resource usage for spark-shell commands

Bharath Bhushan
Is there a way to see the resource usage of each spark-shell command — say time taken and memory used?
I checked the WebUI of spark-shell and of the master and I don’t see any such breakdown. I see the time taken in the INFO logs but nothing about memory usage. It would also be nice to track the time taken in the spark-shell web UI.


Thanks
Reply | Threaded
Open this post in threaded view
|

Re: tracking resource usage for spark-shell commands

Mayur Rustagi
Time taken is shown in Shark shell web ui (hosted on 4040 port).
Also memory used is shown in terms of Storage of RDD, how much shuffle data was written & read during the process is also highlighted thr. 

Mayur Rustagi
Ph: +1 (760) 203 3257


On Tue, Mar 25, 2014 at 6:04 AM, Bharath Bhushan <[hidden email]> wrote:
Is there a way to see the resource usage of each spark-shell command — say time taken and memory used?
I checked the WebUI of spark-shell and of the master and I don’t see any such breakdown. I see the time taken in the INFO logs but nothing about memory usage. It would also be nice to track the time taken in the spark-shell web UI.


Thanks

Reply | Threaded
Open this post in threaded view
|

Re: tracking resource usage for spark-shell commands

Bharath Bhushan
Thanks for the response Mayur. I was seeing the webui of 0.9.0 spark. I see lots of detailed statistics in the newer 1.0.0-snapshot version. The only thing I found missing was the actual code that I had typed in at the spark-shell prompt, but I can always get it from the shell history.

On 26-Mar-2014, at 7:53 am, Mayur Rustagi <[hidden email]> wrote:

Time taken is shown in Shark shell web ui (hosted on 4040 port).
Also memory used is shown in terms of Storage of RDD, how much shuffle data was written & read during the process is also highlighted thr. 

Mayur Rustagi
Ph: +1 (760) 203 3257


On Tue, Mar 25, 2014 at 6:04 AM, Bharath Bhushan <[hidden email]> wrote:
Is there a way to see the resource usage of each spark-shell command — say time taken and memory used?
I checked the WebUI of spark-shell and of the master and I don’t see any such breakdown. I see the time taken in the INFO logs but nothing about memory usage. It would also be nice to track the time taken in the spark-shell web UI.


Thanks