How to log Shuffle read/write size

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to log Shuffle read/write size

Chen Jin
Hi All,

Apart from getting the shuffle read/write size from application UI, is
there any way to enable the metrics in the log file?

Thank you very much,

-chen
Reply | Threaded
Open this post in threaded view
|

Re: How to log Shuffle read/write size

Andrew Or
Hi Chen,

Have you looked at JobLogger.scala? You can register it with SparkContext and it will log task metrics onTaskEnd. Unfortunately, there is currently not a configurable option to enable this. We are looking into a more systematic way to persist the UI information in a more flexible format (JSON).

Andrew


2014-02-11 11:14 GMT-08:00 Chen Jin <[hidden email]>:
Hi All,

Apart from getting the shuffle read/write size from application UI, is
there any way to enable the metrics in the log file?

Thank you very much,

-chen