How to export the Spark SQL jobs from the HiveThriftServer2

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to export the Spark SQL jobs from the HiveThriftServer2

wenxing zheng
Dear all,

I have a HiveThriftServer2 serer running and most of our spark SQLs will go there for calculation. From the Yarn GUI, I can see the application id and the attempt ID of the thrift server. But with the REST api described on the page (https://spark.apache.org/docs/latest/monitoring.html#rest-api), I still can't get the jobs for a given application with the endpoint: /applications/[app-id]/jobs

Can anyone kindly advice how to dump the spark SQL jobs for audit? Just like the one for the MapReduce jobs (https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html).

Thanks again,
Wenxing
Reply | Threaded
Open this post in threaded view
|

Re: How to export the Spark SQL jobs from the HiveThriftServer2

wenxing zheng
the words: [app-id] will actually be [base-app-id]/[attempt-id], where [base-app-id] is the YARN application ID. is not so correct. As after I changed the [app-id] to [base-app-id], it works. 

Maybe we need to fix the document?

From the information of the spark job or the spark stages, I can't see the statistics on the memory part? Appreciated for any hints

On Wed, Dec 6, 2017 at 2:08 PM, wenxing zheng <[hidden email]> wrote:
Dear all,

I have a HiveThriftServer2 serer running and most of our spark SQLs will go there for calculation. From the Yarn GUI, I can see the application id and the attempt ID of the thrift server. But with the REST api described on the page (https://spark.apache.org/docs/latest/monitoring.html#rest-api), I still can't get the jobs for a given application with the endpoint: /applications/[app-id]/jobs

Can anyone kindly advice how to dump the spark SQL jobs for audit? Just like the one for the MapReduce jobs (https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html).

Thanks again,
Wenxing