Get application id when using SparkSubmit.main from java
I am trying to get the application id after I use SparkSubmit.main for a yarn submission.
I am able to make it asynchronous using spark.yarn.watForCompletion=false configuration option, but I can't seem to figure out how I can get the application id for this job. I read both SparkSubmit.scala and Client.scala.
Any thoughts on how I could do it? I'd prefer not to use Client.run directly that returns the application id since there's a lot of environment prep that SparkSubmit actually does.