Spark Streaming job having issue with Java Flight Recorder (JFR)

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark Streaming job having issue with Java Flight Recorder (JFR)

Pramod Biligiri
Hi,
Has anyone successfully used Java Flight Recorder (JFR) with Spark Streaming on Oracle Java 8? JFR works for me on batch jobs but not with Streaming.

I'm running my streaming job on Amazon EMR. I have enabled Java Flight Recorder (JFR) to profile CPU usage. But at the end of the job, the JFR output files get deleted and are not available. This is the error in the JFR trace logs:
[jfr][DEBUG][880.590] Release chunk /mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr.part ref count now 0
[jfr][DEBUG][880.590] Repository chunk /mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr.part deleted.
[jfr][DEBUG][880.590] Repository chunk /mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr deleted.
[jfr][INFO ][880.593] Stopped recording [id=1, name=dump-on-exit-clone-of-0, start=Wed Feb 19 17:26:06 UTC 2020, end=Wed Feb 19 17:40:45 UTC 2020]
[jfr][ERROR][880.593] Could not copy to /tmp/hotspot-pid-8823-id-0-2020_02_19_17_40_45.jfr
[jfr][ERROR][880.593] java.io.FileNotFoundException:No chunks
[jfr][ERROR][880.593]    oracle.jrockit.jfr.ChunksChannel.<init>(ChunksChannel.java:26)
[jfr][ERROR][880.593]    oracle.jrockit.jfr.Recording.copyTo(Recording.java:404)
[jfr][ERROR][880.593]    oracle.jrockit.jfr.JFRImpl.dumpOnExit(JFRImpl.java:802)
[jfr][ERROR][880.593]    oracle.jrockit.jfr.JFRImpl.dumpOnExit(JFRImpl.java:789)
[jfr][ERROR][880.593]    oracle.jrockit.jfr.JFRImpl.destroy(JFRImpl.java:767)
[jfr][ERROR][880.593]    oracle.jrockit.jfr.JFR$2.run(JFR.java:208)
[jfr][ERROR][880.593]    java.lang.Thread.run(Thread.java:748)
[jfr][ERROR][880.593] Could not delete /tmp/hotspot-pid-8823-id-0-2020_02_19_17_40_45.jfr
[jfr][ERROR][880.593] java.io.FileNotFoundException:No chunks
[jfr][ERROR][880.593]    oracle.jrockit.jfr
The rest of the trace can be found in the attachment (stdout.txt).
Also, I've not changed the default value of "spark.streaming.stopGracefullyOnShutdown" (its default value is false)

The Spark driver and executor JVMs are launched with the following options:
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:FlightRecorderOptions=defaultrecording=true,disk=true,repository=/tmp,maxage=2h,dumponexit=true,dumponexitpath=/tmp,loglevel=trace

Pramod


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

stdout.txt (33K) Download Attachment