Tuning G1GC params for aggressive garbage collection?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Tuning G1GC params for aggressive garbage collection?

Akshay Mendole
Hi, 
 
My JVM is basically a spark executor which is running tasks one after another. A task is memory hungry and requires significant memory during its lifecycle.

Screen Shot 2018-12-25 at 3.48.51 PM.png
The above JVM is running on G1GC with default params. As you can see in the VisualVM report on the right hand side between 4:25 to 4:32 PM, the spikes are due to each tasks run by the executor (essentially, each spike is due to the executor picking up new task after the previous one is finished). When I triggered a manual GC at 4:35, I saw a sharp decline the heap usage. Also, as you can see on the left hand side in the JConsole report, the old gen space is never collected by G1GC (the sharp decline in old gen space just before 16:35 is due to the manual GC). 

As my application is a spark batch job application, I am ok if the JVMs spent good amount of time doing GC. But, I am running bit short on memory. So, I wanted to know how I can tune my JVM G1GC params so that there are more frequent GC (with the old gen space also getting collected) and I can get the work done with considerably lesses heap space (XMX).

I tried changing -XX:InitiatingHeapOccupancyPercent . I tried with 0, 5, 10. Lower the value, more frequent was the GC and more CPU it would consume, but the behaviour would not be consistent. After sometime, the heap usage would spike up and if we have set the XMX value lesser (6 GB) than what's there in the picture above, it would throw OOM.

Something that is really working for us is attaching a daemon thread that's calling System.gc() after an interval of 1 minute in each executor. But, I wanted to know how we can gracefully tune our GC for the same that gives the same effect as the manual System.GC() calls.

Thanks,
Akshay

Reply | Threaded
Open this post in threaded view
|

Re: Tuning G1GC params for aggressive garbage collection?

ramannanda9@gmail.com
Hi, 

Did you try increasing concurrentgcthreads for the marking? 

System.gc is not a good way to handle this, as it is not guaranteed and is a high pause,full gc. 

Regards,
Ramandeep Singh

On Tue, Dec 25, 2018, 07:09 Akshay Mendole <[hidden email] wrote:
Hi, 
 
My JVM is basically a spark executor which is running tasks one after another. A task is memory hungry and requires significant memory during its lifecycle.

Screen Shot 2018-12-25 at 3.48.51 PM.png
The above JVM is running on G1GC with default params. As you can see in the VisualVM report on the right hand side between 4:25 to 4:32 PM, the spikes are due to each tasks run by the executor (essentially, each spike is due to the executor picking up new task after the previous one is finished). When I triggered a manual GC at 4:35, I saw a sharp decline the heap usage. Also, as you can see on the left hand side in the JConsole report, the old gen space is never collected by G1GC (the sharp decline in old gen space just before 16:35 is due to the manual GC). 

As my application is a spark batch job application, I am ok if the JVMs spent good amount of time doing GC. But, I am running bit short on memory. So, I wanted to know how I can tune my JVM G1GC params so that there are more frequent GC (with the old gen space also getting collected) and I can get the work done with considerably lesses heap space (XMX).

I tried changing -XX:InitiatingHeapOccupancyPercent . I tried with 0, 5, 10. Lower the value, more frequent was the GC and more CPU it would consume, but the behaviour would not be consistent. After sometime, the heap usage would spike up and if we have set the XMX value lesser (6 GB) than what's there in the picture above, it would throw OOM.

Something that is really working for us is attaching a daemon thread that's calling System.gc() after an interval of 1 minute in each executor. But, I wanted to know how we can gracefully tune our GC for the same that gives the same effect as the manual System.GC() calls.

Thanks,
Akshay



---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Screen Shot 2018-12-25 at 3.48.51 PM.png (818K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Tuning G1GC params for aggressive garbage collection?

Akshay Mendole
Hi,
     Yes. I did try increasing to the value of number of cores. It did not work as expected. I know System.gc is not a proper way. But, since this is a batch application, I'm okay if it spends more time in doing GC and takes considerably more cpu. Frequent system.gc calls with lower xmx (6 GB) gave us more throughout as compared to no manual gc calls with higher xmx (8 GB). If I don't do manual gc calls when xmx is 6 GB the executor would go oom. I was just wondering what Could be the best tuning parameters that would give similar effect. 
Thanks, 
Akshay 

On Tue, 25 Dec 2018, 11:28 pm Ramandeep Singh Nanda <[hidden email] wrote:
Hi, 

Did you try increasing concurrentgcthreads for the marking? 

System.gc is not a good way to handle this, as it is not guaranteed and is a high pause,full gc. 

Regards,
Ramandeep Singh

On Tue, Dec 25, 2018, 07:09 Akshay Mendole <[hidden email] wrote:
Hi, 
 
My JVM is basically a spark executor which is running tasks one after another. A task is memory hungry and requires significant memory during its lifecycle.

Screen Shot 2018-12-25 at 3.48.51 PM.png
The above JVM is running on G1GC with default params. As you can see in the VisualVM report on the right hand side between 4:25 to 4:32 PM, the spikes are due to each tasks run by the executor (essentially, each spike is due to the executor picking up new task after the previous one is finished). When I triggered a manual GC at 4:35, I saw a sharp decline the heap usage. Also, as you can see on the left hand side in the JConsole report, the old gen space is never collected by G1GC (the sharp decline in old gen space just before 16:35 is due to the manual GC). 

As my application is a spark batch job application, I am ok if the JVMs spent good amount of time doing GC. But, I am running bit short on memory. So, I wanted to know how I can tune my JVM G1GC params so that there are more frequent GC (with the old gen space also getting collected) and I can get the work done with considerably lesses heap space (XMX).

I tried changing -XX:InitiatingHeapOccupancyPercent . I tried with 0, 5, 10. Lower the value, more frequent was the GC and more CPU it would consume, but the behaviour would not be consistent. After sometime, the heap usage would spike up and if we have set the XMX value lesser (6 GB) than what's there in the picture above, it would throw OOM.

Something that is really working for us is attaching a daemon thread that's calling System.gc() after an interval of 1 minute in each executor. But, I wanted to know how we can gracefully tune our GC for the same that gives the same effect as the manual System.GC() calls.

Thanks,
Akshay