In this article, I will be sharing some of the best tips to analyse the out of memory issue in java applications. My main focus would be to cover some important details related to garbage collection and its contribution to java.lang.OutOfMemoryError: GC Overhead Limit Exceeded.
Before diving into the issue, let’s understand the common basics terminologies in simplest form to carefully analyse the heap later.
- Heap : Part of memory allocated to JVM used for dynamic allocation of objects which are accessed globally across the threads in the application.
- Heap Allocations : Heap is divided into different parts to manage different generations of objects. The heap parts are: Young Generation, Old or Tenured Generation, and Permanent Generation.
- Garbage collection : It is a process of identifying objects in the heap which are not in use and deleting their reference to reclaim the memory.
- Young Generation : This area holds new objects created in the program.
- Minor GC : JVM periodically triggers Minor GC to clean up the objects created in Young Generation. This is a very small duration event having almost no impact in program processing.
- Old Generation : This area holds the objects which survived the cycles of “minor GC” and have been promoted when the age of objects in Young Generation crossed the threshold.
- Major GC : This is popularly known as “Stop The World” GC , this is triggered to clear the Old Generation space. This GC pauses all the threads of the application and hence for throughput and latency sensitive applications major GC must be minimised.
VisualVM is best tool for debugging as it had some really good default visualisers and plugins to dig deeper into each space of memory as well as CPU. Consider installing Visual GC plugin it helps to visualise the GC happening in real time in each distribution of heap area. It gives a crisp idea of how and where the memory is growing and whether the GC is reclaiming the memory or not.
Analysing JVM memory issues require debugging the heap area in detail and what is interesting and often overlooked is the contribution of CPU cycles on going in the program. We can analyse thread by thread memory and CPU consumption with the help of CPU and memory sampler. It helps us to determine where is the CPU time spent method wise which really benefits to optimise the code.
Talking about java.lang.OutOfMemoryError: GC Overhead Limit Exceeded error, it happens when the application is spending more time doing GC than the actual work. It is an alarming situation because it freezes entire application work for long time. Giving some context of what our application does..it was trying to maintain a queue of incoming messages and publishing those messages to the subscribers in multiple threads over the websocket . The messages were of type byte buffer.
Observations during High GC cycles :
a. During the memory profiling using visual GC we carefully observed the relation between GC cycles and its effect on memory. Every pause had an effect on all the threads of application.
b. Whenever there was full GC, the websocket connection over which the server was receiving messages used to pause, which led to the connection getting restarted due to ping-pong failure as the thread was paused for longer time than the timeout set over connection. There used to be a spike as the paused threads start to retry their connections.
c. On the other side, the connections made by end users to the server starts choking the entire processing of the thread pool which inturn halted the thread to run which was the manager thread.
d. Due to these many halts, the eden space begins to grow quickly so minor GC starts believing these are long-lived objects as the publisher thread was paused and hence it tries to move the objects to the old generation space which started filling it even faster.
The speed with which the queue of incoming messages was getting filled was not getting emptied by the publishers and this resulted in the old generation heap space getting filled completely leading to even longer pauses for full GC and the space was not getting reclaimed at all….ultimately we had to restart our application.
After several tireless rounds of analysis and testing, we found the ray of hope with CPU and memory sampler which accurately helped us to see where most of the CPU time is spent function by function. This was the eureka moment !! We found that the maximum time it was spending in the array list initialisation. So, the part of code which was copying the bytes of message from queue into an array list was the culprit. It was generating way more objects than we imagined and it led to filling up of memory too quickly.
Remedies :
- Removed all the unnecessary object creations from during incoming message processing.
- Removed static declaration of concurrent maps in the code as static objects life cycle is almost same as the application life.
- Started processing batches of connections together in a single thread.
- Sending batches of packets to those users instead of single packet to empty the queue faster using multiple threads.
All these small optimisations, helped us reduce GC cycles + we were able to manage the memory even more efficiently. This single websocket application was able to handle 10k connections with zero to minimum latency in just a heap size of min : 400 mb , max : 2.4 GB. All thanks to visual VM !!
I hope this article helps you and if you find this useful, share it with you team mates and do let me know what challenges you had faced while developing web socket applications :)
