One of the best tools we have today for understanding application behaviour and troubleshooting performance issues are Java Profilers. Java Profilers monitor JVM execution at the bytecode level and can provide information on thread execution and locks, heap memory usage, garbage collection, hot methods, exceptions, class loading, and more.

How Java Profilers Work

Profilers inspect the state of a JVM either by passively listening for events from the JVM or by actively querying the JVM for its state. Profilers can also modify the bytecode of classes to add instrumentation code, like inserting a methodEntered and methodExit event at the beginning and end of a method or inserting an objectCreated event into a constructor.

There are lots of events that are generated by the JVM, and these can be categorized into two types; Instant events and Duration events. Instant events are one time events that have a timestamp and the event data. Examples include; exception events, class load events and object allocation events. Instant events let us know that something has occurred and can be reacted to or simply watched and analyzed. Duration events have a start time and an end time and are therefore used to provide timing for some activity. Examples include Garbage collection which tell you the start and end time of Garbage collection, Monitor wait which tells you how long a thread waited on an object, and Monitor contended which tells you how long a thread was waiting on a lock to be released.

In addition to the generated events, the JVM also has profiling functions that return data about it's internal state. Some commonly used ones are the thread functions like GetThreadState and GetAllThreads, and the stack trace functions like GetStackTrace and GetAllStackTraces. This internal state data is queried periodically by the profiler in a process known as sampling and the sampling period is how often the functions get called to fetch the data. For example, every 20 milliseconds.

Performance Analysis using Java Profilers

When analyzing application performance, there are a number of profiling activities that are commonly performed by programmers and analysts. These include; CPU Profiling, Memory Profiling and Thread Profiling. In some cases, depending on the type of application, I/O Profiling becomes important as well, but the majority of applications are most concerned with the aforementioned 3.

CPU Profiling is primarily concerned with the frequency and length of time of method execution. With CPU profiling you can find out what methods run the most frequently therefore eat up the most CPU time. These are commonly referred to as "hot methods". This gives you an indication of what methods or areas of the code you should focus on optimizing.



There are 2 ways in which CPU profilers get method execution information; by sampling or by instrumentation.

Sampling profilers work by periodically querying the JVM for all the running threads and getting the stack trace for each thread. It then determines what method each thread was executing when the sample was taken and compares the samples to determine how much time was spent in that method. Profiling by sampling has the least amount of overhead and is therefore suitable for production use. However, because it's an indirect way of measuring method execution, it can be error prone or inaccurate depending on the execution flow of the application and how often the profiler samples. To minimize sampling errors your options are to profile over a longer period of time, reduce the interval between samples, or both. However, reducing the interval between samples results in increasing the overhead on the JVM, therefore there's a trade-off that has to be made between accuracy and overhead.

Instrumenting profilers work by modifying the application's bytecode and inserting code for counting invocations or timing methods. This makes them more accurate in determining method invocation counts and potentially more accurate in determining method timing. However, instrumenting application bytecode has a higher potential for introducing performance differences for two reasons. One is that the instrumentation code will likely have some overhead attached to it and the other is that; depending on how the instrumentation is done there may be some optimizations that could have been applied to the non-instrumented code that can't be applied to the instrumented code.

The next common profiling activity is Memory Profiling. Memory profiling is primarily concerned with understanding what objects are using up memory and how memory is being freed by Garbage Collection. With memory profiling, you can monitor the memory usage of your class objects over a period of time, find out what objects are growing and shrinking in size, and where in your code these allocations are taking place. You can also analyze garbage collections. You can see the number and types of garbage collections that have occurred, the length of the pause times for each garbage collection and how much memory they were able to free.



The third common profiling activity is Thread Profiling. Thread Profiling is primarily concerned with understanding what states threads are in and why. With thread profiling you can see if your threads are able to run in parallel or not, find out what how much time your threads spend sleeping, waiting, blocked or executing, and find and analyze cases of high lock contention. Thread profiling is useful when your application is not as performant as it should be yet its CPU usage is low. In this case your application performance could be hindered by locking, and thread profiling helps you find that out.



There are lots of different Java profilers available today. The screenshots in this article are from Java Flight Recorder, a Java profiler developed and recently open sourced by Oracle. To get started with Java Flight Recorder you can checkout this Youtube video from Oracle.

--
For more information on how to tune your Java application for optimal performance, including an example where I use a profiler on a Spring application, discover some hidden performance flaws, and make some tweaks, checkout my Pluralsight course: Java Performance Tuning. In there I cover tools and techniques for analyzing and troubleshooting Java performance issues, JVM tuning topics, Java profiling, and best practices for writing high performance application code.

Garbage collection is the mechanism by which the JVM reclaims memory on behalf of the application when it's no longer needed. At a high level, it consists of finding objects that are no longer in use, freeing the memory associated with those objects, and occasionally compacting the heap to prevent memory fragmentation.

The garbage collector performs it's work using one or more threads. But in order to do the job of tracking down object references and moving objects around in memory it needs to make sure that the application threads are not currently using those objects because if, for example, an application thread is using an object and then the memory location of the object changes due to GC, then bad and unpredictable things could happen. This is why garbage collectors must pause all application threads when performing certain tasks. These pauses are sometimes called Stop-The-World pauses, and the minimization of them is the primary concern of GC tuning, as they can have a huge impact on the performance of a Java application.

Sizing the heap

The first step in garbage collection tuning is tuning the size of the heap. This is because; if the heap is too small, then too many GCs will occur in order to reclaim memory which would reduce overall application throughput. And if the heap is too big heap, there would be fewer GCs but those GCs would take a long time and your response time metrics would take a hit. The parallel collector is especially vulnerable to this problem, so if you're going to need a large heap and low pause times then you should try the G1GC collector.

Side Note: The Concurrent Mark Sweep (CMS) collector has been deprecated since Java 9 and the Shenandoah Garbage Collector is still considered "experimental" as of the time of this writing. Therefore if you are running an online interactive application then the G1GC should be your default choice and if you are running an offline batch application then the Parallel collector should be your first choice.

The size of the heap is controlled by two values: an initial value, specified with the ms flag, and a maximum value, specified with the mx flag.
-Xms1g -Xmx8g

Having an initial and maximum size for the heap allows the JVM to autotune heap size depending on the workload. If the JVM is experiencing memory pressure and observes that it is doing too much GC, it will continually increase the heap until the memory pressure dissipates, or until the heap hits its maximum size. And if memory pressure is low, the JVM can also decide to reduce pause times by shrinking the heap size. This process is called Adaptive sizing and it adjusts not only the overall size of the heap but also the size and ratio of the young and old generations.

If you have taken the time to finely tune your application's GC behavior and sizes, you may choose to turn off adaptive sizing. This can save the JVM, the small period of time that it takes to do the calculation for what the heap size should be. You can do so by setting the flag UseAdaptiveSizePolicy to false.
-XX:-UseAdaptiveSizePolicy
Also, setting the initial heap size to the same value as the max heap size or the initial new gen size to the same value as the max new gen size effectively turns off part of the adaptive sizing behavior.

A strongly recommended guideline for setting the max heap size is that the max heap size should not exceed the amount of physical memory on the machine. And if you have multiple JVMs running, the sum of the max heap sizes should not exceed the machine's physical memory.

A more general recommendation for setting max heap size is that the size should be set so that the heap is about 30% occupied after a full GC. To calculate this, you can look in the GC log for an entry where a full GC takes place and observe how much memory is used when the GC completes. Or you can run the application until it has reached a steady-state, and then force a full GC with jconsole or jcmd.

Tuning GC performance

If adaptive sizing is turned on, then you can use the MaxGCPauseMillis flag to tune GC behavior. This flag sets a target for the maximum GC pause time. When used with the Parallel collector, the JVM will adjust the size of the young and old generations in order to try and meet the goal. It will then adjust the size of the heap so that the time spent in GC does not exceed a certain value, which is by default 1%.

One of the goals of G1GC was that it would need minimal tuning. So in G1GC, the one tuning parameter, MaxGCPauseMillis, performs all the following optimizations in order to try to achieve the specified pause time goal:

• adjust the size of the heap
• start background processing sooner,
• adjust the tenuring threshold for objects to be promoted to the old generation,
• adjust the number of old regions processed during a mixed GC cycle.

In G1GC, default value of the flag is 200 ms. While you may be tempted to set it to something really small like 20 ms, note that to try to achieve this, the garbage collector will contract the young generation to a really small size and collect less of the old generation which would eventually lead to a situation where there's too much garbage in the old generation and the system would have to perform a full GC which is undesirable.

Fixing Concurrent Mode Failures

G1GC is a concurrent collector. This means that some phases of the garbage collection process can be running concurrently while the application threads are still running. And since the running application can continue to produce garbage, we can run into a situation where the application runs out of old generation memory while the garbage collector is still in the middle of the garbage collection process. In other words, the running application is producing garbage faster than it can be cleaned. This situation is known as either a concurrent mode failure, a promotion failure or an evacuation failure depending on when the failure occurs. If you see a lot of these errors in the GC logs; the solution is to either increase the size of the heap, start the G1 background processing earlier, or speed up GC processing by using more background threads.

To performing G1 background activities more frequently, you can reduce the threshold at which a G1 cycle is triggered. This is achieved by reducing the value of the InitiatingHeapOccupancyPercent flag.
-XX:InitiatingHeapOccupancyPercent=45
This flag is set to 45 by default. This means that a GC cycle is triggered when the heap becomes 45% filled. Reducing this value means GC would get triggered earlier and more often. But care should be taken that the value is not set to a number that's too low which would result in GCs happening too frequently.

To increase the number of background threads use the ConcGCThreads flag.
-XX:ConcGCThreads=4
The default value for this flag is set to the value of ParallelGCThreads plus 2, divided by 4. As long as you have sufficient CPU available on the machine, you can increase this value without incurring any performance penalties.

If tuning the heap size and tuning the collector doesn't work for you then you can try another collector. And if you still aren't getting good results then you need to look at tuning the application code itself.

--
For more information on how to tune your Java application for optimal performance, checkout my Pluralsight course: Java Performance Tuning. In there I cover tools and techniques for analyzing and troubleshooting Java performance issues, JVM tuning topics, Java profiling, and best practices for writing high performance application code.