Tuning Java Container Memory for Java apps running in Containers can be quite a daunting task.
There are lot's of guides on this topic. The blog will provide yet another look on them, along with links to all useful articles.
The core problem stems from the fact, that Java has quite a lax allocation of resources, build on relying for years on a dynamic swap file. Now with containers in mind, this all has to go into the container quota. Sadly only part of it is what a typical Java Developer is used to count - namely - the Java Heap. Depending on the scenario, the Java Heap can occupy even less than 50% of the actually required memory.
Read on to understand what is important in measuring the required memory and a proposal for a process how to identify the limits
Key Points
- Error 137 (out of memory) - this error is thrown when the apps in the container try to consume more memory than requested
- Using any version of the Memory Calculator could potentially lead to unexpected results, so the only way to properly estimate the memory usage of the app is to let it run for multiple hours and monitor the metrics at the end.
- Some Java Memory theory (find more details in the references on the bottom):
- The memory used by the Java process in a container can be obtained by SSH-ing into the container and using "ps aux" and observing the "RSS" column (measured in KiB).
- The sum of the RSS columns of all running processes should not go beyond the preset limit for the memory of the container
- The RSS memory of the Java process is the real memory used. A lot can be read about this, but in short:
- The used (RSS) Memory can be split into 3 parts
- Java Heap
- Reserved via e.g -Xmx2000m -Xms2000m -XX:+AlwaysPreTouch (the latter ensures that the memory is indeed consumed by the process and not just reserved
- Off-heap (known to java)
- This is also called Native Memory and can be observed via jcmd `pidof java` VM.native_memory (and -XX:NativeMemoryTracking=summary has to be included as parameter to the java process.
- The top line says "Total: reserved=4556154KB, committed=3574882KB". We are interested in the "committed" value as this is what Java thinks resides in the RAM. This contains the Heap + OffHeap_Java parts
- The off-heap memory is split into various buckets. Some can be restricted, others need to be properly sized, else Java OOMs would come (more on this can be read in the references below)
- From each bucket, only the "committed" parts are what is part of the memory, the rest is just reserved, and may never be claimed
- Off-heap (unknown to java)
- There third part of the memory is not known to java and includes
- Native Memory used by libraries
- Wasted memory due to the way java reserves blocks of memory
- potentially other buckets (there have been also SAPJVAM bugs that result in memory leaks in native mem)
- As a result RSS = JavaHeap + OffHeapJava + OffHeapOthers
- The OffHeapJava memory can be quite unpredictive due to:
- Thread Stack Space - the maximum reserved space per thread (note - not the committed, but reserved) can be limited via -Xss=1m or less.
- One need to estimate how much memory to reserver for thread stack, which can be different for all threads
- Also depending on the application usecase, a thread may use more or less memory at different times
- E.g. An application that has 500 threads, would need to reserve 500mb in this case. But most of the times, threads use not more than 10% of this. So the app would pretty much work well with 50 mb "comitted" ram for threads. And most of the time people would thus overcommit the thread space. Yet if the container is too tightly sized, and can not accommodate increased memory, at some point of time if multiple threads need more memory than the 10% planned - the process may crash
- Garbage Collector type - Serial garbage collector uses 10 mb RAM, but G1 uses 200 mb of ram. They do not grow, but need to be taken in mind
- JIT Compiled classes
- Java optimizes the class bytecode overtime, depending on how much a method is invoked. The compiled classes go into the RAM. The space is limited, but by default it is 240 mb. Some app can get around with 20 mb, others may need more than 240mb, which would like to garbage collecting the excess classes.
- Since this is done over time, the comitted space grows slowly from 0 mb, to the maximum reserved space for this bucket
- The OffHeapOther - can also be prone to growing due to fragmentation of the space. See this [interesting article|https://medium.com/nerds-malt/java-in-k8s-how-weve-reduced-memory-usage-without-changing-any-code-cb....] Depending on the use-case it may grow to a certain extent and settle or grow more
How to properly size the Java Process and the Container
- Some buckets in the memory map outlined above depend on scale, but others - just on the application architecture
- Application Architecture dependent memory sizes
- Number of loaded classes (Metaspace)
- Symbols (interned strings) used
- JIT Compiled classes
- Garbage Collector type
- Maybe some of the native memory used by the apps, depending on the type of library
- Scale dependent memory sizes
- Number of threads active at the same time - they need a certain amount of stack space
- Maybe some of the native memory used by the apps, depending on the type of library
- Heap
- Part of the heap is required just to hold the baseline objects required to run the app. That is - it will never be garbage collected, while the app is in use
- Other part would be required for each thread that is being used to perform some calculations. The more threads, the more memory needed to actually run the app
- And the final part of the heap is the final one, that contains dynamically created objects, that can be safely garbage collected
- Garbage Collectors usually are very quick and manage to recover a lot of objects, but not all. To recover all objects, they need to stop all threads and search for them. This can take several seconds.
- To properly size the application the following considerations need to be made
- Sizing the heap
- It is clear that the minimum size of the heap should be able to contain he basis objects needed by the app, plus other basis objects needed for each thread. Any additional runtime objects needed by the app would be created when needed and then garbage collected
- If Java encounters that garbage collection takes more than 98% of the CPU time, it would throw an OutOfMemoryException.
- But any amount of GC time above 1% is time spent by the app collecting objects instead of doing work
- So a properly sized Heap would allow the app to work under normal load, w/o spending big amount of time in Garbage Collection
- Sizing the OffHeap_Java
- It is best to let the app run under the expected load or even under stress load, for several hours and then using "jcmd <pid> VM.native_memory" monitor the sizes of each bucket. E.g. how much memory went for Class bytecode, how much for Class jit compilation, and Threads. Other buckets are more or less static
- This will give an overview on
- How much Metaspace will be needed - Java will not commit more than it needs, but if needs more than the reserved - it will crash
- How much Jit Compiled space is needed - By default 240 mb are reserved. If java needs more it will just garbage collect some compiled classes. This may lead to performance degradation. So this needs to be estimated well. Also it is important to not be misled, if the committed size is less than the maximum. As this could lead to incorrect computation of RSS needed for the container. This is why it is important that the committed space is monitored over time
- How much Thread Stack Space is used - As already discussed, maybe it is hard to actually judge this by merely sampling data
- Sizing the OffHeap_NonJava
- The difference between RSS and the Commited RAM as per jcmd, is the OffHeap_NonJava.
- It also tends to grow over time, until it platoes after certain load to the app
Once the 3 numbers are clear OffHeap_NonJava, OffHeap_Java, Heap - their sum pretty much will determine the RSS memory needed by the container to run properly (or run with performance bottlenecks if the Heap in not enough (or some of the OffHeap parts)
References
