Java Heap and Thread Dumps on SAP BTP Cloud Foundry

[former_member599840]

Java heap dumps and thread dumps are a powerful way to investigate what's going on in the Java Virtual Machine (JVM). A heap dump allows you to see what is kept in memory after a minor and major garbage collection. Therefore, heap dumps provide indispensable insights which can be used for troubleshooting memory leaks.

This blog post is the third and final part (following part 1 and part 2) of our blog series focusing on Java memory management in container environments. We will show how to create heap dumps and thread dumps on demand for Java applications running on SAP BTP Cloud Foundry.

The cf java plugin

Heap dumps can be created on demand by requesting them from the JVM via tools like jvmmon or jcmd. Which tool and command to use depends on the JVM vendor as well as the Java buildpack which is used on SAP BTP Cloud Foundry. To make it as simple as possible to create heap dumps and thread dumps on demand, we've created the cf java plugin plugin.

The cf java plugin is installed as an addition to the Cloud Foundry Command Line Interface (CF CLI), which is used to manage applications on SAP BTP Cloud Foundry.

You can add the cf java plugin via the command line:

$ cf add-plugin-repo CF-Community http://plugins.cloudfoundry.org

$ cf install-plugin -r CF-Community "java"

Once installed, you can call the cf java plugin. To view all commands and options supported by the plugin, you can run cf java --help:

$ cf java --help 

NAME:

   java - Obtain a heap-dump or thread-dump from a running, SSH-enabled Java application.



USAGE:

   cf java [heap-dump|thread-dump] APP_NAME



OPTIONS:

   --app-instance-index      -i [index], select to which instance of the app to connect

   --container-dir           -cd, the directory path in the container that the heap dump file will be saved to

   --dry-run                 -n, just output to command line what would be executed

   --keep                    -k, keep the heap dump in the container; by default the heap dump will be deleted from the container's filesystem after been downloaded

   --local-dir               -ld, the local directory path that the dump file will be saved to

The cf java plugin uses an SSH tunnel to connect to a selected application APP_NAME and instance --app-instance-index, to trigger a heap dump or thread dump with the JVM specific tools and commands. That said, you need to ensure that your application has the necessary JDK tools installed and SSH access to your application is enabled. By default, SSH access is disabled:

$ cf ssh-enabled APP_NAME

ssh is disabled for app

You can enable SSH access to your application APP_NAME via the following command and a restart of your application:

$ cf enable-ssh APP_NAME

$ cf restart APP_NAME

Another important consideration is the disk space which is available to your application. By default, every application gets a disk of 1 GB attached to the application container. In theory, a heap dump of a Java app can get as large as the maximum heap size (-Xmx) configured for the JVM. Therefore, it's not unusual for a heap dump to exceed 1 GB of the available disk space. You can request more disk space via the following property in your manifest file (with a maximum total disk space of 4 GB):

---

  ...

  disk_quota: 4G

Note that a change to the manifest file requires you to push this change via cf push. If you don't have the manifest file at hand, you can also use the following command to increase the disk quota:

$ cf scale APP_NAME -k 4G

After checking off all the prerequisites, you can create a heap dump using the cf java plugin and transfer it to your local computer:

$ cf java heap-dump APP_NAME --local-dir FOLDER

For example, to create a heap dump of the application with the name spring-music and save it into the current folder on your local computer, you can run:

$ cf java heap-dump spring-music --local-dir .

Successfully created heap dump in application container at: /var/vcap/data/fcb80119-7117-4682-8285-d7d4740cdf66/java_pid12_5.hprof

Heap dump filed saved to: ./spring-music-heapdump-231be9d8-d3a5-403f-aa44-14637c4510d3.hprof

Heap dump filed deleted in app container

With the generated heap dump, you can now use a heap analyser tool of your choice, for example the Eclipse Memory Analyzer (MAT).

It is also possible to create a thread dump by using the thread-dump command:

$ cf java thread-dump APP_NAME > thread.dump

With this setup, you are now able to create heap dumps and thread dumps in no time!

[former_member487841]

Great blog Tim.
Can you suggest the right tool for live monitoring especially to analyze heap, and gc logs.

[former_member599840]

Hi msaadr, thank you for your kind feedback.

There are several ways to get live monitoring of JVM metrics:

• You can use the Application Logging Service and its capability to add custom metrics to include JVM metrics. I will try to get back to you with more documentation or see if we can get one of my colleagues to prepare a follow-up blog post.


• For ad-hoc analysis you can use tools like JConsole or VisualVM via cf ssh after enabling JMX extension through the buildpack configuration (let me know if you want more details as the configuration is a little bit tricky).


• You can set up Prometheus for your Cloud Foundry applications to expose and scrape a variety of metrics from the application and the runtime (a tutorial will be available soon).


• You can use an Application Performance Monitoring (APM) tool. The SAP Java Buildpack has built-in support for Dynatrace, for example. The Cloud Foundry Java Buildpack has even more such integrations for more APM vendors.

Let me know if you have any further questions.

[rampandu]

Hi Tim,

This blog was very helpful in understanding ssh to cf app. However after enable-ssh and restart, I am still getting "ssh is not enabled for app". I have tried restart, restage, and even pushing the app again. But no luck. What else could be wrong?

 

Thanks,

Ram

[rampandu]

After setting the java version in the manifest file and pushing the app again, I am able to collect the dumps.

ex:

JBP_CONFIG_OPEN_JDK_JRE: '{ jre: { repository_root: "https://java-buildpack.cloudfoundry.org/openjdk-jdk/bionic/x86_64", version: 11.+ } }'

Thanks,

Ram

[former_member599840]

Hi Rama,

thanks a lot for your valuable feedback. I have updated the blog and added a link to the JDK tool requirements to make sure that all prerequisites are mentioned in this blog.

Thanks,

Tim

[former_member780640]

Hi tim_gerlach ,

Blog is really good.Can you please provide more details on best Heap analyser tool and live monitoring of JVM metrics.

Thank you.

[former_member599840]

Hi gaurish,

Thanks for your feedback.

If you already have heap dump you can use the Memory Analysis Tool from Eclipse as mentioned at the end of the blog.

Starting with Java 11 you Application Performance Monitoring tools like Dynatrace can also provide you with a continuous (live) heap analysis which make the need for a heap dump a little less important.

Best regards,

Tim

[former_member438762]

Hi Tim.

for my application, in CF, it shows, 2GB is consumed by the application.

 

when I create a heap dump for it . it shows only 450MB used.

do you have any idea how can I get the real memory allocated?

Regards

Eric

[AntalP]

Hi Eric,

 

The "Memory" shows the OS level memory usage, it is allocated/reserved by the JVM.

When a heap dump is generated, the JVM performs a full GC, then it saves the heap content to the dump file.

SAP Note 1824799 - Memory usage of the SAP JVM

provides some details.

 

Best regards,

Antal

[former_member230446]

Hi Tim,

 

Thank you for the blog... All the steps are neatly explained..

 

I tried to take heap dump of our application and incresed the disk space to 4G.

 

After that also I am getting below error:

Dumping heap to /var/vcap/data/xyz/app-nameheapdump-c14e88da-f2f3-4cbd-b71f-d896e5fb3a40.hprof ... Unable to create /var/vcap/data/xyz/app-name-heapdump-c14e88da-f2f3-4cbd-b71f-d896e5fb3a40.hprof: No space left on device

 

Kindly check and help us..

is increasing memory is a option??

 

Regards,

Lekhak Patil

[former_member599840]

Hi Lekhak,

depending on the size of the JVM there should be enough on the disk for at least one heap dump. Can you please check that the app is entitled to use 4G of disk size and verify via ssh if there's already any other heap dump that was created on the disk. You can check via cf ssh APP_NAME. If this doesn't help, can you pleas e let me know how much memory is assigned to your application?

Best regards,

Tim