Websphere JVM hang issue : How to create heap or thread dump

You should check if application server process is running to determine a crash. To do this, you need to know process ID of application server. You can find process ID at server name.pid file in:

<WAS_install_root>/profiles/<profile>/logs/<server> For Exa : /opt/IBM/WebSphere/AppServer/profiles/Profile01/dmgr/logs/dmgr/dmgr.pid

Open the <server_name>.pid file in a text editor. The four-digit number is a process ID. You can use appropriate operating system command to check if process is actively running. If it’s not running, then you have a crash.

What is a Thread Dump? (Java Core Dumps) javacore.<PID><TIME>.txt

A thread dump is a dump of the stacks of all live threads. Thus useful for analysing what an app is up to at some point in time, and if done at intervals handy in diagnosing some kinds of ‘execution’ problems (e.g. thread deadlock).

When to generate ? : If you get unexplained server hangs under WebSphere, you can obtain, from the WebSphere server, a thread dump to help diagnose the problem.
In the case of a server hang, you can force an application to create a thread dump.

If an application server spontaneously dies, look for a file. The JVM creates the file in the product directory structure, with a name like javacore[number].txt

What is a heap dump? heapdump.<PID><TIME>.phd

A heap dump is a “binary dump” of the full memory the JVM is using, and is for example useful if you need to know why you are running out of memory – in the heap dump you could for example see that you have one billion User objects, even though you should only have a thousand, which points to a memory retention problem.

When to generate? : Memory leaks in the Java heap produce java.lang.OutOfMemoryError exceptions in log files. However, not all out-of-memory errors are caused by Java heap memory leaks. Out-of-memory errors can also be caused by the following conditions:
Java heap fragmentation. This fragmentation occurs when no contiguous chunk of free Java heap space is available from which to allocate Java objects. Various causes for this problem exist, including the presence of pinned or dosed objects or because of the repeated allocation of large objects.
Memory leaks in native heap. This problem occurs when a native component, like DB2 connections, is leaking.

How to create Thread Dumps (Java Core Dumps)/Heap Dumps using wsadmin.sh

1. Navigate to cd <WAS_ROOT>/profiles/<PROFILE_NAME>/bin/

2. Connect to deployment manager using wsadmin script
wsadmin.sh <DMGR_HOST> <PORT> -conntype SOAP -username <USERNAME> -password <PASSWORD>

3. Set object variable
wsadmin> set jvm [$AdminControl completeObjectName type=JVM,process=<JVM_NAME>,node=<NODE_NAME>,*]

4. Create HeapDump:

wsadmin>$AdminControl invoke $jvm generateHeapDump
/opt/IBM/WebSphere/AppServer/profiles/Profile01/Node01/./heapdump.20100202.121506.27816.0001.phd

5. Create ThreadDump:

wsadmin>set jvm [$AdminControl completeObjectName type=JVM,process=member2,*]

wsadmin>$AdminControl invoke $jvm dumpThreads

6. Heap or thread dump will be saved to <WAS_ROOT>/profiles/<PROFILE_NAME>/ directory with with respective naming convention

Create Thread dumps using “kill -3″ command

Add following settings:
Navigated to: Servers > Application Servers > Server1 (or the name of  the server to get a heap dump) > Process Definition > Environment Entries

Then set following properties:
IBM_HEAPDUMP = true
IBM_HEAP_DUMP = true
IBM_JAVA_HEAPDUMP_TEXT=true
IBM_HEAPDUMP_OUTOFMEMORY=false
JAVA_DUMP_OPTS=ONANYSIGNAL(JAVADUMP[5],HEAPDUMP[5])

Here export JAVA_DUMP_OPTS=”ONANYSIGNAL(JAVADUMP[n],HEAPDUMP[m])”
- n is the maximum number of javacores that can be generated, and
- m is the maximum number of heapdumps that can be generated

export JAVA_DUMP_OPTS=”ONANYSIGNAL(JAVADUMP[5],HEAPDUMP[5])”
A kill -3 to the java process will generate a maximum of 5 javacore and 5 heapdump files.

Now using “kill -3 <AppServer PID>” should create a HeapDump & ThreadDump