Before you begin
Although heap dumps are generated only in response to a detected memory leak, you must understand that generating heap dumps can have a severe performance impact on WebSphere Application Server for several minutes. When generating multiple heap dumps manually for memory leak analysis, make sure that significant objects are leaked in between the two heap dumps. This approach enables problem determination tools to identify the source of the memory leak.
About this task
You might want to manually generate heap dumps for the analysis of memory leaks. On a Java virtual machines (JVM) in WebSphere Application Server, you cannot enable automated heap dump generation. You might want to designate certain times to take heap dumps because of the overhead involved. On JVM in WebSphere Application Server, you can manually produce heap dumps by using the generateHeapDump operation on WebSphere Application Server managed beans (MBeans) that are special Java beans.
The WebSphere Application Server wsadmin tool provides the ability to run scripts. You can use the wsadmin tool to manage a WebSphere Application Server installation, as well as configuration, application deployment, and server runtime operations. WebSphere Application Server supports the Jacl and Jython scripting languages only. To learn more about the wsadmin tool, see the Starting the wsadmin scripting client topic.
Procedure
Start the wsadmin scripting client. You have several options to run scripting commands, ranging from running them interactively to running them in a profile.
Invoke the generateHeapDump operation on a JVM MBean, for example,
Finding JVM objectName:
WebSphere:type=JVM,process=
Invoking the generateHeapDump operation on JVM MBean:
where,
$
is a Jacl operator for substituting a variable name with its value
invoke
is the command
generateHeapDump
is the operation you are invoking
is the name of the server on which you want to generate a heap dump
is the node to which
Important to AMD 64-bit users: You must specify the -Xtrace option to take heap dumps
Locating and analyzing heap dumps
Use this task to locate and analyze heap dumps.
Before you begin
Do not analyze heap dumps on the WebSphere Application Server machine because analysis is very expensive. For analysis, transfer heap dumps to a dedicated problem determination machine.
About this task
When a memory leak is detected and heap dumps are generated, you must analyze heap dumps on a problem determination machine and not on the application server because the analysis is very central processing unit (CPU) and disk I/O intensive.
Perform the following procedure to locate the heap dump files.
Procedure
On the physical application server where a memory leak is detected, go to the WebSphere Application Server home directory. For example, on the Windows operating system, the directory is:
profile_root\myProfile
IBM heap dump files are usually named in the following way:
heapdump.
Gather all the .phd files and transfer them to your problem determination machine for analysis.
Many tools are available to analyze heap dumps that include Rational Application Developer 6.0. WebSphere Application Server serviceability released a technology preview called Memory Dump Diagnostic For Java. You can download this preview from the product download Web site.
What to do next
When you have a couple of heap dumps, use a number of memory leak problem determination tools to analyze your problem
Problem(Abstract)
This technote describes one way to manually generate heapdumps from WebSphere® Portal.By default, you get heapdumps only on OutOfMemoryErrors. You do not get heapdumps in crashes or through a signal to the JVM.
Cause
Heapdump is an IBM® JVM facility that generates a dump of all the live objects that are on the Java heap; that is, those that are used by the Java application. It shows the objects that are using large amounts of memory on the Java heap, and what is preventing them from being collected by the Garbage Collector.
Resolving the problem
In order to manually generate a WebSphere Portal heapdump, you must first enable signal-based heapdumps by setting the IBM_HEAPDUMP=TRUE environmental variable to the WebSphere Portal (Java) process.
After enabling signal-based heapdumps, you will get heapdumps on OutOfMemoryErrors, crashes, and through signals to the JVM.
Next, you manually generate a heapdump (Java memory dump) in the same way as you manually generate a javacore (Java thread dump), which is by using a signal. The command to send the correct signal to the JVM is dependant on your operating system.
A heapdump is normally produced immediately before a javacore.
Heapdumps now always trigger a garbage collection that occurs before the dump. This means that all the objects that are in the heapdump are live (reachable) objects.
Although heapdumps are generated only in response to a detected memory leak, you must understand that generating heap dumps can have a severe performance impact on WebSphere Portal for several minutes. When generating multiple heap dumps manually for memory leak analysis, make sure that significant objects are leaked in between the two heap dumps. This approach enables problem determination tools to identify the source of the memory leak.
Setting an environment variable in WebSphere Application Server 6.0In the WebSphere Administrative Console, click Application servers -> WebSphere_Portal -> Process Definition -> Custom Properties and add:
IBM_HEAPDUMP=TRUE
Restart WebSphere_Portal.
Setting an environment variable in WebSphere Application Server 5.1In the WebSphere Administrative Console, click Application servers -> WebSphere_Portal -> Process Definition -> Environment Entries and add:
IBM_HEAPDUMP=TRUE
Restart WebSphere_Portal.
Command to manually generate heapdump/javacore on UNIX (AIX, Linux, Solaris, and HP-UX
kill -3 [PID_of_WebSphere_Portal_JVM]
This signal will not end the Java process.
Commands to manually generate heapdump/javacore on Windows
wsadmin.bat [-host host_name] [-port port_number] [-user userid -password password]wsadmin> set jvm [$AdminControl completeObjectName type=JVM,process=WebSphere_Portal,*]wsadmin>$AdminControl invoke $jvm dumpThreads
Taking manual heapdumps in Solaris JVM 1.4.2 version 26-Apr-2007 08:30 (reply 1 of 2)
Top of Form
Bottom of Form
In 1.4.2 you can use the -XX:+HeapDumpOnOutOfMemoryError option to tell the VM to generate a heap dump if OutOfMemoryError is thrown. Another way is to run the application with the HPROF agent or other profiler. Recent 1.4.2 updates have an undocumented/unsupported option to generate a heap dump when ctrl-\ (SIGQUIT).The jmap utility on Solaris can be used to recover a heap dump from a core file or hung process. This is mostly for post mortem diagnostics and there is no guarantee that the heap dump will be readable. As it attaches to the target process via /proc it can take a long time to read the process memory. Recent releases (jdk6) have most improvements in this area (like jmap -dump which allows you to take a snapshot of the heap at any time
Location of the generated heapdumps and javacores for WebSphere Portal
Problem(Abstract)
This technote describes the location of the generated heapdumps and javacores for WebSphere® Portal.
Symptom
Environment
WebSphere Portal running on WebSphere Application Server 6.0 or 5.1 uses Java™ 1.4.2.
Resolving the problem
This information can be found in the IBM Developer Kit and Runtime Environment, Java 2 Technology Edition, Version 1.4.2 Diagnostics Guide. For your convenience, the relevant excepts and links to the topics are provided below.
HeapdumpsThe JVM checks each of the following locations for existence and write-permission, then stores the Heapdump in the first one that is available:
The location that is specified by the IBM_HEAPDUMPDIR environment variable, if set.
The current working directory of the JVM processes:
For WebSphere Portal running on WebSphere Application Server 6.0, this is directory
For WebSphere Portal running on WebSphere Application Server 5.1, this is directory
The location that is specified by the TMPDIR environment variable, if set.
The /tmp directory (X:\tmp for Windows, where X is the current working drive).
Note that enough free disk space (possibly up to 1 GB) must be available for the Heapdump file to be written correctly.On Linux and AIX, a log of Heapdump files is maintained in the file /tmp/javacore_locations.Heapdump filename formats
Windows
heapdump.YYYYMMDD.HHMMSS.PID.phd orheapdump.YYYYMMDD.HHMMSS.PID.txt
Linux & AIX
heapdumpPID.TIME.phd orheapdumpPID.TIME.txt
Solaris
java_pidPID.hprof.TIME
Note: PID is the process ID. TIME is the number of seconds since 1/1/1970.
JavacoresThe JVM checks each of the following locations for existence and write-permission, then stores the Javacore in the first one that is available:
The location specified by the IBM_JAVACOREDIR environment variable if set.
The current working directory of the JVM processes:
For WebSphere Portal running on WebSphere Application Server 6.0, this is directory
For WebSphere Portal running on WebSphere Application Server 5.1, this is directory
The location specified by the TMPDIR environment variable, if set.
The /tmp directory or, on Windows only, the location specified by the TEMP environment variable, if set.
Windows only: If the Javacore cannot be stored in any of the above, it is put to STDERR.
Note that enough free disk space (possibly up to 2.5 MB) must be available for the Javacore file to be written correctly.On Linux and AIX, a log of Javacore files is maintained in the file /tmp/javacore_locations.On Solaris, javacore output is not written to a separate file. Rather, it is appended to the existing native_stdout.log, typically in directory
Windows and Linux
javacore.YYYYMMDD.HHMMSS.PID.txt
AIX
javacorePID.TIME.txt
Solaris
native_stdout.log
Note: PID is the process ID. TIME is the number of seconds since 1/1/1970.
Location of the generated Heapdump
The JVM checks each of the following locations for existence and write-permission, then stores the Heapdump in the first one that is available.
The location that is specified by the IBM_HEAPDUMPDIR environment variable, if set (_CEE_DMPTARG on z/OS(R))
The current working directory of the JVM processes
The location that is specified by the TMPDIR environment variable, if set
The /tmp directory (X:\tmp for Windows(R), where X is the current working drive)
Note that enough free disk space must be available for the Heapdump file to be written correctly
Heapdumps can be very large with millions of items in them. Here is a small sample:
// Version: J2RE 1.4.2 IBM AIX build ca142-20061124 (SR7)
0x30060200 [388832] byte[]
0x300bf0e0 [304] class OOMTest
0x300bf210 [49800] byte[]
0x300cb498 [88] java/lang/Thread
0x0 0x0 0x300d8198 0x0 0x3010f218 0x0 0x300cb4f0 0x300e04f0 0x0
0x300cb4f0 [128] sun/misc/Launcher$AppClassLoader
0x300f89c0 0x300cb570 0x300f8ad8 0x300f8cc8 0x300f8c58 0x301430b8 0x300f8b20 0x0 0x0 0x300f8be8 0x0 0x300f8b78 0x0 0x0 0x300f8a88 0x300f8a18 0x300f8938 0x300f8fb8 0x300f52d
0 0x300f89a0
0x300cb570 [128] sun/misc/Launcher$ExtClassLoader
0x300f5af0 0x0 0x300f5ca8 0x300f5f08 0x300f5e98 0x0 0x300f5d60 0x300f5238 0x0 0x300f5e28 0x0 0x300f5db8 0x0 0x0 0x300f5c58 0x300f5bc0 0x300fb2f8 0x300f5f38 0x300f52d0 0x300
f5ad0 0x300f4f88
0x300cb5f0 [88] java/lang/Thread
0x0 0x0 0x300d8198 0x0 0x300dfa30 0x0 0x0 0x300e04f0 0x0
0x300cb648 [88] java/lang/Thread
0x0 0x0 0x300d8198 0x0 0x300dfa80 0x0 0x0 0x300e04f0 0x0
.
.
.
..........................(more data)........................
// Breakdown - Classes: 321, Objects: 2403, ObjectArrays: 205, PrimitiveArrays: 1671
// EOF: Total 'Objects',Refs(null) : 4600,5249807(925)
The first line in the Heapdump contains the build identifier of the JVM that produced the dump.
In each line of data of the main part of the dump:
The first hex number is the handle of the object.
The number in square brackets is the size of the object.
The remainder of the line is a description of the object. If an object references other objects, those objects are presented as a series of object handles indented in the following line or lines. In the example above, all possible references are dumped, with nulls being explicitly shown as 0x0.
The last two lines in the Heapdump summarize the heap content
