Detecting hung threads in J2EE applications

A common error in J2EE applications is a hung thread. A hung thread can result from a simple software defect (such as an infinite loop) or a more complex cause (for example, a resource deadlock). System resources, such as CPU time, might be consumed by this hung transaction when threads run unbounded code paths, such as when the code is running in an infinite loop. Alternately, a system can become unresponsive even though all resources are idle, as in a deadlock scenario. Unless an end user or a monitoring tool reports the problem, the system may remain in this degraded state indefinitely.
The hang detection option for WebSphere Application Server is turned on by default. You can configure a hang detection policy to accommodate your applications and environment so that potential hangs can be reported, providing earlier detection of failing servers. When a hung thread is detected, WebSphere Application Server notifies you so that you can troubleshoot the problem.
Using the hang detection policy, you can specify a time that is too long for a unit of work to complete. The thread monitor checks all managed threads in the system (for example, Web container threads and object request broker (ORB) threads) . Unmanaged threads, which are threads created by applications, are not monitored.
When WebSphere Application Server detects that a thread has been active longer than the time defined by the thread monitor threshold, the application server takes the following actions:
Logs a warning in the WebSphere Application Server System.Out log file that indicates the name of the thread that is hung and how long it has already been active. The following message is written to the log: WSVR0605W: Thread threadname has been active for hangtime and may be hung. There are totalthreads threads in total in the server that may be hung.
where: threadname is the name that appears in a JVM thread dump, hangtime gives an approximation of how long the thread has been active and totalthreads gives an overall assessment of the system threads.
Issues a Java Management Extensions (JMX) notification. This notification enables third-party tools to catch the event and take appropriate action, such as triggering a JVM thread dump of the server, or issuing an electronic page or e-mail. The following JMX notification events are defined in the com.ibm.websphere.management.NotificationConstants class:
TYPE_THREAD_MONITOR_THREAD_HUNG: This event is triggered by the detection of a (potentially) hung thread.
TYPE_THREAD_MONITOR_THREAD_CLEAR: This event is triggered if a thread that was previously reported as hung completes its work.
Triggers changes in the performance monitoring infrastructure (PMI) data counters. These PMI data counters are used by various tools, such as the Tivoli Performance Viewer, to provide a performance analysis.
False Alarms
If the work actually completes, a second set of messages, notifications and PMI events is produced to identify the false alarm. The following message is written to the System.out log: WSVR0606W: Thread threadname was previously reported to be hung but has completed. It was active for approximately hangtime. There are totalthreads threads in total in the server that still may be hung.
where threadname is the name that appears in a JVM thread dump, hangtime gives an approximation of how long the thread has been active and totalthreads gives an overall assessment of the system threads.
Automatic adjustment of the hang time threshold
If the thread monitor determines that too many false alarms are issued (determined by the number of pairs of hang and clear messages), it can automatically adjust the threshold. When this adjustment occurs, the following message is written to the System.out log: WSVR0607W: Too many thread hangs have been falsely reported. The hang threshold is now being set to thresholdtime.
where: thresholdtime is the time (in seconds) in which a thread can be active before it is considered hung.
You can prevent WebSphere Application Server from automatically adjusting the hang time threshold.