A summary of my recent experience with resolving an 100% CPU utilization issue.
We know well that in Java 5 and above we have the mighty java.lang.management.ThreadMXBean which is able to give us almost all ordinary diagnostics information of a given thread, except more detailed information that only debuggers are able to tell. However, such information is implementation/vendor dependent and some information, like CPU time measurement, might be disabled by default, and moreover, in my humble opinion, these information may be not very accurate since they are reported by the JVM itself, not by the OS.
Luckily, almost every popular OS supports some means of measurements down to the threading level, and on the Java side, we have the thread dump (of course, also in popular JVM implementations). I’m here taking Ubuntu, Windows XP with SUN JVM (5.0 for Windows and 6.0 for Linux) as an example. This example demonstrates the process to find the thread that is utilizing unusual amount of CPU time in a Tomcat instance hosting a very simple web application.
When there’s something wrong caused by the JVM, say, similarly, it simply occupies all CPU time, you can always find the guilty threads by doing so:
1. Getting threading information from the OS.
In most Linux distros, a single ps command can handle this. For instance, ps H -eo user,pid,ppid,tid,time,%cpu,cmd –sort=-%cpu , will print out all the processes and their created threads, sorted by the percentage of overall CPU utilization. We can always RTFM for more information about *nix commands;)
In Windows, however, getting threading information for a process might be harder. I suggest using Process Explorer, which is the most easy to use tool amongst other equivalents.
First of all, open the “Properties” dialog of the JVM process.
Next, click on the “Threads” tab to see threads of the process. You can see that they are sorted by CPU utilization by default.
(I have a dual core CPU and this thread is occupying a whole core)
2. Getting a thread dump from JVM.
This task should be easier for us as Java developers;) In Linux we use kill -3 PID,
(I was running less logs/catalina.out after the thread dump, because I wrongly started Tomcat by bin/catalina.sh start instead of bin/catalina.sh run. Never mind)
and in Windows we can simply hit Ctrl+Break in the console window that is running the JVM process.
3. Finding out the Java thread from
Usually the thread ID printed by the OS are in decimal format, and this value needs to be converted into a hexadecimal. And next, no other magic other than finding “nid=xxx” terms in the thread dump. See highlighted parts:
7772=0x1E5C
2148=0x864
Yep, that’s it, test.Test.test();. I’m not going to amuse you by posting the source of test.Test.test(), since it’s no more than an infinite loop:D
Regards and hope this helps.
Recent Comments