In-depth JVM analysis of Java's thread stack

In this article I will teach you how to analyze the JVM thread stack and how to find the root cause of the problem from the stack information. In my opinion, thread stack analysis technology is a technology that Java EE product support engineers must master. The information stored in the thread stack is usually far beyond your imagination, and we can make good use of this information in our work.

My goal is to share the knowledge and experience I have accumulated in thread analysis over the past dozen years. This knowledge and experience was gained through in-depth analysis of various versions of JVM and JVM vendors from various vendors. In the process, I also summarized a large number of common problem templates.

So, are you ready? Bookmark this article now. I will bring you this series of special articles in the following weeks. What are you waiting for? Please share this thread analysis training plan with your colleagues and friends.

Sounds good, I really should improve my thread stack analysis skills...but where do I start?

My suggestion is to follow me through this thread analysis training program. Below is the training content we will cover. At the same time, I will share with you the actual cases I have handled so that you can learn and understand.

1) Overview and basic knowledge of thread stacks
2) Thread stack generation principles and related tools
3) Differences in the formats of thread stacks in different JVMs (Sun HotSpot, IBM JRE, Oracal JRockit)
4) Thread stack log introduction and analysis method
5) Thread stack analysis and related technologies
6) Common problem templates (thread race state, deadlock, IO call hang, garbage collection/ OutOfMemoryError problems, infinite loops, etc.)
7) Example analysis of thread stack problems

I hope this series of training can bring you real help, so please continue to pay attention to weekly article updates.

But what should I do if I have questions during the learning process or cannot understand the content of the article?

Don't worry, just think of me as your mentor. If you have any questions about the thread stack, you can consult me ​​(provided the problem is not too low). Please feel free to choose one of the following ways to get in touch with me:

1) Leave a comment directly below this article (if you’re sorry, you can remain anonymous)
2) Submit your thread stack data to the Root Cause Analysis forum
3) Send me an email at @phcharbonneau@hotmail.com

Can you help me analyze the problems encountered in our products?

Of course, if you want, you can send me your stack live data via email or the Root Cause Analysis forum. Dealing with practical problems is the best way to learn and improve skills.

I sincerely hope that everyone will like this training. So I will do my best to provide you with high-quality materials and answer your various questions.

Before introducing the thread stack analysis technology and problem model, I must first tell you the basic content. So in this post, I will cover the most basic content first, so that everyone can better understand the interaction between JVM, middleware, and Java EE containers.

Java VM Overview

The Java Virtual Machine is the foundation of the Java EE platform. It is where middleware and applications are deployed and run.

The JVM provides the following things to middleware software and your Java/Java EE programs:

– (in binary form) Java/Java EE program running environment
– Some programs Features and tools (IO infrastructure, data structures, thread management, security, monitoring, etc.)
– Dynamic memory allocation and management with garbage collection

Your JVM can reside on many On top of the operating system (Solaris, AIX, Windows, etc.), and depending on your physical server configuration, you can install 1 to multiple JVM processes on each physical/virtual server.

JVM and Interaction between middleware

The following figure shows the high-level interaction model between JVM, middleware and applications.

The figure shows some simple and typical interactions between JVM, middleware and application software. As you can see, the allocation of threads for standard Java EE applications is done between the middleware kernel and the JVM. (Of course there are exceptions. Applications can directly call the API to create threads. This is not common, and special care must be taken during use)

At the same time, please note that some threads are created internally by the JVM. For management, a typical example is the garbage collection thread. This thread is used internally by the JVM to perform parallel garbage collection processing.

Because most thread allocation is done by the Java EE container, it is important for you to be able to understand and recognize thread stack traces and identify it from thread stack data. This This allows you to quickly know what type of request the Java EE container is executing.

From the analysis perspective of a thread dump stack, you will be able to understand the differences between the thread pools discovered by the JVM. Different and identify the type of request.

The last section will provide you with an overview of what the JVM thread stack is for the HotSop VM, as well as the various threads you will encounter. The details of the IBM VM thread stack form will be in Section 4 is provided to you.

Please note that you can get the thread stack example for this article from the Root Cause Analysis Forum.

JVM Thread Stack - What is it?

The JVM thread stack is a snapshot of a given time that provides you with a complete list of all Java threads that have been created.

Each discovered Java thread will give you the following information:

– The name of the thread; it is often used by middleware manufacturers to identify thread identifiers. It usually also carries the assigned thread pool name and status (running, blocked, etc.)

– Thread type & Priority, for example: daemon prio=3 ** Middleware programs generally create their threads in the form of background daemons, which means that these threads run in the background; they provide services to their users, for example: to you Java EE application **

– Java thread ID, for example: tid=0x000000011e52a800 ** This is the Java thread ID obtained through java.lang.Thread.getId(), which is often used as a long self-increasing Shaping 1..n** Implementation

– Native thread ID, for example: nid=0x251c**, the reason why the key is because the native thread ID allows you to obtain information such as from the operating system’s perspective Information about which thread is using most of the CPU time in your JVM. **

– Java thread status and details, for example: waiting for monitor entry [0xfffffffea5afb000] java.lang.Thread .State: BLOCKED (on object monitor)
** You can quickly understand the thread status and possible reasons for the current blocking**

– Java thread stack trace; this is what you can do from the thread stack so far The most important data found in %Information.

– Java heap memory decomposition; starting from HotSpot VM version 1.6, you can see HotSpot’s memory usage at the end of the thread stack, such as Java’s heap memory (YoungGen, OldGen) & PermGen space. This information is useful when analyzing problems caused by frequent GC. You can use known thread data or patterns to make a quick localization.

Heap
PSYoungGen   total 466944K, used 178734K [0xffffffff45c00000, 0xffffffff70800000, 0xffffffff70800000)
eden space 233472K, 76% used [0xffffffff45c00000,0xffffffff50ab7c50,0xffffffff54000000)
from space 233472K, 0% used [0xffffffff62400000,0xffffffff62400000,0xffffffff70800000)
to  space 233472K, 0% used [0xffffffff54000000,0xffffffff54000000,0xffffffff62400000)
PSOldGen    total 1400832K, used 1400831K [0xfffffffef0400000, 0xffffffff45c00000, 0xffffffff45c00000)
object space 1400832K, 99% used [0xfffffffef0400000,0xffffffff45bfffb8,0xffffffff45c00000)
PSPermGen    total 262144K, used 248475K [0xfffffffed0400000, 0xfffffffee0400000, 0xfffffffef0400000)
object space 262144K, 94% used [0xfffffffed0400000,0xfffffffedf6a6f08,0xfffffffee0400000)

Big dismantling of thread stack information

In order to give you a better understanding, we provide you with the following picture. In this picture, the thread stack information and threads on the HotSpot VM are Pool has made a detailed disassembly, as shown in the following figure:

#It can be seen from the above figure that the thread stack is composed of many different parts. This information is important for problem analysis, but different parts will be used for the analysis of different problem patterns (the problem patterns will be simulated and demonstrated in later articles.)

Now through this analysis Example, let me explain in detail the various components of the thread stack information on HoteSpot:

# Full thread dump标示符

"Full thread dump" is a globally unique keyword, you can use it in the middleware Find it in the output log of the thread stack information of the stand-alone version of Java (for example, use: kill -3 under UNIX). This is the beginning of the thread stack snapshot.

Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.0-b11 mixed mode):

# Threads in Java EE middleware, third-party and custom application software

This part is the core part of the entire thread stack, and it is also the part that usually takes the most analysis time. The number of threads in the stack depends on the middleware you use, third-party libraries (which may have independent threads), and your application (if you create custom threads, this is usually not a good practice).

In our example thread stack, WebLogic is the middleware we use. Starting from Weblogic 9.2, a self-managed thread pool uniquely identified with "'weblogic.kernel.Default (self-tuning)" will be used

"[STANDBY] ExecuteThread: '414' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x000000010916a800 nid=0x2613 in Object.wait() [0xfffffffe9edff000]
  java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0xffffffff27d44de0> (a weblogic.work.ExecuteThread)
    at java.lang.Object.wait(Object.java:485)
    at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:160)
    - locked <0xffffffff27d44de0> (a weblogic.work.ExecuteThread)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

# HotSpot VM Thread
This is an internal thread managed by Hotspot VM and used to perform internal native operations. Generally you don't need to worry too much about this unless you find high CPU usage (via related thread stacks and prstat or native thread IDs).

"VM Periodic Task Thread" prio=3 tid=0x0000000101238800 nid=0x19 waiting on condition

# HotSpot GC Thread
When using HotSpot for parallel GC (common nowadays in environments using multiple physical cores), the HotSpot VM created by default or each JVM manages a GC thread with a specific identity. These GC threads can Let the VM perform its periodic GC cleanup in a parallel manner, which will result in an overall reduction in GC time; at the same time, the cost is an increase in CPU usage time.

"GC task thread#0 (ParallelGC)" prio=3 tid=0x0000000100120000 nid=0x3 runnable
"GC task thread#1 (ParallelGC)" prio=3 tid=0x0000000100131000 nid=0x4 runnable
………………………………………………………………………………………………………………………………………………………………

这事非常关键的数据，因为当你遇到跟GC有关的问题，诸如过度GC、内存泄露等问题是，你将可以利用这些线程的原生Id值关联的操作系统或者Java线程，进而发现任何对CPI时间的高占用. 未来的文章你将会了解到如何识别并诊断这样的问题.

# JNI 全局引用计数
JNI (Java 本地接口)的全局引用就是从本地代码到由Java垃圾收集器管理的Java对象的基本的对象引用. 它的角色就是阻止对仍然在被本地代码使用，但是技术上已经不是Java代码中的“活动的”引用了的对象的垃圾收集.

同时为了侦测JNI相关的泄露而留意JNI引用也很重要. 如果你的程序直接使用了JNI，或者像监听器这样的第三方工具，就容易造成本地的内存泄露.
 
JNI global references: 1925

# Java 堆栈使用视图

这些数据被添加回了 JDK 1 .6 ，向你提供有关Hotspot堆栈的一个简短而快速的视图. 我发现它在当我处理带有过高CPU占用的GC相关的问题时非常有用，你可以在一个单独的快照中同时看到线程堆栈以及Java堆的信息，让你当时就可以在一个特定的Java堆内存空间中解析（或者排除）出任何的关键点. 你如在我们的示例线程堆栈中所见，Java 的堆 OldGen 超出了最大值!

Heap
 PSYoungGen   total 466944K, used 178734K [0xffffffff45c00000, 0xffffffff70800000, 0xffffffff70800000)
 eden space 233472K, 76% used [0xffffffff45c00000,0xffffffff50ab7c50,0xffffffff54000000)
 from space 233472K, 0% used [0xffffffff62400000,0xffffffff62400000,0xffffffff70800000)
 to  space 233472K, 0% used [0xffffffff54000000,0xffffffff54000000,0xffffffff62400000)
 PSOldGen    total 1400832K, used 1400831K [0xfffffffef0400000, 0xffffffff45c00000, 0xffffffff45c00000)
 object space 1400832K, 99% used [0xfffffffef0400000,0xffffffff45bfffb8,0xffffffff45c00000)
 PSPermGen    total 262144K, used 248475K [0xfffffffed0400000, 0xfffffffee0400000, 0xfffffffef0400000)
 object space 262144K, 94% used [0xfffffffed0400000,0xfffffffedf6a6f08,0xfffffffee040000

更多In-depth JVM analysis of Javas thread stack相关文章请关注PHP中文网！