A deep dive into advanced topics about Linux debuggers

Introduction

We finally come to the last article in this series! This time, I'll give a high-level overview of some of the more advanced concepts in debugging: remote debugging, shared library support, expression evaluation, and multithreading support. These ideas are more complex to implement, so I won't go into detail on how to do it, but I'm happy to answer questions about these concepts if you have questions.

Series Index

1. Preparing the environment
2. Breakpoint
3. Registers and Memory
4. Elves and dwarves
5. Source code and signals
6. Source code layer is executed step by step
7. Source code layer breakpoint
8. Call stack
9. Handling variables
10. Advanced Theme

Remote debugging

Remote debugging is very useful for embedded systems or debugging different environments. It also sets a fine line between high-level debugger operations and interaction with the operating system and hardware. In fact, debuggers like GDB and LLDB can run as remote debuggers even when debugging local programs. The general architecture is like this:

debuarch

The debugger is a component that we interact with through the command line. Maybe if you're using an IDE, there's another layer on top of it that communicates with the debugger via the machine interface. On the target machine (probably the same as the native machine) there is a debug stub, which is theoretically a wrapper around a very small operating system debugging library that performs all the low-level debugging tasks like setting breakpoints on addresses. I say "in theory" because debug stubs are getting bigger and bigger these days. For example, the LLDB debug stub size on my machine is 7.6MB. The debug stub communicates with the debugged process and the debugger via the remote protocol by using some operating system-specific functionality (ptrace in our case).
The most common remote debugging protocol is the GDB remote protocol. This is a text-based packet format used to pass commands and information between the debugger and debug stubs. I won't go into detail about it, but you can read further here. If you start LLDB and execute the command log enable gdb-remote packets, you will get a trace of all packets sent over the remote protocol. On GDB, you can do the same thing with set remotelogfile.

As a simple example, this is the packet to set a breakpoint on:

$Z0,400570,1#43

$ Marks the beginning of the packet. Z0 is the command to insert a memory breakpoint. 400570 and 1 are parameters, where the former is the address to set the breakpoint and the latter is the breakpoint type specifier for a specific target. Finally, #43 is a checksum to ensure the data is not corrupted.

The GDB remote protocol is very easy to extend with custom packets, which is useful for implementing platform or language specific functionality.

Shared libraries and dynamic loading support

The debugger needs to know which shared libraries are loaded by the program being debugged so that it can set breakpoints, obtain source code level information and symbols, etc. In addition to finding libraries that are dynamically linked, the debugger must also trace libraries that are loaded via dlopen at runtime. To achieve this purpose, the dynamic linker maintains an intersection structure. This structure maintains a linked list of shared library descriptors, as well as a pointer to a function that is called whenever the linked list is updated. This structure is stored in the .dynamic section of the ELF file and is initialized before program execution.

A simple tracking algorithm:

• The tracer looks for the program's entry in the ELF header (or can use auxiliary vectors stored in /proc//aux).
• The tracking program sets a breakpoint at the entry of the program and starts execution.
• When the breakpoint is reached, the address of the intersection structure is found by searching the load address of .dynamic in the ELF file.
• Check the intersection structure for a list of currently loaded libraries.
• Set a breakpoint on the linker update function.
• The list is updated whenever a breakpoint is reached.
• The tracking program loops infinitely, continuing to execute the program and waiting for signals until the tracking program signal exits.

I wrote a small example of these concepts, which you can find here. I can write in more detail in the future if anyone is interested.

Expression calculation

Expression evaluation is a feature of the program that allows the user to evaluate expressions in the original source language while debugging the program. For example, in LLDB or GDB, you can execute print foo() to call the foo function and print the result.

There are several different calculation methods depending on the complexity of the expression. If the expression is just a simple identifier, the debugger can look at the debug information, find the variable and print out the value, just like we did in the last part of this series. If the expression is somewhat complex, it may be possible to compile the code into an intermediate expression (IR) and interpret it to obtain the result. For example, for some expressions, LLDB will use Clang to compile the expression into an LLVM IR and interpret it. If the expression is more complex, or requires calling certain functions, the code may need to be JITed to the target and executed in the debuggee's address space. This involves calling mmap to allocate some executable memory, then copying the compiled code into that block and executing it. LLDB is implemented using LLVM’s JIT capabilities.

If you want to learn more about JIT compilation, I highly recommend Eli Bendersky's article on the subject.

多线程调试支持

本系列展示的调试器仅支持单线程应用程序，但是为了调试大多数真实程序，多线程支持是非常需要的。支持这一点的最简单的方法是跟踪线程的创建，并解析 procfs 以获取所需的信息。

Linux 线程库称为 pthreads。当调用 pthread_create 时，库会使用 clone 系统调用来创建一个新的线程，我们可以用 ptrace 跟踪这个系统调用（假设你的内核早于 2.5.46）。为此，你需要在连接到调试器之后设置一些 ptrace 选项：

ptrace(PTRACE_SETOPTIONS, m_pid, nullptr, PTRACE_O_TRACECLONE);

现在当 clone 被调用时，该进程将收到我们的老朋友 SIGTRAP 信号。对于本系列中的调试器，你可以将一个例子添加到 handle_sigtrap 来处理新线程的创建：

case (SIGTRAP | (PTRACE_EVENT_CLONE << 8)):
//get the new thread ID
unsigned long event_message = 0;
ptrace(PTRACE_GETEVENTMSG, pid, nullptr, message);
//handle creation
//...

一旦收到了，你可以看看 /proc//task/ 并查看内存映射之类来获得所需的所有信息。

GDB 使用 libthread_db，它提供了一堆帮助函数，这样你就不需要自己解析和处理。设置这个库很奇怪，我不会在这展示它如何工作，但如果你想使用它，你可以去阅读这个教程。

多线程支持中最复杂的部分是调试器中线程状态的建模，特别是如果你希望支持不间断模式或当你计算中涉及不止一个 CPU 的某种异构调试。

最后！

呼！这个系列花了很长时间才写完，但是我在这个过程中学到了很多东西，我希望它是有帮助的。如果你有关于调试或本系列中的任何问题，请在 Twitter @TartanLlama或评论区联系我。如果你有想看到的其他任何调试主题，让我知道我或许会再发其他的文章。

The above is the detailed content of A deep dive into advanced topics about Linux debuggers. For more information, please follow other related articles on the PHP Chinese website!