通过GDB调试内核与模块

说明

在vlinux内核开发过程中不可避免地地会面对如何调试内核的问题。比如printk() 是调试内核代码时最常用的一种技术，但是这种调试方法需要对问题产生的点有个大概的感知，这样才能有的放矢地在合适的代码处添加打印信息。

但是很多情况下，我们对问题产生的原因毫无头绪，也就不可能在茫茫的代码中添加printk，这时候我们就依赖于GDB构建Linux内核调试环境。本文以vlinux 4.1.15为例，介绍调试环境的搭建过程。

vlinux内核文档中的调试说明

在vlinux内核文档中有关于GDB调试内核的说明文档。调试环境搭建的过程也就是一步步实现该文档步骤的过程。

vlinux/Documentation/gdb-kernel-debugging.txt

The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware interfaces allow to debug the Linux kernel and its modules during runtime using gdb. Gdb comes with a powerful scripting interface for python. The kernel provides a collection of helper scripts that can simplify typical kernel debugging steps. This is a short tutorial about how to enable and use them. It focuses on QEMU/KVM virtual machines as target, but the examples can be transferred to the other gdb stubs as well.

Requirements

---

gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true for distributions)

Setup

---

1. Create a virtual Linux machine for QEMU/KVM

(see http://www.linux-kvm.org and http://www.qemu.org for more details).
For cross-development,

http://landley.net/aboriginal/bin keeps a pool of machine images and toolchains that can be helpful to start from.

Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave CONFIG_DEBUG_INFO_REDUCED If your architecture supports CONFIG_FRAME_POINTER, keep it enabled.
Install that kernel on the guest.
Alternatively, QEMU allows to boot the kernel directly using -kernel,

-append, -initrd command line switches. This is generally only useful if you do not depend on modules. See QEMU documentation for more details on this mode.

2. Enable the gdb stub of QEMU/KVM, either

at VM startup time by appending “-s” to the QEMU command line or
during runtime by issuing “gdbserver” from the QEMU monitor console
cd /path/to/linux-build
Start gdb: gdb vmlinux
Note: Some distros may restrict auto-loading of gdb scripts to known safe directories. In case gdb reports to refuse loading vmlinux-gdb.py, add

add-auto-load-safe-path /path/to/linux-build

to ~/.gdbinit. See gdb help for more details.

3. Attach to the booted guest: (gdb) target remote :1234

Examples of using the Linux-provided gdb helpers

---

4. Load module (and main kernel) symbols

(gdb) lx-symbols

loading vmlinux

scanning for modules in /home/user/linux/build

loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko

…

原理

GDB调试过程中需要加载符号信息，即lx-symbols命令，该命令是实现在内核自带python脚本：/vlinux/scripts/gdb/vmlinux-gdb.py，这也是”build gdb 7.2+ (recommended: 7.4+) with python support enabled”的原因。

在开发机上搭建调试环境

1. 检查开发机器上已安装的GDB和python

[admin@rs1f13285 /home/admin/elvis]

$which gdb

/usr/bin/gdb

[admin@rs1f13285 /home/admin/elvis]

$/usr/bin/gdb -v

GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5)

Copyright (C) 2009 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law. Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

[admin@rs1f13285 /home/admin/elvis]

$which python

/usr/ali/bin/python

[admin@rs1f13285 /home/admin/elvis]

$/usr/ali/bin/python-org --version

Python 2.5.4

注意事项

内核文档建议选择GDB 7.4+，但是实际的编译调试过程中发现让GDB加载python脚本需要auto-load功能，GDB 7.5+以后版本才支持该功能。
/vlinux/scripts/gdb/vmlinux-gdb.py脚本使用了python的一个新功能dictionary comprehensions，该功能只能到Python 2.7之后才支持。详情参考https://docs.python.org/2/whatsnew/2.7.html - new-features-added-to-python-2-7-maintenance-releases
由于开发机器上现有的Python 2.5.4版本过低，并且安装在非标准目录/usr/ali，在GDB编译过程中会报错，该问题的workaround方法是：

[admin@rs1f13285 /home/admin/elvis/vlinux]

$mv /usr/ali/bin/python /usr/ali/bin/python-org

[admin@rs1f13285 /home/admin/elvis/vlinux]

$cp /usr/ali/python-2.7/bin/python /usr/ali/bin/python

该方法只是将Python 2.7替换掉原有的Python。如有其它更好办法，请更新此文档。

Build GDB with Python Support

由于开发机上已经安装GDB 7.0.1，为了不影响公用该开发机其它项目的编译，最好自己编译一个仅供自己使用的GDB。这里选择的是支持auto-load功能的GDB 7.5。

首先，编译之前需要检查几个事项：

python是否支持Dictionary Comprehension功能
[admin@rs1f13285 /home/admin/elvis/gdb-7.5]

$python

Python 2.7 (r27, May 27 2013, 20:02:53)

[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2

Type “help”, “copyright”, “credits” or “license” for more information.

d = {n: True for n in range(5)}

d

{0: True, 1: True, 2: True, 3: True, 4: True}

如果d正常打印未告警，说明支持Dictionary Comprehension语法。

编译GDB过程中需要使用到动态链接库7.*.so。动态链接库程序ld.so加载动态库时，搜索的是一个缓存文件/etc/ld.so.cache，这是一个二进制文件，用vim打开确认一下路径中包含libpython2.7.so.1.0，如果未包含该动态库，请将路径“/usr/ali/python-2.7/lib/”添加到/etc/ld.so.conf之后，键入命令ldconfig更新缓存文件ld.so.cache。
[admin@rs1f13285 /home/admin/elvis/gdb-7.5]

$sudo ldconfig
Build GDB with Python Support
[admin@rs1f13285 /home/admin/elvis]

$tar zxvf gdb-7.5.tar.gz

$cd gdb-7.5

再次检查GDB编译所需的头文件和链接选项，与目标Python路径是否一致：

[admin@rs1f13285 /home/admin/elvis/gdb-7.5]

$python ./gdb/python/python-config.py --includes

-I/usr/ali/python-2.7/include/python2.7 -I/usr/ali/python-2.7/include/python2.7

[admin@rs1f13285 /home/admin/elvis/gdb-7.5]

$python ./gdb/python/python-config.py --ldflags

-lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic

检查完成之后开始编译：

$mkdir bin

$cd bin/

$…/configure --with-python

$make

检查编译后的GDB是否支持vlinux自带的vmlinux-gdb.py脚本。

[admin@rs1f13285 /home/admin/elvis/vlinux]

$/home/admin/elvis/gdb-7.5/bin/gdb/gdb

GNU gdb (GDB) 7.5

Copyright (C) 2012 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law. Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-unknown-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

(gdb) set debug auto-load on ---àTo demo “auto-load” function. No need for daily use

(gdb) set auto-load safe-path . ---àSet current Directory as safe-path

auto-load: Updating directories of ".".

auto-load: Using directory ".".

auto-load: And canonicalized as "/apsarapangu/disk1/elvis/vlinux".

(gdb) file ./build/vmlinux     ---àLoad vlinux kernel Symbols

Reading symbols from /apsarapangu/disk1/elvis/vlinux/build/vmlinux...done.

auto-load: Attempted file "/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.gdb" does not exist.

auto-load: Expanded $-variables to "/usr/local/lib/debug:/usr/local/share/gdb/auto-load".

auto-load: Searching 'set auto-load scripts-directory' path "debugdir: d e b u g d i r : datadir/auto-load".

auto-load: Attempted file "/usr/local/lib/debug/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.gdb" does not exist.

auto-load: Attempted file "/usr/local/share/gdb/auto-load/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.gdb" does not exist.

auto-load: Attempted file "/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.py" exists.

auto-load: Loading Python script "/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.py" by extension for objfile "/apsarapangu/disk1/elvis/vlinux/build/vmlinux".

auto-load: Matching file "/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.py" to pattern "."

auto-load: Not matched - pattern ".".

auto-load: Matching file "/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.py" to pattern "/apsarapangu/disk1/elvis/vlinux"

auto-load: Matched - file "/apsarapangu/disk1/elvis/vlinux" to pattern "/apsarapangu/disk1/elvis/vlinux".

auto-load: File "/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.py" matches directory "/apsarapangu/disk1/elvis/vlinux".

(gdb) apropos lx

function lx_current -- Return current task

function lx_module -- Find module by name and return the module variable

function lx_per_cpu -- Return per-cpu variable

function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable

function lx_thread_info -- Calculate Linux thread_info from task variable

lx-dmesg -- Print Linux kernel log buffer

lx-lsmod -- List currently loaded modules

lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules

(gdb)

至此，GDB的环境搭建部分已经完成。

Qemu环境

Qemu环境存放在Git仓库中：git@gitlab.alibaba-inc.com:vLinux/images.git，详细信息请参考http://gitlab.alibaba-inc.com/vLinux/images/wikis/home。克隆该仓库后，只需要在qemu的启动选项里面加上”-s”即可：

[admin@rs1f13285 /home/admin/elvis/images/odps_s20_test/vlinux-stage5.1]

$cat run-vlinux.sh

#! /bin/bash

……

echo Starting VM ...\

set -x

TRACE T R A C E QEMU $* \

   -s \

   -m ${MEM_MB} \

   -smp cpus=${CPUS} \

   -nodefconfig -nodefaults -no-user-config \

   -parallel none \

   -vga none \

   -display none \

-boot menu=off \

……

总结

文本简要介绍了GDB调试内核的主要步骤。