linux性能分析工具之火焰图
一.环境
1.1 jello@jello:~$ uname -a
Linux jello 4.4.0-98-generic #121-Ubuntu SMP Tue Oct 10 14:24:03 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
1.2 jello@jello-Inspiron-N4050:~$ lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
二.准备工作.
2.1安装systemtap (火焰图依赖于此工具)
sudo apt-get install systemtap
2.2 查找内核对应的debug包
jello@jello$ uname -r
4.4.0-98-generic
那么接下来就是去http://ddebs.ubuntu.com/pool/main/l/linux/下载对应的debug包,找4.4.0-98-generic一致的
2.3 下载对应的debug包
wget http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-4.4.0-98-generic-dbgsym_4.4.0-98.121_amd64.ddeb (这是笔者自己找到的对应下载路径)
2.4 安装debug包
sudo dpkg -i linux-image-4.4.0-98-generic-dbgsym_4.4.0-98.121_amd64.ddeb
2.5 安装nginx
sudo apt-get install nginx
此时在浏览器中输入localhost即可出现以下信息表明安装ok
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.
Thank you for using nginx.
2.5 编写systemtap脚本nginx.systemtap,内容如下:
global s;
global quit=0;
probe timer.profile {
if (pid() == target()){
if (quit) {
foreach (i in s-) {
print_ustack(i);
printf("\t%d\n",@count(s[i]));
}
exit();
}
else {
s[ubacktrace()] <<< 1;
}
}
}
probe timer.s(20){
quit = 1
}
2.6 使用systemtap
sudo stap --ldd -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 nginx.systemtap --vp 0001 > nginx.out
各参数解析:
--ldd,添加通过ldd解析出来的所有共享库符号表信息以便为probe到的用户空间二进制提供信息或者以-d选项列出来,注意:这会使得probe模块相当的大
-d /usr/sbin/nginx,为给定的模块(这里是nginx)添加符号表信息到内核对象模块,这可能使能这些模块或者程序的象征性traceback,即使他们没有显式probe到他们里面
--all-modules,相当于为所有当前被加载的模块指定'-dkernel'和‘-d'选项
-D MAXMAPENTRIES=256 ,添加给定的c预处理直接到模块makefile中,这些能被用来限制以下描述的参数,MAXMAPENTRIES=256,设定默认全局数组的最大默认长度为256,通过了解,是用来指定堆栈大小的
-D MAXACTION=20000 设定单个probe hit包含执行的最大语句数目
-D MAXTRACE=100
-D MAXSTRINGLEN=4096 设定字符串最大长度
-D MAXBACKTRACE=100 设定栈帧的最大数目
-x 2082 设定target()的pid为2082
--vp 0001
笔者运行之后出现以下错误信息:
jello@jello:~$ sudo stap --ld -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 nginx.systemtap --vp 0001 > nginx.out
In file included from /usr/share/systemtap/runtime/transport/control.c:14:0,
from /usr/share/systemtap/runtime/transport/transport.c:76,
from /usr/share/systemtap/runtime/linux/print.c:17,
from /usr/share/systemtap/runtime/print.c:17,
from /usr/share/systemtap/runtime/runtime_context.h:22,
from /tmp/stapiPADvo/stap_20923465b7e2011fc7e903a669485b1f_9417_src.c:221:
/usr/share/systemtap/runtime/transport/symbols.c: In function ‘_stp_module_update_self’:
/usr/share/systemtap/runtime/transport/symbols.c:243:44: error: ‘struct module’ has no member named ‘symtab’
if (attr->address == (unsigned long) mod->symtab)
^
/usr/share/systemtap/runtime/transport/symbols.c:245:9: error: ‘struct module’ has no member named ‘num_symtab’
mod->num_symtab * sizeof(mod->symtab[0]);
^
/usr/share/systemtap/runtime/transport/symbols.c:245:34: error: ‘struct module’ has no member named ‘symtab’
mod->num_symtab * sizeof(mod->symtab[0]);
^
make[1]: *** [/tmp/stapiPADvo/stap_20923465b7e2011fc7e903a669485b1f_9417_src.o] Error 1
make: *** [_module_/tmp/stapiPADvo] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_20923465b7e2011fc7e903a669485b1f_9417.ko" in 9530usr/380sys/12191real ms.
Pass 4: compilation failed. [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started
从错误信息来看是某个mod的某个字段不存在,所以断定是内核版本过高,但是systemtap版本过低导致的,因此升级systemtap版本
2.6.1 升级systemtap版本
2.6.1.1获取最新systemtap代码
方案一.git clone git://sourceware.org/git/systemtap.git (下载速度比较慢,虽然配置FQ了,仍然慢,推荐方案二)
方案二.tsocks wget https://sourceware.org/systemtap/ftp/releases/systemtap-3.2.tar.gz (tsocks使能wget通过FQ下载源码,需要配置vpn进行FQ)
以上方案均需FQ,否则下载会非常慢!
2.6.1.2解压systemtap源码
tar xvf systemtap-3.2.tar.gz
2.6.1.3 切换到解压后的systemtap目录
cd systemtap-3.2
2.6.1.4 进行编译
2.6.1.4.1 根据README得知:
需要安装带有libdwfl库的elfutils (版本号需要大于或等于0.151)才能正常配合systemtap;
2.6.1.4.2 安装elfutils
sudo apt-get install elfutils
2.6.1.4.3 验证elfutils的版本信息
eu-objdump -V
2.6.1.4.4 安装必不可少的编译工具
sudo apt-get build-dep systemtap
2.6.1.4.5 创建编译目录
mkdir build
2.6.1.4.6 切换到build目录
cd build
2.6.1.4.7 进行编译前的配置
../configure
2.6.1.4.8 进行正式编译
make all
2.6.1.4.8.1 编译过程中出现以下问题
../bpf-translate.cxx:37:29: fatal error: elfutils/libebl.h: 没有那个文件或目录
2.6.1.4.8.2 查看bpf-translate.cxx
#if _ELFUTILS_PREREQ (0, 167)
#include <elfutils/libdwelf.h>
typedef Dwelf_Strent Stap_Strent;
typedef Dwelf_Strtab Stap_Strtab;
#define stap_strtab_init dwelf_strtab_init
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
#define stap_strtab_free dwelf_strtab_free
#define stap_strtab_finalize dwelf_strtab_finalize
#define stap_strent_offset dwelf_strent_off
#else
#include <elfutils/libebl.h>
typedef Ebl_Strent Stap_Strent;
typedef Ebl_Strtab Stap_Strtab;
#define stap_strtab_init ebl_strtabinit
#define stap_strtab_add(X,Y) ebl_strtabadd(X,Y,0)
#define stap_strtab_free ebl_strtabfree
#define stap_strtab_finalize ebl_strtabfinalize
#define stap_strent_offset ebl_strtaboffset
#endif
2.6.1.4.8.2 修改以上内容为:
#if 1
#include <elfutils/libdwelf.h>
typedef Dwelf_Strent Stap_Strent;
typedef Dwelf_Strtab Stap_Strtab;
#define stap_strtab_init dwelf_strtab_init
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
#define stap_strtab_free dwelf_strtab_free
#define stap_strtab_finalize dwelf_strtab_finalize
#define stap_strent_offset dwelf_strent_off
#else
#include <elfutils/libebl.h>
typedef Ebl_Strent Stap_Strent;
typedef Ebl_Strtab Stap_Strtab;
#define stap_strtab_init ebl_strtabinit
#define stap_strtab_add(X,Y) ebl_strtabadd(X,Y,0)
#define stap_strtab_free ebl_strtabfree
#define stap_strtab_finalize ebl_strtabfinalize
#define stap_strent_offset ebl_strtaboffset
#endif
2.6.1.4.8.3 通过以上修改出现以下错误信息:
make all-recursive
make[1]: Entering directory '/home/jello/data/development/systemtap-3.2/build'
Making all in .
make[2]: Entering directory '/home/jello/data/development/systemtap-3.2/build'
CXX stap-bpf-translate.o
../bpf-translate.cxx:29:9: error: ‘Dwelf_Strent’ does not name a type
typedef Dwelf_Strent Stap_Strent;
^
../bpf-translate.cxx:30:9: error: ‘Dwelf_Strtab’ does not name a type
typedef Dwelf_Strtab Stap_Strtab;
^
../bpf-translate.cxx:1773:3: error: ‘Stap_Strent’ does not name a type
Stap_Strent *name_ent;
^
../bpf-translate.cxx: In constructor ‘bpf::BPF_Section::BPF_Section(const string&)’:
../bpf-translate.cxx:1782:22: error: class ‘bpf::BPF_Section’ does not have any field named ‘name_ent’
: scn(0), name(n), name_ent(0), data(0), free_data(false)
^
../bpf-translate.cxx: At global scope:
../bpf-translate.cxx:1794:3: error: ‘Stap_Strent’ does not name a type
Stap_Strent *name_ent;
^
../bpf-translate.cxx: In constructor ‘bpf::BPF_Symbol::BPF_Symbol(const string&, bpf::BPF_Section*, long int)’:
../bpf-translate.cxx:1801:14: error: class ‘bpf::BPF_Symbol’ does not have any field named ‘name_ent’
: name(n), name_ent(0)
^
../bpf-translate.cxx: At global scope:
../bpf-translate.cxx:1812:3: error: ‘Stap_Strtab’ does not name a type
Stap_Strtab *str_tab;
^
../bpf-translate.cxx: In constructor ‘bpf::BPF_Output::BPF_Output(int)’:
../bpf-translate.cxx:1827:5: error: class ‘bpf::BPF_Output’ does not have any field named ‘str_tab’
str_tab(stap_strtab_init(true))
^
../bpf-translate.cxx:1827:34: error: ‘dwelf_strtab_init’ was not declared in this scope
str_tab(stap_strtab_init(true))
^
../bpf-translate.cxx: In destructor ‘bpf::BPF_Output::~BPF_Output()’:
../bpf-translate.cxx:1835:20: error: ‘str_tab’ was not declared in this scope
stap_strtab_free(str_tab);
^
../bpf-translate.cxx:1835:27: error: ‘dwelf_strtab_free’ was not declared in this scope
stap_strtab_free(str_tab);
^
../bpf-translate.cxx: In member function ‘bpf::BPF_Section* bpf::BPF_Output::new_scn(const string&)’:
../bpf-translate.cxx:1854:6: error: ‘struct bpf::BPF_Section’ has no member named ‘name_ent’
n->name_ent = stap_strtab_add(str_tab, n->name.c_str());
^
../bpf-translate.cxx:1854:33: error: ‘str_tab’ was not declared in this scope
n->name_ent = stap_strtab_add(str_tab, n->name.c_str());
^
../bpf-translate.cxx:32:48: note: in definition of macro ‘stap_strtab_add’
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:32:51: error: ‘dwelf_strtab_add’ was not declared in this scope
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:1854:17: note: in expansion of macro ‘stap_strtab_add’
n->name_ent = stap_strtab_add(str_tab, n->name.c_str());
^
../bpf-translate.cxx: In member function ‘bpf::BPF_Symbol* bpf::BPF_Output::new_sym(const string&, bpf::BPF_Section*, long int)’:
../bpf-translate.cxx:1864:6: error: ‘struct bpf::BPF_Symbol’ has no member named ‘name_ent’
s->name_ent = stap_strtab_add(str_tab, s->name.c_str());
^
../bpf-translate.cxx:1864:33: error: ‘str_tab’ was not declared in this scope
s->name_ent = stap_strtab_add(str_tab, s->name.c_str());
^
../bpf-translate.cxx:32:48: note: in definition of macro ‘stap_strtab_add’
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:32:51: error: ‘dwelf_strtab_add’ was not declared in this scope
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:1864:17: note: in expansion of macro ‘stap_strtab_add’
s->name_ent = stap_strtab_add(str_tab, s->name.c_str());
^
../bpf-translate.cxx: In function ‘void bpf::output_symbols_sections(bpf::BPF_Output&)’:
../bpf-translate.cxx:2183:31: error: ‘struct bpf::BPF_Output’ has no member named ‘str_tab’
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2183:49: error: ‘dwelf_strtab_finalize’ was not declared in this scope
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2190:39: error: ‘struct bpf::BPF_Symbol’ has no member named ‘name_ent’
b->st_name = stap_strent_offset(s->name_ent);
^
../bpf-translate.cxx:2190:47: error: ‘dwelf_strent_off’ was not declared in this scope
b->st_name = stap_strent_offset(s->name_ent);
^
../bpf-translate.cxx:2196:29: error: ‘struct bpf::BPF_Output’ has no member named ‘str_tab’
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2196:47: error: ‘dwelf_strtab_finalize’ was not declared in this scope
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2203:48: error: ‘struct bpf::BPF_Section’ has no member named ‘name_ent’
s->shdr->sh_name = stap_strent_offset(s->name_ent);
^
../bpf-translate.cxx:2203:56: error: ‘dwelf_strent_off’ was not declared in this scope
s->shdr->sh_name = stap_strent_offset(s->name_ent);
从以上信息可知:以上修改反而会引入更多的问题,因此看一下代码
2.6.1.4.8.4 将sytemtap的代码还原
2.6.1.4.8.5 清掉build目录
rm -rf build
2.6.1.4.8.6 重新创建build目录
mkdir build
2.6.1.4.8.7 切换到build目录
cd build
2.6.1.4.8.8 查找elfutils目录
sudo find / |grep elfutils
笔者找到elfutils对应的头文件路径是/usr/include (默认使用apt-get install安装的软件对应的头文件在/usr/include目录下)
2.6.1.4.8.9 指定elfutils的头文件路径进行配置并且指定安装目录为/opt/
rm * -rf (需要清一下之前在build目录下生成的文件)
../configure --with-elfutils=/usr/include --prefix=/opt/
出现以下错误信息:
checking for libvirt... no
checking for libxml2... yes
configure: WARNING: will not build systemtap virt support, cannot find libvirt headers
checking for python-config... /usr/bin/python-config
checking Python.h usability... yes
checking Python.h presence... yes
checking for Python.h... yes
checking for python3-config... /usr/bin/python3-config
checking Python.h usability... yes
checking Python.h presence... yes
checking for Python.h... yes
checking for jsonc... no
checking for ncurses... yes
checking for assembler .section "?" flags support... yes
checking linux/bpf.h usability... yes
checking linux/bpf.h presence... yes
checking for linux/bpf.h... yes
configure: error: No /usr/include/configure, forgot to run autoreconf -i?
从此信息中发现缺少configure,应该与elfutils有关,那么下载与当前elfutils对应的版本源代码
2.6.1.4.8.10 查看当前elfutils版本的方法:
eu-objdump -V
objdump (elfutils) 0.165
Copyright (C) 2012 Red Hat, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Ulrich Drepper.
版本为0.165
2.6.1.4.8.11 获取elfutils的源码
笔者找到的0.165下载地址为:https://sourceware.org/elfutils/ftp/0.165/elfutils-0.165.tar.bz2
2.6.1.4.8.12 下载elfutils源码
tsocks wget https://sourceware.org/elfutils/ftp/0.165/elfutils-0.165.tar.bz2 (FQ下载才快)
2.6.1.4.8.13解压elfutils
tar xvf elfutils-0.165.tar.bz2
2.6.1.4.8.14 指定elfutils源码的路径(笔者的elfutils源码路径在/home/jello/elfutils-0.165下)并指定systemtap的安装路径
../configure --with-elfutils=/home/jello/elfutils-0.165 --prefix=/opt/
2.6.1.4.8.15 开始编译
make
2.6.1.4.8.16 编译又出现错误
systemtap-3.2/build/python/../includes/sys -fPIC -I/usr/include/python2.7 -c HelperSDT/_HelperSDT.c -o /home/jello/systemtap-3.2/build/python/py2build/temp.linux-x86_64-2.7/HelperSDT/_HelperSDT.o
HelperSDT/_HelperSDT.c:10:21: fatal error: sys/sdt.h: 没有那个文件或目录
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Makefile:621: recipe for target 'all-local' failed
make[2]: *** [all-local] Error 1
make[2]: Leaving directory '/home/jello/systemtap-3.2/build/python'
Makefile:2012: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/jello/systemtap-3.2/build'
Makefile:717: recipe for target 'all' failed
make: *** [all] Error 2
从以上信息可知:没有头文件sys/sdt.h,那么就查找sys/sdt.h的路径
2.6.1.4.8.17 查找sys/sdt.h的路径
sudo find /|grep "sys/sdt.h"
发现sys/sdt.h在systemtap-3.2的includes目录下
2.6.1.4.8.18 修改build/python目录下的Makefile
修改前的内容:
AM_CPPFLAGS = -I$(srcdir)/../includes \
-I$(abs_builddir)/../includes/sys修改后的内容:AM_CPPFLAGS = -I$(srcdir)/../includes \
-I$(abs_builddir)/../includes/sys \ -I$(abs_srcdir)/../includes
2.6.1.4.8.19 再次编译又出现以下错误(cd ../../python; CFLAGS="-I../../python/../includes -I/home/jello/systemtap-3.2/build/python/../includes/sys -I/home/jello/systemtap-3.2/build/../python/../includes" /usr/bin/python3 setup.py build \
--build-base /home/jello/systemtap-3.2/build/python/py3build \
--verbose)
Traceback (most recent call last):
File "setup.py", line 9, in <module>
from setuptools import setup, Extension
ImportError: No module named 'setuptools'
Makefile:622: recipe for target 'all-local' failed
make[2]: *** [all-local] Error 1
make[2]: Leaving directory '/home/jello/data/development/systemtap-3.2/build/python'
Makefile:2012: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/jello/data/development/systemtap-3.2/build'
Makefile:717: recipe for target 'all' failed
make: *** [all] Error 2
从以上信息可知:python缺少setuptools模块,所以安装setuptools模块,并且使用python3进行安装
2.6.1.4.8.20 安装setuptools模块
2.6.1.4.8.20.1 获取模块
wget https://bootstrap.pypa.io/ez_setup.py
2.6.1.4.8.20.2 安装模块
sudo python3 ez_setup.py2.6.1.4.8.21 继续编译
make一切ok2.6.1.4.9 安装systemtapsudo make install2.6.1.4.10 卸载原来使用apt-get install安装的systemtapsudo apt-get remove systemtap2.6.1.4.11 设置环境变量使新安装的stap生效
vi /etc/profile往文件最后加入一句:export "PATH=/opt/bin:$PATH"2.6.1.5 验证systap是否安装成功stap -VSystemtap translator/driver (version 3.2/0.165, non-git sources)
Copyright (C) 2005-2017 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 4.11
enabled features: AVAHI PYTHON2 PYTHON3 LIBSQLITE3 LIBXML2 NLS NSS READLINE以上信息中发现版本为3.2/0.165,表明systemtap已经安装成功
2.6.2 启动stap
2.6.2.1 先切换为最高权限
sudo su -l
2.6.2.2 启动stap (之前写过一个nginx.systemtap文件,记得指定该文件路径,笔者将nginx.systemtap放在了当前目录下,因此直接的指定文件)
stap --ld -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 nginx.systemtap --vp 0001 > nginx.out
2.7重开终端进行压力测试(可选)
进行压力测试:
ab -n 900000 -c 50 http://192.168.2.323/index.php /*192.168.2.323为笔者的本地ip地址,注意修改成自己的本地ip地址*/
2.8 FlameFraph
2.8.1 获取FlameGraph
git clone https://github.com/brendangregg/FlameGraph.git
2.8.2
由于2.6.2.2一直未生成nginx.out,因此需要找一下nginx.systemtap是否有错误
未写完,待续...
相关文章
- vmstat和iostat命令进行Linux性能监控
- 【Linux】在linux上java工具jps jstat jinfo等命令找不到怎么办
- linux分享二:Linux如何修改字符集
- Linux基础之查看linux发行版以及内核版本
- Linux基础之linux常用命令之文本替换
- linux性能采用工具oprofile使用
- linux下性能分析命令[总结]
- 【学习总结】快速上手Linux玩转典型应用-第2章-linux简介
- Linux的IO性能监控工具iostat详解
- linux如deepin manjaro对笔记本电脑电池的伤害解决方案:TLP:一个可以延长 Linux 笔记本电池寿命的高级电源管理工具
- linux 错误码 65280,Android:无法在同一部手机上执行通过Linux部署编译的文件
- 【JVM与性能调优】Linux监控命令(与网络相关的命令)复盘
- sar命令,linux中最为全面的性能分析工具之一
- 信号概述 硬件异常将产生信号 进程间通信概述 进程间通信 进程间通信功能 Linux 操作系统支持的主要进程间通信的通信机制 linux 进程间通信(IPC)由以下几个部分发展而来 数据传输 信号
- L74.linux命令每日一练 -- 第十章 Linux网络管理命令 -- nmap和tcpdump
- L73.linux命令每日一练 -- 第十章 Linux网络管理命令 -- dig和host
- L70.linux命令每日一练 -- 第十章 Linux网络管理命令 -- nc和ssh
- L49.linux命令每日一练 -- 第八章 Linux磁盘与文件系统管理命令 -- fdisk和partprobe
- L46.linux命令每日一练 -- 第七章 Linux用户管理及用户信息查询命令 -- w和who
- L3.linux命令每日一练 -- 第一章 Linux命令行简介
- 优化Linux系统性能的必杀技:调整进程优先级!学会使用nice命令,让你的系统飞一般的顺畅!
- linux系统单机性能压测工具Wrk —— 筑梦之路
- linux 下监控某个进程的脚本-以frp为例
- linux线程退出正确姿势demo
- linux===sar命令性能监控
- Linux【实操篇】—— 用户管理、linux系统root密码找回方式
- Linux:性能分析域
- Linux系统知识讲解之二
- Linux stress性能压测工具使用
- Linux CPU性能 系统当中的不可中断进程D和僵尸进程Z
- Linux下的的GDB调试技巧四 —— 常用调试命令(1)
- Linux自动性能调整工具tuned
- 02 从头开始atac项目 ubuntu20 install r4.2 Linux系统环境配置 服务器版本的rstudio r install in linux /ubuntu/centos