进程无法启动，软件运行速度突然变慢，程序的"Segment Fault"等等都是让每个Unix系统用户头痛的问题，本文通过三个实际案例演示如何使用truss、strace和ltrace这三个常用的调试工具来快速诊断软件的"疑难杂症"。
truss和strace用来跟踪一个进程的系统调用或信号产生的情况，而 ltrace用来跟踪进程调用库函数的情况。truss是早期为System V R4开发的调试程序，包括Aix、FreeBSD在内的大部分Unix系统都自带了这个工具；而strace最初是为SunOS系统编写的，ltrace最早出现在GNU/Debian Linux中。这两个工具现在也已被移植到了大部分Unix系统中，大多数Linux发行版都自带了strace和ltrace，而FreeBSD也可通过Ports安装它们。
你不仅可以从命令行调试一个新开始的程序，也可以把truss、strace或ltrace绑定到一个已有的PID上来调试一个正在运行的程序。三个调试工具的基本使用方法大体相同，下面仅介绍三者共有，而且是最常用的三个命令行参数：
-f ：除了跟踪当前进程外，还跟踪其子进程。
-o file ：将输出信息写到文件file中，而不是显示到标准错误输出（stderr）。
-p pid ：绑定到一个由pid对应的正在运行的进程。此参数常用来调试后台进程。
|
使用上述三个参数基本上就可以完成大多数调试任务了，下面举几个命令行例子：
truss -o ls.truss ls -al： 跟踪ls -al的运行，将输出信息写到文件/tmp/ls.truss中。
strace -f -o vim.strace vim： 跟踪vim及其子进程的运行，将输出信息写到文件vim.strace。
ltrace -p 234： 跟踪一个pid为234的已经在运行的进程。
|
三个调试工具的输出结果格式也很相似，以strace为例：
brk(0) = 0x8062aa8
brk(0x8063000) = 0x8063000
mmap2(NULL, 4096, PROT_READ, MAP_PRIVATE, 3, 0x92f) = 0x40016000
|
每一行都是一条系统调用，等号左边是系统调用的函数名及其参数，右边是该调用的返回值。 truss、strace和ltrace的工作原理大同小异，都是使用ptrace系统调用跟踪调试运行中的进程，详细原理不在本文讨论范围内，有兴趣可以参考它们的源代码。
举两个实例演示如何利用这三个调试工具诊断软件的"疑难杂症"：
案例一：运行clint出现Segment Fault错误
操作系统：FreeBSD-5.2.1-release
clint是一个C++静态源代码分析工具，通过Ports安装好之后，运行：
# clint foo.cpp
Segmentation fault (core dumped)
|
在Unix系统中遇见"Segmentation Fault"就像在MS Windows中弹出"非法操作"对话框一样令人讨厌。OK，我们用truss给clint"把把脉"：
# truss -f -o clint.truss clint
Segmentation fault (core dumped)
# tail clint.truss
739: read(0x6,0x806f000,0x1000) = 4096 (0x1000)
739: fstat(6,0xbfbfe4d0) = 0 (0x0)
739: fcntl(0x6,0x3,0x0) = 4 (0x4)
739: fcntl(0x6,0x4,0x0) = 0 (0x0)
739: close(6) = 0 (0x0)
739: stat("/root/.clint/plugins",0xbfbfe680) ERR#2 'No such file or directory'
SIGNAL 11
SIGNAL 11
Process stopped because of: 16
process exit, rval = 139
|
我们用truss跟踪clint的系统调用执行情况，并把结果输出到文件clint.truss，然后用tail查看最后几行。注意看clint执行的最后一条系统调用（倒数第五行）：stat("/root/.clint/plugins",0xbfbfe680) ERR#2 'No such file or directory'，问题就出在这里：clint找不到目录"/root/.clint/plugins"，从而引发了段错误。怎样解决？很简单：mkdir -p /root/.clint/plugins，不过这次运行clint还是会"Segmentation Fault"9。继续用truss跟踪，发现clint还需要这个目录"/root/.clint/plugins/python"，建好这个目录后clint终于能够正常运行了。
案例二：vim启动速度明显变慢
操作系统：FreeBSD-5.2.1-release
vim版本为6.2.154，从命令行运行vim后，要等待近半分钟才能进入编辑界面，而且没有任何错误输出。仔细检查了.vimrc和所有的vim脚本都没有错误配置，在网上也找不到类似问题的解决办法，难不成要hacking source code？没有必要，用truss就能找到问题所在：
# truss -f -D -o vim.truss vim
|
这里-D参数的作用是：在每行输出前加上相对时间戳，即每执行一条系统调用所耗费的时间。我们只要关注哪些系统调用耗费的时间比较长就可以了，用less仔细查看输出文件vim.truss，很快就找到了疑点：
735: 0.000021511 socket(0x2,0x1,0x0) = 4 (0x4)
735: 0.000014248 setsockopt(0x4,0x6,0x1,0xbfbfe3c8,0x4) = 0 (0x0)
735: 0.000013688 setsockopt(0x4,0xffff,0x8,0xbfbfe2ec,0x4) = 0 (0x0)
735: 0.000203657 connect(0x4,{ AF_INET 10.57.18.27:6000 },16) ERR#61 'Connection refused'
735: 0.000017042 close(4) = 0 (0x0)
735: 1.009366553 nanosleep(0xbfbfe468,0xbfbfe460) = 0 (0x0)
735: 0.000019556 socket(0x2,0x1,0x0) = 4 (0x4)
735: 0.000013409 setsockopt(0x4,0x6,0x1,0xbfbfe3c8,0x4) = 0 (0x0)
735: 0.000013130 setsockopt(0x4,0xffff,0x8,0xbfbfe2ec,0x4) = 0 (0x0)
735: 0.000272102 connect(0x4,{ AF_INET 10.57.18.27:6000 },16) ERR#61 'Connection refused'
735: 0.000015924 close(4) = 0 (0x0)
735: 1.009338338 nanosleep(0xbfbfe468,0xbfbfe460) = 0 (0x0)
|
vim试图连接10.57.18.27这台主机的6000端口（第四行的connect（）），连接失败后，睡眠一秒钟继续重试（第6行的nanosleep（））。以上片断循环出现了十几次，每次都要耗费一秒多钟的时间，这就是vim明显变慢的原因。可是，你肯定会纳闷："vim怎么会无缘无故连接其它计算机的6000端口呢？"。问得好，那么请你回想一下6000是什么服务的端口？没错，就是X Server。看来vim是要把输出定向到一个远程X Server，那么Shell中肯定定义了DISPLAY变量，查看.cshrc，果然有这么一行：setenv DISPLAY ${REMOTEHOST}:0，把它注释掉，再重新登录，问题就解决了。
案例三：用调试工具掌握软件的工作原理
操作系统：Red Hat Linux 9.0
用调试工具实时跟踪软件的运行情况不仅是诊断软件"疑难杂症"的有效的手段，也可帮助我们理清软件的"脉络"，即快速掌握软件的运行流程和工作原理，不失为一种学习源代码的辅助方法。下面这个案例展现了如何使用strace通过跟踪别的软件来"触发灵感"，从而解决软件开发中的难题的。
大家都知道，在进程内打开一个文件，都有唯一一个文件描述符（fd：file descriptor）与这个文件对应。而本人在开发一个软件过程中遇到这样一个问题：已知一个fd ，如何获取这个fd所对应文件的完整路径？不管是Linux、FreeBSD或是其它Unix系统都没有提供这样的API，怎么办呢？我们换个角度思考：Unix下有没有什么软件可以获取进程打开了哪些文件？如果你经验足够丰富，很容易想到lsof，使用它既可以知道进程打开了哪些文件，也可以了解一个文件被哪个进程打开。好，我们用一个小程序来试验一下lsof，看它是如何获取进程打开了哪些文件。
/* testlsof.c */
#include #include #include #include #include int main(void) { open("/tmp/foo", O_CREAT|O_RDONLY); /* 打开文件/tmp/foo */ sleep(1200); /* 睡眠1200秒，以便进行后续操作 */ return 0; }
|
将testlsof放入后台运行，其pid为3125。命令lsof -p 3125查看进程3125打开了哪些文件，我们用strace跟踪lsof的运行，输出结果保存在lsof.strace中：
# gcc testlsof.c -o testlsof
# ./testlsof &
[1] 3125
# strace -o lsof.strace lsof -p 3125
|
我们以"/tmp/foo"为关键字搜索输出文件lsof.strace，结果只有一条：
# grep '/tmp/foo' lsof.strace
readlink("/proc/3125/fd/3", "/tmp/foo", 4096) = 8
|
原来lsof巧妙的利用了/proc/nnnn/fd/目录（nnnn为pid）：Linux内核会为每一个进程在/proc/建立一个以其pid为名的目录用来保存进程的相关信息，而其子目录fd保存的是该进程打开的所有文件的fd。目标离我们很近了。好，我们到/proc/3125/fd/看个究竟：
# cd /proc/3125/fd/
# ls -l
total 0
lrwx------ 1 root root 64 Nov 5 09:50 0 -> /dev/pts/0
lrwx------ 1 root root 64 Nov 5 09:50 1 -> /dev/pts/0
lrwx------ 1 root root 64 Nov 5 09:50 2 -> /dev/pts/0
lr-x------ 1 root root 64 Nov 5 09:50 3 -> /tmp/foo
# readlink /proc/3125/fd/3
/tmp/foo
|
答案已经很明显了：/proc/nnnn/fd/目录下的每一个fd文件都是符号链接，而此链接就指向被该进程打开的一个文件。我们只要用readlink()系统调用就可以获取某个fd对应的文件了，代码如下：
#include #include #include #include #include #include int get_pathname_from_fd(int fd, char pathname[], int n) { char buf[1024]; pid_t pid; bzero(buf, 1024); pid = getpid(); snprintf(buf, 1024, "/proc/%i/fd/%i", pid, fd); return readlink(buf, pathname, n); } int main(void) { int fd; char pathname[4096]; bzero(pathname, 4096); fd = open("/tmp/foo", O_CREAT|O_RDONLY); get_pathname_from_fd(fd, pathname, 4096); printf("fd=%d; pathname=%sn", fd, pathname); return 0; }
|
出于安全方面的考虑，在FreeBSD 5 之后系统默认已经不再自动装载proc文件系统，因此，要想使用truss或strace跟踪程序，你必须手工装载proc文件系统：mount -t procfs proc /proc；或者在/etc/fstab中加上一行：
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
lsof 显示进程打开的文件
lslk 显示进程锁定的文件
strace 调试及跟踪 ， 对进程有 strace /truss 等
od 输出文件的内容
2020年12月14日 15:48
This is very educational content and written well for a change. It's nice to see that some people still understand how to write a quality post! soap2 day
2021年1月14日 14:38
It is my first visit to your blog, and I am very impressed with the articles that you serve. Give adequate knowledge for me. Thank you for sharing useful material. I will be back for the more great post. best canadian online casino
2021年1月15日 19:09
Every one of us from time to time is going through serious physical and mental efforts.In such condition it's practically impossible to bring back alertness and freshness with a cup of coffee or a glass of juice. Because of that, Energy drinks like Red Bull become more and more trendy and popular. The million dollar question is: How safe is this magic poison in cans? Energy Drink for College students
2021年1月17日 19:11
Great content material and great layout. Your website deserves all of the positive feedback it’s been getting. soap2da
2021年2月16日 14:49
You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming. 2. cmd 368
2021年2月16日 19:05
I am constantly surprised by the amount of information accessible on this subject. What you presented was well researched and well written to get your stand on this over to all your readers. Thanks a lot my dear. live 22
2021年2月17日 14:44
Yes, I am entirely agreed with this article, and I just want say that this article is very helpful and enlightening. I also have some precious piece of concerned info !!!!!!Thanks. allbet
2021年4月15日 00:32
Cheap Flights Find Cheap Flight For Your Searching for cheap airfare and Last Minte Travel deals your cheap flight deals are almost ready Pay the remaining amount up to 72 hours before your flight's departure. Save money & time on your next flight booking with cheap flights.com Average trip savings calculated based on the price. For domestic flights, mondays showed the highest average airline ticket prices and for international flights, avoid booking on fridays. cheapflights
2021年4月24日 00:02
I have read all the comments and suggestions posted by the visitors for this article are very fine,We will wait for your next article so only.Thanks! Cuet kuet ruet Admission Website
2021年4月24日 00:11
Wow, cool post. I'd like to write like this too - taking time and real hard work to make a great article... but I put things off too much and never seem to get started. Thanks though. hire a hacker
2021年5月06日 05:01
Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with extra information? It is extremely helpful for me. magic mushroom chocolate
2021年5月09日 20:15
Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include. แทงหวย
2021年5月09日 20:21
What a fantabulous post this has been. Never seen this kind of useful post. I am grateful to you and expect more number of posts like these. Thank you very much. บาคาร่าออนไลน์
2021年5月09日 20:29
A very awesome blog post. We are really grateful for your blog post. You will find a lot of approaches after visiting your post. สมัคร UFABET
2021年5月14日 15:14
Just admiring your work and wondering how you managed this blog so well. It’s so remarkable that I can't afford to not go through this valuable information whenever I surf the internet! pitbull puppies near me
2021年5月16日 19:10
Positive site, where did u come up with the information on this posting?I have read a few of the articles on your website now, and I really like your style. Thanks a million and please keep up the effective work. Dank vapes
2021年5月19日 22:53
I got too much interesting stuff on your blog. I guess I am not the only one having all the enjoyment here! Keep up the good work. hairless cat for sale
2022年6月08日 22:18
Personality Disorder Treatment Hillside Horizon is an ideal place for families who seek additional support in helping their children and a better understanding of their child's mental health ... https://hillsidehorizon.com/
2022年6月08日 23:05
Gym in Laguna Hills Fort Fitness is a training ground and Gym in Laguna Hills that empowers physical and mental growth. Call (949) 524-2529 to get started! https://fortfitnessusa.com/
2022年6月14日 20:27
I used to be suggested this website by way of my cousin. I’m no longer positive whether this post is written by means of him as no one else recognize such particular about my difficulty. You’re amazing! Thank you! https://timemuzz.com/
2022年7月05日 19:56
Special BANGALORE Escort and Escort Listing site for elite gentlemen who want to spend time together with a quality Moscow escort lady. Bangalore Escorts service
2022年8月04日 18:57
These hackers use special software that has the capability of "linking" to other web sites. Each time you click a link on one of these sites, the browser will send a request for more information. dark web links
2022年8月04日 19:42
Most of these links contain viruses and other malware that can infect your computer and steal your private information. The only way to protect your computer from these harmful downloads is by running an antivirus program at regular intervals. deep web
2022年8月04日 19:56
By staying abreast of the latest developments, you will be able to make the most of the changes that are happening on the Internet in order to remain a successful Internet marketer. dark web links
2022年8月04日 20:18
If you are thinking; well there are good links on the internet too right? You are absolutely right, but you have to be very careful when you are surfing the dark web. dark web sites
2022年8月04日 20:34
But how do you get started? Most people who are interested in the dark web don't know where to start. dark web
2022年8月04日 20:50
These things will help you succeed in the world of affiliate marketing. Chia has been a successful internet marketer for years, and he has a lot of great advice to share with you. work from home jobs
2022年8月04日 21:06
Either way, you'll need to do some marketing to help sell the product. Once you do that, you'll be on your way to affiliate marketing success. affiliate marketing success
2022年10月01日 16:54
Youre so cool! I dont suppose Ive read anything like this before. So nice to find somebody with original thoughts on this subject. realy i appreciate you for beginning this up. this site is one area that is required on the internet, an individual after some originality. useful job for bringing new stuff to your net! alongnovember.com