strace to debug slowness/hang issues

strace is a Unix diagnostic tool used to trace system calls like read, write, connect etc. It comes handy for analyzing system hang and slowness issues. One of the key advantages is that we can use this tool without root access. But it comes with disadvantage too, it greatly slows down the process we are tracing. So don’t use it in production systems. 

 

strace generates lot of output, at the beginning it will be quiet overwhelming. As you work on problems, you will get comfortable using the tool. Let us see some examples of strace.

 

 

1) To check what system calls a command is making: 

 

strace <command>

e.g., strace ls

 

2) If the process is already started and want to debug diagnose problems: 

 

strace -p <pid>

 

3) We can save the output of strace to a file using -o option

e.g., strace -o out.txt ls

 

4) To trace a bash script:

strace -f -o trace.out bash <script name>

 

5) To trace specific system calls instead tracing all systems calls, we can use -e option. It reduces amount of output generated.

 

 

strace -e <syscall name> <command>
e.g., strace -e open ls

 

 

6) To print timestamp of each system call, use -t option. This helps to calculate how much time a system call took:

 

strace -t -e open ls

 

7)  If a process spawns child processes/threads, these can be traced using -f (stands for follows) option:

 

strace -ft -p <pid>
e.g., strace -ft -p 19516 -o trace.out 

 

-ff can be used to write each process/thread output into a separate file. In this case, output file name will be written in filename.pid form.

 

8) To print summary of time spent on system calls:

 

strace -c -p <pid>

 

9) strace prints only file descriptor numbers. Option “y” can be used to print file name instead of file descriptor. This is available only in latest versions of strace.

10) strace prints strings upto a maximum length of 32. -s strsize option can be used to specify custom length.