What is strace, and how do I make use of it?

What this is#

strace records every system call a process makes, every file it opens, every network connection, every read, write, and wait, the full conversation between a program and the kernel. When logs say nothing and documentation lies, strace shows what a program actually does, which is why it's the tool people reach for last and wish they'd reached for first.

apt install strace

The honest warning first: the output is a firehose, and the skill is entirely in filtering (below). Also, tracing meaningfully slows the traced process, fine for debugging, not something to leave attached to a busy production service for hours.

The three moves that cover most uses#

1. "What files does this thing actually open?" The single most useful filter, watching a program look for its config:

strace -e trace=openat,stat -f myprogram 2>&1 | grep -v ENOENT

or with the failures, because the failures are often the answer:

strace -e trace=openat myprogram 2>&1 | grep ENOENT

ENOENT (no such file) trails show exactly which paths were tried and in what order, "why does it ignore my config" is solved here, and EACCES in the same view is a permissions problem with the exact path attached.

2. "What is this hung process doing?" Attach to a running PID:

strace -p 12345

A healthy busy process scrolls; a hung one shows its last call and sits there, and which call it's stuck in is the diagnosis: stuck in read on a network socket = waiting on a remote end (whose path you can test); stuck in write = a full pipe or slow disk; stuck in futex = waiting on a lock another thread holds. Detach with Ctrl+C, the process continues unharmed.

3. "Where does the time go?" The summary mode:

strace -c myprogram

runs the program and prints a table, calls, errors, and time per syscall type. A program spending 95% of its traced time in poll/select is waiting on the world, not working; thousands of failed openat calls explain a slow startup.

The flags worth knowing#

• -f: follow child processes, essential for anything that forks (which is most things, shells and web servers included).
• -e trace=network / =file / =signal: pre-made filter families ("show me only its network life").
• -o /tmp/trace.txt: write to a file, then grep it at leisure.
• -s 200: show more of each string argument (default truncates at 32 chars, right where the interesting part was).
• -t timestamps each line; -T shows time spent inside each call, the hung-call hunter's flag.

Reading a line#

openat(AT_FDCWD, "/etc/nginx/nginx.conf", O_RDONLY) = 3
connect(4, {sa_family=AF_INET, sin_port=htons(3306), ...}) = -1 ECONNREFUSED

Call, arguments, = result. A number is success (a file descriptor); -1 plus an error name is the story: that second line is a program failing to reach MySQL, connection refused, nothing listening, captured in the act. The error names repay learning: ENOENT (doesn't exist), EACCES (permissions), ECONNREFUSED/ETIMEDOUT (network), ENOSPC (disk full, caught red-handed).

For everything past the common patterns, strace output is superb AI chatbot material, paste the last screen of a hung process's trace and ask what it's waiting on.

(ltrace is the sibling that traces library calls instead of system calls, occasionally the right level, but strace answers the server questions.)

Still need help?#

You can open a support ticket. So we can help on the first reply, it's worth mentioning:

• the VPS hostname or IP,
• the process you're tracing, and the last screen of strace output.

Related questions#

• "How do I see what files a program opens?"
• "How do I find out what a hung process is waiting on?"
• "How do I attach strace to a running process?"
• "What do ENOENT and EACCES mean in strace output?"