gdb-automatic-deadlock-detector
Script adds new GDB command which automatically detects C/C++ thread lockups and deadlocks in GDB debugger.
It automates process of deadlock detection described in this instruction: https://en.wikibooks.org/wiki/Linux_Applications_Debugging_Techniques/Deadlocks
Automation is written as GDB Python script based on Python GBD API: https://sourceware.org/gdb/onlinedocs/gdb/Python-API.html
This script adds new 'blocked' command to GDB. 'blocked' command shows which threads are blocking other threads (or are deadlocked each other) ex.
(gdb) blocked
********************************************************************************
Displaying blocking threads using 'blocked' command
Thread: 7960 waits for thread: 7959 AND DEADLOCKED
Thread: 7959 waits for thread: 7960 AND DEADLOCKED
********************************************************************************
It means that thread 7960 waits for mutex holded by thread 7959 AND thread 7959 waits for mutex holded by thread 7960 - so threas 7959 and 7960 are deadlocked
Preparing sample deadlocked app
- Compile sample application containing thread deadlock.
gcc -pthread deadlockExample.c -o deadlockExample
- Make sure that your Linux OS can dump core files from the processes. Set it up by:
ulimit -c unlimited
- Run compiled application
./deadlockExample
- Notice that this application stucks (because of deadlock). Run another terminal and send SIGABRT signal (signal nr 6) to the running app:
kill -6 $(pidof deadlockExample)
- You should have core file outputed to your PWD dir.
- Also you need to have your GDB compiled with Python support. Check whether you can run 'python' command in your GDB. If not the recompile your GDB using this instruction: https://askubuntu.com/questions/513626/cannot-compile-gdb7-8-with-python-support
Usage
- See content of the 'gdbcommands' file - it contains commands wilchi will be invoked in GDB to run this to detect deadlock automatically (notice import our Python script gdbDisplayLockedThreads and invoking 'blocked' command):
python
import sys
sys.path.append('.')
import gdbDisplayLockedThreads
end
thread apply all bt
blocked
- Run gdb and invoke commands from 'gdbcommands' file automatically:
gdb -c core ./deadlockExample -x ./gdbcommands -batch
- Output shows all backtraces of threads from deadlockExample followed by additional info of which threads are waiting for mutexes holding by other threads ex.
Thread 3 (Thread 0x7efc4958f700 (LWP 7960))
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007efc4a164cfd in __GI___pthread_mutex_lock (mutex=0x601300 <write_mutex>) at ../nptl/pthread_mutex_lock.c:80
#2 0x0000000000400aae in readTest ()
#3 0x00007efc4a1626aa in start_thread (arg=0x7efc4958f700) at pthread_create.c:333
#4 0x00007efc49e97eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 2 (Thread 0x7efc49d90700 (LWP 7959)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007efc4a164cfd in __GI___pthread_mutex_lock (mutex=0x6012c0 <read_mutex>) at ../nptl/pthread_mutex_lock.c:80
#2 0x0000000000400a00 in writeTest ()
#3 0x00007efc4a1626aa in start_thread (arg=0x7efc49d90700) at pthread_create.c:333
#4 0x00007efc49e97eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 1 (Thread 0x7efc4a564700 (LWP 7958)):
#0 0x00007efc4a1638ed in pthread_join (threadid=139622035818960, thread_return=0x0) at pthread_join.c:90
#1 0x0000000000400b95 in main ()
********************************************************************************
Displaying blocking threads using 'blocked' command
Thread: 7960 waits for thread: 7959 AND DEADLOCKED
Thread: 7959 waits for thread: 7960 AND DEADLOCKED
********************************************************************************
It means that thread 7960 waits for mutex holded by thread 7959 AND thread 7959 waits for mutex holded by thread 7960 - so threas 7959 and 7960 are deadlocked
- Now you can detect deadlock for any core using this quick gdb command from point 2 or 'blocked' command from gdb.