Debugging Linux kernel

Performance

There are many factors that can affect the performance of the Linux kernel, including hardware configurations, software configurations, and workload characteristics.

In this context, performance optimization of the Linux kernel involves identifying and addressing performance bottlenecks in the system. This can involve tuning kernel parameters, optimizing system resources, and identifying and fixing bugs and other issues that may be impacting performance.

Given the complexity of the Linux kernel and the wide range of factors that can affect performance, performance optimization can be a challenging task. However, with the right tools and techniques, it is possible to significantly improve the performance and reliability of Linux-based systems.

Perf_events

Perf_events, short for performance events, is a powerful interface that provides detailed insights into the performance characteristics of software running on a system. By analyzing the data collected by perf_events, developers can identify performance bottlenecks and optimize software to improve performance and reduce resource utilization. Perf_events is designed to be a lightweight, low-overhead monitoring solution that has minimal impact on system performance.

🔧 TODO

⚲ Interfaces

man 1 perf – performance analysis tools

Basic commands:

man 1 perf-help – display help information about perf

man 1 perf-top – System profiling tool.

man 1 perf-record – Run a command and record its profile into perf.data

man 1 perf-report – Read perf.data (created by perf record) and display the profile

Other commands:

man 1 perf-annotate – Read perf.data (created by perf record) and display annotated code

man 1 perf-archive – Create archive with object files with build-ids found ...

man 1 perf-arm-spe – Support for Arm Statistical Profiling Extension within...

man 1 perf-bench – General framework for benchmark suites

man 1 perf-buildid-cache – Manage build-id cache.

man 1 perf-buildid-list – List the buildids in a perf.data file

man 1 perf-c2c – Shared Data C2C/HITM Analyzer.

man 1 perf-config – Get and set variables in a configuration file.

man 1 perf-daemon – Run record sessions on background

man 1 perf-data – Data file related processing

man 1 perf-diff – Read perf.data files and display the differential profile

man 1 perf-dlfilter – Filter sample events using a dynamically loaded shared...

man 1 perf-evlist – List the event names in a perf.data file

man 1 perf-ftrace – simple wrapper for kernel's ftrace functionality

man 1 perf-inject – Filter to augment the events stream with additional in...

man 1 perf-intel-pt – Support for Intel Processor Trace within perf tools

man 1 perf-iostat – Show I/O performance metrics

man 1 perf-kallsyms – Searches running kernel for symbols

man 1 perf-kmem – Tool to trace/measure kernel memory properties

man 1 perf-kvm – Tool to trace/measure kvm guest os

man 1 perf-kwork – Tool to trace/measure kernel work properties (latencies)

man 1 perf-list – List all symbolic event types

man 1 perf-lock – Analyze lock events

man 1 perf-mem – Profile memory accesses

man 1 perf-probe – Define new dynamic tracepoints

man 1 perf-sched – Tool to trace/measure scheduler properties (latencies)

man 1 perf-script – Read perf.data (created by perf record) and display tr...

man 1 perf-script-perl – Process trace data with a Perl script

man 1 perf-script-python – Process trace data with a Python script

man 1 perf-stat – Run a command and gather performance counter statistics

man 1 perf-test – Runs sanity tests.

man 1 perf-timechart – Tool to visualize total system behavior during a workload

man 1 perf-trace – strace inspired tool

man 1 perf-version – display the version of perf binary

⚙️ Internals

man 2 perf_event_open – sets up performance monitoring

uapi/linux/perf_event.h _inc

tools/perf _src

linux/perf_event.h _inc

kernel/events/core.c _src

kernel/profile.c _src – simple profiling

📖 References

perf – instruments CPU performance counters, tracepoints, kprobes, and uprobes

https://perf.wiki.kernel.org/

📚 Further reading

perf Examples

The Unofficial Linux Perf Events Web-Page

---

🛠️ Utilities

Performance Co-Pilot, https://pcp.io/ – Performance Co-Pilot

Prometheus, https://prometheus.io/

https://github.com/redhat-nfvpe/container-perf-tools

https://github.com/brendangregg/perf-tools – performance analysis tools based on Linux perf_events (aka perf) and ftrace

readprofile – a tool to read kernel profiling information

📚 Further reading

stress-ng – exercises various kernel interfaces

http://trac.gateworks.com/wiki/linux/profiling

Analyzing application performance in RHEL 9

Monitoring and managing system status and performance in RHEL 9

Real-time Linux

User space debug interfaces

⚲ Interfaces

man 1 dmesg – prints or control the kernel ring buffer

man 2 syslog – system call, which is used to control the kernel printk() buffer

man 1 strace – system calls and signals tracing tool

man 2 ptrace – process trace system call

man 3 klogctl

man 5 core

/sys/kernel/debug/ – debugfs

dmesg --console-level <level>

gdb /usr/src/linux/vmlinux /proc/kcore

/proc/self/stack

dynamic _doc debug

⌨️ hands-on:

echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control

⚙️ Internals

handle_sysrq _id

📚 References

Development tools for the kernel _doc

DebugFS _doc, samples/qmi/qmi_sample_client.c _src

Kprobe-based Event Tracing _doc

Dynamic debug _doc

Linux Magic System Request Key Hacks _doc

Magic SysRq key

Tracing and logging

⚲ API:

User-space interface:

man 1 dmesg – prints or control the kernel ring buffer

man 2 syslog – system call, which is used to control the kernel printk() buffer

/proc/kmsg

man 1 trace-cmd – interacts with Ftrace Linux kernel internal tracer /sys/kernel/debug/tracing/

Most common functions

linux/printk.h _inc

pr_devel _id- conditional debug-level message

pr_debug _id- conditional debug-level or dynamic _doc message

⌨️ hands-on:

echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control

Log messages with other levels:

pr_info _id, pr_notice _id, pr_warn _id, pr_err _id, pr_crit _id, pr_alert _id, pr_emerg _id

asm-generic/bug.h _inc

WARN_ON _id

WARN _id

⚙️ Internals

printk _id

kernel/printk/printk.c _src

arch/x86/kernel/traps.c _src

lib/dump_stack.c _src

kernel/trace _src

scripts/tracing/draw_functrace.py _src

logging _ltp, tracing _ltp

samples/ftrace _src

samples/trace_events _src

samples/trace_printk _src

linux/instrumentation.h _inc

📚 References:

Debugging by printing

Message logging with printk _doc

SystemTap

man 1 stap – systemtap script translator/driver

strace

man 1 strace – trace system calls and signals

LTTng

ftrace

Linux Tracing Technologies _doc

Tracepoint Analysis _doc

Function Tracer _doc – function, latency and event tracing

Event Tracing _doc

Using ftrace to hook to functions _doc

Fprobe - Function entry/exit probe _doc

Kprobes _doc

Kprobe-based Event Tracing _doc

Uprobe-tracer: Uprobe-based Event Tracing _doc

Using the Linux Kernel Tracepoints _doc

Subsystem Trace Points: kmem _doc

Subsystem Trace Points: power _doc

NMI Trace Events _doc

In-kernel memory-mapped I/O tracing _doc

Event Histograms _doc

Histogram Design Notes _doc

Boot-time tracing _doc

Hardware Latency Detector _doc

Intel(R) Trace Hub (TH) _doc

Lockless Ring Buffer Design _doc

System Trace Module _doc

CoreSight - ARM Hardware Trace _doc

🔧 TODO. 🚀 advanced features

linux/kmemleak.h _inc – memory leak detector

pr_cont _id- continues a previous log message in the same line

print_hex_dump_bytes _id

print_hex_dump_debug _id

dump_stack _id

CONFIG_PRINTK_CALLER _id

CONFIG_DEBUG_KERNEL _id

CONFIG_DEBUG_INFO _id

https://git.kernel.org/pub/scm/libs/libtrace/

kgdb and kdb

⚲ Interfaces

linux/kgdb.h _inc

linux/kdb.h _inc

⚙️ Internals

kernel/debug _src

📚 References

Using kgdb, kdb and the kernel debugger internals _doc

kdump

kdump _doc

man 8 crash – Analyze Linux crash dump data or a live system

eBPF

⚲ API:

man 2 bpf ↪ kernel/bpf/syscall.c _src

📖 References

eBPF and BPF _doc

📚 Further reading

man 7 bpf-helpers

Linux Extended BPF (eBPF) Tracing Tools

bpftrace – High-level tracing language for Linux eBPF

BCC – Tools for BPF-based Linux IO analysis, networking, monitoring, and more

Example of trace.py

man 8 stapbpf

eBPF Programming for Linux Kernel Tracing

lockdep - Runtime locking correctness validator _doc

Watchdogs

The Linux Kernel/Softdog Driver

dev_watchdog _id – network device watchdog

The NMI watchdog lockup detectors:

⚲ API

/proc/sys/kernel/nmi_watchdog

/proc/sys/kernel/soft_watchdog

/proc/sys/kernel/watchdog

/proc/sys/kernel/watchdog_cpumask

/proc/sys/kernel/watchdog_thresh

/proc/sys/kernel/hardlockup_all_cpu_backtrace

/proc/sys/kernel/hardlockup_panic

/proc/sys/kernel/softlockup_all_cpu_backtrace

/proc/sys/kernel/softlockup_panic

linux/nmi.h _inc

👁️ Example

./lib/test_lockup.c _src – test module to generate lockups

Provoke NMI watchdog without panic:

echo 0 > /proc/sys/kernel/hardlockup_panic
insmod test_lockup.ko disable_irq=1 time_secs=13

⚙️ Internals

kernel/watchdog.c _src – detects hard and soft lockups on a system

kernel/watchdog_perf.c _src – detects hard lockups on a system using perf

kernel/watchdog_buddy.c _src

📚 References

Documentation for /proc/sys/kernel/ _doc

Softlockup detector and hardlockup detector (aka nmi_watchdog) _doc

kernel parameters:

nmi_watchdog _param

nowatchdog _param

nosoftlockup _param

softlockup_panic _param

---

...

⚙️ Internals

arch/x86/kernel/traps.c _src

📖 References for debugging

Ramoops oops/panic logger _doc

pstore block oops/panic logger _doc

Fault injection _doc

Bisecting a bug _doc

Development tools for the kernel _doc

Kernel Testing Guide _doc

Checkpatch _doc, scripts/checkpatch.pl _src

Selftests _doc, tools/testing/selftests _src

linux/tracepoint.h _inc

📚 Further reading

https://drgn.readthedocs.io/ – programmable debugger

https://crash-utility.github.io/

https://wiki.ubuntu.com/Kernel/Debugging

Linux Applications Debugging Techniques