Debugging Linux kernel

< The Linux Kernel

Performance edit

There are many factors that can affect the performance of the Linux kernel, including hardware configurations, software configurations, and workload characteristics.

In this context, performance optimization of the Linux kernel involves identifying and addressing performance bottlenecks in the system. This can involve tuning kernel parameters, optimizing system resources, and identifying and fixing bugs and other issues that may be impacting performance.

Given the complexity of the Linux kernel and the wide range of factors that can affect performance, performance optimization can be a challenging task. However, with the right tools and techniques, it is possible to significantly improve the performance and reliability of Linux-based systems.

Perf_events, short for performance events, is a powerful interface that provides detailed insights into the performance characteristics of software running on a system. By analyzing the data collected by perf_events, developers can identify performance bottlenecks and optimize software to improve performance and reduce resource utilization. Perf_events is designed to be a lightweight, low-overhead monitoring solution that has minimal impact on system performance.

🔧 TODO

⚲ Interfaces

man 1 perf – performance analysis tools

Basic commands:

man 1 perf-help – display help information about perf

man 1 perf-top – System profiling tool.

man 1 perf-record – Run a command and record its profile into perf.data

man 1 perf-report – Read perf.data (created by perf record) and display th...

Other commands:

man 1 perf-annotate – Read perf.data (created by perf record) and display an...

man 1 perf-archive – Create archive with object files with build-ids found ...

man 1 perf-arm-spe – Support for Arm Statistical Profiling Extension within...

man 1 perf-bench – General framework for benchmark suites

man 1 perf-buildid-cache – Manage build-id cache.

man 1 perf-buildid-list – List the buildids in a perf.data file

man 1 perf-c2c – Shared Data C2C/HITM Analyzer.

man 1 perf-config – Get and set variables in a configuration file.

man 1 perf-daemon – Run record sessions on background

man 1 perf-data – Data file related processing

man 1 perf-diff – Read perf.data files and display the differential profile

man 1 perf-dlfilter – Filter sample events using a dynamically loaded shared...

man 1 perf-evlist – List the event names in a perf.data file

man 1 perf-ftrace – simple wrapper for kernel's ftrace functionality

man 1 perf-inject – Filter to augment the events stream with additional in...

man 1 perf-intel-pt – Support for Intel Processor Trace within perf tools

man 1 perf-iostat – Show I/O performance metrics

man 1 perf-kallsyms – Searches running kernel for symbols

man 1 perf-kmem – Tool to trace/measure kernel memory properties

man 1 perf-kvm – Tool to trace/measure kvm guest os

man 1 perf-kwork – Tool to trace/measure kernel work properties (latencies)

man 1 perf-list – List all symbolic event types

man 1 perf-lock – Analyze lock events

man 1 perf-mem – Profile memory accesses

man 1 perf-probe – Define new dynamic tracepoints

man 1 perf-sched – Tool to trace/measure scheduler properties (latencies)

man 1 perf-script – Read perf.data (created by perf record) and display tr...

man 1 perf-script-perl – Process trace data with a Perl script

man 1 perf-script-python – Process trace data with a Python script

man 1 perf-stat – Run a command and gather performance counter statistics

man 1 perf-test – Runs sanity tests.

man 1 perf-timechart – Tool to visualize total system behavior during a workload

man 1 perf-trace – strace inspired tool

man 1 perf-version – display the version of perf binary

man 2 perf_event_open – sets up performance monitoring

uapi/linux/perf_event.h_inc

🛠️ Utilities

Performance Co-Pilot, https://pcp.io/ – Performance Co-Pilot

Prometheus, https://prometheus.io/

https://github.com/redhat-nfvpe/container-perf-tools

https://github.com/brendangregg/perf-tools – performance analysis tools based on Linux perf_events (aka perf) and ftrace

readprofile – a tool to read kernel profiling information

⚙️ Internals

linux/perf_event.h_inc

kernel/events/core.c_src

kernel/profile.c_src – simple profiling

📚 References

stress-ng – exercises various kernel interfaces

perf – instruments CPU performance counters, tracepoints, kprobes, and uprobes

https://perf.wiki.kernel.org/

http://trac.gateworks.com/wiki/linux/profiling

Analyzing application performance in RHEL 9

Monitoring and managing system status and performance in RHEL 9

The Unofficial Linux Perf Events Web-Page

Real-time Linux

User space debug interfaces edit

⚲ Interfaces

man 1 dmesg – prints or control the kernel ring buffer

man 2 syslog – system call, which is used to control the kernel printk() buffer

man 1 strace – system calls and signals tracing tool

man 2 ptrace – process trace system call

/sys/kernel/debug/ – debugfs

dmesg --console-level <level>

gdb /usr/src/linux/vmlinux /proc/kcore

/proc/self/stack

dynamic_doc debug

⌨️ hands-on:

echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control

⚙️ Internals

handle_sysrq_id

📚 References

Development tools for the kernel_doc

DebugFS_doc, samples/qmi/qmi_sample_client.c_src

Kprobe-based Event Tracing_doc

Dynamic debug_doc

Linux Magic System Request Key Hacks_doc

Magic SysRq key

Tracing and logging edit

⚲ API:

User-space interface:

man 1 dmesg – prints or control the kernel ring buffer

man 2 syslog – system call, which is used to control the kernel printk() buffer

/proc/kmsg

Most common functions

linux/printk.h_inc

pr_devel_id- conditional debug-level message

pr_debug_id- conditional debug-level or dynamic_doc message

⌨️ hands-on:

echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control

Log messages with other levels:

pr_info_id, pr_notice_id, pr_warn_id, pr_err_id, pr_crit_id, pr_alert_id, pr_emerg_id

asm-generic/bug.h_inc

⚙️ Internals

kernel/printk/printk.c_src

arch/x86/kernel/traps.c_src

lib/dump_stack.c_src

kernel/trace_src

scripts/tracing/draw_functrace.py_src

logging_ltp, tracing_ltp

samples/ftrace_src

samples/trace_events_src

samples/trace_printk_src

linux/instrumentation.h_inc

📚 References:

Debugging by printing

Message logging with printk_doc

man 1 stap – systemtap script translator/driver

man 1 strace – trace system calls and signals

Linux Tracing Technologies_doc

Tracepoint Analysis_doc

Function Tracer_doc – function, latency and event tracing

Event Tracing_doc

Using ftrace to hook to functions_doc

Fprobe - Function entry/exit probe_doc

Kprobe-based Event Tracing_doc

Uprobe-tracer: Uprobe-based Event Tracing_doc

Using the Linux Kernel Tracepoints_doc

Subsystem Trace Points: kmem_doc

Subsystem Trace Points: power_doc

NMI Trace Events_doc

In-kernel memory-mapped I/O tracing_doc

Event Histograms_doc

Histogram Design Notes_doc

Boot-time tracing_doc

Hardware Latency Detector_doc

Intel(R) Trace Hub (TH)_doc

Lockless Ring Buffer Design_doc

System Trace Module_doc

CoreSight - ARM Hardware Trace_doc

🔧 TODO. 🚀 advanced features

linux/kmemleak.h_inc – memory leak detector

pr_cont_id- continues a previous log message in the same line

print_hex_dump_bytes_id

print_hex_dump_debug_id

CONFIG_PRINTK_CALLER_id

CONFIG_DEBUG_KERNEL_id

CONFIG_DEBUG_INFO_id

kgdb and kdb edit

⚲ Interfaces

linux/kgdb.h_inc

linux/kdb.h_inc

⚙️ Internals

kernel/debug_src

📚 References

Using kgdb, kdb and the kernel debugger internals_doc

man 8 crash – Analyze Linux crash dump data or a live system

eBPF edit

Retrieved from "https://en.wikibooks.org/w/index.php?title=The_Linux_Kernel/Debugging&oldid=4370759"