Embedded Linux systems can range from consumer electronics like smartphones and smart TVs to industrial machinery, medical devices, automotive systems, communication equipment and more. Linux brings the power, flexibility, and stability to these resource-constrained devices.

Core components of an Linux on embedded systems are boot loader, custom Linux kernel and root file system. Source code is compiled by GCC cross compiler. Common boot loader is Das U-Boot – the Universal Boot Loader. Root FS usually is based on BusyBox – software suite that provides many Unix utilities in a single executable file. Local debugging often is limited, so gdbserver can be used.

Basic embedded Linux system can be compiled manually from modules above. There are tools for complete system generation from old and simple Buildroot to huge Yocto Project, which creates customized embedded Linux distributions. PC distributions Debian, Ubuntu ant others have build for ARM processors.

Raspberry Pi, BeagleBoard and Nvidia Jetson are examples of popular single-board computers supporting embedded Linux.

Unlike regular desktop or server distributions of Linux these versions of Linux usually have much fewer libraries and application, firmware is smaller and typically boot from flash memory or SD card.


Projects

edit

Build frameworks and environments

edit
Wear OS – Android for smartwatches and other wearables.
AOSP – Android Open Source Project
Buildroot – old simple patch and make based build environment
Yocto Project and OpenEmbedded – a BitBake based build framework and environment
BitBake – make-like build tool
OpenWrt – open wireless router
Embeddable Linux Kernel Subset – for ancient 16-bit x86 machines and emulators


Components

edit
The Linux Kernel
"init" implementations: OpenRC, runit
GUIs: LXQt, LXDE, Xfce, FLTK
coreboot – lightweight BIOS


General purpose GNU C Library glibc, 7.9 MB
uClibc, 560 KB
musl, 527 KB
dietlibc, 185 KB
Newlib
klibc – primarily for booting Linux systems
Bionic – originally developed by Google for the Android embedded system operating system


Further reading:

Comparison of C/POSIX standard library implementations for Linux
Choosing System C library

Distributions

edit

Distributions for ARM processors:

Arch Linux ARM – port of Arch Linux for ARM processors
ArmbianDebian and Ubuntu based
RedSleeve – port of Red Hat Enterprise Linux
emteria.OS – proprietary Android based


Lightweight cloud oriented systems:

Alpine Linux – uses musl, BusyBox and OpenRC.
Fedora CoreOS
RHCOS – Red Hat Enterprise Linux CoreOS for OpenShift Container Platform
https://MicroShift.io/ – lightweight Kubernetes distribution designed for edge and IoT computing environments


Lightweight multi-platform distributions:

Gentoo Linux
Tiny Core Linux
Kali Linux
Lubuntu
Slackware
Void Linux
Xubuntu


Common multi-platform distributions:

Debian
Fedora
openSUSE
Ubuntu

Features of Linux kernel for embedded systems

edit
I/O and buses:
GPIO
drivers/gpio src, tools/gpio src, General Purpose Input/Output doc
SPI
drivers/spi src, tools/spi src, Serial Peripheral Interface doc
I2C
⚲ UAPI: Instantiate I2C devices from user-space doc
drivers/i2c src, https://i2c.wiki.kernel.org
I2C/SMBus Subsystem doc
drivers/net/can srcController Area Network bus
drivers/mfd srcMulti-Function Devices
drivers/usb/gadget src – peripheral USB device implementation
drivers/iio src – Industrial I/O
drivers/pwm srcPulse-width modulation
drivers/regulator src – Generic Voltage and Current Regulator support
drivers/fpga src
drivers/slimbus src
SLOB – Simple List Of Blocks memory allocator
Devicetree – describes hardware components for the kernel


Multimedia

edit
Audio
ASoC – ALSA System on Chip
snd_soc_card id
ASoC – ALSA SoC Layer doc
https://www.alsa-project.org/wiki/ASoC
TFT (LCD) – Thin-film-transistor liquid-crystal display
fbtft_display id


File systems and storage

edit
UBIFS – Unsorted Block Image FS
SquashFS – a compressed read-only FS with low overhead
drivers/mtd srcMemory Technology Device block device
More: YAFFS, JFFS2


MMU less

edit

Linux was originally designed on a processor with a memory management unit (MMU). Most embedded systems do not have a MMU, as we discussed earlier (Embedded Systems/Memory) and are called microcontrollers. Microcontrollers without a MMU are cheaper consumes less power.


Benefits of using a processor with a MMU:

can isolate running "untrusted" machine code from running "critical" code, so the "untrusted" code is guaranteed (in the absence of hardware failures) not to interfere with the "critical" code
makes it easier for the OS to present the illusion of virtual memory
can run "normal" Linux (could also run "μClinux", but what's the point?)


μClinux – "MicroController Linux" is a version of the Linux kernel that supports Altera NIOS, ARM, Freescale M68K (including DragonBall, ColdFire, PowerQUICC and others), Hitachi H8, MIPS, and Xilinx MicroBlaze processors.

Disabling parameter CONFIG_MMU enables nommu mode.


Real-time

edit

People use a variety of methods to combine real-time tasks with Linux:

  1. Run the real-time tasks on a dedicated microcontroller; communicate with a (non-real-time) PC that handles non-real-time tasks. This is pretty much your only choice if you need real-time response times below 1 microsecond.
  2. Run the real-time tasks in a "underlying" dedicated real-time operating system; run Linux as a "nested operating system" inside one low-priority task on top of the real-time operating system. Some of these systems claim real-time response times below 500 microseconds.
  3. use a Linux kernel designed to emphasize real-time tasks, and run the real-time tasks with a high priority (perhaps even as a kernel thread).

Examples of RT distributions:

Xenomai
RTAI
RTLinux
MontaVista Linux
Red Hat Enterprise Linux for Real Time
Real-time Ubuntu


Several RT features provide real time capabilities:

RT preemption
RT scheduling
CPU partitioning and isolation
Dynticks (Tickless, nohz)


RT preemption

edit

The Linux Foundation's Real-Time Linux (RTL) collaborative project is focused on improving the real-time capabilities of Linux and advancing the adoption of real-time Linux in various industries, including aerospace, automotive, robotics, and telecommunications.

Parameter CONFIG_PREEMPT_RT id enables real-time preemption.

RT scheduling policies

edit

Scheduling policies for RT:

SCHED_FIFO id, SCHED_RR id
implemented in kernel/sched/rt.c src
SCHED_DEADLINE
implemented in kernel/sched/deadline.c src


API:

man 1 chrt – manipulate the real-time attributes of a process
man 2 sched_rr_get_interval – get the SCHED_RR interval for the named process
man 2 sched_setscheduler, sched_getscheduler – set and get scheduling policy/parameters
man 2 sched_get_priority_min, sched_get_priority_max – get static priority range

Testing RT capabilities

edit

The testing process for Real-Time Linux typically involves several key aspects. First and foremost, it is crucial to verify the accuracy and stability of the system's timekeeping mechanisms. Precise time management is fundamental to real-time applications, and any inaccuracies can lead to timing errors and compromise the system's real-time capabilities.

Another essential aspect of testing is evaluating the system's scheduling algorithms. Real-Time Linux employs advanced scheduling policies to prioritize critical tasks and ensure their timely execution. Testing the scheduler involves assessing its ability to allocate resources efficiently, handle task prioritization correctly, and prevent resource contention or priority inversion scenarios.

Furthermore, latency measurement is a critical part of Real-Time Linux testing. Latency refers to the time delay between the occurrence of an event and the system's response to it. In real-time applications, minimizing latency is crucial to achieving timely and predictable behavior. Testing latency involves measuring the time it takes for the system to respond to various stimuli and identifying any sources of delay or unpredictability.

Additionally, stress testing plays a significant role in assessing the system's robustness under heavy workloads. It involves subjecting the Real-Time Linux system to high levels of concurrent activities, intense computational loads, and input/output operations to evaluate its performance, responsiveness, and stability. Stress testing helps identify potential bottlenecks, resource limitations, or issues that might degrade the real-time behavior of the system.


RTLA – The realtime Linux analysis tool:
rtla timerlat doc – CLI for the kernel's timerlat tracer doc
rtla osnoise doc – CLI for the kernel's osnoise tracer doc. Kernel function run_osnoise id measures time with function trace_clock_local id in loop.
rtla hwnoise doc – CLI for the osnoise tracer doc with interrupts disabled
Implementation: tools/tracing/rtla src and kernel/trace/trace_osnoise.c src
Linux scheduling latency debug and analysis
RT-Tests, source
cyclictest
some RT-Tests man pages:
cyclictest – measures man 2 clock_nanosleep or man 2 nanosleep delay
hackbench – scheduler benchmark/stress test
hwlatdetect – CLI for /sys/kernel/tracing/hwlat_detector doc / kernel/trace/trace_hwlat.c src. Kernel function kthread_fn id measures time delays with function trace_clock_local id in loop.
oslat – measures delay with RDTSC in busy loop
RT Tracing Tools with eBPF
realtime ltp


Further reading about real-time Linux:

linux/spinlock_rt.h inc used via linux/spinlock_types.h inc
linux/rtmutex.h inc used via linux/mutex_types.h inc
linux/rwbase_rt.h inc used via linux/rwlock_types.h inc
Introduction to Real-Time Linux: Unleashing Deterministic Computing
Power Management and Scheduling in the Linux Kernel (OSPM)
the Real-Time Linux wiki
CPU partitioning and isolation
Realtime@LWN
linux-stable-rt.git
linux-rt-devel.git
Realtime kernel patchset, Arch Linux
https://www.kernel.org/pub/linux/kernel/projects/rt/ - RT patches for upstream kernel
High Precision Event Timer (HPET)
Demystifying the Real-Time Linux Scheduling Latency
Real-time kernel tuning in RHEL 9
Linux subsystems related to real-time
Linux kernel scheduling and preemption
Interrupts
Deferred works
Non-maskable interrupt handler (NMI)
System management interrupt (SMI)
man 7 sched
latency @ LKML
PREEMPT_RT @ LKML
QA about PREEMP_RT, LPC'23, State of the onion, pdf

💾 Historical

The PREEMPT_RT patch has been partially merged into the mainline Linux kernel, starting from version 5.15. Lazy preemption (CONFIG_PREEMPT_LAZY) remains in the external patchset.

Further readings about Embedded Linux:

Embedded Linux Conference (ELC) @ EOSS 2023
https://elinux.org
Category:Embedded_Linux
Linux kernel interfaces
ARM support
arch/arm src, ARM Architecture doc
https://lwn.net/Kernel/Index/#Architectures-ARM
arch/arm64 src, ARM64 Architecture doc
architecture-specific initialization
Learning the vi Editor