Serial Programming/IP Over Serial Connections

General

Prerequisites

This page assumes the reader is familiar with the TCP/IP protocol suite and IP communication in general.

A Note on Terminology

This module uses the normal TCP/IP terminology, where the terms indicate the following:

TCP: The TCP protocol as such, at the transport layer.
IP: The IP protocol as such, at the internetworking layer.
TCP/IP: The complete internet protocol suite, encompassing many more protocols than just TCP or IP.

Also, this module uses the TCP/IP notation of protocol layers, not the OSI model, unless stated otherwise:

+--------------------+
| Application Layer  |  e.g. SMTP
+--------------------+
| Transport Layer    |  e.g. TCP
+--------------------+
| Internetwork Layer |  e.g. IP
+--------------------+
| Link Layer         |  e.g. SLIP or PPP
+--------------------+

The Problem

General

Transmitting IP data over a serial (RS-232) line is haunted by several problems, among them:

lack of framing
lack of flow control
lack of session management

Of which lack of framing is the most serious one. All mentioned problems are explained as it follows.

Lack of Framing

Different layers have different tasks in the TCP/IP protocol stack. Higher layers expect particular services from their lower layers. One of the services the IP protocol expects from the link layer is the framing, deframing of IP packets. This means, the link layer is, among other tasks, responsible for properly separating each packet and identifying an IP packet's start and end when it moves the packet over some communication link. The link layer is supposed to do this, so when packets are placed in some data stream they can be properly extracted from the stream.

In cases where a 'naked' serial connection is used as the link layer, this creates a problem, because a simple serial connection does not provide any framing facilities. Instead, the data is just a stream. The receiving side would not be able to explicitly identify the start and end of a packet.

In a 100% reliable serial connection this would be tolerable, because length information in the IP packets could be used to separate packets from the serial data stream. However, a connection is usually not 100% reliable, particularly not if it is in fact done via a modem over a public telephone line. If a packet would be corrupted, partly lost, etc. the receiver would run out of sync, and the transmission completely garbled. In case a broken IP packet could be reliably identified, a higher layer protocol like TCP can handle the problem and initiate a retransmission. Therefore, proper identification, and thus framing/deframing of IP packets is essential.

Lack of Flow Control

Due to the nature of serial connections it is desirable to exercise flow control over the serial connection hop. However, IP does not provide any facility for controlling the flow on the link layer, and a protocol like TCP uses a much different way for end-to-end flow control, and does not deal with a single hop at all.

Lack of Session Management

It is typically necessary to do some additional 'housekeeping' when running IP over a serial line, particularly when that serial line is backed up by a modem connection. This includes setting up such a modem connection (dialing into some remote site), and authenticating oneself at that remote site. IP as such does not support these tasks.

Solutions

General

The solution to the above mentioned problems is to introduce an additional protocol at the link layer. That protocol in fact represents the link layer for the higher layer protocols and uses the serial connection internally.

Two such protocols are in common usage. A very simple one, called SLIP, and a more complex, but much more powerful one, called PPP. New services typically use PPP, while SLIP can still some times be found with older services.

+-------------+
| TCP         |
+-------------+
| IP          |
+-------------+---------------+
| Link Layer  | Modem Control |
+-------------+---------------+
| Serial Line                 |
+-----------------------------+
| Modem                       |
+-----------------------------+
| Phone Line                  |
+-----------------------------+

SLIP

SLIP fixes the most serious problem for sending IP datagrams over a serial line: Lack of framing. Flow control handling is left to the modem transmission protocols and higher layer TCP/IP protocols. Session management is supposed to be handled outside of SLIP, e.g. set up ("dial-in") of a modem connection is not done by SLIP. SLIP assumes the serial connection has already been established.

SLIP is standardized in RFC 1055, A Nonstandard for transmission of IP datagrams over serial lines: SLIP. The term Nonstandard in the title is there because of historical reasons. SLIP wasn't even described for a long time, and when it was finally described in RFC 1055 that description was only intended as an informal summary of the current practice. Only later it was decided to give that RFC the status of an official IETF standard - but one which should not be the first choice - because of SLIP's limitations.

SLIP is so simple, it is almost embarrassing. It requires a serial port configuration of 8 data bits, no parity, and either hardware flow control, or CLOCAL mode (3-wire null-modem). It defines only four special 8-bit sequences (characters) and little functionality over the standard serial line protocol.

Hex value	Octal value	Abbreviation	Description
0xC0	0300	END	Frame End
0xDB	0333	ESC	Escape
0xDC	0334	ESC_END	Transposed Frame End, when used in an escape sequence
0xDD	0335	ESC_ESC	Transposed Escape, when used in an escape sequence

When the SLIP driver receives an IP packet from the interworking layer, it simply copies it octet by octet (with a slight exception, see below) and appends an additional octet with the value 0xC0 (hex). This additional octet marks the end of an IP packet, and is therefore called the END character.

To avoid random line noise being interpreted as the beginning of a packet, it is customary to prepend the END character to an IP packet if packets are not continuously and with full speed send over the line and transmission is resumed. This way possible random noise is itself framed as a packet, and the higher layers (particularly the IP layer) will discard such 'packets', because they will be identified as illegal (wrong checksum, wrong size, below minimum size).

Frames when sending continuous

+-----------------+-----------------+-----------------+
| IP datagram |END| IP datagram |END| IP datagram |END| ...
+-----------------+-----------------+-----------------+

Frames when sending with delay between packages, prepending additional END characters to exclude line noise

+-----------------+            +----+-----------------+            +----+-----------------+
| IP datagram |END| line noise |END | IP datagram |END| line noise |END | IP datagram |END| ...
+-----------------+            +----+-----------------+            +----+-----------------+

The END characters marked in red in the above figure are END characters prepended to the IP datagrams which follow them. But they effectively act as end marks for the preceding line noise, separating potential random line noise from real data.

The framing with the 0xC0 END character creates a slight problem. If the character itself appears in an IP packet, SLIP would wrongly interpret this as the end of the packet. This is fixed by introducing another special character, the ESC character with the value 0xDB (hex). This is used to build two escape sequences:

0xDB 0xDC: Replacement for 0xC0 values within an IP datagram
This ensures there will never be an END character in the transmitted datagram
0xDB 0xDD: Replacement for 0xDB values within an IP datagram
This is the 'escape of the escape' character.

SLIP inserts these escape sequences into the IP datagrams it receives for transmission from the interworking layer, and it replaces the escape sequences with the original values when forwarding a received datagram to the interworking layer.

That's all that SLIP does. There is no error detection, compression or device (modem) control. There is also no support for host configuration, security, session management, and other control functions. Therefore SLIP on its own is not satisfactory over an error-prone dial-up connection, although it operates reliably over MNP4 (or higher) error correction modem links.

PPP

PPP, the Point-to-Point Protocol, provides a standard method for transporting datagrams over point-to-point links, including standard serial lines. The specification of PPP defines the following three main components:

A method for encapsulating datagrams in PPP frames. It supports IP datagrams as well as datagrams of other networking protocols.
A Link Control Protocol (LCP). This protocol defines procedures for establishing, configuring, and testing the datalink connection.
A family of Network Control Protocols (NCPs) for establishing and configuring different network-layer protocols.

PPP is a rather complex protocol, consisting of mandatory and optional parts, as well as extensions. A likely incomplete list of PPP-related RFC standards is given below:

RFC 1661 - The Point-to-Point Protocol (PPP)
RFC 1332 - The PPP Internet Protocol Control Protocol (IPCP)
RFC 1334 - PPP Authentication Protocols
RFC 1377 - The PPP OSI Network Layer Control Protocol (OSINLCP)
RFC 1378 - The PPP AppleTalk Control Protocol (ATCP)
RFC 1552 - The PPP Interworking Packet Exchange Control Protocol (IPXCP)
RFC 1570 - PPP LCP Extensions
RFC 1618 - PPP over ISDN
RFC 1662 - PPP in HDLC-like Framing
RFC 1962 - The PPP Compression Control Protocol (CCP)
RFC 1968 - The PPP Encryption Control Protocol (ECP)
RFC 1973 - PPP in Frame Relay
RFC 1989 - PPP Link Quality Monitoring
RFC 1990 - The PPP Multilink Protocol (ML)
RFC 1994 - PPP Challenge Handshake Authentication Protocol (CHAP)
RFC 2043 - The PPP SNA Control Protocol (SNACP)
RFC 2097 - The PPP NetBIOS Frames Control Protocol (NBFCP)
RFC 2125 - The PPP Bandwidth Allocation Protocol (BAP) / The PPP Bandwidth Allocation Control Protocol (BACP)
RFC 2290 - Mobile-IPv4 Configuration Option for PPP IPCP
RFC 2364 - PPP over AAL5
RFC 2472 - IP Version 6 over PPP
RFC 2516 - A Method for Transmitting PPP over Ethernet (PPPoE)
RFC 2615 - PPP over SONET / SDH

To do:
Somehow mark the RFCs which are relevant for serial line communication. Possibly don't mention the others at all.

In short, implementing PPP is not for the faint of heart. It is a serious job, requiring solid programming, protocol and telecommunication knowledge - even if only the parts relevant for serial line communication are taken into account. Programmers wishing to implement PPP should already reserve weeks to study the relevant standards. It is also a good idea to define clear requirements which optional PPP parts need to be implemented. It should also be considered obtaining a compatibility test suite to verify standards compliance of the own implementation. Simple trial-and-error with some existing consumer implementation of the protocol usually doesn't cut it. This can only 'guarantee' to a certain extent that the PPP implementation is compatible with that particular other implementation in a particular configuration. It does not guarantee that the implementation is compatible with a wide range of existing PPP implementations.

If it is a viable option to buy a PPP implementation for a particular platform instead of doing an own implementation, then that option should seriously be considered.

HDLC

HDLC (High-Level Data Link Control) is another Layer 2 protocol which can be used to carry IP traffic over serial (and other) connections. HDLC predates PPP and is a generic Layer 2 protocol. It can be found in industrial applications and is also popular for serial point-to-point links between IP routers. It is less common in end-user applications, like dialup connections to an ISP. The Wikipedia article about HDLC provides a good overview about HDLC, and provides references to the relevant standards.

HDLC is mainly concerned with framing, and provides a few additional control messages (procedures). It is relatively easy to implement over serial lines.

Serial Programming: Introduction and OSI Network Model -- RS-232 Wiring and Connections -- Typical RS232 Hardware Configuration -- 8250 UART -- DOS -- MAX232 Driver/Receiver Family -- TAPI Communications In Windows -- Linux and Unix -- Java -- Hayes-compatible Modems and AT Commands -- Universal Serial Bus (USB) -- Forming Data Packets -- Error Correction Methods -- Two Way Communication -- Packet Recovery Methods -- Serial Data Networks -- Practical Application Development -- IP Over Serial Connections