C Programming/Serialization

Serialization

It is often necessary to send or receive complex data structures to or from another program that may run on a different architecture or may have been designed for different version of the data structures in question. A typical example is a program that saves its state to a file on exit and then reads it back when started.

The 'send' function will typically start by writing a magic identifier and version to the file or network socket and then proceed to write all the data members one by one (i.e. in serial). If variable length arrays are encountered (e.g. strings), it will either write a length followed by the data or it will write the data followed by a special terminator. The format is often XML or binary; in the latter case the htonl() set of macros may come in handy.

The 'receive' function will be nearly identical: it will read all the items one by one. Variable length arrays are either handled by reading the count followed by the data, or by reading the data until the special terminator is reached.

Since these two functions often follow the same pattern as the declaration of the data(structures), it would be nice if they could all be generated from a common definition.

X-Macros

X-Macros uses the preprocessor to force the compiler to compile the same piece of text more than once. Sometimes a special file (with extension .def) is included multiple times. For example variables.def may look like this :

INT(value)
INT(shift)

In this example the C programming will then look like this :

...
#define INT(var) int var;
#include "variables.def"
#undef INT
...
printf ("version=1\n");
#define INT(var) printf (#var "=%d\n", var);
#include "variables.def"
#undef INT
...

If including a separate file multiple times is undesirable, another macro can be used. For example :

#define VARIABLES INT(value) \
                  INT(shift)

The #includes can then be replaced with calls to the macro.

Using this method, one can also pass in the name(s) of (an)other macro(s) that can operate on the list of values. For example:

#define VAR_LIST(_) _(value) \
                    _(shift)
...
#define VAR_INT_DECL(var) int var;
VAR_LIST(VAR_INT_DECL)
...
printf ("version=1\n");
#define VAR_INT_PRINTF(var) printf (#var "=%d\n", var);
VAR_LIST(VAR_INT_PRINTF)
...

This does not require the redefinition of macros and can make the code easier to understand and maintain.

X-Macros are also particularly useful for keeping mappings between strings and enumerated types synchronized.

Serialization with versioning

Suppose we want to add additional variables to the above example, but we still want the program to be able to read the old version 1 files. Then we would add a version parameter and a default value parameter to the list processing macros:

#define VAR_LIST(_) _(value,1,0) \
                    _(shift,1,0) \
                    _(mask,2,0xffff)
...
int inputVer;
#define VAR_INT_DECL(var,varVer,default) int var;
VAR_LIST(VAR_INT_DECL)
...
scanf ("version=%d", &inputVer);
#define VAR_INT_SCN(var,varVer,default) if (varVer <= inputVer) scanf (#var "=%d", &var); else var = default;
VAR_LIST(VAR_INT_SCN)
...
printf ("version=2\n"); /* Always output at highest known version */
#define VAR_INT_PRT(var,varVer,default) printf (#var "=%d\n", var);
VAR_LIST(VAR_INT_PRT)
...