Wikibooks

C++ Language/Preprocessing/Tokenization

< C++ Language‎ | Preprocessing
C++ Language
  1. Overview
  2. Preprocessing
    1. Macros
    2. Tokenization
    3. Preprocessing Strings
    4. Compile-time Constants
    5. Compile-time Assertions
    6. Conditional-Compilation
  3. Namespace
    1. Namespace Shortcuts
    2. Nesting
    3. Splitting across Files
    4. Named and Unnamed
  4. Type
    1. Hungarian Prefixes
    2. Boolean
    3. Literals
    4. Bitwise Representation
    5. Max Value
    6. Conversion
    7. Enumeration
    8. Shortcuts
    9. Variant
    10. RTTI: RunTime Type Information
  5. Variables
    1. Lifetime and Scope
    2. Initialization
    3. Uniform Initialization
    4. Initialize without Constructor
    5. Decomposing-Auto
    6. Define at Same Time as Type
  6. Indirection
    1. Pointers
    2. Typed NULL
    3. Arrays
    4. Reference Variables
    5. Rvalue-References
    6. Smart Pointers
  7. Expressions
    1. Precedence
    2. Chain of Assignment
  8. Program Flow
    1. Loops
    2. The break Statement
    3. Return Value
    4. Parameters
    5. Function Overloading
    6. Function-Pointer
    7. Deprecating a Function
  9. Objects
    1. Accessibility
    2. Unwanted Compiler-Generated Functions
    3. Constructors
    4. Destructors
    5. Const Members
    6. Static Members
    7. Inheritance
    8. Operator Overloading
    9. Nested Classes
    10. Pointer to Members
    11. Low-Level C Structures
  10. Templates
    1. Templated-Functions
    2. Templated-Classes
    3. Templated-Variables
    4. Template-Parameters
    5. Traits
    6. Metaprogramming
    7. Concepts
  11. Error Handling
    1. Returning an Error Code
    2. Assertions
    3. C++ Exception Handling
  12. Standard Libraries
    1. C-Runtime
    2. STL: Standard Template Library
    3. Strings
    4. IOStreams
    5. Filesystem Access
    6. Multithreading
    7. RNG: Random Number Generator
  13. Obsolete Features

When your C++ source file is being compiled, one of the first things that happens is your compiler uses its preprocessor to "tokenize" your program.

Tokenization would process code like iA+++iB by splitting it into four tokens (variable identifier iA, then operator ++, then operator +, then variable identifier iB).

A preprocessor macro is able to use ## within its definition, to construct a compilation-token while the macro expands.

Additional information about tokenization (includes interactive examples)

Retrieved from "https://en.wikibooks.org/w/index.php?title=C%2B%2B_Language/Preprocessing/Tokenization&oldid=4034950"