Tokens, Lexemes, and Patterns
You'll learn the three fundamental building blocks that lexical analyzers use to transform source code into structured units.
The Core Idea
When a compiler reads your source code, it doesn't process raw text—it first breaks it into meaningful chunks. Think of reading a sentence: you don't process individual letters; you recognize words and punctuation. Similarly, a lexical analyzer recognizes three related concepts:
- Tokens are the categories or types of elements (like "identifier," "number," or "operator")
- Lexemes are the actual strings found in the source code (like
count,42, or+) - Patterns are the rules that define what strings belong to each token category
A Concrete Analogy
Imagine sorting mail:
- Token = the category (letter, postcard, package)
- Lexeme = the actual item in your hand
- Pattern = the rules to classify it ("if it's thin and rectangular with a stamp, it's a letter")
Example from Code
Consider this line: int x = 10;
- Lexeme
int→ Token KEYWORD (pattern: reserved word list) - Lexeme
x→ Token IDENTIFIER (pattern: letter followed by letters/digits) - Lexeme
=→ Token ASSIGNMENT (pattern: exactly the character=) - Lexeme
10→ Token NUMBER (pattern: sequence of digits) - Lexeme
;→ Token SEMICOLON (pattern: exactly the character;)
The pattern is often expressed as a regular expression that matches valid lexemes for that token type.
Key Takeaway: Tokens classify what kind of element you have, lexemes are the concrete text instances, and patterns are the recognition rules—together they form the foundation of how compilers understand source code structure.