Course contentsShow
← Computer Science
Lesson 1939 of 2,87244. Lexical Analysis and ParsingFree lesson

Tokens, Lexemes, and Patterns

Defining the fundamental concepts: tokens as categories, lexemes as actual strings, and patterns as token rules.

Tokens, Lexemes, and Patterns

You'll learn the three fundamental building blocks that lexical analyzers use to transform source code into structured units.

The Core Idea

When a compiler reads your source code, it doesn't process raw text—it first breaks it into meaningful chunks. Think of reading a sentence: you don't process individual letters; you recognize words and punctuation. Similarly, a lexical analyzer recognizes three related concepts:

  • Tokens are the categories or types of elements (like "identifier," "number," or "operator")
  • Lexemes are the actual strings found in the source code (like count, 42, or +)
  • Patterns are the rules that define what strings belong to each token category

A Concrete Analogy

Imagine sorting mail:

  • Token = the category (letter, postcard, package)
  • Lexeme = the actual item in your hand
  • Pattern = the rules to classify it ("if it's thin and rectangular with a stamp, it's a letter")

Example from Code

Consider this line: int x = 10;

  • Lexeme int → Token KEYWORD (pattern: reserved word list)
  • Lexeme x → Token IDENTIFIER (pattern: letter followed by letters/digits)
  • Lexeme = → Token ASSIGNMENT (pattern: exactly the character =)
  • Lexeme 10 → Token NUMBER (pattern: sequence of digits)
  • Lexeme ; → Token SEMICOLON (pattern: exactly the character ;)

The pattern is often expressed as a regular expression that matches valid lexemes for that token type.

Key Takeaway: Tokens classify what kind of element you have, lexemes are the concrete text instances, and patterns are the recognition rules—together they form the foundation of how compilers understand source code structure.