Course contentsShow
← Computer Science
Lesson 1940 of 2,87244. Lexical Analysis and ParsingFree lesson

Token Classification

Categorizing tokens into keywords, identifiers, literals, operators, and punctuation with examples from real languages.

Token Classification

What you'll learn: How lexical analyzers group tokens into distinct categories—keywords, identifiers, literals, operators, and punctuation—so the parser can understand the structure of source code.

The Core Idea

Once a lexical analyzer (lexer) has broken source code into tokens, it must classify each token into a specific category. Think of this like sorting mail: every piece goes into a bin (junk, bills, personal letters) based on its characteristics. Similarly, every token belongs to exactly one category that defines its role in the program.

The Five Main Token Categories

1. Keywords

Reserved words with special meaning in the language. They cannot be used as variable names.

Examples: if, while, class, return, void

2. Identifiers

Names chosen by the programmer for variables, functions, classes, etc.

Examples: userName, calculateTotal, myArray

3. Literals

Actual constant values appearing directly in code.

Examples: 42 (integer), 3.14 (float), "hello" (string), true (boolean)

4. Operators

Symbols that perform operations on operands.

Examples: +, -, *, ==, &&, =

5. Punctuation (Delimiters)

Symbols that structure the code but don't perform operations.

Examples: {, }, (, ), ;, ,

Real-World Example

Consider this Java snippet: int count = 42;

  • int → Keyword (declares integer type)
  • count → Identifier (variable name)
  • = → Operator (assignment)
  • 42 → Literal (integer constant)
  • ; → Punctuation (statement terminator)

Each category tells the parser what grammatical role the token plays in the language's syntax rules.

Key Takeaway: Token classification organizes tokens into keywords, identifiers, literals, operators, and punctuation, enabling the parser to validate syntax and build meaningful program structures.