Token Classification
What you'll learn: How lexical analyzers group tokens into distinct categories—keywords, identifiers, literals, operators, and punctuation—so the parser can understand the structure of source code.
The Core Idea
Once a lexical analyzer (lexer) has broken source code into tokens, it must classify each token into a specific category. Think of this like sorting mail: every piece goes into a bin (junk, bills, personal letters) based on its characteristics. Similarly, every token belongs to exactly one category that defines its role in the program.
The Five Main Token Categories
1. Keywords
Reserved words with special meaning in the language. They cannot be used as variable names.
Examples: if, while, class, return, void
2. Identifiers
Names chosen by the programmer for variables, functions, classes, etc.
Examples: userName, calculateTotal, myArray
3. Literals
Actual constant values appearing directly in code.
Examples: 42 (integer), 3.14 (float), "hello" (string), true (boolean)
4. Operators
Symbols that perform operations on operands.
Examples: +, -, *, ==, &&, =
5. Punctuation (Delimiters)
Symbols that structure the code but don't perform operations.
Examples: {, }, (, ), ;, ,
Real-World Example
Consider this Java snippet: int count = 42;
int→ Keyword (declares integer type)count→ Identifier (variable name)=→ Operator (assignment)42→ Literal (integer constant);→ Punctuation (statement terminator)
Each category tells the parser what grammatical role the token plays in the language's syntax rules.
Key Takeaway: Token classification organizes tokens into keywords, identifiers, literals, operators, and punctuation, enabling the parser to validate syntax and build meaningful program structures.