Error Detection and Recovery in Lexing
You'll learn how a lexer identifies invalid input and strategically recovers to continue finding valid tokens instead of crashing on the first error.
The Challenge: Bad Input Happens
Even with perfect regular expressions and finite automata, real-world source code contains typos and illegal character sequences. A basic lexer might halt at the first @ symbol in a language that doesn't allow it, or crash on 123abc where identifiers can't start with digits. This isn't helpful—compilers should report all errors in one pass, not force programmers to play "fix one error at a time."
Detecting Invalid Sequences
Your lexer's finite automaton has no valid transitions for certain inputs from certain states. When the DFA can't consume the next character and hasn't reached an accepting state, you've detected a lexical error. The current lexeme being built doesn't match any token pattern.
Panic-Mode Recovery
The most common recovery strategy is panic mode: when an error occurs, the lexer discards characters until it finds a safe "synchronization point" to resume. Typical synchronization points include:
- Whitespace: Skip to the next space or newline
- Statement delimiters: Skip until
;or} - Next recognizable token: Discard characters until the automaton can successfully transition again
Think of it like reading garbled text: when you hit gibberish, you skip ahead to the next recognizable word instead of giving up entirely.
The Recovery Process
- Detect error (invalid character or no valid transition)
- Report error with position information
- Enter recovery mode: discard characters
- Stop at synchronization point
- Resume normal lexical analysis
- Continue finding tokens and errors
This allows one compiler run to report multiple lexical errors, dramatically improving the development experience.
Key Takeaway: Error detection identifies invalid character sequences when the automaton has no valid transitions; panic-mode recovery skips to synchronization points (whitespace, delimiters) to continue lexing and report multiple errors in one pass.