Understanding the TokenStream API

In this lesson, you will learn how to interface with the TokenStream type to bridge the gap between raw compiler input and high-level abstract syntax trees (AST).

From Raw Streams to Meaningful Code

Up to this point, we have primarily dealt with macro_rules!, which handles token manipulation for us behind the scenes using pattern matching. When we move toward procedural macros, we work directly with the proc_macro::TokenStream type. Think of a TokenStream as an opaque, sequential iterator of TokenTree objects. A TokenTree is either a single Token (like a keyword or operator), a Literal (like 42 or "hello"), or a Group (tokens nested inside delimiters like {}, [], or ()).

The TokenStream API is intentionally restrictive; it does not provide an AST. It is a linear view of the source code. To make sense of it, you must treat the stream as a "cursor" that you consume, token by token.

Bridging the Gap

To transform a TokenStream into something actionable, you typically follow a process of parsing:

1. Consumption: You iterate through the stream, grouping tokens into logical units.
2. Structural Validation: You check if the sequence matches your expected domain-specific syntax.
3. AST Transformation: You map those validated token groups into your own internal data structures (structs or enums representing the AST).

For example, if your macro receives my_macro!(struct User { id: u32 }), the TokenStream contains the tokens for struct, User, {, id, :, u32, and }. You must manually verify that the Group contains a valid field-type pair.

use proc_macro::TokenStream;

pub fn my_proc_macro(input: TokenStream) -> TokenStream {
    // We treat the input as an iterator to inspect the stream
    for token in input {
        match token {
            proc_macro::TokenTree::Group(g) => { /* Handle code blocks */ }
            proc_macro::TokenTree::Ident(i) => { /* Handle identifiers */ }
            _ => { /* Handle operators or literals */ }
        }
    }
    // Return a new TokenStream as output
    "fn generated_code() {}".parse().unwrap()
}

By manually traversing these tokens, you gain total control over the compilation process, allowing you to interpret source code in ways that standard pattern matching cannot capture.

Key Takeaway: The TokenStream API provides a low-level, sequential interface that requires you to manually parse and group tokens before they can be modeled as a semantic AST.