The Procedural Macro Pipeline

In this lesson, you will learn how to orchestrate the transformation of Rust source code into AST nodes by traversing the multi-stage procedural macro pipeline.

From Tokens to Abstract Syntax Trees

While macro_rules! operates on pattern matching at the token level, procedural macros allow us to hook directly into the compiler’s internal processing stages. The "pipeline" is essentially a series of hand-offs where raw text is refined into structured logic.

The pipeline begins when the compiler encounters a procedural macro invocation (like a custom derive or a function-like macro). At this moment, the compiler freezes compilation and hands control to your macro crate. The process generally follows these three transitions:

1. TokenStream Intake: The input arrives as a TokenStream. This represents the raw, unparsed surface syntax of your code.
2. Parsing into AST: You use a parser (typically via the syn crate) to map these tokens into an Abstract Syntax Tree. This creates a semantic representation of your code—identifying structs, enums, fields, and attributes as distinct, inspectable nodes rather than just symbols.
3. Code Generation: Once you have manipulated or analyzed these AST nodes, you emit a new TokenStream. This output is then injected back into the compiler’s pipeline as if you had hand-written the code yourself.

A Conceptual Illustration

Imagine you are writing a decorator that generates a new method for a struct. The pipeline looks like this:

// 1. Intake: TokenStream (e.g., struct User { id: u32 })
// 2. Parsing: Syn::parse_macro_input!() turns tokens into a DeriveInput struct
// 3. Transformation: You analyze the fields and construct a new TokenStream
// 4. Output: The compiler splices your generated code back into the crate

By interacting with the pipeline here, you aren't just doing string substitution; you are performing structural engineering on the code before the compiler performs type-checking or borrow-checking. You are providing the compiler with "pre-compiled" structural instructions, ensuring that the final binary is generated as if your expanded code were always present in the source file.

Key Takeaway: The procedural macro pipeline transforms raw TokenStreams into semantically rich AST nodes, allowing you to programmatically rewrite the structure of your crate before the final compilation phases.