Model Inputs and Attention Masks
You'll learn: How to properly format inputs for model forward passes using input_ids, attention_mask, and token_type_ids.
What the Model Actually Receives
When you tokenize text, you're not quite done preparing data for the model. The tokenizer returns a dictionary with special keys that tell the model exactly what to process and what to ignore.
The Three Key Components
input_ids
These are the actual token numbers representing your text. Think of them as the "words" the model reads—but in number form. Every token from your vocabulary gets converted to its unique ID.
attention_mask
This binary array (0s and 1s) tells the model which tokens to pay attention to. Why? Because sequences get padded to equal lengths in batches. If one sentence is 10 tokens and another is 15, you pad the shorter one with special padding tokens. The attention_mask marks real tokens as 1 and padding as 0, so the model ignores the filler.
Analogy: Imagine reading a form where some fields are filled and others say "N/A"—the attention mask tells you which fields actually matter.
token_type_ids
Used when you have two sequences (like question + context). This marks which tokens belong to sentence A (all 0s) versus sentence B (all 1s). Many single-sequence tasks don't need this.
Why This Matters
Models expect these exact inputs. When you call tokenizer(text, return_tensors="pt"), it automatically generates all three. Understanding them helps you debug shape mismatches and batch processing issues.
Key Takeaway: The model needs input_ids (what to read), attention_mask (what to focus on), and sometimes token_type_ids (which sentence is which)—all provided automatically by your tokenizer.