Tokenizer Encoding and Decoding APIs
You'll learn: How to use tokenizer methods to convert text to tokens and back, including the metadata models need for attention.
What Encoding and Decoding Do
Once you've chosen or trained a tokenizer, you need practical ways to convert text into tokens (encoding) and tokens back into text (decoding). Modern tokenizer libraries provide clean APIs that handle these operations and generate additional model inputs.
Think of encoding like translating a sentence into a numbered list your model can read, and decoding like translating those numbers back into human-readable text.
Core Methods
encode(text) converts a single text string into a list of token IDs:
token_ids = tokenizer.encode("Hello world")
# Returns: [101, 7592, 2088, 102]
decode(token_ids) reverses this process:
text = tokenizer.decode([101, 7592, 2088, 102])
# Returns: "Hello world"
batch_encode(texts) efficiently processes multiple texts at once, crucial for training batches.
Tokenizer Output Formats
When you use methods like encode_plus or __call__, tokenizers return dictionaries with:
input_ids: The token ID sequencesattention_mask: Binary flags (1s and 0s) indicating which tokens are real vs. padding (remember PAD tokens from lesson 1242)token_type_ids: Distinguishes between different segments (like question vs. context in BERT)
These extra fields tell the model which tokens to pay attention to and how to treat different parts of the input.
Key Takeaway: Tokenizer APIs provide encode/decode for basic conversion, with extended methods returning attention masks and token type IDs that models require for proper processing of padded and multi-segment inputs.