Course contentsShow
← AI Engineering
Lesson 52 of 1,8862. Working with Pre-trained ModelsFree lesson

Tokenizers: Encoding and Decoding

Understand how tokenizers convert text to input IDs and back, including special tokens and padding.

Tokenizers: Encoding and Decoding

What you'll learn: How tokenizers translate between human-readable text and the numerical IDs that models actually process.

The Translation Layer

Think of a tokenizer as a bilingual interpreter between you and the model. Models can't read text directly—they only understand numbers. The tokenizer encodes (converts) your text into numerical IDs, the model processes those numbers, and then the tokenizer decodes (converts back) the result into readable text.

Encoding: Text to Numbers

When you encode text, the tokenizer:

  1. Splits your text into tokens (words, subwords, or characters)
  2. Looks up each token in its vocabulary
  3. Returns the corresponding numerical ID for each token
  4. Adds special tokens like [CLS] (start), [SEP] (separator), or [PAD] (padding)

For example, "Hello world" might become [101, 7592, 2088, 102] where 101 is [CLS] and 102 is [SEP].

Decoding: Numbers to Text

Decoding reverses the process. Given [7592, 2088], the tokenizer looks up those IDs and reconstructs "Hello world". It typically removes special tokens automatically so you get clean output.

Special Tokens and Padding

Special tokens are markers the model needs to understand structure. Different models use different special tokens (BERT uses [CLS], GPT uses <|endoftext|>).

Padding ensures all sequences in a batch have the same length. Short sequences get padding tokens ([PAD]) added until they match the longest sequence in the batch.

Why This Matters

You'll use the same tokenizer for both encoding inputs before the model and decoding outputs after. Mismatched tokenizers cause gibberish output—always use the tokenizer that matches your model.

Key Takeaway: Tokenizers are the essential bridge between text and models, converting text to input IDs (encoding) and model outputs back to text (decoding), while managing special tokens and padding for proper model input.