Course contentsShow
← Machine Learning and Deep Learning
Lesson 1178 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

Handling Long Documents

Use sliding window, hierarchical, or truncation strategies when input exceeds BERT's 512 token limit.

Handling Long Documents

You'll learn: How to work with documents longer than BERT's 512-token limit using practical chunking strategies.

The 512-Token Wall

BERT models have a hard maximum input length of 512 tokens due to their fixed positional embeddings and computational constraints. But what happens when you need to classify a 2,000-word research paper or extract information from a long legal document? You hit a wall — literally.

Three Core Strategies

1. Truncation (The Simple Cut)

Keep only the first 512 tokens and discard the rest. Think of reading only a book's first chapter to guess its genre. Works surprisingly well when critical information appears early (like news articles), but risks missing important context.

2. Sliding Window (The Overlapping Scan)

Break the document into overlapping 512-token chunks, process each separately, then aggregate results. Imagine reading a book through a moving window, with each view slightly overlapping the previous one. You might average predictions from all windows or take the maximum confidence score.

3. Hierarchical Approach (The Two-Pass System)

First, encode chunks independently to get representations, then use another layer to combine these chunk-level embeddings. Like writing chapter summaries first, then synthesizing them into a book report. More sophisticated but requires additional architecture.

Choosing Your Strategy

  • Truncation: Fast baseline when key info is at the start
  • Sliding window: Better coverage, moderate complexity
  • Hierarchical: Most powerful but computationally expensive

Some practitioners combine strategies — for example, taking the first 256 and last 256 tokens to capture both introduction and conclusion.

Key Takeaway: BERT's 512-token limit requires chunking strategies — truncation for speed, sliding windows for coverage, or hierarchical processing for sophisticated long-document understanding.