Using ELMo Embeddings in Downstream Tasks
What you'll learn: How to integrate pretrained ELMo representations into your own models for tasks like text classification and named entity recognition.
The Integration Approach
Now that you understand ELMo produces rich, contextualized embeddings, the key question is: how do you actually use them? Think of ELMo as a sophisticated feature extractor. Just as you might use pretrained image features from a convolutional network, you extract ELMo embeddings and feed them into your task-specific model.
Two Common Integration Patterns
1. Feature-Based Approach
You freeze the pretrained ELMo model and use its outputs as additional input features. For each token in your sequence, ELMo gives you a context-aware representation. You then concatenate these with other features (like your own embeddings) and pass them to your classification or tagging layers.
# Conceptual flow (not runnable)
# 1. Get ELMo embeddings for input text
elmo_embeddings = elmo_model(tokens) # Shape: [batch, seq_len, 1024]
# 2. Combine with your task-specific model
combined = concatenate([elmo_embeddings, other_features])
# 3. Pass to your classifier
output = your_task_model(combined)
2. Layer Weighting
ELMo produces embeddings from multiple bidirectional LSTM layers. You can learn a weighted combination of these layers for your specific task—some tasks benefit more from lower layers (syntax), others from higher layers (semantics).
Why This Works
ELMo's contextual representations capture nuances that static embeddings miss. The word "bank" gets different vectors in "river bank" vs "savings bank," providing your downstream model with much richer information to work with.
Key Takeaway: ELMo embeddings serve as powerful pretrained features that you integrate into task-specific models, either by concatenating them with other inputs or by learning task-optimized combinations of ELMo's internal layers.