Course contentsShow
← Machine Learning and Deep Learning
Lesson 1114 of 3,53824. The Transformer ArchitectureFree lesson

Fixed Computation per Layer

Understanding why transformer layers have predictable compute versus RNN state-dependent complexity.

Fixed Computation per Layer

What you'll learn: Why each transformer layer performs the same amount of computation regardless of sequence length, unlike RNNs where computation depends on internal state complexity.

The Core Idea

In a transformer, every layer performs a fixed set of operations determined only by the model's architecture (dimensions, number of heads, etc.). Once you define your model with d_model=512 and num_heads=8, every token at every position goes through exactly the same computational pipeline: multi-head attention, layer norm, feedforward network, another layer norm.

Contrast this with RNNs and LSTMs: their computation depends on the hidden state size and gating complexity. Each timestep involves matrix multiplications whose cost varies with how information flows through gates. The LSTM's forget, input, and output gates create state-dependent branching in computation.

Why This Matters

Predictability for hardware: GPUs and TPUs can optimize transformer layers aggressively because the operation count is known at compile time. You can pre-allocate memory and schedule parallel operations perfectly.

RNN unpredictability: An RNN's computation varies subtly based on gate activations—while the parameter count is fixed, the effective "work" done by gates can differ between sequences, making hardware optimization harder.

Simple Analogy

Think of a transformer layer as an assembly line where every car gets exactly 10 bolts tightened, no matter what. An RNN is like a quality control station where some cars need 5 checks, others need 15, depending on what the previous station flagged—same inspector, but unpredictable workload.

Key Takeaway: Transformer layers have deterministic, fixed computation per token determined solely by model dimensions, enabling superior hardware utilization compared to RNNs where state-dependent operations create variable computational complexity.