Dynamic vs Static Computational Graphs
What you'll learn: You'll understand the key difference between frameworks that build computation graphs before execution versus those that build graphs on-the-fly during execution.
The Core Distinction
In deep learning frameworks, computational graphs can be constructed in two fundamentally different ways:
Static Graphs (Define-and-Run) — exemplified by TensorFlow 1.x — require you to first define the entire computational graph structure, then separately run data through it. Think of it like writing a complete blueprint before construction begins. You declare all operations upfront, compile the graph, then execute it multiple times with different data.
Dynamic Graphs (Define-by-Run) — the approach PyTorch pioneered — build the computational graph as operations execute. The graph is created on-the-fly during the forward pass. It's like drawing the blueprint while simultaneously building: each line of Python code immediately executes and records what happened for later backpropagation.
Why This Matters
Static graphs enable powerful optimizations because the framework sees the entire structure beforehand. However, they're less intuitive—debugging feels disconnected from your Python code, and control flow (if statements, loops) requires special graph operations.
Dynamic graphs feel more "Pythonic." You can use normal Python control flow, print intermediate values naturally, and each forward pass can have a completely different structure (useful for variable-length sequences or conditional architectures). The trade-off: slightly less opportunity for graph-level optimization.
The Analogy
Static: Writing a entire recipe, then cooking it. You must specify every step before starting.
Dynamic: Cooking while deciding what to do next. Each action happens immediately and you adapt as you go.
Key Takeaway: Static graphs (define-and-run) separate graph construction from execution for optimization, while dynamic graphs (define-by-run) build the graph during execution for flexibility and debugging ease.