Graph Attention Networks (GAT)
What you'll learn: How GAT uses attention mechanisms to dynamically weight neighbor contributions during message passing, letting the model learn which edges matter most.
The Core Idea
In standard graph convolutions (like GCN), all neighbors contribute equally to a node's updated representation—or at most, they're weighted by graph structure alone (like node degree). Graph Attention Networks introduce learnable attention weights that determine how much influence each neighbor should have. Instead of treating all edges equally, GAT asks: "Which neighbors are most relevant for this particular node?"
Think of it like a conversation at a party: you don't listen equally to everyone around you. You pay more attention to certain people based on what they're saying and how relevant it is to you. GAT does the same for graph nodes.
How Attention Works on Graphs
For each node, GAT:
- Computes attention scores between the node and each of its neighbors using a learned attention mechanism (typically a small neural network)
- Normalizes scores across neighbors (usually with softmax) to get attention weights that sum to 1
- Aggregates neighbor features using these weights—important neighbors contribute more
The attention coefficients are computed from both the node's own features and its neighbor's features, making them content-based rather than purely structural.
Why It Matters
GAT automatically learns which connections are important during training. In a citation network, for example, it might learn that recent papers or papers from certain authors deserve more attention than others—without you manually specifying these rules.
Key Takeaway: Graph Attention Networks replace fixed aggregation weights with learned, content-dependent attention scores, allowing the model to dynamically focus on the most relevant neighbors for each node during message passing.