Dueling DQN: Advantage Identifiability
What you'll learn: Why splitting Q-values into state-value and advantage streams creates an ambiguity problem, and how to fix it with mathematical constraints.
The Identifiability Problem
In Dueling DQN, we decompose the Q-function into two parts: a state-value stream V(s) and an advantage stream A(s,a). The architecture outputs:
Q(s,a) = V(s) + A(s,a)
Here's the catch: this decomposition is fundamentally ambiguous. Imagine V(s) increases by +5 while all A(s,a) values decrease by -5. The final Q-values remain identical! The network could learn any arbitrary split between value and advantage, making it impossible to know which component is truly responsible for each prediction.
Think of it like splitting a restaurant bill: if the total is 60 and Bob 70 and Bob $30. Without additional constraints, there are infinite valid splits.
Enforcing Identifiability
To make the decomposition unique and stable, we enforce constraints that "anchor" the advantage stream:
Zero-Mean Constraint
Subtract the mean advantage from all actions:
Q(s,a) = V(s) + (A(s,a) - mean_a'(A(s,a')))
This forces advantages to be centered around zero—positive for better-than-average actions, negative for worse.
Max Constraint (More Common)
Alternatively, force the best action to have zero advantage:
Q(s,a) = V(s) + (A(s,a) - max_a'(A(s,a')))
Now V(s) represents the value when taking the best action, and advantages show relative losses.
Both constraints ensure V(s) learns meaningful state values while A(s,a) captures only action-specific differences.
Key Takeaway: Raw advantage streams in Dueling DQN are unidentifiable without constraints; applying zero-mean or max operations ensures stable, interpretable learning by eliminating the ambiguity in the value-advantage decomposition.