One potential flaw of Q-Learning is that the maximization over the actions in the update:
leads to a systematic bias in the estimated action values .
Consider two actions that provide the same average reward, but one is stochastic and the other deterministic. The stochastic reward will exceed the average roughly half of the time and be chosen by the maximum operation, causing the corresponding action value be over-estimated. A similar argument can be made about random inaccuracies in the output of the network or random initializations of the q-function.
The underlying problem is that the same network both selects the target (by the maximization operation) and updates the values. Double Q-Learning tackles this problem by training two models and simultaneously:
Now the choice of the target and the target itself are decoupled, which helps prevent these biases. In practice, new tuples are randomly assigned to update one model or another. This is double Q-learning.
Double deep Q-networks or double DQNs use deep networks and to estimate the action values, and the update becomes: