With on-policy methods, the current best policy is used to guide the agent through the environment.
- This policy is based on the observed action values in every state, but of course, it’s not possible to estimate the value of actions that haven’t been used, and there is nothing to encourage the algorithm to explore these.
- One solution is to use exploring starts. Here, episodes with all possible state-action pairs are initiated, so every combination is observed at least once. However, this is impractical if the number of states is large or the starting point cannot be controlled.
- Another approach is to use an epsilon greedy policy, in which a random action is taken with probability , and the optimal action is allotted the remaining probability . The choice of trades off exploitation and exploration. Here, an on-policy method will seek the best policy from this epsilon-greedy family, which will generally not be the best overall policy.
In off-policy methods, the optimal policy (the target policy) is learned based on episodes generated by a different behavior policy . Typically, the target policy is deterministic, and the behavior policy is stochastic (e.g., and epsilon-greedy policy). Hence, the behavior policy can explore the environment, but the learned target policy remains efficient.
- Some off-policy methods explicitly use importance sampling to estimate the action value under policy using samples from .
- Others, such as Q-Learning, estimate the values based on the greedy action, even though this is not necessarily what was chosen.