Value Function

The return $G_t$ depends on the state $s_t$ and the policy $\pi [a|s]$. From this state, the agent will pass through a sequence of states, taking actions and receiving rewards. This sequence differs every time the agent starts in the same place since, in general, the policy $\pi [a_t|s_t]$, the state transitions $Pr(s_{t+1}|s_t,a_t)$, and the rewards issued $Pr(r_{t+1}|s_t,a_t)$ are all stochastic.

We can characterize how “good” a state is under a given policy by considering the expected return $v[s_t|\pi ]$. This is the return that would be received on average from this state and is termed the state value or state-value function:

$$v[s_t|\pi ]=\mathbb{E}[G_t|s_t,\pi ]$$

• Informally, the state value tells us the long-term reward we can expect on average if we start in this state and follow the specified policy thereafter.
• It is highest for states where it’s probable that subsequent transition will bring large rewards soon (assuming the discount factor $\gamma$ is less than one)

Similarly, the action value or state-action value function $q[s_t,a_t|\pi ]$ is the expected return from executing action $a_t$ in state $s_t$:

$$q[s_t,a_t|\pi ]=\mathbb{E}[G_t|s_t,a_t,\pi ]$$

• The action value tells us the long-term reward we can expect on average if we start in this state, take this action, and follow the specified policy thereafter.
• Through this quantity, reinforcement learning algorithms connect future rewards to current actions (i.e resolve the temporal credit assignment problem)

rl
State-value function / State value
?
Characterizes how good a state is. It is the expected reward if we start in this state and follow the specified policy thereafter.

$$v[s_t|\pi ]=\mathbb{E}[G_t|s_t,\pi ]$$

+++

State-action value function / Action value
?

$$q[s_t,a_t|\pi ]=\mathbb{E}[G_t|s_t,a_t,\pi ]$$

Characterizes how good an action is based on the given state. It is the expected return if we start in this state, take this action, and follow the specified policy thereafter.

+++