Value Function

The return $G_t$ depends on the state $s_t$ and the policy $\pi [a|s]$. From this state, the agent will pass through a sequence of states, taking actions and receiving rewards. This sequence differs every time the agent starts in the same place since, in general, the policy $\pi [a_t|s_t]$, the state transitions $Pr(s_{t+1}|s_t,a_t)$, and the rewards issued $Pr(r_{t+1}|s_t,a_t)$ are all stochastic.

We can characterize how “good” a state is under a given policy by considering the expected return $v[s_t|\pi ]$. This is the return that would be received on average from this state and is termed the state value or state-value function:

$$v[s_t|\pi ]=\mathbb{E}[G_t|s_t,\pi ]$$

• Informally, the state value tells us the long-term reward we can expect on average if we start in this state and follow the specified policy thereafter.
• It is highest for states where it’s probable that subsequent transition will bring large rewards soon (assuming the discount factor $\gamma$ is less than one)

Similarly, the action value or state-action value function $q[s_t,a_t|\pi ]$ is the expected return from executing action $a_t$ in state $s_t$:

$$q[s_t,a_t|\pi ]=\mathbb{E}[G_t|s_t,a_t,\pi ]$$

• The action value tells us the long-term reward we can expect on average if we start in this state, take this action, and follow the specified policy thereafter.
• Through this quantity, reinforcement learning algorithms connect future rewards to current actions (i.e resolve the temporal credit assignment problem)

rl
State-value function / State value
?
Characterizes how good a state is. It is the expected reward if we start in this state and follow the specified policy thereafter.

$$v[s_t|\pi ]=\mathbb{E}[G_t|s_t,\pi ]$$

+++

State-action value function / Action value
?

$$q[s_t,a_t|\pi ]=\mathbb{E}[G_t|s_t,a_t,\pi ]$$

Characterizes how good an action is based on the given state. It is expected return if we start in this state, take this action, and follow the specified policy thereafter.
+++