GP for Control

Control problems are attractive for GP because the controller is naturally a program. A controller maps state to action:

$$u_t=f(s_t)$$

Thus, we can use GP to evolve threshold logic, arithmetic combinations of sensor readings, nonlinear feedback laws, and conditional policies.

Consider the classic cartpole problem:

Failure occurs if $\left|{\theta }_t\right|>{\theta }_{\text{max}}$ or $\left|x_t\right|>x_{\text{max}}$. Basically, the pole must stay near upright and within the track.

The state is:

$$s=(x,\dot{x},\theta ,\dot{\theta })$$

GP evolves a controller of the form:

$$u=f(x,\dot{x},\theta ,\dot{\theta })$$

We can have the output either be a discrete force $\{-10,+10\}$, or a continuous form in a bounded interval.

We can then define a survival-based fitness:

$$F=\text{number of timesteps before failure}$$

or a reward-based fitness:

$$\begin{array}{rl}F& =\sum _{t=0}^T{r}_t\\ r_t& =1-\alpha \left|{\theta }_t\right|-\beta \left|x_t\right|\end{array}$$

where we are basically rewarding 1 for each step staying alive, with penalties for large angles and displacements. This encourages both stability and control stability.