An RBM network consists of:
- A “hidden layer”:
- A “visible layer”: , because this layer interacts with the environment.
Connections between layers are symmetric, represented by weight matrix .
RBM energy
Similar Hopfield Networks, an RBM is characterized by an energy:
This can be rewritten as:
where . The terms of this correspond to:
- Discordance cost:
- Operating cost:
Boltmann Probability
The RBM network states are visited with the Boltmznn probability:
where the partition function is defined as
Since lower energy states are visited more frequently:
Training an RBM as a Generative Model
Suppose our inputs . We want an RBM to behave as a generative model such that:
Let the loss function be:
Rewriting:
Thus, we can decompose the loss into .