Restricted Boltzmann machine

---

A restricted Boltzmann machine, commonly abbreviated as RBM, is a neural network where neurons beyond the visible have probabilitistic outputs. The machine is restricted because connections are restricted to be from one layer to the next, that is, having no intra-layer connections.

As with contrastive Hebbian learning, there are two phases to the model, a positive phase, or wake phase, and a negative phase, or sleep phase.

Model

Model of a neuron. j is the index of the neuron when there is more than one neuron. For the RBM, the activation function is logistic, and the activation is actually the probability that the neuron will fire.

We use a set of binary-valued neurons. Given a set of k-dimensional inputs represented as a column vector

生成缩略图出错：无法将缩略图保存到目标地点

, and a set of m neurons with (initially random, between -0.01 and 0.01) synaptic weights from the inputs, represented as a matrix formed by m weight column vectors (i.e. a k row x m column matrix):

生成缩略图出错：无法将缩略图保存到目标地点

where

生成缩略图出错：无法将缩略图保存到目标地点

is the weight between input i and neuron j.

During the positive phase, the output of the set of neurons is defined as follows:

生成缩略图出错：无法将缩略图保存到目标地点

where

生成缩略图出错：无法将缩略图保存到目标地点

is a column vector of probabilities, where element i indicates the probability that neuron i will output a 1.

生成缩略图出错：无法将缩略图保存到目标地点

is the logistic sigmoidal function:

生成缩略图出错：无法将缩略图保存到目标地点

During the negative phase, from this output, a binary-valued reconstruction of the input

生成缩略图出错：无法将缩略图保存到目标地点

is formed as follows. First, choose the binary outputs of the output neurons

生成缩略图出错：无法将缩略图保存到目标地点

based on the probabilities

生成缩略图出错：无法将缩略图保存到目标地点

. Then:

生成缩略图出错：无法将缩略图保存到目标地点

Then the reconstructed binary inputs

生成缩略图出错：无法将缩略图保存到目标地点

based on the probabilities

生成缩略图出错：无法将缩略图保存到目标地点

. Next, the binary outputs

生成缩略图出错：无法将缩略图保存到目标地点

are computed again based on the probabilities

生成缩略图出错：无法将缩略图保存到目标地点

, but this time from the reconstructed input:

生成缩略图出错：无法将缩略图保存到目标地点

This completes one wake-sleep cycle.

To update the weights, a wake-sleep cycle is completed, and weights updated as follows:

生成缩略图出错：无法将缩略图保存到目标地点

where η is some learning rate. In practice, several wake-sleep cycles can be run before doing the weight update. This is known as Gibbs sampling.

A batch update can also be used, where some number of patterns less than the full input set (a mini-batch) are uniformly randomly presented, the wake and sleep results recorded, and then the updates done as follows:

生成缩略图出错：无法将缩略图保存到目标地点

where

生成缩略图出错：无法将缩略图保存到目标地点

is an average over the input presentations. This method is called contrastive divergence.

References

• Hinton, Geoffrey (August 2, 2010). "A practical guide to training restricted Boltzmann machines". University of Toronto Department of Computer Science.

• Cho, KyungHyun (March 14, 2011) "Improved Learning Algorithms for Restricted Boltzmann Machines". Aalto University.