Oja's rule

Oja's rule, developed by Finnish computer scientist Erkki Oja in 1982, is a stable version of Hebb's rule.

Model


As with Hebb's rule, we use a linear neuron. Given a set of k-dimensional inputs represented as a column vector $$\vec{x} = [x_1, x_2, \cdots, x_k]^T$$, and a linear neuron with (initially random) synaptic weights from the inputs $$\vec{w} = [w_1, w_2, \cdots, w_k]^T$$ the output the neuron is defined as follows:

$$y = \vec{w}^T \vec{x} = \sum_{i=1}^k w_i x_i$$

Oja's rule gives the update rule which is applied after an input pattern is presented:

$$\Delta \vec{w} = \eta y (\vec{x} - y \vec{w})$$

Oja's rule is simply Hebb's rule with weight normalization, approximated by a Taylor series with terms of $$\eta^n$$ ignored for n>1 since $$\eta$$ is small.

It can be shown that Oja's rule extracts the first principal component of the data set. If there are many Oja's rule neurons, then all will converge to the same principal component, which is not useful. Sanger's rule was formulated to get around this issue.