Difference between revisions of "Hebb's rule"

From Eyewire
Jump to: navigation, search
 
(8 intermediate revisions by 3 users not shown)
Line 1: Line 1:
'''Hebb's Rule''' or Hebb's postulate attempts to explain "associative learning", in which simultaneous activation of cells leads to pronounced increases in synaptic strength between those cells.
+
<translate>
  
:Let us assume that the persistence or repetition of a reverberatory activity (or "trace") tends to induce lasting cellular changes that add to its stability.… When an axon of cell ''A'' is near enough to excite a cell ''B'' and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that ''A'''s efficiency, as one of the cells firing ''B'', is increased.<ref>{{cite book|last1=Hebb|first1=D. O.|title=The Organization of Behavior: A Neuropsychological Theory|year=1949|isbn=978-0805843002}}</ref>
+
'''Hebb's Rule''' or Hebb's postulate attempts to explain "associative learning", in which simultaneous activation of cells leads to pronounced increases in [[Synapse|synaptic]] strength between those cells. Hebb stated:
 +
 
 +
:Let us assume that the persistence or repetition of a reverberatory activity (or "trace") tends to induce lasting cellular changes that add to its stability.… When an [[Axon|axon]] of cell ''A'' is near enough to excite a cell ''B'' and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that ''A'''s efficiency, as one of the cells firing ''B'', is increased.<ref>Hebb, D. O. (1949). <em>The Organization of Behavior: A Neuropsychological Theory</em> ISBN 978-0805843002.</ref>
  
 
==Model==
 
==Model==
  
[[File:ArtificialNeuronModel english.png|thumb|right|400px|Model of a neuron. For a linear neuron, the activation function is not present (or simply the identity function).]]
+
[[File:ArtificialNeuronModel english.png|thumb|right|400px|Model of a neuron. <i>j</i> is the index of the neuron when there is more than one neuron. For a linear neuron, the activation function is not present (or simply the identity function).]]
  
Given a set of k-dimensional inputs represented as a column vector <math>\vec{x} = [x_1, x_2, \cdots, x_k]^T</math>, and a linear neuron with (initially random) synaptic weights from the inputs <math>\vec{w} = [w_1, w_2, \cdots, w_k]^T</math> the output the neuron is defined as follows:
+
Given a set of k-dimensional inputs represented as a column vector:
  
<center><math>y = \vec{w}^T \vec{x} = \sum_{i=1}^k w_i x_i</math></center>
+
[[File:Hebb1.png|center]]
 +
 
 +
and a linear neuron with (initially random, uniformly distributed between -1 and 1) synaptic weights from the inputs:
 +
 
 +
[[File:Hebb2.png|center]]
 +
 
 +
then the output the neuron is defined as follows:
 +
 
 +
[[File:Hebb3.png|center]]
  
 
Hebb's rule gives the update rule which is applied after an input pattern is presented:
 
Hebb's rule gives the update rule which is applied after an input pattern is presented:
  
<center><math>\Delta \vec{w} = \eta \vec{x} y</math></center>
+
[[File:Hebb4.png|center]]
  
where <math>\eta</math> is some small fixed learning rate.
+
where η is some small fixed learning rate.
  
 
It should be clear that given the same input applied over and over, the weights will continue to grow without bound. One solution is to limit the size of the weights. Another solution is to normalize the weights after every presentation:
 
It should be clear that given the same input applied over and over, the weights will continue to grow without bound. One solution is to limit the size of the weights. Another solution is to normalize the weights after every presentation:
  
<center><math>\vec{w} \leftarrow \vec{w} / \left \| \vec{w} \right \|</math></center>
+
[[File:Hebb5.png|center]]
  
 
Normalizing the weights leads to [[Oja's rule]].
 
Normalizing the weights leads to [[Oja's rule]].
 +
 +
==Hebb's rule and correlation==
 +
 +
Instead of updating the weights after each input pattern, we can also update the weights after all input patterns. Suppose that there are <em>N</em> input patterns. If we set the learning rate η equal to 1/<em>N</em>, then the update rule becomes
 +
 +
[[File:Hebb6.png|center]]
 +
 +
where <em>n</em> is the pattern number, and [[File:Hebb7.png]] is the average over N input patterns. This is convenient, because we can now substitute [[File:Hebb8.png]]:
 +
 +
[[File:Hebb9.png|center]]
 +
 +
<em>C</em> is the correlation matrix for [[File:Hebb10.png]], provided that [[File:Hebb10.png]] has mean zero and variance one. This means that strong correlation between elements of [[File:Hebb10.png]] will result in a large increase in the weights from those elements, which is what Hebb's rule is all about.
 +
 +
Note that if [[File:Hebb10.png]] does not have mean zero and variance one, then the relationship holds up to a factor. Similarly, if the learning rate is not equal to 1/<em>N</em>, then the relationship is still true up to a factor.
  
 
==References==
 
==References==
 
<references/>
 
<references/>
 +
 +
[[Category: Neural computational models]]
 +
 +
</translate>

Latest revision as of 03:15, 24 June 2016

Hebb's Rule or Hebb's postulate attempts to explain "associative learning", in which simultaneous activation of cells leads to pronounced increases in synaptic strength between those cells. Hebb stated:

Let us assume that the persistence or repetition of a reverberatory activity (or "trace") tends to induce lasting cellular changes that add to its stability.… When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased.[1]

Model

Model of a neuron. j is the index of the neuron when there is more than one neuron. For a linear neuron, the activation function is not present (or simply the identity function).

Given a set of k-dimensional inputs represented as a column vector:

Error creating thumbnail: Unable to save thumbnail to destination

and a linear neuron with (initially random, uniformly distributed between -1 and 1) synaptic weights from the inputs:

Error creating thumbnail: Unable to save thumbnail to destination

then the output the neuron is defined as follows:

Error creating thumbnail: Unable to save thumbnail to destination

Hebb's rule gives the update rule which is applied after an input pattern is presented:

Error creating thumbnail: Unable to save thumbnail to destination

where η is some small fixed learning rate.

It should be clear that given the same input applied over and over, the weights will continue to grow without bound. One solution is to limit the size of the weights. Another solution is to normalize the weights after every presentation:

Error creating thumbnail: Unable to save thumbnail to destination

Normalizing the weights leads to Oja's rule.

Hebb's rule and correlation

Instead of updating the weights after each input pattern, we can also update the weights after all input patterns. Suppose that there are N input patterns. If we set the learning rate η equal to 1/N, then the update rule becomes

Error creating thumbnail: Unable to save thumbnail to destination
where n is the pattern number, and
Error creating thumbnail: Unable to save thumbnail to destination
is the average over N input patterns. This is convenient, because we can now substitute
Error creating thumbnail: Unable to save thumbnail to destination
:
Error creating thumbnail: Unable to save thumbnail to destination
C is the correlation matrix for
Error creating thumbnail: Unable to save thumbnail to destination
, provided that
Error creating thumbnail: Unable to save thumbnail to destination
has mean zero and variance one. This means that strong correlation between elements of
Error creating thumbnail: Unable to save thumbnail to destination
will result in a large increase in the weights from those elements, which is what Hebb's rule is all about. Note that if
Error creating thumbnail: Unable to save thumbnail to destination
does not have mean zero and variance one, then the relationship holds up to a factor. Similarly, if the learning rate is not equal to 1/N, then the relationship is still true up to a factor.

References

  1. Hebb, D. O. (1949). The Organization of Behavior: A Neuropsychological Theory ISBN 978-0805843002.