I'm beginning to write a neural network simulator in Java and thinking of Hebbian Learning but I'm stuck at one thing:

What causes two neurons to fire at the same interval while only one of them is input and the other is not? Does it fire itself?

Hebbian Learning rule says "those fire together will wire together" but since I'm activating only input neuron, what makes the other interior neuron fire? Do I need to make them fire randomly at times instead of just using weights matrix $W_{ij}$?

I'm a fan of unsupervised learning and need to start with a simple case such as I give one of 3-5 neurons a "1" and expect from another one a "0" as a "NOT" operator.


You basically have 2 options:

  1. Manually fire both neurons together that you want to pair - do this as many times as needed to pair them. After learning, it should be sufficient to fire only one neuron for the second neuron to fire as desired.
  2. Assign a starting weight to the connection between the neurons such that firing one will trigger the other. This is basically shortcutting the above learning process.

Normally, the learning process in neural networks is based on multiple simultaneous inputs - not just one. The network learns the statistical probability of input neurons firing together, and is eventually able to predict them without them firing. So a test case of a single neuron firing can be used after learning is complete (the first option) or when you restore a previously saved matrix of a trained network (the second option).

| improve this answer | |
  • $\begingroup$ I understand that, for the second option, I start with random multipliers for all inputs of all neurons, then they have higher probability for firing when they have higher inputs? So I throw dice(0.0 - 1.0) if it crosses some level (total input sum divided by barrier value) then it fires and repeating parallelly for all neurons? I have some gpu+cpu accelerated opencl java class that can throw about a billion dices per second(randomness quality somewhat low, does this matter?). Thank you for helping mr Arnon. $\endgroup$ – huseyin tugrul buyukisik Jun 11 '15 at 17:33
  • $\begingroup$ That's right, normally the weights matrix is initialized with random values such that different (internal) neurons will fire for different combinations of input neurons. As the learning process progresses, the random values tend to stabilize. The randomness quality doesn't usually matter that much, since it only affects the initial state. $\endgroup$ – Arnon Weinberg Jun 11 '15 at 19:29

Probably you already know most of the stuff i will talk about. But I want to make it clear any way.

First from the nonscientific perspective of hebian learning

I think when there are just 2 neurons and you want to "wire them together" i suspect it wouldn't be learning anything. In the and you may have a post-synaptic neuron fires whenever the pre-synaptic fires.

But when there are at least 2 pre-synaptic neurons it begins to make sense. (And you may avoid giving random numbers)

Long Term Potantiation (LTP) is usually discussed when hebian learning in process. To make a clear example lets assume 3 pre-synaptic neurons one of then (S1) has a stronger connection (weight) to post synaptic neuron while other two (W1 and W2) has a weak connection(weight).

To make it more concrete i will give (neuroscientifically non-valid) meanings to these neurons. Lets say post-synaptic neuron is a neuron that recognized motorbikes. S1 fires when you see a bike, W1 fires when there is a motor sound and W2 (lets make it arbitrary) fires when it smells cherries.

In the begining you don't have the idea how a motorbike sounds like. But when you see it S1 fires, and because you hear it W1 also fires. However W1's contribution is very weak S1 may produce enough inout for post-synaptic neuron to fire. Since W1 was firing when post-synaptic neuron is firing the connection is strengthened. Anf if you have enouh inputs after some points even without the presence of S1 post-synaptic neuron can fire. Since it didn't smell cheries when you say the motorbike W2 remained still the same.

So the message to take home is it is meaningful when there are multiple pre-synaptic inputs and the effect is to one side only.

I am quoting from "Cognitive Neuroscience" book of Gazzangia

three rules for associative LTP have bee n drawn:

  1. Cooperativity . More than one input must be active at the same time.
  2. Associativity . Weak inputs are potentiated when co-occurring with stronger inputs.
  3. Specificity . Only the stimulated synapse shows potentiation.

From the perspective of machine learning:

For unsupervised learning in neural networks you may want to look at:

  • Boltzmann Machines
  • Stochastic Maximum Likehood Learning
| improve this answer | |
  • $\begingroup$ Thank you for explanation, it seems this type neural network is used more like a classifier than a function approximator. $\endgroup$ – huseyin tugrul buyukisik Jun 13 '15 at 5:30
  • $\begingroup$ I think its both yes and no. Yes because neuron models usually have a a treshold value and the output is fired or not.This is not really substitute for a continious function No because I) In theory considering each neuron does a linear regression (summing up inputs with sum weights its like ax +by+...+z) and combining these would allow you to estimate complex functions you can build a neuron model. II) considering the neuron model is digital in the sense that it fires or it doesn't, if you can convert your function to a discrete (digital) one you can estimate it with number of neuron outputs. $\endgroup$ – Sinan Onur ALTINUÇ Jun 13 '15 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.