0
$\begingroup$

I'm attempting to find an example of a learning machine/neural network that achieves zero variance, but I am having a hard time finding an example anywhere.

Variance is defined as the generalization error for a neural network or in other words the difference between the learning function derived from a specific sample and the function derived from all training samples.

Does anyone know an example that they can share? Any help would be greatly appreciated.

$\endgroup$
  • 1
    $\begingroup$ I'm not an expert in the field of machine learning, but I don't think such a thing is possible unless you're overlearning. In such cases you'll correctly classify every one of your samples, but you'll fail to generalize to new samples... Or am I misunderstanding your question? $\endgroup$ – blz Oct 14 '14 at 10:10
  • $\begingroup$ If you overlearn you will most likely drive variance up by fitting the function too much to one training sample and then getting a significantly different (idiosyncratic) function for the other training samples, IMO. $\endgroup$ – user6682 Oct 14 '14 at 12:52
  • $\begingroup$ By the way, this question should probably be migrated to stats.SE, since they officially deal with machine learning questions and we aren't exploring a specifically 'cognitive' topic $\endgroup$ – user6682 Oct 14 '14 at 16:36
2
$\begingroup$

First, a definition, which is basically what you said but refers to elements in the domain of the function: "A learning algorithm has high variance for a particular input x if it predicts different output values when trained on different training sets." So, in order to have zero variance, the machine/NN must output the exact same value for x across training sets. The only way to achieve zero variance with respect to all possible x is if you have a constant (non-learning) machine/NN (because then it will trivially maintain its output/function), or if you have training samples perfectly sampled from the same (computable in your machine/NN!) function without noise. An example of such training data would be samples consisting of elements (x,y) all sampled from the same linear function. However, in most real world applications this won't be the case. Furthermore, it isn't necessarily advisable to minimise variance because you will overgeneralize (assume a function for the underlying distribution that is too simple).

Source:

S. Geman, E. Bienenstock, and R. Doursat (1992). Neural networks and the bias/variance dilemma. Neural Computation 4, 1–58.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.