I'm attempting to find an example of a learning machine/neural network that achieves zero variance, but I am having a hard time finding an example anywhere.
Variance is defined as the generalization error for a neural network or in other words the difference between the learning function derived from a specific sample and the function derived from all training samples.
Does anyone know an example that they can share? Any help would be greatly appreciated.
First, a definition, which is basically what you said but refers to elements in the domain of the function: "A learning algorithm has high variance for a particular input x if it predicts different output values when trained on different training sets." So, in order to have zero variance, the machine/NN must output the exact same value for x across training sets. The only way to achieve zero variance with respect to all possible x is if you have a constant (non-learning) machine/NN (because then it will trivially maintain its output/function), or if you have training samples perfectly sampled from the same (computable in your machine/NN!) function without noise. An example of such training data would be samples consisting of elements (x,y) all sampled from the same linear function. However, in most real world applications this won't be the case. Furthermore, it isn't necessarily advisable to minimise variance because you will overgeneralize (assume a function for the underlying distribution that is too simple).
Source:
S. Geman, E. Bienenstock, and R. Doursat (1992). Neural networks and the bias/variance dilemma. Neural Computation 4, 1–58.
