I've read theoretical papers on why constraints are a necessary part of any learning system. The papers were aimed at human learning. I'm having trouble finding relevant references now.

What papers deal with the theoretical issue of whether or not learning must be constrained?

  • 1
    $\begingroup$ Constrained in what way? $\endgroup$ – Seanny123 Aug 28 '15 at 0:24
  • 1
    $\begingroup$ In any way. The basic argument is that it is impossible to have learning unless there is some limit to what can be learned, if I remember correctly. $\endgroup$ – Josh de Leeuw Aug 28 '15 at 2:07
  • $\begingroup$ By 'constraints', are you referring to inter-subject (i.e. Calculus) constraints or constraints on learning as a whole? $\endgroup$ – Sydney Maples Aug 28 '15 at 3:03
  • $\begingroup$ Learning as a whole. It's a very broad, general argument. I just can't remember where I found it! $\endgroup$ – Josh de Leeuw Aug 28 '15 at 12:45

If that suffices, I can give you the classical article from the domain of language learning: Gold 1967.

The basic intuition is that an infinite number of grammars could explain any given set of strings. Analogously, you can probably consider fitting a polynomial to a set of points, or instances of the inverse problem (reconstructing a source from observations). In language acquisition, without some prior constraint on the to be acquired grammar, therefore, no unique convergence will happen. This has led to the Poverty of the Stimulus class of arguments, and the wide range of responses to it. There is a wide range of responses to the Gold paper - if you want more, it really depends on what specifically you're looking for.

In practice, I don't think anyone can imagine a learner in the real world that is without constraints, first and foremost those of storage and computational power.

  • $\begingroup$ This is definitely along the lines of what I was looking for, and is very helpful. $\endgroup$ – Josh de Leeuw Aug 31 '15 at 18:04
  • $\begingroup$ Glad I could help. If you specify your question, I'll specify my answer. $\endgroup$ – jona Aug 31 '15 at 19:34

There's a related saying in statistics:

you cannot do inference without making assumptions.

From Information Theory, Inference, and Learning Algorithms (page 26) by David J. C. Mackay.

Inference about the world is learning, and if you don't assume anything about the world, you can't learn to generalize. Assumptions limit your hypothesis space.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.