What you are actually asking about is the debate surrounding the question:
Can psychological quantities be measured?
Up until about 1800 psychological questions where discussed by philosophers. A separate psychological discipline did not yet exist. Answers to questions relating to perception, emotion and cognition where attempted on the basis of religious beliefs, philosophical tradition and unsystematic observation and introspection.
Over the course of the eighteenth century, the body-mind dualism was slowly being resolved, which allowed first physiologists and then physicists over the course of the nineteens century to begin to apply their methodology of experimenting with plants and animals and mathematical principles, as they had been developed within the natural sciences, to psychological questions. In the middle of the nineteenth century, the first psychological institutes were founded, the first by Wundt (in Leipzig, Germany), and the discipline of psychology came into existence.
Looking at the history of psychology, you will realize that while psychological questions are as old as humanity, the academic discipline of psychology came into existence when scholars began to measure, i.e. quantify, the psyche. This was in accordance with the rise of the natural sciences during the Enlightenment and their dominance over our whole worldview (almost?) until today.
In his classice monography, Psychometric Methods (1936), J. P. Guilford writes:
The progress and maturity of a science are often judged by the extent to which it has succeeded in the use of mathematics. The “psychometric methods” are procedures for psychological measurement. Measurement means the description of data in terms of numbers and this, in turn, means taking advantage of the many benefits that operations with and mathematical thinking provide.
Thus, while psychological questions have traditionally been considered in the humanities and arts, the academic discipline of psychology has been created and finds its home within the natural sciences and shares their basic approach of quantitative measurement and statistical analysis. Philosophical and artistic "psychology" have continued to exist parallel to institutionalized academic psychology and have given rise to psychoanalysis (Freud), humanistic psychology (Rogers) or critical psychology (Holzkamp), among others.
The principles and standards of psychological experimentation and statistical analysis as we know them today where not discovered instantaneously but developed and refined until the middle of the twentieth century. In 1936 Guilford found that the natural sciences lacked a common theory of measurement:
Measurement in the physical sciences has come so naturally that in that connection little thought has had to be given to what measurement really is. There are some physical scientists who maintain that what is called measurement in psychology is not measurement at all. It is true that the term measurement is sometimes defined in such a way that it does not apply to most of the operations in psychology commonly known as measurement. Definition of an abstract term such as measurement is an arbitrary matter, however, and psychologists will either define it to cover what they are doing under that concept or they will invent a new term for what they are doing with numbers and arithmetical operations. Fortunately, measurement can be defined sufficiently broadly to include the operations known as psychological measurement.
Four years before this statement, in 1932, the British Association for the Advancement of Science had established a committee of scientists that should consider the problem of measurement in the empricial disciplines. The committee was especially asked to resolve the question, whether a measurement of sensation was possible. The answer to this question would decide wether psychology was a natural science coequal to physics or chemistry. Psychologist S. S. Stevens (1946) writes about the committee's discussions:
For seven years a committee of the British Association for the Advancement of Science debated the problem of measurement. Appointed in 1932 to represent Section A (Mathematical and Physical Sciences) and Section J (Psychology), the committee was instructed to consider and report upon the possibility of “quantitative estimates of sensory events” — meaning simply: Is it possible to measure human sensation? Deliberation led only to disagreement, mainly about what is meant by the term measurement. An interim report in 1938 found one member complaining that his colleagues “came out by that same door as they went in,” and in order to have another try at agreement, the committee begged to be continued for another year. For its final report (1940) the committee chose a common bone for its contentions, directing its arguments at a concrete example of a sensory scale. This was the Sone scale of loudness (S.S. Stevens and H. Davis, Hearing. New York: Wiley, 1938), which purports to measure the subjective magnitude of an auditory sensation against a scale having the formal properties of other basic scales, such as those used to measure length and weight. Again the 19 members of the committee came out by the routes they entered, and their views ranged widely between two extremes. One member submitted “that any law purporting to express a quantitative relation between sensation intensity and stimulus intensity is not merely false but is in fact meaningless unless and until a meaning can be given to the concept of addition as applied to sensations.”
So, while the committee could not agree on the question wether sensation can be measured, they made it clear that an empirical equivalent to arithmetic addition would constitute measurement. This refers to the foundation of the measurement of physical quantities by Hermann von Helmholtz (1887) and defined axiomatically by Hölder (1901).
I don't want to go into the details of this argument, because it goes beyond the scope of this question. In short, Helmholtz sought to base measurement on the empirical comparison of so-called "extensive quantities", for example physical length or mass. He correlated arithmetic addition to physical combination (e.g. lineal apposition of staves to measure lengths) and the equality of two numbers to a method of "physical comparison", for example the comparison of weights on a beam balance).
How these principles of extensive measurment can be applied in psychology, and if it should, has been heavily debated, and different "test theories" have been developed, such as classical test theory, latent trait theory, or the theory of knowledge spaces. Foundational texts for a representational theory of measurement are those by Krantz, Luce, Suppes, and Tversky (1971, 1989, 1990) and Robers (1979), if you are interested in this aspect. All books feature some general introduction but require mathematical understanding to follow.
This requirement of a broad and deep understanding of mathematics is the reason why all psychology looks the same to you.
Most currently publishing psychologists have only a superficial understanding of methodology. Gigerenzer, Krauss and Vitouch (2004) explain some of the methodological shortcomings and mistakes:
One of us once had a student who ran an experiment for his thesis. Let us call him Pogo. Pogo had an experimental group and a control group and found that the means of both groups were exactly the same. He believed it would be unscientific to simply state this result; he was anxious to do a significance test. The result of the test was that the two means did not differ significantly, which Pogo reported in his thesis.
In 1962, Jacob Cohen reported that the experiments published in a major psychology journal had, on average, only a 50 : 50 chance of detecting a medium-sized effect if there was one. That is, the statistical power was as low as 50%. This result was widely cited, but did it change researchers’ practice? Sedlmeier and Gigerenzer (1989) checked the studies in the same journal, 24 years later, a time period that should allow for change. Yet only 2 out of 64 researchers mentioned power, and it was never estimated. Unnoticed, the average power had decreased (researchers now used alpha adjustment, which shrinks power). Thus, if there had been an effect of a medium size, the researchers would have had a better chance of finding it by throwing a coin rather than conducting their experiments. When we checked the years 2000 to 2002, with some 220 empirical articles, we finally found 9 researchers who computed the power of their tests. Forty years after Cohen, there is a first sign of change.
Editors of major journals such as A. W. Melton (1962) made null hypothesis testing a neces- sary condition for the acceptance of papers and made small p-values the hallmark of excellent experimentation. The Skinnerians found themselves forced to start a new journal, the Journal of the Experimental Analysis of Behavior, to publish their kind of experiments (Skinner, 1984, p. 138). Similarly, one reason for launching the Journal of Mathematical Psychology was to escape the edi- tors’ pressure to routinely perform null hypothesis testing. One of its founders, R. D. Luce (1988), called this practice a “wrongheaded view about what constituted scientific progress” and “mind- less hypothesis testing in lieu of doing good research: measuring effects, constructing substantive theories of some depth, and developing probability models and statistical procedures suited to these theories” (p. 582).
The student, the researchers, and the editors had engaged in a statistical ritual rather than sta- tistical thinking. Pogo believed that one always ought to perform a null hypothesis test, without exception. The researchers did not notice how small their statistical power was, nor did they seem to care: Power is not part of the null ritual that dominates experimental psychology.
The articel runs on and explains what the p-value can and cannot do (read it, if you are a psychologist). I only quoted the beginning in length to show you that a methodologically savvy minority among psychologists believes that the majority of scholars publishing in their discipline does not know what they do. They follow a "ritual" of blindly applying the same methods to all problems, thus hampering the progress of the discipline and reducing the value of their findings.
Nevertheless, while widely ignored, progress has been made (see my brief remarks about different theories of measurement above).
If you want to learn more about the development of psychology as a natural science, grab any book on the history of psychology from a nearby university library. Make sure you pick one that starts at least in the nineteenth century, but for a better understanding it is necessary to look back until Leibnitz and his contemporaries.
If you want to learn more about measurement in psychology, books such as Joel Michell's Measurement in Psychology: A Critical History of a Methodological Concept provide a good introduction. I have not read any of those books and cannot recommend one. My own knowledge (and the summary given here) stems from a couse in psychometrics I took while studying psychology.
To summarize, academic psychology uses experimentation and statistical analysis because it is a natural science and adheres to the basic principles that define these disciplines. The belief that the universe is probabilistic (and events are distributed and these distributions can be described through arithmetic means and such) rules most of the natural sciences today and is not unique to psychology.
Beyond this unifying foundation, the different fields of psychology have quite different goals and methods. For example, general psychology (including the psychology of perception) is not interested in the distribution of a trait (such as intelligence) across a population, but in understanding, for example, the principles that govern learning in all individuals.
It is not true that experiments have to be performed on a group of subjects. In the psychology of perception, for example, which does not deal with interspecies differences but with the basic principles of human psychology, it is common to use only one single subject, because it is assumed that certain aspects of perception, such as trichromatic vision, are the same in all members of the population and can be investigated in any (healthy) individual. But of course this one test subject has to perform multiple trials, which are then analysed statistically.
Statistical analysis and the application of mathematics to reality itself goes back to ancient times. Bakker (2003) finds predecessors of contemporary statistical concepts as far back as ancient Indian texts:
In an ancient Indian story, Rtuparna estimated the number of leaves and fruit on two great branches of a spreading tree (Hacking 1975). He estimated the number on the basis of one single twig, which he multiplied by the estimated number of twigs on the branches and found a number, which after a night of counting turned out to be very close to the real number.
Aristotle, in the Nichomachean Ethics calculated the mean as "a point equally distant from either extreme", and this was developed further by Ticho Brahe and others until Legendre introduced the least squares method, that we use today, in 1805.
If you want to know about the history of statistics, that is beyond the scope of this site.
The answer to your question in detail:
Why group level statistics?
Because there is no other way of measurement. As a physicist you know that every measurement is a sample from a population. If, for example, you measure weight, each measurement of the same weight will be different, if your scale is sensitive enough. You will then calculate the distribution and its characteristics (mean, slope, range etc.) to describe that distribution. If you do only one measurement, then that is because the differences are negligible for your purpose (e.g. human body weight) or measurement is too expensive and time consuming.
Since when group level statistics?
The ancient Indian example shows that group level statistics has been done since the dawn of time: a representative sample of leaves where drawn from a tree to understand the whole population of leaves. Of course that Indian "mathematician's" estimations where primitive, since the relevant mathematics hadn't been invented yet. The arithmetic mean was defined as late as 1805. But the basic idea and principles are identical to ours: estimate the properties of a population from a representative sample.
Why group level statistics in psychology?
The reason is not a psychological but a methodological one, and it is given in the quote by Guilford: It is the dominating belief of our culture that a true science must employ mathematics. So as soon as psychology was created as a separate scientific discipline, it had to use statistics. Otherwise it would have remained an area of speculation, in the eyes of the scientific community.
Since when did psychology employ group level statistics?
Building on the groundbreaking work of Ernst Heinrich Weber, Gustav Fechner defined the absolute threshold as the statistical mean of all threshold values in the method of limits in his work Elemente der Psychophysik (1860). Fechner also developed the notion of the median (in 1878).
But my answer describes that the use of mathematical methods in psychology was not so much a singular event that affected the rest of the science like an explosion, but a gradual process that happened as psychology was slowly detached from philosophy and grew into a natural science from around 1800 to around 1950, when the process was largely finished (with Stevens' publication).
And again, you must understand that the theory of measurement – and even statistics itself! – was in development in all of the natural sciences at the same time. For example, the arithmetic mean was defined in 1805, Legendre published the least squares method in the same year, and Gauss defined the normal distribution and maximum-likelihood estimation in 1809. This ongoing development was why the British committee on measurement was set up in 1932: because no satisfying theory of measurement had been created until then!
Why the popularity of the mean?
Because most psychologists don't understand statistics well enough to do more than that.
Why population average instead of effects that are present in every single individual in the population, or a specific sub-population?
You obviously did not read a lot of psychological texts. The field of general psychology focusses on effects that are present in every single individual and often does experiments of one proband. Only the field of differential psychology explores the distribution of traits among the population. Other fields look at sub-populations (e.g. children or old people, psychiatric patients, workers of a certain occupation, and so on).
In the history of psychology, the first studies (all through the 1800s) explored general psychology: perception, cognitition, memory. Beginning around 1900, differential psychology began first with the measurement of intelligence, and then the study of medical psychology in psychiatric patients. Around the First World War occupational psychology began to study work behaviour and pedagogical psychology began to apply the findings to the training of future workers and soldiers. Workers and soldiers are groups, so there is also an economic and political reason for group-focussed psychology. Advertising and Human Resources are still today the largest areas of psychology beside clinical psychology, which partially explains the continuing focus on population based studies.
Finally, of course there are scholars that employ a more refined statistics and do not focus on the mean so much, and there are scholars that believe there are aspects of human psychology that are not covered by a statistical approach. If you are interested in them, go deeper into psychology or create your own alternative methodology that represents what you believe about the human psyche.
All of this is of course my personal opinion. I am no historian of psychology and do not have the overview to judge the correctness of my interpretation. I used the small sample of publications that my professor for methodology provided and that I hope is representative. From this sample, I estimated the history of psychology, and my estimation may, of course, be wrong.
Sources:
- Bakker, A. (2003). The early history of average values and implications for education. Journal of Statistics Education, 11(1), 17-26.
- Fechner, G. Th. (1860). Elemente der Psychophysik [Elements of Psychophysics]. Leipzig: Breitkopf und Härtel.
- Gigerenzer, G., Krauss, S., & Vitouch, O. (2004). The null ritual. The Sage book of quantitative methodology for the social sciences, 391-408.
- Guilford, J. P. (1936). Psychometric Methods. New York: McGraw-Hill.
- Helmholtz, H. (1887). Zählen und Messen erkenntnistheoretisch betrachtet. In Philosophische Aufsätze Eduard Zeller zu seinem fünfzigjährigen Doctor-Jubiläum gewidmet. Leipzig: Fues.
- Hölder, O. (1901). Die Axiome der Quantität und die Lehre vom Mass". Berichte über die Verhandlungen der Koeniglich Sächsischen Gesellschaft der Wissenschaften zu Leipzig, Mathematisch-Physikaliche Klasse, 53, 1–46.
- Krantz, D.H., Luce, R.D., Suppes, P. & Tversky, A. (1971). Foundation of measurement. Vol. I. Additive and polynomial representations. New York: Academic Press.
- Luce, R.D., Krantz, D.H., Suppes, P. & Tversky, A. (1990). Foundation of measurement. Vol. III. Representation, axiomatization, and invariance. New York: Academic Press.
- Roberts, F.S. (1979). Measurement theory with applications to decisionmaking, utility, and the social sciences. Reading: Addison-Wesley.
- Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103, 667-680.
- Suppes, P., Krantz, D.H., Luce, R.D. & Tversky, A. (1989). Foundation of measurement. Vol. II. Geometrical, threshold, and probabilistic representations. New York: Academic Press.
