Procedures behind establishing the IQ scale

Question

There are some good answers about testing individual's intelligence, and about mathematics side of the IQ scale, e.g:

When current IQ tests are developed, the median raw score of the norming sample is defined as IQ 100 and scores each standard deviation (SD) up or down are defined as 15 IQ points greater or less,[2] although this was not always so historically. By this definition, approximately two-thirds of the population scores between IQ 85 and IQ 115, and about 5 percent of the population scores above 125.[3][4]

What I wanted to ask though, is the organizational/"political" side of establishing the parameters of the test, the "calibration" procedure.

• Who (organization/person/entity) establishes or updates the coefficients which make the average test score for given age to correspond to IQ of 100, and the standard deviation increments of 15?

• Who is the test group: how is the tested population providing the raw data for establishing the above selected? (in particular, how to avoid selection bias: people wanting to take IQ tests, because they are (rightly) expecting getting good result and thus bragging rights of high IQ; people of low IQ avoiding the expected effort, frustration and shame).

• How does that look on the international landscape; are the coefficients established "per nation" and computed by national organizations, or are they global, or some other form yet?

Answer 1

Due to fact, that I can´t give a comment, I will post an answer, that is not sufficient. However, I hope this might give you little more insight. Furthermore, my knowledge mainly builds on german IQ Tests.

1.) Usually this is done by the institution/person etc., that has developed the IQ test. However, in some cases the parameters aren´t updated. So, you should always check for this. Especially in regard of very old IQ Tests and the Flynn effect

2.) First, the type of test-sample usually depends on the population, for which the IQ test is being developed (if you want an IQ test for intelligent people in their 20´s, you should go and try it at different universities). In case of Germany: a lot of IQ Tests usually take samples in different parts of Germany, with different demographic attributes. Usually people get paid for participating. Still, in most cases it is hard to sample a representative sample, due to fact that people with higher social status and intelligence tend to agree more easily in participating (“sadly”, you can´t force someone to take part). So, the aspects you have mentioned are indeed a huge problem, with which usually can´t be dealt perfectly. Sometimes people are told, that they won´t get feedback on their performance or only, if they wish to. As you might guess, this barely counters the problem. Furthermore, huge representative samples are expensive to gather. Therefore, the sad truth is, that in most cases it comes down to representative approximations (at least in Germany). These are the reasons why a lot of (german) IQ Tests are biased upwards (f.e. in the IST-2000R more people have been graduated from high School (="Abitur") than the average; this is even worse for the extended version - to counter that, the subsamples are weighted differently). Just as a little note: IQ tests for students are usually better normed, due to the fact, that they can be accessed more easily. Long story short: Researchers often try to cope with selection bias, but often can´t really avoid it.

3.) to be honest: I have no knowledge on the international landscape

Just as a little note: You will usually find more detailed information in the specific IQ manual. And: When you are using an IQ test, you should always think of whether or not your test-participant can be compared with the test-sample (you shouldn´t have any problems in comparing IQ performance of a 35 year old participant when the test-sample is representative for ages between 20-30, due to the fact, that the decrease in the average (fluid) intelligence from 20 to 35 is not that huge (as compared to the increase in the younger age, see f.e. Baltes (1997) Figure 3). But then of course, you can only make judgments relative to the (unrepresentative) test-sample, not to the whole population itself. This is also the reason, why i don´t really like statments like "my IQ is ...", because such statements always depend on the test-sample and test itself, that was being used.

An article, that is dealing (along with other aspects of the human development) with the development of fluid and crystallized intelligence across the lifespan: Baltes, P. B. (1997). On the incomplete architecture of human ontogeny: Selection, optimization, and compensation as foundation of developmental theory. American psychologist, 52(4), 366-380.

Answer 2

I've been involved in norming the German version of the WAIS-IV in 2012. The procedure for norming a psychological test is always the same:

1. A representative sample of the whole population is drawn.

   What differs between tests is how the sample is drawn. For the WAIS-IV, equal numbers of persons where recruited for each age group (in, I believe, 5 year increments), gender, and educational level in four regions of Germany. So while, as @bucky has explained, certain groups of people are more willing to participate, we have filled each "bucket" of the sample with the same number of people. The effort to fill certain buckets was larger, but no subpopulation is overrepresented.

   Other tests use different procedures to draw their samples, and this is one of the distinguishing criteria between different tests, and the reason why some tests (such as the Wechsler tests) are preferred above others.

2. The norming is usually done by the test author him- or herself when the test is first published. Later revisions are often normed by the test publisher with the help of "professional testers".

   In the case of the WAIS-IV, the sample was tested by psychology students from four German universities who had been schooled in applying the test. The testing was supervised by specially schooled grad students. The norms where calculated from the sample test results by an experienced statistician. The whole procedure was supervised by psychologists working at the test publisher (Pearson).

   Sometimes individuals (e.g. scholars) or organisations (e.g. universities) create norms for a population independent of the test author or publisher, and sometimes these norms are published (as scientific papers) for other scholars to use.

   Sometimes norms for subpopulations that have not been sampled during the official norming are mathematically extrapolated, e.g. if persons over a certain age have not been part of the norming sample the curve of the age-by-result distribution is extended mathematically to include the missing ages.

3. Norms are always only applicable to the population the sample was drawn from. That is why the population is always mentioned along with the norms.

   Large (in the sense of popularity and publication effort) tests such as the Wechsler tests are normed for each country, sometimes even for different subpopulations within a country (e.g. by gender, by age, for different occupational groups, for different test situations [e.g. there are 10- and 20- minute normes for the Raven Progressive Matrices, depending on how much time you want to allow your probands to answer], and so on).

   If norms for one population that you would like to test (as a test user) do not exist, an experienced psychologist might estimate the test results from norms from another population when the relation between the two populations with regard to the tested construct is known.

   (There is an ongoing intense debate about this procedure. Using intelligence as an example, when you use different norms for different populations, all populations will be equally intelligent. Norming means that the average person in that population is defined as having an IQ of 100. Their raw scored might differ! So when you use, for example, an African norm for Africans and a European norm for Europeans, Africans and Europeans appear to be equally intelligent. But when you use the same norm for both populations, the African population seems to have lower IQ. There are many different interpretations of this result – that Africans are less intelligent, that the tests culturally biased, that "intelligence", as tested by an IQ test, is dependent on schooling, that intelligence differs between cultures, and so on. What is for certain is that norming, even if it could be done perfectly, doesn't tell you what the test results mean.)

Answer 3

I was never working on developing intelligence tests, but as much as I have read and heard from my professors, could be stated in the short words considering international comparision of the results: For every test you are going to make, you should at first align at some theory of intelligence, according to which you are going to make the questions. Tnen, you give a bunch of these question to the sample. Acoording to their answers, you coose the questions on acording to their difficulty that some are so easy that almost everybody will be able to resolve them, some should be of the average difficulty and some very difficult. Than, you should calculate how many of these question should be in order to enable that the results of some population would take shape of the normal curve. Than you make that the aritmetic mean of this curve is marked with IQ 100, and one standard deviation of 15 IQ points. That is how is calculated. Of course you should assign to the arithmetic mean a value of 200, but it could be sligthly comlplcated for calculation. So, if you take the samples for each test from one state, as is usually done, than the results could not be compared among the members of diferent staes. However, if you could take the sample from several states, the results should be compared between these states. There is one problem considering the state which is known as cultural influence. The question whether the items could be choosen in a way that culture where people live are not influencing how . Than you can think of other possible generalziation of this problem, where people may live into the same country but be influenced by different things, or people who live in the distant countries but use internet may become one specific sample.

More about test you could read in Spearmans books, which are available on the internet. I would recommand you from several reasons, apart that I like them and find them very easily for reading. One of them is that he is one of the first who developed theories of the intelligence after which are made the first tests. I have also found, I think in one of his books, but I am not apsolutery sure, the referece towards Galton paper when he introduced normal curve. Unfortunately I have not yet found it, so if you, or someone else find this paper, I would like to ask you to send me if possible. And that paper would give you the full answer on your question (considering the Bell's curve)