Beyond a certain level, is reading comprehension affected more by psychological intuition than by verbal intelligence?

Question

Certainly, you have reading comprehension passages on the SAT, GRE, and LSAT. But beyond that point, is it really possible to measure reading comprehension through multiple-choice tests? There are multiple motivations for why an author may make argument X, and multiple interpretations of his exact argument, which may be affected by the immediate circumstances that made him write that argument at that particular point in time (and while we may know a bit about the society that he lived in, chances are that we'll probably never know who he intended the book to be written for - hell - some people probably intended to write a book to impress some potential lover [and hell, maybe some passages in the book are motivated to attract the sympathy of that lover] even though they pretended to write it for the greater good of society).

I mean, all you have to do is to look at courtrooms. There, people pretty much argue about what people (in a case with precedent) actually meant, and there's usually no concrete (and correct) answer. And people who study the life of a particular philosopher may have multiple interpretations of why philosopher X wrote passage Y, and what X really meant in writing passage Y.

Plus, it's oftentimes very interesting to read the writing of someone you know REALLY well because you know their life so well that you can dream up hypotheses of what prompted them to write passage X at time Y (and you also know his social network so you could imagine what audience he had in mind, and what audience he might have secretly hoped to impress without mentioning it). And perhaps because of this, reading comprehension is not a skill with a single scale, since people are better at understanding some people than they are at understanding others.

I mean, I guess the ultimate test of reading comprehension would be to test someone on one of Immanuel Kant's books, or on some recent postmodernist book (made artificially difficult to read/parse), and then ask him "well, what is Kant trying to get at"? But is that really a valid test? Or maybe one where you have to dissect the entire structure of a book where X is pretending to be Y pretending to be Y pretending to be X (of course, such books are rarely written, but theoretically possible).

Alternatively, wouldn't someone with exceptional reading comprehension be better at reading crappy writing? (because they're probably more able to process multiple thought-streams at once - which includes possible fragmentary/jumbled ideas?) A common assumption in reading comprehension tests is that the writing is "perfect", but this is frequently not the case, and people (especially older people) can get attention lapses when writing

Answer 1

The assessment of reading comprehension is not my main area of interest, but some of the many articles on the topic of assessment of reading comprehension may be reflevant. See "measurement of reading comprehension" on Google Scholar (e.g., a couple of the pdfs of reviews here, and here).

This entry in the literacy encyclopedia on assessment of reading comprehension might be a good starting place. The article has a the following quote of a definition of reading comprehension:

The National Assessment of Educational Progress (NAEP) 2009 Reading Framework Committee defines reading comprehension as ...“an active and complex process that involves understanding written text, developing and interpreting meaning, and using meaning as appropriate to type of text, purpose and situation (National Center for Educational Statistics, 2005, p. 2).

Here are a few general points:

• Multiple choice testing has the benefit of being reliable given a sufficient number of items. Of course, questions could be raised over the breadth of what it measures (i.e., validity issues).
• Familiarity with the content should increase reading comprehension. Thus, to build on your question, a philosophy student should comprehend more Kant, than a non-philosophy student holding general reading ability constant. A similar argument could be made where the reader has insider knowledge about what the author is trying to say.
• Some texts are more difficult to comprehend in general, whether it be because the text is poorly written (e.g., grammar, spelling, stylistic issues) or because of the complexity of the material.

Implication of the Rasch model to questions of measurement:

A basic model of responding to multiple choice questions based on Item Response Theory (IRT) is the Rasch model. It basically says that the probability of an individual getting an item correct is a function of their ability and item difficulty.

So according to this model, increasing the difficulty of the reading comprehension items would affect all individuals roughly equally. As can be shown with IRT the reliability of measurement can be increased by setting the difficulty of items to a level that discriminates between individuals. I.e., if items are too difficult, everyone gets the item wrong, and we don't learn much; likewise, if the item is too easy, everyone gets the item right, and we don't learn much. Relating this to your question, including poorly worded writing or really difficult subject matters would only increase difficulty, which would not directly affect measurement of individual differences.

However, the Rasch model is generally simplification of the real world. In general, I think it takes you a long way in understanding the factors driving test performance. However, in some cases you might have reading items which some individuals understand better than others and this factor is unrelated to ability (e.g., a reading that embodies cultural assumptions shared by only some individuals). It is for this reason that reading comprehension tests should probably avoid content where a subgroup is likely to possess substantially more domain specific knowledge on the topic.