Judgments of similarity between samples of writing

Question

I was thinking last night about the possibility of an experiment that investigates the factors contributing to peoples' judgments of 'stylistic similarity' between two samples of writing. For example, such an experiment might consist of multiple trials for each subject, each of which would present a different set of say 3 short excerpts of writing (text randomly sampled from a corpus of authors' writings... maybe no more than 500 words each). Each participant would be asked to make a judgment of either "Which 2 of these writings are most similar in style?" OR 'Which 2 of these writings were most likely created by the same person?".

Part of the data analysis and interpretation might involve the use of programs similar to this one: http://www.hackerfactor.com/GenderGuesser.php . Researchers could form hypotheses about relatively low-level factors that might be important in peoples' judgments of stylistic similarity, and then write programs that would make judgments based on these low-level factors.

I'm assuming that research resembling or even perfectly matching what I've described above has been conducted in the past. After reading what I've written above, does anyone have any suggestions for relevant readings?

Answer 1

It sounds like you're talking about Latent Semantic Analysis. Here's their rundown of what it is.

However, LSA as currently practiced has some additional limitations. It makes no use of word order, thus of syntactic relations or logic, or of morphology. Remarkably, it manages to extract correct reflections of passage and word meanings quite well without these aids, but it must still be suspected of incompleteness or likely error on some occasions.

It's used in a lot of automated grading programs, or systems designed to give a score to paper based on some criteria. However, because order doesn't matter, it makes syntactic and grammatical coherence much harder to measure, so quantifying 'style' might be a little hard, unless by 'style' you mean something like word choice, phrase choice, etc.

Here's a gscholar search. The first three should give a good overview.