Manipulating the difficulty of guessing a correlation

Question

I am trying to generate correlations between random variables (two dimensional) with a defined linear relationship (in the $r$ sense), but with different visual patterns when plotted. I am trying to create a 'guess the correlation' task where I can systematically manipulate the difficulty for an observer to guess the linear relationship.

What I am doing now is given a correlation $r$ I generate the first and second values, $X_1$ and $X_2$, with $n$ samples from the standard normal distribution. Then from there I make $X_3$ a linear combination of the two $X_3 = r X_1 + \sqrt{1-r^2}\,X_2$

Then: $Y_1 = \mu_1 + \sigma_1 X_1, \quad Y_2 = \mu_2 + \sigma_2 X_3$

And now $Y_1$ and $Y_2$ have a correlation $r$.

For manipulating the difficulty I've been playing with the parameters of the distribution and $n$, however, I am not satisfied with the results.

Any idea on how to systematically increase the difficulty of the task? (i.e., adding outliers, for instance etc).

Note: Difficulty is a cognitive/psychology question rather than a statistical one. I intend to test the notion of difficulty empirically (i.e., under a specific parameter combinations, people tend to do worse). The idea is to generate plots with varying parameters for a given correlation value (i.e., changing the number of points, the variance, outlier, functional form? etc). What are the parameters and what would be a systematic way to manipulate them.

Answer 1

You will probably need to do pilot testing to ascertain the difficulty of a given correlational task.

From pilot testing, you would need to quantify the difficulty of the task. One option would be the mean discrepancy between actual and predicted correlation. However, there would be other metrics of both degree of error and difficulty.

I would hypothesise that the easiest correlations to guess based on scatterplots would be strong linear relationships involving bivariate normal distributions and lots of data (e.g., n > 1000).

There are a whole bunch of things that you could try to make the task more difficult. Here's what comes to my mind:

• Outliers (both outliers that increase the correlation and outliers that decrease the correlations); vary the number of outliers in a particular region; combine outliers that increase and decrease the correlation; make outliers even more extreme
• Non-linear relationships (e.g., mixtures of linear and another function such as quadratic, cyclical, stepwise; power functions, logistic fucntions, circular, etc.)
• bimodal distributions on one or both variables
• Highly skewed distributions on one or both variables
• Fewer data points
• data that yield correlations only a little bit above or below zero
• where there is a function like a quadratic, make the x depend on y rather than y depend on x.

More generally, I think that practice and feedback effects will also be relevant. I.e., it may be relevant to ascertain whether the differences in difficulty with novices also correspond to relative differences in difficulty with participants who have been exposed to the full range of items you have generated.