2
$\begingroup$

In a pilot study we want to give participants 3 different problems. Those come in two versions, one (supposedly) easier than the other. Participants will randomly be assigned one version of the problems. The goal is to compare the performance on each version of each of the problems. What might be problematic however, is that if it is true that one version is easier, giving some participants three times the easy version and others three times the hard version might affect their motivation. The participants with the difficult problems might get more exhausted and demotivated and will possibly not try as hard on each problem that follows.

One solution would be to randomize the assignment for each problem independently, so that participants might receive two times the easy version and one time the harder one, or any other combination.

From a methodological standpoint, would that be recommendable? Could the observations still be considered independent when comparing the two conditions? Or is there any other way to circumvent that problem?

$\endgroup$
2
$\begingroup$

It depends on what you care to investigate.

Do you care what happens when one person gets two hard questions as compared to someone who gets two easy ones? If so, record it for analysis. If not, control for it by ensuring that each participant receives the same number of easy and difficult problems.

Do you care if there are ordering effects? If so, record that data for analysis. If not, then control for it by ensuring that each participant gets a set of questions classified in the same order. E.g., standardize on EASY-HARD-EASY, and randomly assign problems of each classification.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.