Is it recommendable to put participants in different conditions across trials?

Question

In a pilot study we want to give participants 3 different problems. Those come in two versions, one (supposedly) easier than the other. Participants will randomly be assigned one version of the problems. The goal is to compare the performance on each version of each of the problems. What might be problematic however, is that if it is true that one version is easier, giving some participants three times the easy version and others three times the hard version might affect their motivation. The participants with the difficult problems might get more exhausted and demotivated and will possibly not try as hard on each problem that follows.

One solution would be to randomize the assignment for each problem independently, so that participants might receive two times the easy version and one time the harder one, or any other combination.

From a methodological standpoint, would that be recommendable? Could the observations still be considered independent when comparing the two conditions? Or is there any other way to circumvent that problem?

Answer 1

It depends on what you care to investigate.

Do you care what happens when one person gets two hard questions as compared to someone who gets two easy ones? If so, record it for analysis. If not, control for it by ensuring that each participant receives the same number of easy and difficult problems.

Do you care if there are ordering effects? If so, record that data for analysis. If not, then control for it by ensuring that each participant gets a set of questions classified in the same order. E.g., standardize on EASY-HARD-EASY, and randomly assign problems of each classification.