11
$\begingroup$

I am planning on running an online psychological experiment where participants must learn about various simulated environments and then make inferences about them. Because completing the experiment would require a nontrivial amount of attention and time, I have been toying with the idea of giving personal feedback at the end of the experiment (i.e., using gamification).

I am considering giving participants feedback regarding how their responses compare to our computational model's predictions and/or to the average. The hope is that receiving a "score", even when it's made clear that such a score is not evaluative or absolute, will get people to pay more attention and will make them more likely to complete the task. The feedback would only be offered following the experiment itself, and its precise nature would not be revealed until that time. Moreover, I wouldn't specify how our predicted scores are derived. Calculating the exact values by hand would be extremely time consuming, so even if the derivation is explained I don't see how anyone could realistically bias their responses with that knowledge.

Questions

  • Does giving personal feedback raise any ethical problems?
  • How might it affect response patterns?
  • Will giving personal feedback elicit higher completion rates and more attentive responding, without any significant complications?
$\endgroup$
5
  • $\begingroup$ "Making Surveys More Fun" blog post might be relevant. $\endgroup$ Jul 10, 2012 at 20:28
  • 1
    $\begingroup$ Interesting question, but I don't think it's possible to answer with the information given. If you give us a detailed explanation of the experiment and what you hope to find, perhaps we can help. My intuitions: 1) is there any reason to suspect this is unethical? 2) people might try to change their answers to match feedback, even if they don't know how the feedback was derived. this in confusing though, because you say feedback is only given at the end of the experiment. so i'm not sure how that would help your cause. $\endgroup$
    – Jeff
    Nov 20, 2012 at 6:23
  • $\begingroup$ 3) yes, feedback can engage attention. whether that mitigates attrition in your experiment is unclear. $\endgroup$
    – Jeff
    Nov 20, 2012 at 6:23
  • $\begingroup$ Paying it forward because I think this could be a very valuable question in the same vein as cogsci.stackexchange.com/questions/9584/… $\endgroup$ Apr 25, 2015 at 9:10
  • $\begingroup$ ^sir, answer me these questions, and i'll nail your bounty like a fi-dolla whore: 1) how many hours are we talking per subject? 2) what is the sample -- undergrad students? 3) is your research related in any way to meta-cognitive ability? $\endgroup$
    – faustus
    Apr 25, 2015 at 14:47

1 Answer 1

7
+100
$\begingroup$

Ethics of Feedback:

The APA's code of ethics (2010) as well as the Advisory Group on Conducting Research on the Internet (AGCRI) report (2004) summarize ethical issues related to conducting offline and online psychological research. Feedback is normal in psychological experiments, and researchers are encouraged to debrief participants, before, immediately after, or as soon as the research conducted reasonably allows without risking an effect on results.

Research proposals should undergo an Institutional Review Board (IRB) review prior to initiation. A consideration with respect to feedback is if it might reasonably be construed by participants as being on a positive-negative dimension (ie , the feedback could be considered negative), then you may want to reconsider providing it, whereas if the feedback is on a neutral dimension (an example would be a personality trait such as placement on an introvert-extrovert dimension), then there is unlikely to be any concern.

Effect of Motivation:

Of course, feedback provided after task completion cannot affect task performance. Presumably however, the promise or expectation of feedback at the end of a task may have an effect. Without knowing the nature of the research, the general findings on the effect of incentives on results are that they are negligible if any. For example, Mason and Watts (2009) find that "Accuracy ... is not reliably different for different wages," and similarly, a review of online studies conducted on Mechanical Turk (2011) reports that "most workers are not motivated primarily by the financial returns and genuinely care about the quality of their work," and result validity is quite high after some basic spam filtering is applied.

Nonetheless, I would recommend testing the effect of your chosen incentive if at all possible. This can be done either by comparing results from unincentivized participants to those of incentivized participants, or through validity checks - for example in surveys common practice is to include multiple questions with similar meanings but different wordings to check that they all get similar responses as a way of confirming that answers are legitimate and not just trying to get through the survey as fast as possible.

Completion Rates:

The effect of incentives on task completion in general is very mixed. Online studies do consistently see greater participation for higher paid experiments, but for non-monetary incentives the results are more complex. The overjustification effect is an example where incentives may actually reduce task completion rates or performance quality, though this effect is not very robust in practice. Houmanfar and Hayes (1998) report that "the results indicated no effects of either private feedback or public feedback on task completion," Conrad et al (2010) find that "feedback can improve their experience though not necessarily their completion rates."

The difference between a simple score at the end of the experiment, and more typical notions of gamification may be considerable however. Gamification in general does have an overall positive impact. Gamification typically involves a variety of interactive features that come into play throughout the task. So if at all possible, I would recommend incorporating say, a progress indicator, badge system, storyline, or any other gamification feature that can reasonably be added. Note that gamification features are not uniformly successful, so some experimentation may be necessary to determine what works best in your particular situation.

Non-contingent Incentives:

One final note: You didn't mention if the feedback would be contingent on completion of the experiment. While the idea of providing incentives only on task completion may seem intuitively sound, there is substantial evidence that non-contingent incentives are generally more effective. For example, if you promise to provide a score only on task completion, then fewer people might attempt the task than if you promise to provide a score regardless, and even if completion rates are much lower you might get higher participation overall. Say only 10 people attempt the task for a contingent incentive, and have a 50% completion rate, then you get 5 results. On the other hand if 100 people attempt the task for a non-contingent incentive, and have a much lower 10% completion rate, then you get 10 results - twice as many. Worth considering I'd say.

$\endgroup$
2
  • $\begingroup$ +1 Thanks for a broad and well-sourced answer, Arnon. It seems to revolve more around incentives than feedback, though, so in some sense it's (partially) a great answer to a different question. The question as I read it was about the ethical and methodological implications/effects of, whether or not, for example to display a %correct metric during a working memory task. I have personal experience with this where we didn't find any differences for feedback-condition Ss in exploratory analysis, but nothing direct like Conrad 2010. $\endgroup$ Apr 26, 2015 at 10:44
  • $\begingroup$ The OP seemed specific about providing feedback only at the end of the experiment, so that it served as an incentive to complete the task. There are quite a few studies on the effects of on-going feedback that I intentionally excluded from the answer. The results there vary as well, so I certainly wouldn't recommend adding such feedback if there was a risk of undesirable effects on the results. $\endgroup$
    – Arnon Weinberg
    Apr 26, 2015 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.