Imagine a population of people in an office environment. There is a small war going on about whether this brand of coffee machine is better than that one.

How would you go about testing which coffee maker is truly most preferred?

My current plan is to take 10 people out of that population and make them each drink two cups of coffee, one made by each machine. People will be given the choice of preparation styles for their individual trial, black, milk, sugar, or both. Both cups will be made in the exact same way for that individual trial. Whether a given cup will have coffee from machine "A" or "B" will be randomized to avoid people's preference for their first taste being taken into account. Neither the participants, nor the administrator of the test will know which cup was made with which machine, to avoid unconscious biases. Lastly, each participant will be independent of each other to avoid a bias forming from everyone else choosing a particular cup. (Though the randomized nature of the study should make this a moot issue, but still.)

Is there anything I'm forgetting to make this a more perfect study? I would love to have a larger sample than 10, but alas, that's not an option. Also, I'm worried that allowing people to choose how their coffee will be prepared will skew the results, but as long as both cups within a trial are made the same way, it shouldn't be a problem, right?

  • $\begingroup$ Allowing people to choose their style of coffee preparation would improve generalizability but might increase variance...or it might decrease it, since people's preferences might be a little more systematic within their area of familiarity. $\endgroup$ May 11 '14 at 1:38
  • $\begingroup$ Beware of the "Pepi Challenge" effect. Preferences reported during single tastings of small quantities vary greatly from preferences reported from continuous consumption over a period of weeks. IIRC, short taste tests tend to bias responses in favor of sweeter and more sour flavors, for instance. You may wish to give people a cup of coffee every morning over a period of days, if such a design is feasible. $\endgroup$ Jun 7 '14 at 16:14

The study design sounds pretty good. Some of the good things you are proposing:

  • Using a repeated measures design will give you more statistical power than a between subjects design, which is particularly useful when your sample size is small.
  • Randomising or counterbalancing for order should mostly control for order effects.
  • Double blind will focus the test on taste rather than branding and as you say limit experimenter effects

A few further suggestions:

  • You could provide a little bread and water between the two tastings or separate the tastings by some longer period of time to limit any residual taste effects from the first taste.
  • And yes, as you say, increasing sample size will increase your statistical power. That said, if your sample is your population (e.g., you have an office of 10 people), then the smaller sample size doesn't matter.
  • You should consider how the response scale you will use for rating taste. You could simple get a binary preference. But I think it might be better to use some form of scale that rates preference in addition to an explicit question at the end about preference.
  • $\begingroup$ I hadn't thought to make them clean their palate before trying the second cup. Would salty pretzels work? Not only do we have a near infinite supply of pretzels, but I feel the salt would contrast well with the coffee. $\endgroup$
    – Soviero
    May 11 '14 at 4:27
  • $\begingroup$ The salt might also temper the taste of the second cup, so it may be best to keep the flavours bland and neutral between tastings. $\endgroup$
    – Adb
    May 12 '14 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.