In behavioral economics and psychology, there are many experiments where participants receive variations of a piece of text depending on the experimental condition they are assigned to. For example, Beenen et al. [1] "used different messages in [an] invitation email to vary [an online community users'] perceptions of their uniqueness and who would benefit from their contributions. ... then tracked their contribution behavior following the invitation."
What are the most important factors to consider when designing phrases for such experimental conditions?
We're planning to conduct an experiment on motivations to join an emerging online community. We have envisioned two types of incentives to join:
- Engaging in high social activities, e.g., joining discussions or finding friends on Wikipedia Talk pages.
- Engaging in low social activities, e.g., individually editing an article or creating a new article.
We've designed emails to send to a group of people inviting them to join the community. We've randomized people between the low and high social experimental conditions. Depending on their conditions, we randomly add one of the following phrases in their invitation email:
High Social Activity Condition:
- Organize content with others
- Collaboratively summarize content
- Collaboratively visualize complex content
- Study with others
- Receive constructive feedback
- Build a reputation within a group
- Share your expertise
- Contribute skills to a group
- Supply knowledge to a group
- Collaboratively help others learn
- Get constructive feedback and collaboratively help others
- Build a reputation and share knowledge
Low Social Activity Condition:
- Summarize content
- Organize and link knowledge
- Visualize complicated concepts
- Improve your learning
- Study/Acquire/Absorb new information
- Visualize and make new connections
- Contribute knowledge
- Make knowledge a public good
- Make knowledge easier to understand
Ideally these phrases should be the same in every aspect other than their meaning that should reflect the experimental condition. However, these phrases are currently different in the number of characters, number of words, number of syllables of words, how words sound, and shapes of characters. Each of these differences may confound our experiment design. Also, there may be other differences between these phrases that we are not aware of.
Please let us know of best practices/rules/studies about the design of phrases for experimental conditions.
Also, I'd appreciate it if you inform us of other similar experimental studies about motivations to join communities.
[1] Ling, K., Beenen, G., Ludford, P., Wang, X., Chang, K., Li, X., ... & Kraut, R. (2005). Using social psychology to motivate contributions to online communities. Journal of Computer‐Mediated Communication, 10(4), 00-00.
