I am doing a study in which I intend to study if the response of chatGPT corresponds to responses by human participants. I will ask human participants (n = 50) to rate the correctness of some sentences on a scale of 1-10 and also ask the same questions to chatGPT. If I were to compare these two responses (humans vs chatGPT), what kind of statistics should I use since there is only one response from chatGPT whereas 50 responses from human participants? This is done to see if the response patterns are similar.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You could use a one-sample Wilcoxon signed-rank test to ask whether the median of the human participant responses is different from the ChatGPT response. That's only appropriate for a single sentence, though; if you have a question about the sentences as a group you'll need to come up with an appropriate hypothesis that applies to the whole group before you bother testing anything at all. Hypothesis first!!
However, even for the one question this tests a pretty narrow hypothesis (whether the human responses are on average greater or less than the ChatGPT answer) that may not actually be relevant to your research question. From your description, you already know that the response patterns are not similar if you only have one ChatGPT response that is fixed and human responses that vary: those are completely different distributions. (You might also check whether this assumption of yours is actually correct or if it's possible to get different ChatGPT answers) So, I would be careful about what your actual hypothesis is and make sure that descriptive statistics aren't a better fit. Not every question is appropriate for null hypothesis significance testing. If you have a large enough human sample, you will always find statistical significance eventually, so it's probably more important that you characterize how different the responses are.
answered Jul 15 at 13:20