Preregistration questions sample/outliers

Question

HI I'm extremely new to research and my professor asked me to complete a preregistration for a project he wants to co author. However, I am stuck because the preregistration he asked me to complete requests a sample size and how I will define outliers.

The study is a pretty straight forward moderation model with each of the three variables measured by a psychometric assessment. I have been trying to teach myself more about statistics and how to define outliers using a median absolute deviation and determine a sample size a priori using gpower software but I'm in over my head and keep running into walls. Like the gpower method requires you a desired effect size and due to my lack of experience I'm struggling to know what is a "good" effect size for a moderation model? How would you determine that? I'm trying to look at other studies using moderation models but I'm still quite lost. As for outliers I think I might just be overthinking this as every other preregistration I've referenced has only defined them using criteria based on completion time, and incomplete data, I thought I might need to use some more rigorous qualifiers based on like distance from the mean or median.

I feel like I may be overthinking a lot of this but any direction would be greatly appreciated. Thank you.

Answer 1

In leu of a proper answer, here are a few quick thoughts for you to consider.

Power analysis

Power analysis is a notoriously tricky thing to do well. Your efforts to gather information on effect sizes from previous research make sense to me. Look for other analyses that have used similar variables in similar populations (as close as you can get). Sometimes, you'll need to translate this into an effect that can be read by G*Power (I'm glancing through my options in G*Power wondering which one you'd use for a moderation effect: maybe The $F$ test for $R^2$ increase?). You could always just use a default "moderate" effect size (many do), though this reduces the reliability of the power analysis estimates. If you're new to this, consider using at most an $\alpha = 0.05$ and least a power of $0.80$.

I'll admit, though, that I'm biased against such canned power analyses (perhaps another contributor who is a fan could provide you with better guidance). They make a lot of implicit assumptions. The only ones I've ever trusted were simulations based on empirical data, and even those I take with a grain of salt.

Outliers

There's several ways to do outlier detection, depending on your data collection method. I'd a) specify a reasonable range per variable (i.e., an age of 135 would be excluded), b) specify some details regarding the data collection procedure (this is what you reference), and c) consider if an analytic method that you have access to can handle outlier removal.

There are some classic outlier removal methods (e.g., 1.5*IQR or standardized residuals >3), but there are more modern "robust" methods that diminishes the impact of outliers without complete removal. Some (including myself) believe that should a datapoint be validated and within the plausible range (defined in a) for a variable, removal of the outlier is questionable. In these cases, perhaps you can consider presenting the analyses both ways: using analysis that includes the outlier(s), and using an analysis that removed/diminished it/them.