Sample size for a survey or questionnaire study: what to power for and what to report

Start with one comparison

A questionnaire study can support dozens of comparisons, and a sample size cannot be calculated for all of them at once. Choose the primary one: the outcome score, the groups or exposure being compared, and the statistical test that will answer it. Everything else in the study is secondary or exploratory and does not drive the sample size, although it should be labelled that way in the analysis plan.

If the study is descriptive, for example estimating the prevalence of a symptom or the mean score in a population, the calculation targets the precision of that estimate instead. You choose how wide a confidence interval you can accept and solve for the sample that delivers it.

Choose an effect size you can defend

The effect size is the input that matters most and the one most often chosen carelessly. For a validated instrument, the best anchor is a published minimal clinically important difference for that score. Failing that, use the difference or correlation reported in prior studies of the same population. A generic small, medium, or large effect from a textbook is the weakest choice and should be a last resort, stated as such.

Along with the effect size you need its scale: the standard deviation of the score in a comparable sample for a continuous outcome, or the expected proportions in each group for a binary one. Instrument development papers and prior studies usually supply these.

Inflate for the realities of survey data

The number the calculator returns is the number of analyzable responses, not the number of people to invite. Divide by the expected response rate to get invitations. Then allow for participants who start but do not finish, and for questionnaires that cannot be scored because too many items are missing. Response and completion rates from your own pilot or from a comparable study in the same setting are the right inputs.

If participants were recruited in groups, such as clinics, teams, or classrooms, responses within a group tend to be more alike than responses across groups. That clustering reduces the effective sample size and the calculation should account for it with a design effect based on the expected within-cluster correlation and cluster size.

• Analyzable responses from the power calculation.
• Divided by the expected response rate for the number to invite.
• Plus an allowance for incomplete and unscorable questionnaires.
• Times a design effect if recruitment was clustered.

Validation studies follow different rules

If the purpose is to validate or adapt an instrument rather than to test a hypothesis, the sample size is governed by the psychometric methods. Factor analysis and item response models need larger samples than a group comparison, and the guidance is usually stated as a ratio of participants to items or as an absolute minimum. Reviews of published validation studies show wide variation in practice, so cite the specific guidance you followed.

Write it into the methods

One paragraph is enough: the primary outcome and comparison, the effect size and where it came from, the standard deviation or proportions used, alpha and power, the resulting analyzable sample, the inflation factors applied, and the software or formula. A reader should be able to reproduce the number from that paragraph alone. If the study was completed with fewer participants than planned, say so and report the achieved power or, better, focus on the confidence intervals.