Participant sampling for online studies

(partly paraphrased)

Number of participants
We recommend at least 50-100 participants at a minimum per distinct user segment. (…) Whether or not you’ll see a difference [between smaller and larger groups] depends on the following:

  • Magnitude of the effect; highly significant flaws will be easily detected in small samples
  • Significance level, aka alpha level, power, confidence level, ‘probability level’ and expressed in numbers such as ‘p<0.01'; This is the degree to which any differences seen may be due to chance. It's basically how much error you are willing to sneak into your data as noise
  • Repeated measures/pairing; ‘within subjects’ studies only
  • Expected variation of the sample; refers to how differently you expect participants to behave. If there is a wide variety of user types and characteristics, there is likely to be a larger variance and therefore more participants will be needed. […] Smaller variance in data will lend itself better to seing any statistical differences between versions of a design.

Sampling techniques

Use probability sampling techniques if all unique members of the population in question is known.

  • Simple random sampling: you take a random sample by some means, eg. excel formula or a simple ‘pick out of a hat’ lottery
  • Stratified sampling: Form meaningful groups out of your sample and then select individuals randomly from each group. You will get a representation from each ‘strata’
  • Systematic (random) sampling: Put members in a random order and then pick every *nth* individual
  • Multistage sampling: Take different samples using different techniques in order to eliminate bias

Use nonprobability sampling techniques if you do not know every unique member, i.e. members do not have an equl chance to be invited to the sample.

  • Convenience sampling: Invite as many as you can; introduces bias ‘by design’.
  • Snowball sampling: a flavour of convenience sampling where invited individuals invite other participants and so on to create a pyramid effect.
  • Quota sampling: Set up meaningful groups and use convenience sampling to get representatives from each group

Albert, Tullis, Tedesco (2010), p.42-44