Participant sampling for online studies

June 1, 2010

(partly paraphrased)

Number of participants
We recommend at least 50-100 participants at a minimum per distinct user segment. (…) Whether or not you’ll see a difference [between smaller and larger groups] depends on the following:

  • Magnitude of the effect; highly significant flaws will be easily detected in small samples
  • Significance level, aka alpha level, power, confidence level, ‘probability level’ and expressed in numbers such as ‘p<0.01'; This is the degree to which any differences seen may be due to chance. It's basically how much error you are willing to sneak into your data as noise
  • Repeated measures/pairing; ‘within subjects’ studies only
  • Expected variation of the sample; refers to how differently you expect participants to behave. If there is a wide variety of user types and characteristics, there is likely to be a larger variance and therefore more participants will be needed. […] Smaller variance in data will lend itself better to seing any statistical differences between versions of a design.

Sampling techniques

Use probability sampling techniques if all unique members of the population in question is known.

  • Simple random sampling: you take a random sample by some means, eg. excel formula or a simple ‘pick out of a hat’ lottery
  • Stratified sampling: Form meaningful groups out of your sample and then select individuals randomly from each group. You will get a representation from each ‘strata’
  • Systematic (random) sampling: Put members in a random order and then pick every *nth* individual
  • Multistage sampling: Take different samples using different techniques in order to eliminate bias

Use nonprobability sampling techniques if you do not know every unique member, i.e. members do not have an equl chance to be invited to the sample.

  • Convenience sampling: Invite as many as you can; introduces bias ‘by design’.
  • Snowball sampling: a flavour of convenience sampling where invited individuals invite other participants and so on to create a pyramid effect.
  • Quota sampling: Set up meaningful groups and use convenience sampling to get representatives from each group

Albert, Tullis, Tedesco (2010), p.42-44

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s