Sample Size Considerations

Note:  This is the first in a series of six posts on sampling, sampling size, and sampling methods.

I am frequently asked how large a sample should be, and there is usually no perfect answer. Instead, there is a long list of questions and considerations to take into account. Taking some time to talk these issues over with a statistician or sampling expert will give you a sample that will get closer to perfect.

Representativeness

Before talking about size of the sample per se, I just want to emphasize that the end goal of any sample is representativeness of the population under study. The population is everyone with a given set of characteristics (kindergarten children in Chicago, infants born in Lake County, plants that grow in Michigan ponds, etc.). A sample is a small selection of those individuals or things. 

We often need to sample because we do not have the resources to get information on all persons or things in a population.  And sometimes, if we know a lot about the population already, we do not need to gather data from the whole population to learn what we want to know - a sample will do.

A good sample will always represent the population it is intended to. That means it should be just as complex and should represent all the varieties of individuals in the population it represents. If the age of the population fits along a normal curve (below), your goal should be that your sample would fit the same curve. You can see in Figure 1, only one of the samples comes close to representing the population reasonably well.

Part of achieving representativeness has to do with how much we already know about the population under study. Researchers now know so much about American voting behavior that a sample of 500 voters, if selected in a truly random way, typically represents well the voting preferences of all of the voters in United States.

But there are not many phenomena we know so much about. Most of the time, we struggle to decide if our sample is large enough and if it actually represents well the population we care about. And the biggest issue is that we will not know the answers to these questions until our data are collected.

Key considerations

These are the questions that go through my mind when someone asks me about sample size.

1. Do you need to sample, or should you just use the whole population? In pediatrics, the population of interest can be quite small, especially with the rare genetic conditions children may have. When working with a small population, each case you miss might impact the representativeness of the sample. In that case, you probably would not want to exclude anyone from your sample, and instead try to capture the whole population in a region or in your clinical practice.

2. As a very rough rule of thumb, statistical procedures require a sample of about 50 to produce reliable results when comparing two groups. You never want to go below that. If you want to compare responses to a survey between girls and boys, you would need 25 girls and 25 boys in your sample (50 in total). If your questions and comparisons are more complicated, you need to go higher (unless the population is very small to start with, see #1).

3. How much do you know about the population you want to study? The less you know, the larger you will want your sample to be. This is because once you get into the analysis, you may learn things that you did not expect to and may then want to make more complex comparisons. A larger sample gives you more room to do this, as well as more space to deal with any kind of unexpected finding.

4. Do you want to say a lot about the population, or just one thing? You need a smaller sample to estimate the percent of individuals with blue eyes than to estimate the percent of individuals with blue eyes, brown hair, freckles and purple shoes. Every additional characteristic you want to compare requires additional sample.

5. Is the thing you are studying quite variable? The higher the variance in key metrics under study, the more sample size you will need to show a difference or make a comparison. Studying health outcomes over ten years would require a larger sample than a two-week follow up because of the huge variance in possible outcomes that could occur over such a long time. Or studying relatively short term outcomes for a condition like cerebral palsy would require more sample because the outcomes are incredibly variable, unlike outcomes from chicken pox, for instance.

6. Do you want to show a statistically significant difference between two groups or the same group over time? If so, you will want to conduct what is called a ‘power analysis.’ Such a procedure provides you with the minimal sample size required to show statistical significance, with (usually) an 80% probability.

7. If you are tracking a single group over time, will you be able link an individual’s data from the first time point to later ones? If you can, then you may get by with a smaller sample because the variance will be more limited than if you cannot actually link the two time points for each participant. (Conducting a paired t-test requires less sample than an independent group t-test.)

8. Consider the cost of obtaining a representative sample. Some groups are very hard to sample and will cost a lot more. For example, if a part of your sample do not have phones to answer a survey, you will need to hire a survey administrator to go to the participants’ residences to get the responses. This would drive the cost of your survey up, but help make sure it is representative. A smaller representative sample is much better than a larger unrepresentative one, no matter what you are trying to learn.

I hope this gives you some ideas to chew on as you think through sample size issues. It is always a good idea to talk these issues out with a sampling expert. If you do not have access to one, spend some time in the research literature. You may get guidance from others who have conducted a similar study with a population similar to yours.

Next time: Power analysis.

 

---