Conducting a power analysis

 

For some reason, power analysis has a certain cloud of mystery around it.  The goal of this post is to take the mystery out of this key study preparation tool.

What is power analysis?

Power analysis is a statistical procedure that computes the likelihood that the study you are setting up will result in findings that would be similar to those of the population. A .8 power tells you that if you conducted your study over and over again, 80% of the time it would result in the same findings as the population. Where statistical significance testing examines the extent to which the two groups you are comparing are different, power analysis produces a probability that the difference you observe actually exists in the population (probability of a Type II error in hypothesis testing).

Don't forget that a sample often has quite a different mean and distribution than the population (Figure 1), so in layman's terms, power analysis is tool to understand how big that difference can be without misleading you.

Power is strongly associated with sample size, so many researchers and quality improvement specialists use power analysis as a means of estimating a sample size that is associated with the power they would like their analysis or study to have.  And this is the use we will make of power analysis in this post.

Figure 1. Potential population and sample distribution differences (Source: Master Black Belt, http://slideplayer.com/slide/7488193/)

 

Estimating sample size with power analysis

First, before getting to power analysis, let me start by saying that the easiest way to think of estimating a sample size (in my mind) is to think of it as working backwards from your study results to your sample size.  There is more complexity than that, but if you have ever run a t-test or chi-square, you know that sample size is very important to whether or not your analysis produces a statistically significant finding.  Think of solving for sample size from a desired t-test result. 

Having said this, power analysis can be a lot more complicated because your final analysis might be a lot more complicated than a t-test.  But the principles are the same.  All the things that affect your results when you are doing your analysis (e.g., the number of co-variates, use of hierarchical sampling, high variability in the dependent variable) will also play into estimating your needed sample size.

What this post provides you with are the basic principles of power analysis and an Excel spreadsheet with formulas built in to you can roll up your sleeves and give it a try.  But before concluding that my estimated sample size is the one to go with, I always check-in with a statistician, because it can be far more complicated than this Excel tool suggests.

Principles of power analysis

Let's start with a power analysis for a t-test (a comparison of two means). The t-test is one of the first procedures we reach for and is the basis for most statistical procedures using linear data.

As you know from the basic statistics posts in the blog, statistics revolves around the twin notions of central tendency and variability.  Tests of statistical significance consider both of these, and so does power analysis.  There are several things you need to know or to estimate in order to compute a sample size:

1. The anticipated mean of the control group (or pretest). This may be available from your own previous studies, or from similar studies in the research literature.
2. The standard deviation of the mean of the control group.  This is the measure of variability around your mean.  Like the mean, this may be available from your previous studies, from pilot studies or from the research literature.
3. The anticipated mean of the treatment or demonstration group (or post-test).
4. The statistical significance level you will use in your study.
5. The power that you expect to use.  It is usually set at .8, but can be set as low as .7 and as high as .9 (it depends on assumptions you make about your study).
6. Your knowledge of the outcome and population under study.

When to use power analysis

• For estimating the sample size: Power analysis is extremely useful to estimate how large your recruitment effort needs to be for a certain study.  Collecting data is expensive.  If you can show an important difference with fewer data, then it makes sense to do so.

Power analysis also helps you understand the nature of the variability in your sample, so that you have a better handle on the sort of heterogeneity you may see in your sample.  It is a great learning tool.

• For estimating the population size: In addition to being used to estimate sample size, power analysis can be used to estimate population size you will need to compute a 'precise' mean. Many times in pediatrics, diagnoses are rare.  It may be that a single clinic sees all the cases of a certain condition in a region, so that the mean of the clinic patients is essentially the mean of the population. But the variability of the population can be quite wide.  How large would the population need to be to have a reasonably 'precise' mean?

Another instance of the need for a population power analysis would be when there is an emerging health problem (like a new virus causing hospitalizations).  You might want to know how many cases will need to be seen before you have your arms around the probabilities of different outcomes.  For example, when HIV-AIDs and Ebola began to occur in the United States, it took several months or even years of cases for epidemiologists to advise clinicians and families on the range of outcomes for the diseases.

Finally, there may be times you are tracking the population over time, and you want to set some expectations on what the range of the mean of the population could be in the future.  Power analysis can provide the population size you will need to avoid making too strong of an interpretation when a mean falls outside the confidence intervals.

Sample size calculations

UseSample Size Determination simplified v1.xlsx to play with sample size issues.  The calculator provided here offers sample size calculations for the most common situations:

1. Mean of a paired sample: This calculation is used when you are comparing means for the same study participants over time, and where the data are identified enough for you to link the pretest and the posttest. 
2. Mean of two independent samples: Use this when you have two separate samples (different patients are in each sample) and you need to compare means. If you have not sampled the same ratio of participants from each group, an additional factor need to be added, but this formula is a good start.
3. A proportion from a sample: Use this when you are comparing the proportions (percent) of two different samples.
4. Confidence interval of a mean of a population: Use this when you want to anticipate the population size required for a certain the confidence interval for a population mean.
5. Confidence interval for a proportion of a population: Use this when you want to anticipate the population size required for a certain the confidence interval for a population proportion (percent).

Some caveats

Power analysis presumes that you have employed random sampling, which is very hard to do in many clinical situations. If you cannot use random sampling, or your design involves stratified or clustered sampling, a statistician will be the person you reach out to in order to estimate the sample size you will need. 

Power analysis is something that I play with a lot when I am designing a study (it gives mean lot of information that helps me lay out a strong design), but I always have a statistician review my assumptions and calculations before I begin work or submit a research proposal. Even if you are very confident in your ability to use power analysis to estimate your needed sample size, sample size is so critical to your study that a second set of eyes is worthwhile.  

 

---