The use of exclusion criteria is commonplace in psychiatric research. Researchers justify these selective criteria by noting that they increase the study’s internal validity by producing a study population that is targeted and appropriate. A new study, published in the April American Journal of Psychiatry, has shown that while exclusion criteria may optimize internal validity, they may also negatively affect a study’s external validity, that is, the ability to generalize the study’s results to the clinical setting.

The new study looked at the use of exclusion criteria in alcohol treatment outcome studies. The most common exclusion criteria found in alcohol studies include concurrent psychiatric/emotional problems, concurrent medical problems, drug dependence, noncompliance/lack of motivation, unsuccessful prior alcohol treatment, living a great distance from a treatment facility, residential instability, and social instability.

The research was conducted by Keith Humphreys, Ph.D., an assistant professor (research) of psychiatry at Stanford University, and Constance Weisner, Dr.P.H., an adjunct professor of public health at the University of California at Berkeley. The investigators followed 593 patients seeking alcohol treatment at one of eight participating treatment programs. The clinical setting included patients from both public and private treatment programs. The clinical population was analyzed and matched to the eight criteria listed above. The criteria were then used to produce a valid hypothetical research population. The resulting research population was compared with the clinical population with regard to sex, race, income, and the presence of additional drug, alcohol, and/or psychiatric problems.

Humphreys and Weisner reported that exclusion criteria did significantly alter the makeup of the research population from that of the 593 patients who were seeking treatment in the clinical setting. In particular, the exclusion criteria produced research samples in which African-American, low-income, and severely disabled patients were significantly underrepresented. All three of these variables have previously been shown to affect alcohol treatment outcome.

The differences between included and excluded patients were more pronounced in the private clinical setting; however, they were significantly different in the public setting as well. Using the commonly accepted criteria, African-American, low-income, and severely disabled patients were present in the clinical setting, but largely excluded from the research population. By employing the accepted exclusion criteria, subjects were excluded from the study who have been shown to be at higher risk for the disorder being researched.

Outcome treatment studies are meant to validate particular methods of treating patients with specific diagnoses. This effort leads to the development of diagnosis-driven, efficacious treatments. These treatment plans are a major recent advance in psychiatry. By matching a patient with a specific diagnosis to a proven treatment, chances of a successful outcome are maximized.

The current study sheds doubt on the ability of researchers to generalize the results of their treatment outcome studies, specifically with regard to alcohol treatment, to the population encountered in the clinical setting. This produces a significant barrier to the development of appropriate diagnosis-specific, proven efficacious treatments.

Humphries and Weisner noted that if the clinical population is significantly different from the research population, it may follow that the treatment outcome in the two groups would also be different. This difference is often referred to as the "research-practice gap."

The authors cautioned that it is unlikely that alcohol treatment research is the only area where exclusion criteria may create a research-practice gap. They urge all investigators to include systematic analysis in all fields of treatment research in order to protect generalization of results.