Posted by Larry Hoover on January 27, 2009, at 17:59:31
In reply to Re: Stats...ooops, posted by linkadge on January 27, 2009, at 10:59:37
There are two sequential statistical procedures used in this sort of study. You calculate the average change resulting from the drug and placebo groups, and then you determine if the difference between groups (if any) is large enough to be unlikely to have occurred by chance.
Here is an image showing difference in mean change between the drug and placebo arms from the published and unplished antidepressant trials recently studied by Kirsch:
The zero line across the chart indicates no difference between the groups. Plots below the zero line indicate that placebo was superior. Plots above the zero line indicate that the drug was superior. Don't be confused by the dark horizontal line at 3. That's not relevant at this point. What is very clear is that far more often than not, drug response was superior to placebo, in absolute terms.
The second step is to determine if the difference is significant. That's really an unfortunate choice of language, because it has nothing to do with how meaningful the difference is. What significance means is nothing more than determining if the observed difference in group means is or is not likely to be due to chance, with typical significance thresholds set at 5% or less that chance findings explain the difference.
The basic assumption in all such studies is that there is no difference. That's called the null hypothesis. In order to reject the null hypothesis, calculated differences in group means must exceed a specific threshold, taking into account variability in the size of the groups, and the variability in scores within each group. There are tables of standard values with thresholds for 95% and 99% probabilities that the observed difference is a true difference, for different sizes of groups.
This article references an earlier Kirsch meta-analysis, using most of the same data as the table I provided, which found that 57% of the time the difference between the placebo and drug response was not large enough, taking into account the size and variability in individual response within the two groups, to reach the threshold required to reject the null hypothesis. If you go back to that image I linked to earlier, now that line at 3 has meaning. It is a slightly different representation of the threshold of statistical significance, but it's similar enough. Above that line, the difference is significant, i.e. drug was better than placebo, with strong likelihood that the observed difference is not due to chance (I count 16 such studies). There are no studies in which the contrary finding is true, i.e. no studies indicated that placebo was superior to drug (nothing below -3). For all the rest, those below the line at difference = 3 and above difference = -3, the null hypothesis was not rejected. For those studies (57% according to Kirsch), the assumption that there was no difference is not invalidated STATISTICALLY. The chart clearly shows otherwise, IMHO, but not to the standard required by science.
Here's a link that shows graphical representations of the influence of within-group variability on the test for a significant difference between two groups:
http://www.socialresearchmethods.net/kb/stat_t.phpIt doesn't matter if you use a t-test, as the latter link demonstrates, or ANOVA. The results should be identical.
Lar
poster:Larry Hoover
thread:876214
URL: http://www.dr-bob.org/babble/20090104/msgs/876620.html