AIDS vaccine numbers off, statistician says
Effectiveness for minorities may be overstated
Andrew Pollack, New York Times
VaxGen may have overstated the effectiveness of its AIDS vaccine
because it did not make the proper statistical adjustments to its
data, an expert consulted by the company said Wednesday.
Steven Self, a professor of biostatistics at the University of
Washington, said the company should have lowered the level of
confidence with which it said the vaccine appeared to protect blacks,
Asians and other non-Hispanic minorities from infection by HIV.
Dr. Donald Francis, the president of VaxGen, said there was some
debate among statisticians about the proper adjustments but called it
"a tangential issue." Even with the adjustments, he said, the results
showing efficacy in minorities would still be statistically
significant.
VaxGen reported Monday that its vaccine was ineffective overall in a
trial of 5,400 participants. But it said in a subset of 500
non-Hispanic minorities, the vaccine reduced infection by 66.8
percent, a statistically significant result. The company has since
been criticized by outside scientists and AIDS activists for stressing
conclusions based on a small number of patients.
VaxGen reported that its finding for the minority subgroup had a
confidence interval of 30.2 percent to 84.2 percent. That meant there
was a 95 percent probability that the actual efficacy of the vaccine
in the subgroup was between those points.
But Self said that by his calculation, the low end of the confidence
interval should be just above zero. That is because the company should
have lowered its confidence to account for the fact that it analyzed
multiple subgroups of patients. If the data are cut enough ways, some
effectiveness can almost always be found, he said, so such "penalties"
to the interval are taken to guard against false positive conclusions.
Self said VaxGen asked him for advice after other statisticians began
criticizing it for failing to make such adjustments.
"It’s probably an honest error," he said. But he said the fact that
the lower bound was still above zero still provided "some marginal
statistical evidence that there is some efficacy in that subgroup."
VaxGen’s Francis said the important thing now is to determine whether
there is some biological explanation for why the vaccine appeared to
work in some groups.