Difference between revisions of "Sandbox"

From ChanceWiki
Jump to: navigation, search
Line 1: Line 1:
AIDS Vaccine
+
Disc Fragments
  
“Hardly ever believe what you read” is a maxim that will stand you in good steadGoogling “aids vaccine Thailand” will get 248,000 hits, most of which are misleadingIn essence, the URLs say that for the first time an effective vaccine against AIDS has been manufacturedBut that was last month. Reality has now set in.
+
The dream of every clinical trial is to come up with something which is inexpensive, definitive and likely to result in media publicity“Improved outcome after lumbar microdiscectomy in patients shown their excised disc fragments: a prospective, double blind, randomized, controlled trial” by M.J. Tait, et al [http://www.ncbi.nlm.nih.gov/pubmed/19684238 here] fulfills the desire.   
 +
 +
According to local Twin Cities website with the heading, [http://www.minnpost.com/healthblog/2009/09/03/11307/seeing_it_appears_is_believing_when_it_comes_to_back_surgery Seeing, it appears, is believing when it comes to back surgery]: “British surgeons report that patients who underwent a surgical procedure (lumbar microdiscectomy) for back pain caused by a spinal disc tear (“slipped disc”) had better outcomes when they received fragments of their removed disc after the operationThat’s right. Simply taking home a souvenir of the operation in a pot of saline solution improved the patients’ recovery. They reported less leg and back pain, less leg weakness and less “pins and needles” sensations (paresthesia). They also took fewer pain medications after the surgery.
  
The following chart found in [http://online.wsj.com/article/SB125511780864976689.html the Wall Street Journal of October 9, 2009] paints a different picture. “New infections occurred in 51 of the 8,197 people who got the vaccine, compared with 74 of the 8,198 volunteers who got placebo shots.”  Note that the “125” infections represent “51 + 74.”
+
The surgeons “said they decided to do the study for two main reasons: They knew that a patient’s anxiety and depression going into surgery for a spinal disc tear has a big impact on the recovery process. They had also noticed, anecdotally, that many of their patients who responded best to the surgery — and who seemed to experience the least anxiety and depression afterwards — were those who had been given their disc fragments.”
  
 +
The abstract of the journal article notes low p-values to make their case “that presenting the removed disc material to patients after LMD improves patient outcome.”:
  
<center> http://www.dartmouth.edu/~chance/forwiki/HiV.gif</center>
+
Lumbar microdiscectomy (LMD) is a commonly performed neurosurgical procedure. We set up a prospective, double blind, randomised, controlled trial to test the hypothesis that presenting the removed disc material to patients after LMD improves patient outcome. METHODS: Adult patients undergoing LMD for radiculopathy caused by a prolapsed intervertebral disc were randomised into one of two groups, termed experimental and control. Patients in the experimental group were given their removed disc fragments whereas patients in the control group were not. Patients were unaware of the trial hypothesis and investigators were blinded to patient group allocation. Outcome was assessed between 3 and 6 months after LMD. Primary outcome measures were the degree of improvement in sciatica and back pain reported by the patients. Secondary outcome measures were the degree of improvement in leg weakness, paraesthesia, numbness, walking distance and use of analgesia reported by the patients. RESULTS: Data from 38 patients in the experimental group and 36 patients in the control group were analysed. The two groups were matched for age, sex and preoperative symptoms. More patients in the experimental compared with the control group reported improvements in leg pain (91.5 vs 80.4%; p<0.05), back pain (86.1 vs 75.0%; p<0.05), limb weakness (90.5 vs 56.3%; p<0.02), paraesthesia (88 vs 61.9%; p<0.05) and reduced analgesic use (92.1 vs 69.4%; p<0.02) than preoperatively. CONCLUSION: Presentation of excised disc fragments is a cheap and effective way to improve outcome after LMD.
  
The announcement on September 24, 2009 indicated that the p-value is 3.9%A Minitab run shows that, in fact, the p-value is higher (i.e., worse) as indicated by the Fisher exact test.  However, the .048 is still under the mystical .05:
+
The entire paper is only three pages in length and so its calculations can be checkedBelow are the calculation results for the three secondary outcomes for which the paper claims statistical significance:
  
'''Test and CI for Two Proportions'''
+
1.  Improved Leg Weakness--the paper states that the p-value is less that .02.  Minitab shows that the p-value from Fisher’s exact test is .024.
  
<table width="50%" border="1">
+
Test and CI for Two Proportions     [leg weakness]
  <tr>
 
     <td><div align="center">Sample</div></td>
 
    <td><div align="center">X</div></td>
 
    <td><div align="center">N</div></td>
 
    <td><div align="center">Sample p</div></td>
 
  </tr>
 
  <tr>
 
    <td><div align="center">1</div></td>
 
    <td><div align="center">51</div></td>
 
    <td><div align="center">8197</div></td>
 
    <td><div align="center">0.006222</div></td>
 
  </tr>
 
  <tr>
 
    <td><div align="center">2</div></td>
 
    <td><div align="center">74</div></td>
 
    <td><div align="center">8198</div></td>
 
    <td><div align="center">0.009027</div></td>
 
  </tr>
 
</table>
 
  
Difference = p (1) - p (2)<br>
+
Sample  X  N      Sample p
Estimate for difference:  -0.00280480<br>
+
1             9 16      0.562500
95% CI for difference: (-0.00546736, -0.000142249)<br>
+
2           19 21      0.904762
Test for difference = 0 (vs not = 0):  Z = -2.06 P-Value = 0.039<br>
 
  
Fisher's exact test: P-Value = 0.048<br>
 
  
“Efficacy” of 31.2seems to be determined from<br>
+
Difference = p (1) - p (2)
(74 - 51)/ 74 = .310<br>
+
Estimate for difference:  -0.342262
 +
95% CI for difference: (-0.615844, -0.0686795)
 +
Test for difference = 0 (vs not = 0):  Z = -2.45  P-Value = 0.014
  
In the final column of the chart--“Strictly adheres to trial design”--appears the unreleased<br> “per protocol” version. According to<br> [http://blogs.sciencemag.org/scienceinsider/2009/10/unrevealed-anal.html Science Magazine]:
+
Fisher's exact test: P-Value = 0.024
  
::The second analysis is called “per protocol” and adheres strictly to how the trial was designed by only including the study participants who got the full regimen of vaccine shots at the right time. Because it excludes study participants who didn't get the full vaccine regimen, it usually provides corroboration to the looser “intent to treat” findings.
+
2. Parathaesia--The paper states that the p-value is less that .05.  Minitab shows that the p-value is from Fisher’s exact test is .08.
  
The article doesn’t say what the breakdown of the 86 infections is.  Nevertheless, it indicates that the p-value of 16% puts a damper on enthusiasm for the vaccine.
+
Test and CI for Two Proportions    [parathaesia]
  
::The press conference was not a scholarly, rigorously honest presentation,” said one leading HIV/AIDS investigator, who like others asked that his name not be used. “It doesn’t meet the standards that have been set for other trials, and it doesn’t fully present the borderline results. It’s wrong.
+
Sample  X  N    Sample p
 +
1            22  25    0.880000
 +
2            13  21    0.619048
 +
 
 +
 
 +
Difference = p (1) - p (2)
 +
Estimate for difference:  0.260952
 +
95% CI for difference:  (0.0173021, 0.504603)
 +
Test for difference = 0 (vs not = 0): Z = 2.10  P-Value = 0.036
 +
 
 +
Fisher's exact test: P-Value = 0.080
 +
 
 +
3. Reduced Analgesic Use--The paper states that the p-value is less that .02.  Minitab shows that the p-value from Fisher’s exact test is .017.
 +
 
 +
Test and CI for Two Proportions
 +
 
 +
Sample  X  N    Sample p
 +
1            35  38    0.921053
 +
2            25  36    0.694444
 +
 
 +
 
 +
Difference = p (1) - p (2)
 +
Estimate for difference:  0.226608
 +
95% CI for difference:  (0.0534229, 0.399793)
 +
Test for difference = 0 (vs not = 0):  Z = 2.49  P-Value = 0.013
 +
 
 +
Fisher's exact test: P-Value = 0.017
 +
 
 +
The primary outcomes, leg pain and (low) back pain for the treatment vs. the control were not calculated in a similar manner to the way the secondary outcomes were.  Instead of using a two-sample test of proportions, the results for “pain” were calculated by having five categories: “Much better,” Little better,” “Same,” “Little worse,” and “Much worse.”  That is, an ordinal scale was employed.  Because the accompanying graphs, Figure 1A and 1B in the paper, are not precise enough to determine the number in each category, a nonparametric calculation is hard to carry out.
 +
 
 +
Nevertheless, ignoring the breakdown into five categories, here are Minitab results for leg pain and back pain, respectively; note that the p-values are much different from the claimed <.05:
 +
 
 +
Test and CI for Two Proportions [leg pain]
 +
Sample  X  N    Sample p             
 +
1            35  38    0.921053
 +
2            29  36    0.805556
 +
 +
 
 +
Difference = p (1) - p (2)
 +
Estimate for difference:  0.115497
 +
95% CI for difference:  (-0.0396318, 0.270626)
 +
Test for difference = 0 (vs not = 0):  Z = 1.46  P-Value = 0.144
 +
 +
Fisher's exact test: P-Value = 0.185
 +
 
 +
Test and CI for Two Proportions [back pain]
 +
Sample  X  N    Sample p           
 +
1            33  38  0.868421
 +
2            27  36  0.750000
 +
 +
 
 +
Difference = p (1) - p (2)
 +
Estimate for difference:  0.118421
 +
95% CI for difference:  (-0.0592271, 0.296069)
 +
Test for difference = 0 (vs not = 0):  Z = 1.31  P-Value = 0.191
 +
 +
Fisher's exact test: P-Value = 0.242 
  
 
Discussion
 
Discussion
  
1.  “Strictly adheres to trial design” has an efficacy of 26.2% and 86 infections. Show that this leads approximately to 36 and 50 infections, respectively. 
+
1.  Why might an individual report a better outcome because he was handed his disc fragment?  Why might he feel worse?
 +
 
 +
2. Assuming that the p-values reported in the article are correct, what criticism might still remain?
  
2. The articles fail to tell us the number of participants in the “per protocol” situationHowever, use the 36 and 50 cited above and show via a statistics package such as Minitab that the Fisher exact test comes up with about 16% for the p-value regardless of whether the sample sizes are the original ones or 4000 each, 5000 each, etc.
+
3. A disc fragment is one form of excised body partWhat other excised body part might have a similar positive result?  What other excise body part might have a distinctly negative result?
  
3. The “researchers with the U.S. Army who helped run the study, strongly objected to the assertion that they gave the data a positive spin… The debate over the way the results were presented will have no immediate practical impact because even under the most optimistic assessment, the vaccine offered too little protection to be a serious candidate for widespread use. If this is so, why was there so much positive publicity in September?
+
4. This study took place in London, EnglandWhy might patient reaction be different in, let us say, Asia or Africa?

Revision as of 12:17, 18 October 2009

Disc Fragments

The dream of every clinical trial is to come up with something which is inexpensive, definitive and likely to result in media publicity. “Improved outcome after lumbar microdiscectomy in patients shown their excised disc fragments: a prospective, double blind, randomized, controlled trial” by M.J. Tait, et al here fulfills the desire.

According to local Twin Cities website with the heading, Seeing, it appears, is believing when it comes to back surgery: “British surgeons report that patients who underwent a surgical procedure (lumbar microdiscectomy) for back pain caused by a spinal disc tear (“slipped disc”) had better outcomes when they received fragments of their removed disc after the operation. That’s right. Simply taking home a souvenir of the operation in a pot of saline solution improved the patients’ recovery. They reported less leg and back pain, less leg weakness and less “pins and needles” sensations (paresthesia). They also took fewer pain medications after the surgery.”

The surgeons “said they decided to do the study for two main reasons: They knew that a patient’s anxiety and depression going into surgery for a spinal disc tear has a big impact on the recovery process. They had also noticed, anecdotally, that many of their patients who responded best to the surgery — and who seemed to experience the least anxiety and depression afterwards — were those who had been given their disc fragments.”

The abstract of the journal article notes low p-values to make their case “that presenting the removed disc material to patients after LMD improves patient outcome.”:

Lumbar microdiscectomy (LMD) is a commonly performed neurosurgical procedure. We set up a prospective, double blind, randomised, controlled trial to test the hypothesis that presenting the removed disc material to patients after LMD improves patient outcome. METHODS: Adult patients undergoing LMD for radiculopathy caused by a prolapsed intervertebral disc were randomised into one of two groups, termed experimental and control. Patients in the experimental group were given their removed disc fragments whereas patients in the control group were not. Patients were unaware of the trial hypothesis and investigators were blinded to patient group allocation. Outcome was assessed between 3 and 6 months after LMD. Primary outcome measures were the degree of improvement in sciatica and back pain reported by the patients. Secondary outcome measures were the degree of improvement in leg weakness, paraesthesia, numbness, walking distance and use of analgesia reported by the patients. RESULTS: Data from 38 patients in the experimental group and 36 patients in the control group were analysed. The two groups were matched for age, sex and preoperative symptoms. More patients in the experimental compared with the control group reported improvements in leg pain (91.5 vs 80.4%; p<0.05), back pain (86.1 vs 75.0%; p<0.05), limb weakness (90.5 vs 56.3%; p<0.02), paraesthesia (88 vs 61.9%; p<0.05) and reduced analgesic use (92.1 vs 69.4%; p<0.02) than preoperatively. CONCLUSION: Presentation of excised disc fragments is a cheap and effective way to improve outcome after LMD.

The entire paper is only three pages in length and so its calculations can be checked. Below are the calculation results for the three secondary outcomes for which the paper claims statistical significance:

1. Improved Leg Weakness--the paper states that the p-value is less that .02. Minitab shows that the p-value from Fisher’s exact test is .024.

Test and CI for Two Proportions [leg weakness]

Sample X N Sample p 1 9 16 0.562500 2 19 21 0.904762


Difference = p (1) - p (2) Estimate for difference: -0.342262 95% CI for difference: (-0.615844, -0.0686795) Test for difference = 0 (vs not = 0): Z = -2.45 P-Value = 0.014

Fisher's exact test: P-Value = 0.024

2. Parathaesia--The paper states that the p-value is less that .05. Minitab shows that the p-value is from Fisher’s exact test is .08.

Test and CI for Two Proportions [parathaesia]

Sample X N Sample p 1 22 25 0.880000 2 13 21 0.619048


Difference = p (1) - p (2) Estimate for difference: 0.260952 95% CI for difference: (0.0173021, 0.504603) Test for difference = 0 (vs not = 0): Z = 2.10 P-Value = 0.036

Fisher's exact test: P-Value = 0.080

3. Reduced Analgesic Use--The paper states that the p-value is less that .02. Minitab shows that the p-value from Fisher’s exact test is .017.

Test and CI for Two Proportions

Sample X N Sample p 1 35 38 0.921053 2 25 36 0.694444


Difference = p (1) - p (2) Estimate for difference: 0.226608 95% CI for difference: (0.0534229, 0.399793) Test for difference = 0 (vs not = 0): Z = 2.49 P-Value = 0.013

Fisher's exact test: P-Value = 0.017

The primary outcomes, leg pain and (low) back pain for the treatment vs. the control were not calculated in a similar manner to the way the secondary outcomes were. Instead of using a two-sample test of proportions, the results for “pain” were calculated by having five categories: “Much better,” Little better,” “Same,” “Little worse,” and “Much worse.” That is, an ordinal scale was employed. Because the accompanying graphs, Figure 1A and 1B in the paper, are not precise enough to determine the number in each category, a nonparametric calculation is hard to carry out.

Nevertheless, ignoring the breakdown into five categories, here are Minitab results for leg pain and back pain, respectively; note that the p-values are much different from the claimed <.05:

Test and CI for Two Proportions [leg pain] Sample X N Sample p 1 35 38 0.921053 2 29 36 0.805556


Difference = p (1) - p (2) Estimate for difference: 0.115497 95% CI for difference: (-0.0396318, 0.270626) Test for difference = 0 (vs not = 0): Z = 1.46 P-Value = 0.144

Fisher's exact test: P-Value = 0.185

Test and CI for Two Proportions [back pain] Sample X N Sample p 1 33 38 0.868421 2 27 36 0.750000


Difference = p (1) - p (2) Estimate for difference: 0.118421 95% CI for difference: (-0.0592271, 0.296069) Test for difference = 0 (vs not = 0): Z = 1.31 P-Value = 0.191

Fisher's exact test: P-Value = 0.242

Discussion

1. Why might an individual report a better outcome because he was handed his disc fragment? Why might he feel worse?

2. Assuming that the p-values reported in the article are correct, what criticism might still remain?

3. A disc fragment is one form of excised body part. What other excised body part might have a similar positive result? What other excise body part might have a distinctly negative result?

4. This study took place in London, England. Why might patient reaction be different in, let us say, Asia or Africa?