I was quoting the statistics, I wasn't pretenting to be a statisitcian: Difference between revisions

From ChanceWiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(6 intermediate revisions by the same user not shown)
Line 2: Line 2:
BBC News, 15 July 2005
BBC News, 15 July 2005


[http://pass.maths.org.uk/issue21/features/clark/  Beyound reasonable doubt]<br>
[http://pass.maths.org.uk/issue21/features/clark/  Beyond reasonable doubt]<br>
Plus Magazine, 2002<br>
Plus Magazine, 2002<br>
Helen Joyce
Helen Joyce
Line 9: Line 9:
Paediatric and Perinatal Epidemiology 2004, 18, 320-326<br>
Paediatric and Perinatal Epidemiology 2004, 18, 320-326<br>
Roy Hill
Roy Hill
___________________________________________________________________________________
___________________________________________________________________________________


Sir Roy Meadow is a pediatrician well known for his research in child abuse.  The  BBC article reports that the UK General Medical Council (GMC) has found Sir Roy guilty of serious professional misconduct has "struck him off" the medical registry which if upheld under appeal will prevent Meadow from practicing  medicine in the UK.
Sir Roy Meadow is a pediatrician well known for his research in child abuse.  The  BBC article reports that the UK General Medical Council (GMC) has found Sir Roy guilty of serious professional misconduct has "struck him off" the medical registry.  If upheld under appeal, this will prevent Meadow from practicing  medicine in the UK.


This decision was based on flawed statistical estimates that Meadow made while testifying as an expert witness in a 1999 case in which a Sally Clark was found guilty of murdering her two baby boys and given a lifetime sentence.
This decision was based on flawed statistical estimate that Meadow made while testifying as an expert witness in a 1999 court case in which a Sally Clark was found guilty of murdering her two baby boys and given a life sentence.


The first death was recordered as a natural cot death.  Å "cot death" or "crib death" are other names for SIDS (sudden infant death syndrome).   
The death of Sally Clark's first child was recorded as a natural cot death.  Å "cot death" is other names for SIDS (sudden infant death syndrome).   


The name SIDS was proposed by the pathologist Bruce Beckwih at a conference in 1969 and the definition, which is still current, was formulated at the conference by Beckwith and others as follows:  
The name SIDS was proposed by the pathologist Bruce Beckwih at a conference in 1969 and the definition, which is still current, was formulated at the conference by Beckwith and others as follows:  
<blockquote>The suddent death of a baby that is unexpected by history and in whom a<br> thorough post-mortem examination fails to demonstrate an adequate cause of death. </blockquote>


Roy Meadow testified in Sally Clark's trial and stated that chance of two natural cot deaths was 1 in 73 million suggesting that two natural cot deaths was beyond belief.  This estimate came from a study called the "Confidential Inquiry for Stillbirths and Deaths in Infancy" (CESDI). The study gave detailed information about the deaths of all babies in five regions of England between 1993 and 1996.  The report of the study estimated that the chance of a cot death was 1 in 1,303. But if the child was from an afluint non-smoking family with the mother aged over 26, then the chance decreased to 1 in 8,543. Since Sally Clark was in the this group, Meadow assumed that the chance of the first child was a cot death was 1/8,543 and squaring this he obtained the 1 in 73  million estimate for the chance of two cot deaths.  
<blockquote>The sudden death of a baby that is unexpected by history and in whom a<br> thorough post-mortem examination fails to demonstrate an adequate cause of death. </blockquote>
 
Testifying in Sally Clark's trial Meadow stated that chance of two natural cot deaths was 1 in 73 million suggesting that two natural cot deaths was beyond belief.  This estimate came from a study called the "Confidential Inquiry for Stillbirths and Deaths in Infancy" (CESDI). The study gave detailed information about the deaths of all babies in five regions of England between 1993 and 1996.  Using this data the study estimated that the chance of a cot death was 1 in 1,303. But if the child was from an affluent non-smoking family with the mother aged over 26, then the chance decreased to 1 in 8,543. Since Sally Clark was in the this group, Meadow assumed that the chance of the first child was a cot death was 1/8,543 and squaring this he obtained the 1 in 73  million estimate for the chance of two cot deaths. Giving evidence, Sir Roy defended the calculations he used to arrive at the 73 million to one figure.  He compared the odds of two cot deaths in the same family to that of a punter successfully backing an 80-1 shot at the Grand National four years in a row.
 
There are three obvious  problems with this 1 in 73 million statistic: (1)  Because of environmental and genetical effects it seems very unlikely that the a SIDS death for a families the first baby and for their second baby are independent and (2) giving such odds with no further explanation mighr suggest to the jury that their is a 1 in 74 million chance that Sally Clark is innocent and (3) neither the defence or the procecustion suggested that the deaths were SIDS deaths--the proceducion claiming that they were murders, and the defence that they were natural deaths explained by the medical evidence.  Ås far as we can tell only problem (1) was seriously discussed in the news.  The first  article to discuss this issue appears to be the article  "False statistics may have led to solictor;s murder conviction" by Sherry Norton in The Independent (London), December 31, 1999. This article reports on and article  by ÍDr Stephen Watkins, director of public health at Stockport health entitled "Conviction by mathematical error? (BMJ, 2000, 320, 2-3).  Norton writes:
 
<blockquote>In Britain, there are 344 cot deaths each year, with the risk for the whole population being one cot death in 2.75 million. There are 378,000 second or subsequent births in England each year, and Dr Watkins has calculated that, if cot death is random, two such deaths will occur in the same family somewhere in England once every seven years.
 
However, because cot deaths are not random and can be linked to genetic defects, social class and lifestyle, recurrence rates have been found to be about five times the general rate. Therefore Dr Watkins has concluded that two cot deaths in the same family will, on average, occur at least once every one and a half years. </blockquote>
 
 
 


Of cource, soon after the trial it was pointed out that there are a number of problems with this estimate. These were nicely spelled out by a letter from the Royal Statistical Society to the Lord Chanceller dated 23 January 2002. This letter included the following discussion of the statistical issues involved:
These were nicely spelled out by a letter from the Royal Statistical Society to the Lord Chanceller dated 23 January 2002. This letter included the following discussion of the statistical issues involved:


<blockquote>You will be aware of the considerable public attention aroused by the recent conviction, confirmed on appeal, of Sally Clark for the murder of her two infants. One focus of the public attention was the statistical evidence given by a medical expert witness, who drew on a published study to obtain an estimate of the frequency of sudden infant death syndrome (SIDS, or "cot death") in families having some of the characteristics of the defendant's family. The witness went on to square this estimate to obtain a value of 1 in 73 million for the frequency of two cases of SIDS in such a family. This figure had an immediate and dramatic impact on all media reports of the trial, and it is difficult to believe that it did not also influence jurors.<br><br>
<blockquote>You will be aware of the considerable public attention aroused by the recent conviction, confirmed on appeal, of Sally Clark for the murder of her two infants. One focus of the public attention was the statistical evidence given by a medical expert witness, who drew on a published study to obtain an estimate of the frequency of sudden infant death syndrome (SIDS, or "cot death") in families having some of the characteristics of the defendant's family. The witness went on to square this estimate to obtain a value of 1 in 73 million for the frequency of two cases of SIDS in such a family. This figure had an immediate and dramatic impact on all media reports of the trial, and it is difficult to believe that it did not also influence jurors.<br><br>
 
The calculation leading to 1 in 73 million is invalid. It would only be valid if SIDS cases arose independently within families, an assumption that would need to be justified empirically. Not only was no such empirical justification provided in the case, but there are very strong reasons for supposing that the assumption is false. There may well be unknown genetic or environmental factors that predispose families to SIDS, so that a second case within the family becomes much more likely than would be a case in another, apparently similar, family.<br><br>
The calculation leading to 1 in 73 million is invalid. It would only be valid if SIDS cases arose independently within families, an assumption that would need to be justified empirically. Not only was no such empirical justification provided in the case, but there are very strong reasons for supposing that the assumption is false. There may well be unknown genetic or environmental factors that predispose families to SIDS, so that a second case within the family becomes much more likely than would be a case in another, apparently similar, family.<br><br>
 
A separate concern is that the characteristics used to classify the Clark family were chosen on the basis of the same data as was used to evaluate the frequency for that classification. This double use of data is well recognize by statisticians as perilous, since it can lead to subtle yet important biases.<br><br>
A separate concern is that the characteristics used to classify the Clark family were chosen on the basis of the same data as was used to evaluate the frequency for that classification. This double use of data is well recognize by statisticians as perilous, since it can lead to subtle yet important biases.<br><br>
 
For these reasons, the 1 in 73 million figure cannot be regarded as statistically valid. The Court of Appeal recognized flaws in its calculation, but seemed to accept it as establishing "... a very broad point, namely the rarity of double SIDS" [AC judgment, para 138]. However, not only is the error in the 1 in 73 million figure likely to be very large, it is almost certainly in one particular direction - against the defendant. Moreover, following from the 1 in 73 million figure at the original trial, the expert used a figure of about 700,000 UK births per year to conclude that "... by chance that happening will occur every 100 years". This conclusion is fallacious, not only because of the invalidity of the 1 in 73 million figure, but also because the 1 in 73 million figure relates only to families having some characteristics matching that of the defendant. This error seems not to have been recognised by the Appeal Court, who cited it without critical comment [AC judgment para 115]. Leaving aside the matter of validity, figures such as the 1 in 73 million are very easily misinterpreted. Some press reports at the time stated that this was the chance that the deaths of Sally Clark's two children were accidental. This (mis-)interpretation is a serious error of logic known as the Prosecutor's Fallacy (1). The jury needs to weigh up two competing explanations for the babies' deaths: SIDS or murder. The fact that two deaths by SIDS is quite unlikely is, taken alone, of little value. Two deaths by murder may well be even more unlikely. What matters is the relative likelihood of the deaths under each explanation, not just how unlikely they are under one explanation. </blockquote>
For these reasons, the 1 in 73 million figure cannot be regarded as statistically valid. The Court of Appeal recognized flaws in its calculation, but seemed to accept it as establishing "... a very broad point, namely the rarity of double SIDS" [AC judgment, para 138]. However, not only is the error in the 1 in 73 million figure likely to be very large, it is almost certainly in one particular direction - against the defendant. Moreover, following from the 1 in 73 million figure at the original trial, the expert used a figure of about 700,000 UK births per year to conclude that "... by chance that happening will occur every 100 years". This conclusion is fallacious, not only because of the invalidity of the 1 in 73 million figure, but also because the 1 in 73 million figure relates only to families having some characteristics matching that of the defendant. This error seems not to have been recognized by the Appeal Court, who cited it without critical comment [AC judgment para 115]. Leaving aside the matter of validity, figures such as the 1 in 73 million are very easily misinterpreted. Some press reports at the time stated that this was the chance that the deaths of Sally Clark's two children were accidental. This mis-interpretation is a serious error of logic known as the Prosecutor's Fallacy (1). The jury needs to weigh up two competing explanations for the babies' deaths: SIDS or murder. The fact that two deaths by SIDS are quite unlikely is, taken alone, of little value. Two deaths by murder may well be even more unlikely. What matters is the relative likelihood of the deaths under each explanation, not just how unlikely they are under one explanation. </blockquote>
 




To be continued
To be continued

Latest revision as of 15:52, 24 July 2005

Sir Roy Meadow struck off by GMC
BBC News, 15 July 2005

Beyond reasonable doubt
Plus Magazine, 2002
Helen Joyce

Muliple sudden infant deaths--coincidence or beyound coincidence
Paediatric and Perinatal Epidemiology 2004, 18, 320-326
Roy Hill

___________________________________________________________________________________

Sir Roy Meadow is a pediatrician well known for his research in child abuse. The BBC article reports that the UK General Medical Council (GMC) has found Sir Roy guilty of serious professional misconduct has "struck him off" the medical registry. If upheld under appeal, this will prevent Meadow from practicing medicine in the UK.

This decision was based on flawed statistical estimate that Meadow made while testifying as an expert witness in a 1999 court case in which a Sally Clark was found guilty of murdering her two baby boys and given a life sentence.

The death of Sally Clark's first child was recorded as a natural cot death. Å "cot death" is other names for SIDS (sudden infant death syndrome).

The name SIDS was proposed by the pathologist Bruce Beckwih at a conference in 1969 and the definition, which is still current, was formulated at the conference by Beckwith and others as follows:

The sudden death of a baby that is unexpected by history and in whom a
thorough post-mortem examination fails to demonstrate an adequate cause of death.

Testifying in Sally Clark's trial Meadow stated that chance of two natural cot deaths was 1 in 73 million suggesting that two natural cot deaths was beyond belief. This estimate came from a study called the "Confidential Inquiry for Stillbirths and Deaths in Infancy" (CESDI). The study gave detailed information about the deaths of all babies in five regions of England between 1993 and 1996. Using this data the study estimated that the chance of a cot death was 1 in 1,303. But if the child was from an affluent non-smoking family with the mother aged over 26, then the chance decreased to 1 in 8,543. Since Sally Clark was in the this group, Meadow assumed that the chance of the first child was a cot death was 1/8,543 and squaring this he obtained the 1 in 73 million estimate for the chance of two cot deaths. Giving evidence, Sir Roy defended the calculations he used to arrive at the 73 million to one figure. He compared the odds of two cot deaths in the same family to that of a punter successfully backing an 80-1 shot at the Grand National four years in a row.

There are three obvious problems with this 1 in 73 million statistic: (1) Because of environmental and genetical effects it seems very unlikely that the a SIDS death for a families the first baby and for their second baby are independent and (2) giving such odds with no further explanation mighr suggest to the jury that their is a 1 in 74 million chance that Sally Clark is innocent and (3) neither the defence or the procecustion suggested that the deaths were SIDS deaths--the proceducion claiming that they were murders, and the defence that they were natural deaths explained by the medical evidence. Ås far as we can tell only problem (1) was seriously discussed in the news. The first article to discuss this issue appears to be the article "False statistics may have led to solictor;s murder conviction" by Sherry Norton in The Independent (London), December 31, 1999. This article reports on and article by ÍDr Stephen Watkins, director of public health at Stockport health entitled "Conviction by mathematical error? (BMJ, 2000, 320, 2-3). Norton writes:

In Britain, there are 344 cot deaths each year, with the risk for the whole population being one cot death in 2.75 million. There are 378,000 second or subsequent births in England each year, and Dr Watkins has calculated that, if cot death is random, two such deaths will occur in the same family somewhere in England once every seven years. However, because cot deaths are not random and can be linked to genetic defects, social class and lifestyle, recurrence rates have been found to be about five times the general rate. Therefore Dr Watkins has concluded that two cot deaths in the same family will, on average, occur at least once every one and a half years.



These were nicely spelled out by a letter from the Royal Statistical Society to the Lord Chanceller dated 23 January 2002. This letter included the following discussion of the statistical issues involved:

You will be aware of the considerable public attention aroused by the recent conviction, confirmed on appeal, of Sally Clark for the murder of her two infants. One focus of the public attention was the statistical evidence given by a medical expert witness, who drew on a published study to obtain an estimate of the frequency of sudden infant death syndrome (SIDS, or "cot death") in families having some of the characteristics of the defendant's family. The witness went on to square this estimate to obtain a value of 1 in 73 million for the frequency of two cases of SIDS in such a family. This figure had an immediate and dramatic impact on all media reports of the trial, and it is difficult to believe that it did not also influence jurors.

The calculation leading to 1 in 73 million is invalid. It would only be valid if SIDS cases arose independently within families, an assumption that would need to be justified empirically. Not only was no such empirical justification provided in the case, but there are very strong reasons for supposing that the assumption is false. There may well be unknown genetic or environmental factors that predispose families to SIDS, so that a second case within the family becomes much more likely than would be a case in another, apparently similar, family.

A separate concern is that the characteristics used to classify the Clark family were chosen on the basis of the same data as was used to evaluate the frequency for that classification. This double use of data is well recognize by statisticians as perilous, since it can lead to subtle yet important biases.

For these reasons, the 1 in 73 million figure cannot be regarded as statistically valid. The Court of Appeal recognized flaws in its calculation, but seemed to accept it as establishing "... a very broad point, namely the rarity of double SIDS" [AC judgment, para 138]. However, not only is the error in the 1 in 73 million figure likely to be very large, it is almost certainly in one particular direction - against the defendant. Moreover, following from the 1 in 73 million figure at the original trial, the expert used a figure of about 700,000 UK births per year to conclude that "... by chance that happening will occur every 100 years". This conclusion is fallacious, not only because of the invalidity of the 1 in 73 million figure, but also because the 1 in 73 million figure relates only to families having some characteristics matching that of the defendant. This error seems not to have been recognized by the Appeal Court, who cited it without critical comment [AC judgment para 115]. Leaving aside the matter of validity, figures such as the 1 in 73 million are very easily misinterpreted. Some press reports at the time stated that this was the chance that the deaths of Sally Clark's two children were accidental. This mis-interpretation is a serious error of logic known as the Prosecutor's Fallacy (1). The jury needs to weigh up two competing explanations for the babies' deaths: SIDS or murder. The fact that two deaths by SIDS are quite unlikely is, taken alone, of little value. Two deaths by murder may well be even more unlikely. What matters is the relative likelihood of the deaths under each explanation, not just how unlikely they are under one explanation.


To be continued