I was quoting the statistics, I wasn't pretending to be a statistician: Difference between revisions

From ChanceWiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(43 intermediate revisions by 3 users not shown)
Line 3: Line 3:


[http://pass.maths.org.uk/issue21/features/clark/  Beyond reasonable doubt]<br>
[http://pass.maths.org.uk/issue21/features/clark/  Beyond reasonable doubt]<br>
Plus Magazine, 2002<br>
Plus magazine, 2002<br>
Helen Joyce
Helen Joyce
[http://www.sciencemag.org/cgi/content/summary/309/5734/543a Flawed Statistics in Murder Trial May Cost Expert His Medical License]<br>
Science, 22 July 2005<br>


Multiple sudden infant deaths--coincidence or beyond coincidence<br>
Multiple sudden infant deaths--coincidence or beyond coincidence<br>
Line 15: Line 12:
___________________________________________________________________________________
___________________________________________________________________________________


Sir Roy Meadow is a pediatrician, well known for his research in child abuse.  The BBC article reports that the UK General Medical Council (GMC) has found Sir Roy guilty of serious professional misconduct and has "struck him off" the medical registry.  If upheld under appeal, this will prevent Meadow from practicing medicine in the UK.
Sir Roy Meadow is a pediatrician, well known for his research in child abuse.  The BBC article reports that the UK General Medical Council (GMC) has found Sir Roy guilty of serious professional misconduct and has "struck him off" the medical registry.  If upheld under appeal this will prevent Meadow from practicing medicine in the UK.


This decision was based on a flawed statistical estimate that Meadow made while testifying as an expert witness in a 1999 trial in which a Sally Clark was found guilty of murdering her two baby boys and given a life sentence.   
This decision was based on a flawed statistical estimate that Meadow made while testifying as an expert witness in a 1999 trial in which a Sally Clark was found guilty of murdering her two baby boys and given a life sentence.   


To understand Meadow's testimony we need to know what a SIDS (sudden infant death syndrome) is.  The name SIDS was proposed by the pathologist Bruce Beckwih at a conference in 1969 and the definition, which is still current, along with many others,was formulated at the conference by Beckwith and others as follows:  
To understand Meadow's testimony we need to know what SIDS (sudden infant death syndrome) is.  The standard definition of SIDS is:


<blockquote>The sudden death of a baby that is unexpected by history and in whom a thorough post-mortem examination fails to demonstrate an adequate cause of death. </blockquote>
<blockquote>The sudden death of a baby that is unexpected by history and in whom a thorough post-mortem examination fails to demonstrate an adequate cause of death. </blockquote>
Line 25: Line 22:
The death of Sally Clark's first baby was reported as a cot death, which is another name for SIDS.  Then when her second baby died she was arrested and tried for murdering both her children.
The death of Sally Clark's first baby was reported as a cot death, which is another name for SIDS.  Then when her second baby died she was arrested and tried for murdering both her children.


We were not able to find a transcript for the original trial but from Lexis Nexis we found transcripts of two appeals that the Clarks made, one in October 2000 that they lost, and the other in April 2003 which they won releasing Sally after four anda half years in jail. The 2003 transcript reported on the statistical testimony in the original trial as follows:
We were not able to find a transcript for the original trial, but from Lexis Nexis we found transcripts of two appeals that the Clarks made, one in October 2000 and the other in April 2003. The 2003 transcript reported on the statistical testimony in the original trial as follows:


<blockquote> Professor Meadow was asked about some statistical information as to the happening of two cot deaths within the  
<blockquote> Professor Meadow was asked about some statistical information as to the happening of two cot deaths within the same family, which at that time was about to be published in a report of a government funded multi-disciplinary research  
same family, which at that time was about to be published in a report of a government funded multi--disciplinary research  
team, the Confidential Enquiry into Sudden Death in Infancy (CESDI) entitled 'Sudden Unexpected Deaths in Infancy' to which the professor was then writing a Preface. Professor Meadow said that it was 'the most reliable study and easily the largest and in that sense the latest and the best' ever done in this country. <br><br>
team, the Confidential Enquiry into Sudden Death in Infancy (CESDI) entitled 'Sudden Unexpected Deaths in Infancy' to  
which the professor was then writing a Preface. Professor Meadow said that it was 'the most reliable study and easily the  
largest and in that sense the latest and the best' ever done in this country. <br><br>


It was explained to the jury that there were factors that were suggested as relevant to the chances of a SIDS death within a given family; namely the age of the mother, whether there was a smoker in the household and the absence of a wage-earner in the family.<br><br>
It was explained to the jury that there were factors that were suggested as relevant to the chances of a SIDS death within a given family; namely the age of the mother, whether there was a smoker in the household and the absence of a wage-earner in the family.<br><br>


None of these factors had relevance to the Clark family and Professor Meadow was asked if a figure of 1 in 8,543 reflected the risk of there being a single SIDS within such a family. He agreed that it was. A table  
None of these factors had relevance to the Clark family and Professor Meadow was asked if a figure of 1 in 8,543 reflected the risk of there being a single SIDS within such a family. He agreed that it was. A table from the CESDI report was placed before the jury. He was then asked if the report calculated the risk of two infants dying of SIDS in that family by chance. His reply was: 'Yes, you have to multiply 1 in 8,543 times 1 in 8,543 and I think it gives that in the penultimate paragraph. It points out that it's approximately a chance of 1 in 73 million.' <br><br>
from the CESDI report was placed before the jury. He was then asked if the report calculated the risk of two infants dying  
of SIDS in that family by chance. His reply was: 'Yes, you have to multiply 1 in 8,543 times 1 in 8,543 and I think it  
gives that in the penultimate paragraph. It points out that it's approximately a chance of 1 in 73 million.' <br><br>


It seems that at this point Professor Meadow's voice was dropping and so the figure was repeated and then  
It seems that at this point Professor Meadow's voice was dropping and so the figure was repeated and then Professor Meadow added: 'In England, Wales and Scotland there are about say 700,000 live births a year, so it is saying by chance that happening will occur about once every hundred years.' <br><br>
Professor Meadow added: 'In England, Wales and Scotland there are about say 700,000 live births a year, so it is saying  
by chance that happening will occur about once every hundred years.' <br><br>


Mr. Spencer [for the prosecution] then pointed to the suspicious features alleged by the Crown in this present case and asked: 'So is  
Mr. Spencer [for the prosecution] then pointed to the suspicious features alleged by the Crown in this present case and asked: 'So is this right, not only would the chance be 1 in 73 million but in addition in these two deaths there are features, which would be regarded as suspicious in any event?' He elicited the reply 'I believe so'. <br><br>
this right, not only would the chance be 1 in 73 million but in addition in these two deaths there are features, which wuld
be regarded as suspicious in any event?' He elicited the reply 'I believe so'. <br><br>


All of this evidence was given without objection from the defence but Mr. Bevan (who represented the  
All of this evidence was given without objection from the defence but Mr. Bevan (who represented the Appellant at trial and at the first appeal but not at ours) cross--examined the doctor. He put to him figures from other research that suggested that the figure of 1 in 8,543 for a single cot death might be much too high. He then dealt with the chance of two cot deaths and Professor Meadow responded: 'This is why you take what's happened to all the children into account, and that is why you end up saying the chance of the children dying naturally in these circumstances is very, very long odds indeed one in 73 million.'  
Appellant at trial and at the first appeal but not at ours) cross--examined the doctor. He put to him figures from other  
research that suggested that the figure of 1 in 8,543 for a single cot death might be much too high. He then dealt with the  
chance of two cot deaths and Professor Meadow responded:  
'This is why you take what's happened to all the children into account, and that is why you end up saying the chance  
of the children dying naturally in these circumstances is very, very long odds indeed one in 73 million.'  
He then added: <br><br>
He then added: <br><br>


'. . . it's the chance of backing that long odds outsider at the Grand National, yu know; let's say it's a 80 to 1 chance,  
'. . . it's the chance of backing that long odds outsider at the Grand National, you know; let's say it's a 80 to 1 chance,  
you back the winner last year, then the next year there's another horse at 80 to 1 and it is still 80 to 1 and you back it again  
you back the winner last year, then the next year there's another horse at 80 to 1 and it is still 80 to 1 and you back it again  
and it wins. Now here we're in a situation that, you know, to get to these odds of 73 million you've got to back that 1 in 80  
and it wins. Now here we're in a situation that, you know, to get to these odds of 73 million you've got to back that 1 in 80  
Line 64: Line 46:
very, very unlikely.' <br><br>
very, very unlikely.' <br><br>


The trial judge clearly tried to divert the jury away from reliance on this statistical evidence. He said:  
The trial judge clearly tried to divert the jury away from reliance on this statistical evidence. He said: 'I should, I think, members of the jury just sound a word of caution about the statistics. However compelling you may find them to be, we do not convict people in these courts on statistics. It would be a terrible day if that were so. If there is one SIDS death in a family, it does not mean that there cannot be another one in the same family.' </blockquote>
'I should, I think, members of the jury just sound a word of caution about the statistics. However compelling you may  
find them to be, we do not convict people in these courts on statistics. It would be a terrible day if that were so. If there is  
one SIDS death in a family, it does not mean that there cannot be another one in the same family.' </blockquote>


Note that Meadow obtained the odds of 73 million to one from the CESDI report so there is some truth to the statement "I was quoting the statistics, I wasn't pretending to be a statistician” that Meadow made to the General Medical Council. Note also that both Meadow and the Judge took this statistic seriously and must have felt that it was evidence that Sally Clark was guilty.  This was also true of the press.  The Sunday Mail (Queenstand, Australia) had an article titled "Mum killed her babies" in which we read:  
Note that Meadow obtained the odds of 73 million to one from the CESDI report so there is some truth to the statement "I was quoting the statistics, I wasn't pretending to be a statistician" that Meadow made to the General Medical Council. Note also that both Meadow and the Judge took this statistic seriously and must have felt that it was evidence that Sally Clark was guilty.  This was also true of the press.  The Sunday Mail (Queenstand, Australia) had an article titled "Mum killed her babies" in which we read:  


<BLOCKQUOTE>Medical experts gave damning evidence that the odds of both children dying from cot death were 73 million to one.</blockquote>
<BLOCKQUOTE>Medical experts gave damning evidence that the odds of both children dying from cot death were 73 million to one.</blockquote>


There are two obvious problems with this 1 in 73 million statistic: (1) Because of environmental and genetics effects it seems very unlikely that the a SIDS death for a families the first baby and for their second baby are independent and (2) These odds might suggest to the jury that there is a 1 in 74 million chance that Sally Clark is innocent. This is, the well-known prosecuter's paradox.  The medical experts testimonies were very technical and some were contradictory. The 1 in 73 million odds were something the jury would at least feel that they understood.  
There are two obvious problems with this 1 in 73 million statistic: (1) Meadow assumed that in a family like the Clarks the events the "first child has a SIDS death" and "the second child has a SIDS death" are independent events.  Because of environmental and genetics effects it seems very unlikely this is the case. (2) The  73 million to 1 odds might suggest to the jury that there is a 1 in 73 million chance that Sally Clark is innocent. The medical experts testimonies were very technical and some were contradictory. The 1 in 73 million odds were something the jury would at least feel that they understood.  If you gave these odds to your Uncle George and asked him if Sally Clark is guilty he will very likely say "yes".


Of course these odds  of 73 million to 1 for SIDS teaths are useless to the jury in assessing guilt unless they are also given the corresponding odds that the deaths were the result of murders. In his article on the Sally Clark trial Professor Roy Hill estimated that, for a randomly chosen family with two baby deaths, the probability that the deaths are the result of SIDS is about 10 times more likely than the probability that they are the result of murders.  Thus, if this kind of statistical evidence means anything, it suggests that Sally Clark is innocent.  
The 73 million to 1 odds for SIDS deaths are useless to the jury in assessing guilt unless they are also given the corresponding odds that the deaths were the result of murders. We shall see later that, in this situation, SIDS deaths are about 9 time more likely than murders suggesting that Sally Clark is innocent rather than guilty.  


The Clarks had their first appeal in 2 October 2000.  By this time they realized that they had to have there own statisticians as expert witnesses.  They chose Ian Evett from the Forensic Science Service and Philip Dawid, Professor of Statistics at the Department of Statistical Science, University College London.  Both of these statisticians have specialized in statistical evidence in the courts.  In his report Dawid gave a very clear description of what would be required to obtain a reasonable  estimate of the probability of two SIDS deaths in a randomly chosen family with two babies.  He emphasized that it would be important also to have some estimate of the variability of this estimate.  Then he gave an equally clear discussion on the relevance of this probability, emphasizing the need for the corresponding probability of two murders in a family with two children.  His conclusion was:
The Clarks had their first appeal in 2 October 2000.  By this time they realized that they had to have their own statisticians as expert witnesses.  They chose Ian Evett from the Forensic Science Service and Philip Dawid, Professor of Statistics at University College London.  Both of these statisticians have specialized in statistical evidence in the courts.  In his report Dawid gave a very clear description of what would be required to obtain a reasonable  estimate of the probability of two SIDS deaths in a randomly chosen family with two babies.  He emphasized that it would be important also to have some estimate of the variability of this estimate.  Then he gave an equally clear discussion on the relevance of this probability, emphasizing the need for the corresponding probability of two murders in a family with two children.  His conclusion was:


<blockquote>The figure “1 in 73 million” quoted in Sir Roy Meadow's testimony at trial, as the probability of two babies both dying of SIDS in a family like Sally Clark's, was highly misleading and prejudicial.  The value of this probability has not been estimated with anything like the precision suggested, and could well be very much higher.  But, more important, the figure was presented with no explanation of the logically correct use of such information - which is very different from what a simple intuitive reaction might suggest.  In particular, such a figure could only be useful if compared with a similar figure calculated under the alternative hypothesis that both babies were murdered.  Even though assessment of the relevant probabilities may be difficult, there is a clear and well-established statistical logic for combining them and making appropriate inferences from them, which was not appreciated by the court. </blockquote>
<blockquote>The figure ''1 in 73 million'' quoted in Sir Roy Meadow's testimony at trial, as the probability of two babies both dying of SIDS in a family like Sally Clark's, was highly misleading and prejudicial.  The value of this probability has not been estimated with anything like the precision suggested, and could well be very much higher.  But, more important, the figure was presented with no explanation of the logically correct use of such information - which is very different from what a simple intuitive reaction might suggest.  In particular, such a figure could only be useful if compared with a similar figure calculated under the alternative hypothesis that both babies were murdered.  Even though assessment of the relevant probabilities may be difficult, there is a clear and well-established statistical logic for combining them and making appropriate inferences from them, which was not appreciated by the court. </blockquote>


These two statisticians were not allowed to appear in the court proceedings but only to have their reports read.   
These two statisticians were not allowed to appear in the court proceedings but only to have their reports read.   


The Clarks Ground 3a for the appeal dealt with the statistical issues including Meadow's incorrect calculation and the Judge failing to warn the jury against the "prosecutor's fallacy".
The Clarks' grounds for appeal included medical and statistical errors.  In particular they included Meadow's incorrect calculation and the Judge's failing to warn the jury against the "prosecutor's fallacy".
 
Concerning the miscalculation of the odds for two SIDS in a family of two children, the judge remarks that this was already known and all that really mattered was that appearance of two SIDS deaths is unusual.
 
The judge then dismisses the prosecutor's fallacy with the remark:
 
He [Everett] makes the obvious point that the evidential material in Table 3.58 tells us nothing whatsoever as to the guilt or innocence of the appellant.
 
The judge concludes:
 
<blockquote> Thus we do not think that the matters raised under Ground 3(a) (the statistical issues) are capable of affecting the safety of the convictions. They do not undermine what was put before the jury or cast a fundamentally different light on it. Even if they had been raised at trial, the most that could be expected to have resulted would be a direction to the jury that the issue was the broad one of rarity, to which the precise degree of probability was unnecessary. </blockquote>
 
The Judge dismissed the appeal.
 
After this the mathematics and statistical communities realized that it was necessary to explain these statistical issues to the legal community and the press.  On 23 October Royal Statistics Society addressed these issues in a press release and in January 2002 they sent a letter to the Lord Chamberllor. Both of these are available [http://www.rss.org.uk/main.asp?page=1225 here]. Here is the letter to the Lord Chancelor:
 
<blockquote> Dear Lord Chancellor, <br><br>
 
I am writing to you on behalf of the Royal Statistical Society to express the Society's concern about
some aspects of the presentation of statistical evidence in criminal trials. <br><br>
 
You will be aware of the considerable public attention aroused by the recent conviction, confirmed on
appeal, of Sally Clark for the murder of her two infants. One focus of the public attention was the
statistical evidence given by a medical expert witness, who drew on a published study to obtain an
estimate of the frequency of sudden infant death syndrome (SIDS, or "cot death") in families having
some of the characteristics of the defendant's family. The witness went on to square this estimate to
obtain a value of 1 in 73 million for the frequency of two cases of SIDS in such a family. This figure had
an immediate and dramatic impact on all media reports of the trial, and it is difficult to believe that it did
not also influence jurors. <br><br>
 
The calculation leading to 1 in 73 million is invalid. It would only be valid if SIDS cases arose
Independently within families, an assumption that would need to be justified empirically. Not only was
no such empirical justification provided in the case, but there are very strong reasons for supposing that
the assumption is false. There may well be unknown genetic or environmental factors that predispose
families to SIDS, so that a second case within the family becomes much more likely than would be a
case in another, apparently similar, family.<br><br>
 
A separate concern is that the characteristics used to classify the Clark family were chosen on the basis
of the same data as was used to evaluate the frequency for that classification. This double use of data is
well recognized by statisticians as perilous, since it can lead to subtle yet important biases.
 
<br><br>
For these reasons, the 1 in 73 million figure cannot be regarded as statistically valid. The Court of
Appeal recognized flaws in its calculation, but seemed to accept it as establishing "... a very broad point,
namely the rarity of double SIDS" [AC judgment, para 138]. However, not only is the error in the 1 in
73 million figure likely to be very large, it is almost certainly in one particular direction - against the
defendant. Moreover, following from the 1 in 73 million figure at the original trial, the expert used a
figure of about 700,000 UK births per year to conclude that "... by chance that happening will occur
every 100 years". This conclusion is fallacious, not only because of the invalidity of the 1 in 73 million
figure, but also because the 1 in 73 million figure relates only to families having some characteristics
matching that of the defendant. This error seems not to have been recognized by the Appeal Court, who
cited it without critical comment [AC judgment para 115]. Leaving aside the matter of validity, figures
such as the 1 in 73 million are very easily misinterpreted. Some press reports at the time stated that this
was the chance that the deaths of Sally Clark's two children were accidental. This (mis-)interpretation is
a serious error of logic known as the Prosecutor's Fallacy . The jury needs to weigh up two competing
explanations for the babies' deaths: SIDS or murder. The fact that two deaths by SIDS is quite unlikely
is, taken alone, of little value. Two deaths by murder may well be even more unlikely. What matters is
the relative likelihood of the deaths under each explanation, not just how unlikely they are under one
explanation. <br><br>
 
The Prosecutor's Fallacy has been well recognized in the context of DNA profile evidence. Its
commission at trial has led to successful appeals (R v. Deen, 1993; R v. Doheny/Adams 1996). In the
latter judgment, the Court of Appeal put in place guidelines for the presentation of DNA evidence.
However, we are concerned that the seriousness of the problem more generally has not been sufficiently
recognized. In particular, we are concerned that the Appeal Court did not consider it necessary to
examine the expert statistical evidence, but were content with written submissions. <br><br>
 
The case of R v. Sally Clark is one example of a medical expert witness making a serious statistical
error. Although the Court of Appeal judgment implied a view that the error was unlikely to have had a
profound effect on the outcome of the case, it would be better that the error had not occurred at all.
Although many scientists have some familiarity with statistical methods, statistics remains a specialized
area. The Society urges you to take steps to ensure that statistical evidence is presented only by
appropriately qualified statistical experts, as would be the case for any other form of expert evidence.
Without suggesting that there are simple or uniform answers, the Society would be pleased to be
involved in further discussions on the use and presentation of statistical evidence in courts, and to give
advice on the validation of the expertise of witnesses.<br><br>


Concerning the miscalulation of the odds for two SIDS in a family of two children, the judge remarks that this was already known and all that really mattered was that appearance of two SIDS deaths is unusual.
Yours sincerely<br>
Professor Peter Green, <br>
President, Royal Statistical Society.  
</blockquote>


The judge dismisses the prosecutors fallacy with the remark:
Now that we all agree that we need to know the relative probability that a family with two babies loses them by SIDS deaths or by murder, what are these probabilities?.  Roy Hill, Professor of Mathematics at the University of Salford tackled this question.  His results were first given in an unpublished paper "Cot death or Murder-weighing the probabilities" presented to the Developmental Physiology Conference, June 2002 (available from the author). Hill published his results in the his article "Multiple sudden infant deaths--coincidence or beyond coincidence?, "Paediatric and Perinatal Epidemiology" 2004, 18, 320-326.


He [Everett] makes the obvious point that the evidential material in Table 3.58 tell us nothing whatsoever as to the guilt or innocence of the appellant.  
In the trial the news media made frequent references to "Meadow's law".  This law is: "One cot death is a tragedy, two cot deaths is suspicious and three cot deaths is murder". This motivated Hill to test this law by estimating the relative probability of SIDS and murder deaths for the case of 1 baby, 2 babies, and 3 babies.  This is a difficult problem since data from different studies give different estimates, the estimates differ over time etc.  However Hill did a heroic job of combining all data he could find to come up with reasonable estimates. Here is what he found:


He concludes:
Regarding the issue of independence Hill concludes:


<blockquote> Thus we do not think that the matters raised under Ground 3(a) are capable of affecting the safety of the
<blockquote>In the light of all the data, it seems reasonable to estimate that the risk of SIDS is  between 5 and 10 times greater for infants where a sibling has already been a SIDS victim.</blockquote>
convictions. They do not undermine what was put before the jury or cast a fundamentally different light on it. Even if
they had been raised at trial, the most that could be expected to have resulted would be a direction to the jury that the issue
was the broad one of rarity, to which the precise degree of probability was unnecessary. </blockquote>


After this the mathematics and statistical communities felt obliged to explain the statistical issues involved in Meaadow's famous 1 in 73 odds.  On 23 October Royal Stistical Society put out the following news release:
As to the relative probabilities of SIDS deaths and murders Hill provides the following estimates:


<blockquote>The Royal Statistical Society today issued a statement, prompted by issues raised by the Sally Clark
(1) An infant is about 17 times more likely to be a SIDS victim than a homicide victim.
case, expressing its concern at the misuse of statistics in the courts. <br><br>
"In the recent highly-publicised case of R v. Sally Clark, a medical expert witness drew on published
studies to obtain a figure for the frequency of sudden infant death syndrome (SIDS, or "cot death") in
families having some of the characteristics of the defendant's family. He went on to square this figure to
obtain a value of 1 in 73 million for the frequency of two cases of SIDS in such a family. <br><br>
"This approach is, in general, statistically invalid. It would only be valid if SIDS cases arose
independently within families, an assumption that would need to be justified empirically. Not only was
no such empirical justification provided in the case, but there are very strong a priori reasons for
supposing that the assumption will be false. There may well be unknown genetic or environmental
factors that predispose families to SIDS, so that a second case within the family becomes much more  
likely. <br><br>
"The well-publicised figure of 1 in 73 million thus has no statistical basis. Its use cannot reasonably be  
justified as a "ballpark" figure because the error involved is likely to be very large, and in one particular
direction. The true frequency of families with two cases of SIDS may be very much less incriminating
than the figure presented to the jury at trial. <br><br>
http://www.therss.org.uk/archive/evidence/sclark.html (1 of 2)08/12/2004 13:59:31
RSS Statement regarding statistical issues in the Sally Clark case, October 23rd 2001
"Aside from its invalidity, figures such as the 1 in 73 million are very easily misinterpreted. Some press
reports at the time stated that this was the chance that the deaths of Sally Clark's two children were
accidental. This (mis-)interpretation is a serious error of logic known as the Prosecutor's Fallacy. The
jury needs to weigh up two competing explanations for the babies' deaths: SIDS or murder. Two deaths
by SIDS or two murders are each quite unlikely, but one has apparently happened in this case. What
matters is the relative likelihood of the deaths under each explanation, not just how unlikely they are
under one explanation (in this case SIDS, according to the evidence as presented). <br><br>
"The Court of Appeal has recognised these dangers (R v. Deen 1993, R v. Doheny/Adams 1996) in
connection with probabilities used for DNA profile evidence, and has put in place clear guidelines for
the presentation of such evidence. The dangers extend more widely, and there is a real possibility that
without proper guidance, and well-informed presentation, frequency estimates presented in court could
be misinterpreted by the jury in ways that are very prejudicial to defendants. <br><br>
"Society does not tolerate doctors making serious clinical errors because it is widely understood that
such errors could mean the difference between life and death. The case of R v. Sally Clark is one
example of a medical expert witness making a serious statistical error, one which may have had a
profound effect on the outcome of the case. <br><br>
"Although many scientists have some familiarity with statistical methods, statistics remains a specialised
area. The Society urges the Courts to ensure that statistical evidence is presented only by appropriately
qualified statistical experts, as would be the case for any other form of expert evidence."


(2) Two infants are about 9 times more likely to be SIDS victims than homicide victims.


References
(3) Three infants have about the same probability of being SIDS victims or homicide victims.


Transcript for the 2000 and 2003 appeals can be obtained from Lexis Nexis following the following route:
These estimates do not support Meadow's Law.  Despite many references to Meadow's Law in the medical journals and the news media, the editor for Hill's article comments that it appears to be due to D.J. and F.J. M Di Maio and seems not to appear in any of Meadow's writings.
 
Hill's analysis is used in the very nice article on  the Sally Clark case [http://pass.maths.org.uk/issue21/features/clark/  "Beyond reasonable doubt"] by Helen Joyce in ''Plus magazine''.  This is a great article to have students read. 
 
After the failure of their appeal, the Clarks started a campaign to get the news media to support their campaign. They also continued to search for medical explanations for their children's deaths.  In the process they found that the prosecutor's pathologist who had performed the autopsies for the two children had withheld the information that their second child had been suffering from a bacterial infection which could have been the cause of a natural death.  Recall that his first opinion had been that the first child also was a natural death.  This information and the flawed statistics led the Criminal Cases Review Commission, which investigates possible miscarriages of justice, to refer the case back to the courts for another appeal.
 
In this appeal the judge ruled that if the bacterial infection information had been known in the original trial, the Sally Clark would probably not have been convicted, and so he allowed the appeal and quashed the convictions freeing Sally Clalrk after two and a half years in jail.. 
 
The judge also agreed that the statistical evidence was seriously flawed and conclued:
 
<blockquote>Thus it seems likely that if this matter had been fully argued before us we would, in all probability, have considered that the statistical evidence provided a quite distinct basis upon which the appeal had to be allowed. </blockquote>
 
Thus we cannot say that the famous 1 chance in 73 million statistic was responsible for Sally Clark being freed from jail but it is very likely the reason she spent two and a half years in jail.
 
For the complete Sally Clark story we recommend the book "Stolen Innocence" by John Batt available at U.K. Amazon.  Batt is a lawyer and good friend of the Clarks.  He attended the trials and his book tells the Sally Clark story from beginning to end.  We also found it interesting to read the transcripts of the 2000 and 2003 appeal. These were not easy to find so we include at the end of this article  the  Lexis Nexis path to these transcrips.
 
Sir Roy Meadow was also the key prosecution witness in two other cases similar to the Sally Clark case: the Angela Canning case and the Trupti Patel case.  Roy Hill also wrote an intersting article "Reflections on the cot death cases", ''Significance'', volume 2 (2005), issue 1 in which he discusses the statistical issues in all three cases.
 
Finding the 2000 and 2003 transcripts in Lexis Nexis.


Open Lexis Nexis
Open Lexis Nexis
Line 151: Line 188:
Choose" Commonwealth and Foreign Nations" from the sidebar
Choose" Commonwealth and Foreign Nations" from the sidebar


Choose  "Sally Clark” for the " Keyword"
Choose  "Sally Clark" for the "Keyword"


Choose "UK Cases" for the "Source"
Choose "UK Cases" for the "Source"
Line 157: Line 194:
Choose "Previous five years” for the "Date."
Choose "Previous five years” for the "Date."


The two " r v Clarks" are the appeals.
The two "r v Clarks" are the appeals.
 
DISCUSSION QUESTIONS:


To be continued
(1)  Do you think it is easonable not to allow a doctor to practice because of a statistical error in a court case?


Retrieved from "http://chance.dartmouth.edu/chancewiki/index.php/I_was_quoting_the_statistics%2C_I_wasn%27t_pretending_to_be_a_statisitcian"
(2)  Do you agree with the RSS that statistical evidence should only be provided by statistical experts?
The two " r v Clarks" are the appeals.
To be continued


Retrieved from "http://chance.dartmouth.edu/chancewiki/index.php/I_was_quoting_the_statistics%2C_I_wasn%27t_pretending_to_be_a_statisitcian"
(3) Give the 73 million to 1 odds to a few of your friends and ask them if they were on the jury, in the absence of any other information, would they think that this makes it is very likely that Sally Clark is guilty?

Latest revision as of 18:20, 6 August 2005

Sir Roy Meadow struck off by GMC
BBC News, 15 July 2005

Beyond reasonable doubt
Plus magazine, 2002
Helen Joyce

Multiple sudden infant deaths--coincidence or beyond coincidence
Paediatric and Perinatal Epidemiology 2004, 18, 320-326
Roy Hill

___________________________________________________________________________________

Sir Roy Meadow is a pediatrician, well known for his research in child abuse. The BBC article reports that the UK General Medical Council (GMC) has found Sir Roy guilty of serious professional misconduct and has "struck him off" the medical registry. If upheld under appeal this will prevent Meadow from practicing medicine in the UK.

This decision was based on a flawed statistical estimate that Meadow made while testifying as an expert witness in a 1999 trial in which a Sally Clark was found guilty of murdering her two baby boys and given a life sentence.

To understand Meadow's testimony we need to know what SIDS (sudden infant death syndrome) is. The standard definition of SIDS is:

The sudden death of a baby that is unexpected by history and in whom a thorough post-mortem examination fails to demonstrate an adequate cause of death.

The death of Sally Clark's first baby was reported as a cot death, which is another name for SIDS. Then when her second baby died she was arrested and tried for murdering both her children.

We were not able to find a transcript for the original trial, but from Lexis Nexis we found transcripts of two appeals that the Clarks made, one in October 2000 and the other in April 2003. The 2003 transcript reported on the statistical testimony in the original trial as follows:

Professor Meadow was asked about some statistical information as to the happening of two cot deaths within the same family, which at that time was about to be published in a report of a government funded multi-disciplinary research

team, the Confidential Enquiry into Sudden Death in Infancy (CESDI) entitled 'Sudden Unexpected Deaths in Infancy' to which the professor was then writing a Preface. Professor Meadow said that it was 'the most reliable study and easily the largest and in that sense the latest and the best' ever done in this country.

It was explained to the jury that there were factors that were suggested as relevant to the chances of a SIDS death within a given family; namely the age of the mother, whether there was a smoker in the household and the absence of a wage-earner in the family.

None of these factors had relevance to the Clark family and Professor Meadow was asked if a figure of 1 in 8,543 reflected the risk of there being a single SIDS within such a family. He agreed that it was. A table from the CESDI report was placed before the jury. He was then asked if the report calculated the risk of two infants dying of SIDS in that family by chance. His reply was: 'Yes, you have to multiply 1 in 8,543 times 1 in 8,543 and I think it gives that in the penultimate paragraph. It points out that it's approximately a chance of 1 in 73 million.'

It seems that at this point Professor Meadow's voice was dropping and so the figure was repeated and then Professor Meadow added: 'In England, Wales and Scotland there are about say 700,000 live births a year, so it is saying by chance that happening will occur about once every hundred years.'

Mr. Spencer [for the prosecution] then pointed to the suspicious features alleged by the Crown in this present case and asked: 'So is this right, not only would the chance be 1 in 73 million but in addition in these two deaths there are features, which would be regarded as suspicious in any event?' He elicited the reply 'I believe so'.

All of this evidence was given without objection from the defence but Mr. Bevan (who represented the Appellant at trial and at the first appeal but not at ours) cross--examined the doctor. He put to him figures from other research that suggested that the figure of 1 in 8,543 for a single cot death might be much too high. He then dealt with the chance of two cot deaths and Professor Meadow responded: 'This is why you take what's happened to all the children into account, and that is why you end up saying the chance of the children dying naturally in these circumstances is very, very long odds indeed one in 73 million.' He then added:

'. . . it's the chance of backing that long odds outsider at the Grand National, you know; let's say it's a 80 to 1 chance, you back the winner last year, then the next year there's another horse at 80 to 1 and it is still 80 to 1 and you back it again and it wins. Now here we're in a situation that, you know, to get to these odds of 73 million you've got to back that 1 in 80 chance four years running, so yes, you might be very, very lucky because each time it's just been a 1 in 80 chance and you know, you've happened to have won it, but the chance of it happening four years running we all know is extraordinarily unlikely. So it's the same with these deaths. You have to say two unlikely events have happened and together it's very, very, very unlikely.'

The trial judge clearly tried to divert the jury away from reliance on this statistical evidence. He said: 'I should, I think, members of the jury just sound a word of caution about the statistics. However compelling you may find them to be, we do not convict people in these courts on statistics. It would be a terrible day if that were so. If there is one SIDS death in a family, it does not mean that there cannot be another one in the same family.'

Note that Meadow obtained the odds of 73 million to one from the CESDI report so there is some truth to the statement "I was quoting the statistics, I wasn't pretending to be a statistician" that Meadow made to the General Medical Council. Note also that both Meadow and the Judge took this statistic seriously and must have felt that it was evidence that Sally Clark was guilty. This was also true of the press. The Sunday Mail (Queenstand, Australia) had an article titled "Mum killed her babies" in which we read:

Medical experts gave damning evidence that the odds of both children dying from cot death were 73 million to one.

There are two obvious problems with this 1 in 73 million statistic: (1) Meadow assumed that in a family like the Clarks the events the "first child has a SIDS death" and "the second child has a SIDS death" are independent events. Because of environmental and genetics effects it seems very unlikely this is the case. (2) The 73 million to 1 odds might suggest to the jury that there is a 1 in 73 million chance that Sally Clark is innocent. The medical experts testimonies were very technical and some were contradictory. The 1 in 73 million odds were something the jury would at least feel that they understood. If you gave these odds to your Uncle George and asked him if Sally Clark is guilty he will very likely say "yes".

The 73 million to 1 odds for SIDS deaths are useless to the jury in assessing guilt unless they are also given the corresponding odds that the deaths were the result of murders. We shall see later that, in this situation, SIDS deaths are about 9 time more likely than murders suggesting that Sally Clark is innocent rather than guilty.

The Clarks had their first appeal in 2 October 2000. By this time they realized that they had to have their own statisticians as expert witnesses. They chose Ian Evett from the Forensic Science Service and Philip Dawid, Professor of Statistics at University College London. Both of these statisticians have specialized in statistical evidence in the courts. In his report Dawid gave a very clear description of what would be required to obtain a reasonable estimate of the probability of two SIDS deaths in a randomly chosen family with two babies. He emphasized that it would be important also to have some estimate of the variability of this estimate. Then he gave an equally clear discussion on the relevance of this probability, emphasizing the need for the corresponding probability of two murders in a family with two children. His conclusion was:

The figure 1 in 73 million quoted in Sir Roy Meadow's testimony at trial, as the probability of two babies both dying of SIDS in a family like Sally Clark's, was highly misleading and prejudicial. The value of this probability has not been estimated with anything like the precision suggested, and could well be very much higher. But, more important, the figure was presented with no explanation of the logically correct use of such information - which is very different from what a simple intuitive reaction might suggest. In particular, such a figure could only be useful if compared with a similar figure calculated under the alternative hypothesis that both babies were murdered. Even though assessment of the relevant probabilities may be difficult, there is a clear and well-established statistical logic for combining them and making appropriate inferences from them, which was not appreciated by the court.

These two statisticians were not allowed to appear in the court proceedings but only to have their reports read.

The Clarks' grounds for appeal included medical and statistical errors. In particular they included Meadow's incorrect calculation and the Judge's failing to warn the jury against the "prosecutor's fallacy".

Concerning the miscalculation of the odds for two SIDS in a family of two children, the judge remarks that this was already known and all that really mattered was that appearance of two SIDS deaths is unusual.

The judge then dismisses the prosecutor's fallacy with the remark:

He [Everett] makes the obvious point that the evidential material in Table 3.58 tells us nothing whatsoever as to the guilt or innocence of the appellant.

The judge concludes:

Thus we do not think that the matters raised under Ground 3(a) (the statistical issues) are capable of affecting the safety of the convictions. They do not undermine what was put before the jury or cast a fundamentally different light on it. Even if they had been raised at trial, the most that could be expected to have resulted would be a direction to the jury that the issue was the broad one of rarity, to which the precise degree of probability was unnecessary.

The Judge dismissed the appeal.

After this the mathematics and statistical communities realized that it was necessary to explain these statistical issues to the legal community and the press. On 23 October Royal Statistics Society addressed these issues in a press release and in January 2002 they sent a letter to the Lord Chamberllor. Both of these are available here. Here is the letter to the Lord Chancelor:

Dear Lord Chancellor,

I am writing to you on behalf of the Royal Statistical Society to express the Society's concern about some aspects of the presentation of statistical evidence in criminal trials.

You will be aware of the considerable public attention aroused by the recent conviction, confirmed on appeal, of Sally Clark for the murder of her two infants. One focus of the public attention was the statistical evidence given by a medical expert witness, who drew on a published study to obtain an estimate of the frequency of sudden infant death syndrome (SIDS, or "cot death") in families having some of the characteristics of the defendant's family. The witness went on to square this estimate to obtain a value of 1 in 73 million for the frequency of two cases of SIDS in such a family. This figure had an immediate and dramatic impact on all media reports of the trial, and it is difficult to believe that it did not also influence jurors.

The calculation leading to 1 in 73 million is invalid. It would only be valid if SIDS cases arose Independently within families, an assumption that would need to be justified empirically. Not only was no such empirical justification provided in the case, but there are very strong reasons for supposing that the assumption is false. There may well be unknown genetic or environmental factors that predispose families to SIDS, so that a second case within the family becomes much more likely than would be a case in another, apparently similar, family.

A separate concern is that the characteristics used to classify the Clark family were chosen on the basis of the same data as was used to evaluate the frequency for that classification. This double use of data is well recognized by statisticians as perilous, since it can lead to subtle yet important biases.



For these reasons, the 1 in 73 million figure cannot be regarded as statistically valid. The Court of Appeal recognized flaws in its calculation, but seemed to accept it as establishing "... a very broad point, namely the rarity of double SIDS" [AC judgment, para 138]. However, not only is the error in the 1 in 73 million figure likely to be very large, it is almost certainly in one particular direction - against the defendant. Moreover, following from the 1 in 73 million figure at the original trial, the expert used a figure of about 700,000 UK births per year to conclude that "... by chance that happening will occur every 100 years". This conclusion is fallacious, not only because of the invalidity of the 1 in 73 million figure, but also because the 1 in 73 million figure relates only to families having some characteristics matching that of the defendant. This error seems not to have been recognized by the Appeal Court, who cited it without critical comment [AC judgment para 115]. Leaving aside the matter of validity, figures such as the 1 in 73 million are very easily misinterpreted. Some press reports at the time stated that this was the chance that the deaths of Sally Clark's two children were accidental. This (mis-)interpretation is a serious error of logic known as the Prosecutor's Fallacy . The jury needs to weigh up two competing explanations for the babies' deaths: SIDS or murder. The fact that two deaths by SIDS is quite unlikely is, taken alone, of little value. Two deaths by murder may well be even more unlikely. What matters is the relative likelihood of the deaths under each explanation, not just how unlikely they are under one explanation.

The Prosecutor's Fallacy has been well recognized in the context of DNA profile evidence. Its commission at trial has led to successful appeals (R v. Deen, 1993; R v. Doheny/Adams 1996). In the latter judgment, the Court of Appeal put in place guidelines for the presentation of DNA evidence. However, we are concerned that the seriousness of the problem more generally has not been sufficiently recognized. In particular, we are concerned that the Appeal Court did not consider it necessary to examine the expert statistical evidence, but were content with written submissions.

The case of R v. Sally Clark is one example of a medical expert witness making a serious statistical error. Although the Court of Appeal judgment implied a view that the error was unlikely to have had a profound effect on the outcome of the case, it would be better that the error had not occurred at all. Although many scientists have some familiarity with statistical methods, statistics remains a specialized area. The Society urges you to take steps to ensure that statistical evidence is presented only by appropriately qualified statistical experts, as would be the case for any other form of expert evidence. Without suggesting that there are simple or uniform answers, the Society would be pleased to be involved in further discussions on the use and presentation of statistical evidence in courts, and to give advice on the validation of the expertise of witnesses.

Yours sincerely
Professor Peter Green,
President, Royal Statistical Society.

Now that we all agree that we need to know the relative probability that a family with two babies loses them by SIDS deaths or by murder, what are these probabilities?. Roy Hill, Professor of Mathematics at the University of Salford tackled this question. His results were first given in an unpublished paper "Cot death or Murder-weighing the probabilities" presented to the Developmental Physiology Conference, June 2002 (available from the author). Hill published his results in the his article "Multiple sudden infant deaths--coincidence or beyond coincidence?, "Paediatric and Perinatal Epidemiology" 2004, 18, 320-326.

In the trial the news media made frequent references to "Meadow's law". This law is: "One cot death is a tragedy, two cot deaths is suspicious and three cot deaths is murder". This motivated Hill to test this law by estimating the relative probability of SIDS and murder deaths for the case of 1 baby, 2 babies, and 3 babies. This is a difficult problem since data from different studies give different estimates, the estimates differ over time etc. However Hill did a heroic job of combining all data he could find to come up with reasonable estimates. Here is what he found:

Regarding the issue of independence Hill concludes:

In the light of all the data, it seems reasonable to estimate that the risk of SIDS is between 5 and 10 times greater for infants where a sibling has already been a SIDS victim.

As to the relative probabilities of SIDS deaths and murders Hill provides the following estimates:

(1) An infant is about 17 times more likely to be a SIDS victim than a homicide victim.

(2) Two infants are about 9 times more likely to be SIDS victims than homicide victims.

(3) Three infants have about the same probability of being SIDS victims or homicide victims.

These estimates do not support Meadow's Law. Despite many references to Meadow's Law in the medical journals and the news media, the editor for Hill's article comments that it appears to be due to D.J. and F.J. M Di Maio and seems not to appear in any of Meadow's writings.

Hill's analysis is used in the very nice article on the Sally Clark case "Beyond reasonable doubt" by Helen Joyce in Plus magazine. This is a great article to have students read.

After the failure of their appeal, the Clarks started a campaign to get the news media to support their campaign. They also continued to search for medical explanations for their children's deaths. In the process they found that the prosecutor's pathologist who had performed the autopsies for the two children had withheld the information that their second child had been suffering from a bacterial infection which could have been the cause of a natural death. Recall that his first opinion had been that the first child also was a natural death. This information and the flawed statistics led the Criminal Cases Review Commission, which investigates possible miscarriages of justice, to refer the case back to the courts for another appeal.

In this appeal the judge ruled that if the bacterial infection information had been known in the original trial, the Sally Clark would probably not have been convicted, and so he allowed the appeal and quashed the convictions freeing Sally Clalrk after two and a half years in jail..

The judge also agreed that the statistical evidence was seriously flawed and conclued:

Thus it seems likely that if this matter had been fully argued before us we would, in all probability, have considered that the statistical evidence provided a quite distinct basis upon which the appeal had to be allowed.

Thus we cannot say that the famous 1 chance in 73 million statistic was responsible for Sally Clark being freed from jail but it is very likely the reason she spent two and a half years in jail.

For the complete Sally Clark story we recommend the book "Stolen Innocence" by John Batt available at U.K. Amazon. Batt is a lawyer and good friend of the Clarks. He attended the trials and his book tells the Sally Clark story from beginning to end. We also found it interesting to read the transcripts of the 2000 and 2003 appeal. These were not easy to find so we include at the end of this article the Lexis Nexis path to these transcrips.

Sir Roy Meadow was also the key prosecution witness in two other cases similar to the Sally Clark case: the Angela Canning case and the Trupti Patel case. Roy Hill also wrote an intersting article "Reflections on the cot death cases", Significance, volume 2 (2005), issue 1 in which he discusses the statistical issues in all three cases.

Finding the 2000 and 2003 transcripts in Lexis Nexis.

Open Lexis Nexis

Choose "Legal Research" from the sidebar

From "Case Law" choose "Get a Case"

Choose" Commonwealth and Foreign Nations" from the sidebar

Choose "Sally Clark" for the "Keyword"

Choose "UK Cases" for the "Source"

Choose "Previous five years” for the "Date."

The two "r v Clarks" are the appeals.

DISCUSSION QUESTIONS:

(1) Do you think it is easonable not to allow a doctor to practice because of a statistical error in a court case?

(2) Do you agree with the RSS that statistical evidence should only be provided by statistical experts?

(3) Give the 73 million to 1 odds to a few of your friends and ask them if they were on the jury, in the absence of any other information, would they think that this makes it is very likely that Sally Clark is guilty?