Sandbox: Difference between revisions

From ChanceWiki
Jump to navigation Jump to search
No edit summary
 
Line 1: Line 1:
==Do Oscar Winners Live Longer?==
If you put "Oscar winners live longer" in Google you will get  over 7,000 hits.  Here is one from the January 23, 2007 issue of [http://www.healthandage.com/public/health-center/37/news/7655/Oscar-winners-live-longer.html Health and Aging]


Oscar winners live longer: Reported by Susan Aldridge, PhD, medical journalist.<br>


<blockquote>It is Oscar season again and, if you're a film fan, you'll be following proceedings with interest. But did you know there is a health benefit to winning an Oscar? Doctors at Harvard Medical School say that a study of actors and actresses shows that winners live, on average, for four years more than losers. And winning directors live longer than non-winners.
==Forsooth==
Source: "Harvard Health Letter" March 2006.</blockquote>


The assertion that Oscar winners live longer was based on an article by Donald Redelmeier, and Sheldon Singh: "Survival in Academy Award-winning actors and actresses". ''Annals of Internal medicine'', 15 May, 2001, Vol. 134, No. 10, 955-962.
==Quotations==
“We know that people tend to overestimate the frequency of well-publicized, spectacular
events compared with more commonplace ones; this is a well-understood phenomenon in
the literature of risk assessment and leads to the truism that when statistics plays folklore,
folklore always wins in a rout.”
<div align=right>-- Donald Kennedy (former president of Stanford University), ''Academic Duty'', Harvard University Press, 1997, p.17</div>


This is the kind of study the media loves to report and medical journals enjoy the publicity they get. Another such claim, in the news as this is written, is that the outcome  of the Super Bowl football game determines whether the stock market will go up or down this year.  Unlike the Oscar winners story the author of this claim, Leonard Koppet, admited that it was all a joke, see [http://www.dartmouth.edu/~chance/chance_news/recent_news/chance_news_13.04.html#item11 Chance News 13.04].
----


A recent paper by James Hanley, Marie-Pierre Sylvestre and Ella Huszti, "Do Oscar winners live longer than less successful peers? A reanalysis of the evidence," ''Annals of Internal medicine'', 5 September 2006, Vol 145, No. 5, 361-363, claims that the Redelmeier, Singh paper was flawed. They provided a reanalysis of the data showing that it does not  support the claim that Oscar winners live longer.
"Using scientific language and measurement doesn’t prevent a researcher from conducting flawed experiments and drawing wrong conclusions — especially when they confirm preconceptions."


For their study, Redelmeier and Singh identified all actors and actresses ever nominated for an academy award in a leading or a supporting role up to the time of the study (n = 762).  Among these there were 235 Oscar winners.  For each nominee another cast member of the same sex who was in the same film and was born in the same era was identified (n= 887) and these were used as controls. 
<div align=right>-- Blaise Agüera y Arcas, Margaret Mitchell and Alexander Todoorov, quoted in: The racist history behind facial recognition, ''New York Times'', 10 July 2019</div>


The authors  used the Kaplan-Meier method to provide a life table for the Oscar winners and the control group. A life table estimates for each x the probability of living x years.  In [http://www.dartmouth.edu/~chance/chance_news/recent_news/chance_news_10.06.html#item10  Chance News 10.06] We illustrated the Kaplan-Meier method, using data obtained from Dr. Radelmeier.
==In progress==
[https://www.nytimes.com/2018/11/07/magazine/placebo-effect-medicine.html What if the Placebo Effect Isn’t a Trick?]<br>
by Gary Greenberg, ''New York Times Magazine'', 7 November 2018


In their paper Redelmeier and Singh provided the following graph showing the life tables of the two goups.
[https://www.nytimes.com/2019/07/17/opinion/pretrial-ai.html The Problems With Risk Assessment Tools]<br>
by Chelsea Barabas, Karthik Dinakar and Colin Doyle, ''New York Times'', 17 July 2019


<center> http://www.dartmouth.edu/~chance/forwiki/oscar.jpg </center>
==Hurricane Maria deaths==
Laura Kapitula sent the following to the Isolated Statisticians e-mail list:


The areas under the two curves  are estimates for the life expectance for the two groups. Using a test called the "log-rank test" they concluded that the overall difference in life expectancy was 3.9 years (79.7 vs. 75.8 years; P) = .003.
:[Why counting casualties after a hurricane is so hard]<br>
:by Jo Craven McGinty, Wall Street Journal, 7 September 2018


While the life tables look like  standard life tables there is one big difference. We note that 100 percent of the Oscar winners live to be at least 30 years old. Of course this is not surprising because they are known to be Oscar winners. Thus we know ahead of time that the Oscar winners will live longer than a traditional life table would predict. This gives them an advantage in their life expectancy. This is called a selection bias. Of course the controls also have an advantage because we know that were in a movie at about the same age as a nominee. But there is no reason to believe that these advantages are the same.
The article is subtitled: Indirect deaths—such as those caused by gaps in medication—can occur months after a storm, complicating tallies
   
Laura noted that  
:[https://www.washingtonpost.com/news/fact-checker/wp/2018/06/02/did-4645-people-die-in-hurricane-maria-nope/?utm_term=.0a5e6e48bf11 Did 4,645 people die in Hurricane Maria? Nope.]<br>
:by Glenn Kessler, ''Washington Post'', 1 June 2018


Here is a more obvious example of selection bias discussed in Robert Abelson's book "Statistics as Principled Argument' and reported in Chance News 4.05.
The source of the 4645 figure is a [https://www.nejm.org/doi/full/10.1056/NEJMsa1803972 NEJM article]. Point estimate, the 95% confidence interval ran from 793 to 8498.


<Blockquote>A study found that the average life expectancy
President Trump has asserted that the actual number is
    of famous orchestral conductors was 73.4 years,
[https://twitter.com/realDonaldTrump/status/1040217897703026689 6 to 18].
    significantly higher than the life expectancy for
The ''Post'' article notes that Puerto Rican official had asked researchers at George Washington University to do an estimate of the death toll.  That work is not complete.
    males, 68.5,  at the time of the study.  Jane
[https://prstudy.publichealth.gwu.edu/ George Washington University study]
    Brody in her "New York Times" health column
    reported that this was thought to be due to arm
    exercise.  J. D  Caroll gave an alternative
    suggestion, remarking that it was reasonable to
    assume that a famous orchestra conductor was
    at least 32 years old. The life expectancy for
    a 32 year old male was 72 years making the 73.4
    average not at all surprising. </blockquote>


To avoid the possible of selective bias, Redelmeier and Singh did an analysis using time-dependent covariates, in which winners were counted as controls until the time of first they won the Oscar.  This  resulted in a difference in life expentance of 20% (CI, 0% to 35%). Since the confidence interval includes 0 the difference is not significant. So one might wonder why they made their claim that Oscar winners live longer.
:[https://fivethirtyeight.com/features/we-still-dont-know-how-many-people-died-because-of-katrina/?ex_cid=538twitter We sttill don’t know how many people died because of Katrina]<br>
:by Carl Bialik, FiveThirtyEight, 26 August 2015


In a letter to the editor in response to the study by Hanley et al., Redelmeier and Singh report that they did the same analysis with one more year's data and obtained a result even more clearly not significant.  
----
[https://www.nytimes.com/2018/09/11/climate/hurricane-evacuation-path-forecasts.html These 3 Hurricane Misconceptions Can Be Dangerous. Scientists Want to Clear Them Up.]<br>
[https://journals.ametsoc.org/doi/abs/10.1175/BAMS-88-5-651 Misinterpretations of the “Cone of Uncertainty” in Florida during the 2004 Hurricane Season]<br>
[https://www.nhc.noaa.gov/aboutcone.shtml Definition of the NHC Track Forecast Cone]
----
[https://www.popsci.com/moderate-drinking-benefits-risks Remember when a glass of wine a day was good for you? Here's why that changed.]
''Popular Science'', 10 September 2018
----
[https://www.economist.com/united-states/2018/08/30/googling-the-news Googling the news]<br>
''Economist'', 1 September 2018


To avoid selection bias, Sylvester and colleagues analyzed the data by comparing the life expectancy of the winners from the moment they win with others alive at that age.  In the McGill Press Release,  Hanley remarks "The results are not as, shall we say, dramatic, but they're more accurate." We recommend reading this [http://www.mcgill.ca/newsroom/news/?ItemID=21645 press release] for more information about the study by Sylvester et al.
[https://www.cnbc.com/2018/09/17/google-tests-changes-to-its-search-algorithm-how-search-works.html We sat in on an internal Google meeting where they talked about changing the search algorithm — here's what we learned]
----
[http://www.wyso.org/post/stats-stories-reading-writing-and-risk-literacy Reading , Writing and Risk Literacy]


When the Redelmeier and Singh paper came out, our colleague Peter Doyle was skeptical of the results (he called their paper "a crock"). In the fall of 2003 he wrote to the editor of the Annals of Internal Medicine, asking if the article
[http://www.riskliteracy.org/]
was a hoax:
-----
[https://twitter.com/i/moments/1025000711539572737?cn=ZmxleGlibGVfcmVjc18y&refsrc=email Today is the deadliest day of the year for car wrecks in the U.S.]


<blockquote>It has finally dawned on me that, like the screenwriters study in the
==Some math doodles==
British Medical Journal, the original OSCAR study (Annals of Internal
<math>P \left({A_1 \cup A_2}\right) = P\left({A_1}\right) + P\left({A_2}\right) -P \left({A_1 \cap A_2}\right)</math>
Medicine, 15 May 2001) may not have been a travesty of statistics after all,
but simply a joke.  Here's why I think so.


(1) The conclusion---which has been colorfully paraphrased as saying
<math>P(E)  = {n \choose k} p^k (1-p)^{ n-k}</math>
that winning an OSCAR is better for your health then being
made immune to cancer---is inherently preposterous.
The suggestion seems really `too stupid to discuss'.  This
should alert the reader that the article may be a hoax.


(2) The fallacy upon which the conclusion is based is well-known, and
<math>\hat{p}(H|H)</math>
instantly obvious in the case of similar claims, like Mark Mixer's
keen observation that `breaking your hip increases your life expectancy'.
We're not talking here about something really subtle---we're talking about
a common statistical fallacy.  The authors could reasonably expect
that the fallacy would be evident (if not instantly then upon a little
reflection) to anyone with basic statistical savvy---or just plain
common sense.


(3) The article itself implicitly recognizes the fallacy behind the method;
<math>\hat{p}(H|HH)</math>
describes how to modify the method to avoid the fallacy (`time-dependent
covariates'); and indicates that when the method is modified appropriately,
the results lapse into statistical insignificance.  However, all this is
done subtly, so that it doesn't leap out at you.  Unless you are on the
lookout, you're likely either to miss the point entirely, or fail to
appreciate its significance.  The effect here is much the same as if
the authors had dated the article `1 April 2001': Once you identify
the fallacy that is at work here, you will find within the article
itself the evidence to confirm that this is indeed a hoax.


(4)  The article claims that data will be made available online, but
==Accidental insights==
when you look for the data, you find only a lame excuse for why it isn't
available (something about `confidentiality').  The statement that
data will be made available is something you would normally
include in an article like this:  It's all part of the masquerade.
The authors didn't bother to make the data available
because they didn't think anyone would be dense enough to actually
want to look it.


In light of all this, it seems plausible that this article was intended
My collective understanding of Power Laws would fit beneath the shallow end of the long tail. Curiosity, however, easily fills the fat end.  I long have been intrigued by the concept and the surprisingly common appearance of power laws in varied natural, social and organizational dynamics.  But, am I just seeing a statistical novelty or is there meaning and utility in Power Law relationships? Here’s a case in point.
to be a joke---at least by the authors.
</blockquote>


The editor assured Peter that the article was not a hoax, and that it had been submitted to a "painstaking review".
While carrying a pair of 10 lb. hand weights one, by chance, slipped from my grasp and fell onto a piece of ceramic tile I had left on the carpeted floor. The fractured tile was inconsequential, meant for the trash.
<center>[[File:BrokenTile.jpg | 400px]]</center>
As I stared, slightly annoyed, at the mess, a favorite maxim of the Greek philosopher, Epictetus, came to mind: “On the occasion of every accident that befalls you, turn to yourself and ask what power you have to put it to use.”  Could this array of large and small polygons form a Power Law? With curiosity piqued, I collected all the fragments and measured the area of each piece.


To demonstrate the fallacy behind the article,
<center>
Peter wrote a computer simulation, using Oscar data compiled by Mark Mixer.
{| class="wikitable"
The program generated fake Oscar winners by choosing at random from among the nominees.
|-
These fake winners exhibited the same apparent tendency to live longer that is exhibited by the genuine winners.
! Piece !! Sq. Inches !! % of Total
|-
| 1 || 43.25 || 31.9%
|-
| 2 || 35.25 ||26.0%
|-
|  3 || 23.25 || 17.2%
|-
| 4 || 14.10 || 10.4%
|-
| 5 || 7.10 || 5.2%
|-
| 6 || 4.70 || 3.5%
|-
| 7 || 3.60 || 2.7%
|-
| 8 || 3.03 || 2.2%
|-
| 9 || 0.66 || 0.5%
|-
| 10 || 0.61 || 0.5%
|}
</center>
<center>[[File:Montante_plot1.png | 500px]]</center>
The data and plot look like a Power Law distribution. The first plot is an exponential fit of percent total area. The second plot is same data on a log normal format. Clue: Ok, data fits a straight line. I found myself again in the shallow end of the knowledge curve. Does the data reflect a Power Law or something else, and if it does what does it reflect?  What insights can I gain from this accident? Favorite maxims of Epictetus and Pasteur echoed in my head:
“On the occasion of every accident that befalls you, remember to turn to yourself and inquire what power you have to turn it to use” and “Chance favors only the prepared mind.”


Despite the fact that, in their paper  Redelmeier and Singh said the data they used would be available on their website, it never was. That is why Peter had to rely on Mark for the data in his simulation. Mark's data is available  [http://www.dartmouth.edu/~chance/forwiki/ocar4.nb here] embedded in Peter's Mathematica program. If you do not have Mathematica you can read this using the free [http://www.wolfram.com/products/mathreader/ MathReader].  (We will also make Mark's data available in Excel format.)  For the paper by Hanley and his colleagues, Redelmeier and Singh did make their data available, though it was not the original data since it included the results of one more year of Oscars winners. This data is available [http://www.dartmouth.edu/~chance/forwiki/OscarData.xls here].
<center>[[File:Montante_plot2.png | 500px]]</center>
My “prepared” mind searched for answers, leading me down varied learning paths. Tapping the power of networks, I dropped a note to Chance News editor Bill Peterson. His quick web search surfaced a story from ''Nature News'' on research by Hans Herrmann, et. al. [http://www.nature.com/news/2004/040227/full/news040223-11.html Shattered eggs reveal secrets of explosions].  As described there, researchers have found power-law relationships for the fragments produced by shattering a pane of glass or breaking a solid object, such as a stone. Seems there is a science underpinning how things break and explode; potentially useful in Forensic reconstructions.
Bill also provided a link to [http://cran.r-project.org/web/packages/poweRlaw/vignettes/poweRlaw.pdf a vignette from CRAN] describing a maximum likelihood procedure for fitting a Power Law relationship. I am now learning my way through that.


Peter simulation can also be used to show that even a supposedly correct application of the Kaplan-Meier test fails to eliminate selection bias completely:  Effects to which the corrected test assigns a p-value less than .05 happen more than 5 percent of the time.  Let's try to see why.  Peter describes the corrected form of the Kaplan-Meier test as follows:
Submitted by William Montante
<blockquote>We decide to compare those who have
won an Oscar (call them ‘winners’) with those who have merely been nominated
(call them ‘also-rans’). Our ‘null hypothesis’ is that having won an Oscar doesn’t
help your health. We create a contest by associating a point to the death of
anyone who has ever been nominated for an Oscar. Points are bad: the winners
get a point if the deceased was a star; the also-rans get a point if the deceased
was an also-ran. Suppose that the deceased died in their dth day of life. Over
the course of history, some number a of nominees will have made it to the kth.
day of their lives, and been a winner on that day; some number b of nominees
will have made it to the dth day of their lives, and been an also-ran on that day.
If our hypothesis is correct, and having won an Oscar really doesn’t help your
health, then the probability that the winners get this point should be a/(a+b).
So now we’ve got a game of points. with known probability of winning for each
point.</blockquote>
The corrected Kaplan-Meier statistic is obtained by analyzing this game as if it had been determined beforehand exactly how many points would be played, together with the associated parameters a and b.  But of course this was not the case.  There are subtle interdependencies which cause the method to generate misleading p-values.


This defect of the Kaplan-Meier test is known to experts in survival analysis.  It would be worrisome, were it not for the fact that the errors it introduces are tiny compared to those resulting from the failure to understand that, as Mark Mixer so aptly observed, "breaking your hip increases your life expectancy."
----
 
 
 
 
 
===Homework===
 
Write a computer simulation showing why "breaking your hip increases your life expectancy", based on the simplest probability model you can come up with.
Analyze the results of your simulation using the corrected Kaplan-Meier test, and observe that deviations to which the corrected test ascribes p<.05 happen more than 5 percent of the time.

Latest revision as of 20:58, 17 July 2019


Forsooth

Quotations

“We know that people tend to overestimate the frequency of well-publicized, spectacular events compared with more commonplace ones; this is a well-understood phenomenon in the literature of risk assessment and leads to the truism that when statistics plays folklore, folklore always wins in a rout.”

-- Donald Kennedy (former president of Stanford University), Academic Duty, Harvard University Press, 1997, p.17

"Using scientific language and measurement doesn’t prevent a researcher from conducting flawed experiments and drawing wrong conclusions — especially when they confirm preconceptions."

-- Blaise Agüera y Arcas, Margaret Mitchell and Alexander Todoorov, quoted in: The racist history behind facial recognition, New York Times, 10 July 2019

In progress

What if the Placebo Effect Isn’t a Trick?
by Gary Greenberg, New York Times Magazine, 7 November 2018

The Problems With Risk Assessment Tools
by Chelsea Barabas, Karthik Dinakar and Colin Doyle, New York Times, 17 July 2019

Hurricane Maria deaths

Laura Kapitula sent the following to the Isolated Statisticians e-mail list:

[Why counting casualties after a hurricane is so hard]
by Jo Craven McGinty, Wall Street Journal, 7 September 2018

The article is subtitled: Indirect deaths—such as those caused by gaps in medication—can occur months after a storm, complicating tallies

Laura noted that

Did 4,645 people die in Hurricane Maria? Nope.
by Glenn Kessler, Washington Post, 1 June 2018

The source of the 4645 figure is a NEJM article. Point estimate, the 95% confidence interval ran from 793 to 8498.

President Trump has asserted that the actual number is 6 to 18. The Post article notes that Puerto Rican official had asked researchers at George Washington University to do an estimate of the death toll. That work is not complete. George Washington University study

We sttill don’t know how many people died because of Katrina
by Carl Bialik, FiveThirtyEight, 26 August 2015

These 3 Hurricane Misconceptions Can Be Dangerous. Scientists Want to Clear Them Up.
Misinterpretations of the “Cone of Uncertainty” in Florida during the 2004 Hurricane Season
Definition of the NHC Track Forecast Cone

Remember when a glass of wine a day was good for you? Here's why that changed. Popular Science, 10 September 2018

Googling the news
Economist, 1 September 2018

We sat in on an internal Google meeting where they talked about changing the search algorithm — here's what we learned

Reading , Writing and Risk Literacy

[1]

Today is the deadliest day of the year for car wrecks in the U.S.

Some math doodles

<math>P \left({A_1 \cup A_2}\right) = P\left({A_1}\right) + P\left({A_2}\right) -P \left({A_1 \cap A_2}\right)</math>

<math>P(E) = {n \choose k} p^k (1-p)^{ n-k}</math>

<math>\hat{p}(H|H)</math>

<math>\hat{p}(H|HH)</math>

Accidental insights

My collective understanding of Power Laws would fit beneath the shallow end of the long tail. Curiosity, however, easily fills the fat end. I long have been intrigued by the concept and the surprisingly common appearance of power laws in varied natural, social and organizational dynamics. But, am I just seeing a statistical novelty or is there meaning and utility in Power Law relationships? Here’s a case in point.

While carrying a pair of 10 lb. hand weights one, by chance, slipped from my grasp and fell onto a piece of ceramic tile I had left on the carpeted floor. The fractured tile was inconsequential, meant for the trash.

BrokenTile.jpg

As I stared, slightly annoyed, at the mess, a favorite maxim of the Greek philosopher, Epictetus, came to mind: “On the occasion of every accident that befalls you, turn to yourself and ask what power you have to put it to use.” Could this array of large and small polygons form a Power Law? With curiosity piqued, I collected all the fragments and measured the area of each piece.

Piece Sq. Inches % of Total
1 43.25 31.9%
2 35.25 26.0%
3 23.25 17.2%
4 14.10 10.4%
5 7.10 5.2%
6 4.70 3.5%
7 3.60 2.7%
8 3.03 2.2%
9 0.66 0.5%
10 0.61 0.5%
Montante plot1.png

The data and plot look like a Power Law distribution. The first plot is an exponential fit of percent total area. The second plot is same data on a log normal format. Clue: Ok, data fits a straight line. I found myself again in the shallow end of the knowledge curve. Does the data reflect a Power Law or something else, and if it does what does it reflect? What insights can I gain from this accident? Favorite maxims of Epictetus and Pasteur echoed in my head: “On the occasion of every accident that befalls you, remember to turn to yourself and inquire what power you have to turn it to use” and “Chance favors only the prepared mind.”

Montante plot2.png

My “prepared” mind searched for answers, leading me down varied learning paths. Tapping the power of networks, I dropped a note to Chance News editor Bill Peterson. His quick web search surfaced a story from Nature News on research by Hans Herrmann, et. al. Shattered eggs reveal secrets of explosions. As described there, researchers have found power-law relationships for the fragments produced by shattering a pane of glass or breaking a solid object, such as a stone. Seems there is a science underpinning how things break and explode; potentially useful in Forensic reconstructions. Bill also provided a link to a vignette from CRAN describing a maximum likelihood procedure for fitting a Power Law relationship. I am now learning my way through that.

Submitted by William Montante

Retrieved from "https://www.causeweb.org/wiki/chance/index.php?title=Sandbox&oldid=22455"