Chance News 14

Quotation

The [Supreme] Court concluded that mental health professionals' predictions were "not always wrong...only most of the time."

Gerd Gigerenzer

Gerd Gigerenzer's Calculated Risks Revisited

In 2002 http://www.dartmouth.edu/%7Echance/chance_news/recent_news/chance_news_11.03.html#item10 Chance News 11/03] had a lengthy and very positive review of Gerd Gigerenzer's book, Calculated Risks: How To Know When Numbers Deceive You, . Readers are urged to download that excellent review because of the information contained. However, the book is so good and so persuasive that it is worth another look in order to alert readers to some other aspects of the book and how it relates to subsequent events.

The aforementioned review did not mention the abundant number of actual, real-world incidents cited in which doctors, lawyers and social workers, not to mention patients, clients and jurors, were unable to unscramble the difference between P(X| Y) and P(Y| X). Also not mentioned was Gigerenzer's dim view of screening for breast cancer and prostate cancer. Screening may be defined, according to H. Gilbert Welch, as "the systematic examination of asymptomatic people to detect and treat disease." See Chance News 12 for a review of Welch's 2004 book, Should I Be Tested For Cancer? Maybe Not And Here's Why. Welch echoes and amplifies Gigerenzer contention that (mass) screening is counterproductive, especially when there is little evidence that a cure exists. Just to complicate matters, however, see a 2005 Chance News wiki which indicates that mammography screening does reduce the death rate of breast cancer, http://chance.dartmouth.edu/chancewiki/index.php/Chance_News_8#Mammograms_Validated_as_Key_in_Cancer_Fight. Unfortunately, the article in the New England Journal of Medicine does not explain why mammogram screening is deemed responsible for 28 to 65% of the 24% drop in the breast cancer death rate. Gigerenzer would prefer, and this is one of his main points, that any statistical data be given in counts rather than in percentages, especially percentages without a base rate, such as relative risk which he views as the most misleading. On page 60 is found the following table to illustrate this point

                       Treatment                                               Deaths per 1000 women

No mammography screening 4 Mammography screening 3 Consequently, there is "a 25 percent relative risk reduction." He would prefer focusing on the difference in the number of deaths which yields the more revealing and perhaps more honest statement: "The absolute risk reduction is 4 minus 3, that is, 1 out of 1000 women (which corresponds to .1 percent)." However, "Counting on their clients' innumeracy, organizations that want to impress upon clients the benefits of treatment generally report them in terms of relative risk reduction...applicants [for grants] often feel compelled to report relative risk reductions because they sound more impressive." Although he did not use this example, one's relative "risk" of winning the lottery is infinitely greater if one buys a ticket, yet one's absolute "risk" of winning has hardly improved at all. Most of his numerical examples are typified by his discussion of the picture given below which indicates the superiority of dealing with counts. Note that "H" represents having the disease and "D" represents a diagnosis having the symptom as seen by testing positive. Characteristically, there is a large number in the population who do not have the disease and because of the possibility of a wrong classification, the number of false positives (99) outweighs the number of true positives (8) resulting in P(disease| symptom) being much lower (8/(8+99)) than P(symptom| disease) (.8). This type of result, low probability of disease given symptom, is true even when ".8" is replaced by a number much closer to 1 provided there are many more who do not have the disease.

Here is an example he did not consider but it also illustrates the superiority of dealing with counts. Instead of two populations--diseased and healthy--which are greatly different in size, consider Boys and Girls and the desire to predict gender based on some simple test. Assume that 50% of births are Boys so that P(Boy) = P(Girl) = 1/2. A simple, inexpensive, non-invasive gender-testing procedure indicates that it is "perfect" for boys, P(Test Boy| Boy) = 1, implying P(Test Girl| Boy) = 0. Unfortunately, this simple, inexpensive, non-invasive gender-testing procedure for girls is a "coin toss," P(Test Girl| Girl) = P(Test Boy| Girl) = 1/2. Application of Bayes theorem yields what seems to be a strange inversion, P(Boy| Test Boy) = 2/3 and P(Girl| Test Girl) = 1. That is, somehow, "perfection" switched from Boy to Girl. The test is perfect in "confirming" that a Boy is a Boy and has a 50% error rate in confirming that a Girl is a Girl. The test is perfect in "predicting" that a person who tests as a girl is in fact a girl but has 33% error rate in predicting that a person who tests as a Boy is in fact a Boy. Thus, the term perfect is ambiguous. Perfection in confirmation, i.e., the test conditional on the gender, does not mean perfection in prediction, i.e., the gender conditional on the test.

Some of the puzzlement disappears if we deal with counts; the table below is equivalent to Gigerenzer's "tree" diagram. Assume 50 Boys and 50 Girls to start with. Every one of the 50 Boys will test as a Boy--none of the Boys test as a Girl; of the 50 Girls, 25 will test as a Boy and 25 will test as a girl. Therefore, P(Girl| Test Girl) = 1. One is tempted to to explain the switch by using the lingo of medical testing: false positives, false negatives, sensitivity, specificity, positive predictive value, negative predictive value. However, one hesitates to designate either gender as diseased even though the mathematics is the same.

                               Test Boy                   Test Girl              Total

Boy 50 0 50

           Girl                             25                             25                      50

Gigerenzer rightly concludes that the language of statistics is not natural for most individuals. Perhaps the puzzlement in this specific example is at least partly due to the natural language known as English. Boys, Girls, Test Boys and Test Girls are too confusing. . Replace "Boy" by "Norwegian" and "Girl" by "German" and assume that there are as many Norwegians as Germans. Let every Norwegian be "Blond," so that P(Blond| Norwegian) = 1 and only half the Germans are Blond. Thus, P(German| Not Blond) =1; the switch, P(German| Not Blond) = P(Blond| Norwegian) = 1, is rather obvious. Is the this situation easier to understand because of the linguistics--hair color and ethnicity are easily distinct as Test Boy and Boy are not?

Discussion Questions 1. Gigerenzer has a chapter entitled, "(Un)Informed Consent." Based on your experience, what do you imagine the chapter contains? 2. A drawing of two tables (that is, physical tables on which things are placed) appears on page 10. He claims the tables (due to Roger Shepard) are identical in size and shape. After staring at them in disbelief of the claim, how would you verify the contention? 3. Physicians sometime make the following type of statement:"Never mind the statistics, I treat every patient as an individual." Defend this assertion. Criticize this assertion. 4. The physicist, Lord Rutherford, is reputed to have said, " If your experiment, needs statistics you ought to have done a better experiment." Defend and criticize Lord Rutherford. 5. Assume an asymptomatic woman has a mammogram which looks suspicious and then a biopsy which is negative. Would she be grateful for the clean bill of health or would she become an advocate who opposes (mass) screening? Suppose instead we assume a man has a suspiciously high PSA and the painful multiple biopsies (6-12 "sticks") are all negative. Would he be grateful for the clean bill of health or would he become an advocate who opposes (mass) screening? 6. Calculated Risks also deals with the risk to the physician making a recommendation and a diagnosis. Discuss why in our present-day litigious society the risks to the physician (who may or may not recommend a test or may or may not make a diagnosis) are not symmetrical. Along these lines, who are the vested interests involved in maintaining screening and testing? 7. Revisit the Boy/Girl scenario but now the test always says Boy regardless of gender, P(Test Boy| Boy) = P(Test Boy| Girl) = 1. Complete the table for this version. Obviously, this test has the advantage of being extremely simple, cost-free and non-invasive. Use either the Probability Format or the Frequency Format to comment on the statistical worthiness of this test.