All Statistics are Wrong; Some Statistics are Useful

Project MOSAIC with support from NSF DUE-0920350

What do our students need to know to make informed decisions?

• Personal decisions — e.g. medical, financial
• Professional decisions — e.g. what skills to seek

What broad skills will our students need?

• In the workplace
• In interpreting the news
• In relating to scientific findings

What is the bottleneck for our students?

• Not finding a p-value

Huge amounts of data are being generated

• Outside of experimental settings
• Often without a design

Huge amounts of data are being generated

• Outside of experimental settings
• Often without a design

Students need to be prepared for a world in which:

• The economy is more invested in drawing useful conclusions from data than ever before
• Science is more driven by large amounts of data
• Personal decisions — medical, educational — connect with the research literature

Huge amounts of data are being generated

Students need to be prepared for a new world

Individuals and the media believe that data is knowledge

• They want to know how to extract useful knowledge from data
• They generally are not aware of the limitations of observational data

Work is based in teams

• Collaboration, evaluation, specialization
• The model of exchanged notes (e.g. email) has broken down

Work is based in teams

• Collaboration, evaluation, specialization
• The model of exchanged notes (e.g. email) has broken down

Publication is instant

• Old model: Get data, draft, redraft, publish
• New model: Get data, draft, publish, comment, revise, publish, new data and comment, revise and update, publish, …

Work is based in teams

• Collaboration, evaluation, specialization
• The model of exchanged notes (e.g. email) has broken down

Publication is instant

• Old model: Get data, draft, redraft, publish
• New model: Get data, draft, publish, comment, revise, publish, new data and comment, revise and update, publish, …

Topics are more complex

• From gene to genomics
• From inventory to logistics
• From treatment to medical systems

The need to compute with data has increased dramatically.

What do people need to know about computing?

• The organization of data
• A basis for learning what else they need to know about computing

But who is teaching computing?

Everyone teaches narrowly to their purposes:

• Computer scientists and programming
• Mathematicians and computer algebra
• Statistics and p-values

Laboratory Skills

• Traditional motivation (e.g. Fisher's “Techniques”)
• Now available by software

Statistical Concepts

• Evidence from Data
• Covariation and Adjustment
• Inference about causation

Trends in the evolving world position the concepts of statistics more centrally than ever

• But these changes have not been driven by statistics

Still rooted in the needs of the laboratory and in the pedagogy of traditional mathematics

• An interest in provable statements and traditional curricula
• “The Ptolemaic Curriculum” of t-tests and tables
• “No causation without experimentation”

You have a handout.

• Draw a star by points you agree with and an X through points you disagree with

• Feel free to list other trends you think are important and should inform the design of statistics education

Take a Couple of Minutes

Talk to your neighbors!

The reason we ask …

• If not now, when?
• If not us, who?

If we are not the agents of change, who will be?

• The computer scientists should teach computation
• Pedagogy should follow historical development: ontogeny recapitulates phylogeny
• Statistics receiving appropriate status a peer of the classical disciplines
• There is only so much we can do. Others have to step up to the plate as well
• We will need more resources to do …

Let's start by looking at where we are.

Then we can decide if this leads where we want to be.

[Source: Introduction to the Practice of Statistics (Moore, McCabe and Craig)]

[Source: Introduction to the Practice of Statistics (Moore, McCabe and Craig)]

Turn over your handout

• Sketch out a graphic you think is iconic.
  

Take a Couple of Minutes

Share this with your neighbors!

The statistician who supposes that his main contribution to the planning of an experiment will involve statistical theory, finds repeatedly that he makes his most valuable contribution simply by persuading the investigator to explain why he wishes to do the experiment. — Gertrude Cox

• Even when you do an experiment, you may want to use covariates.

• Especially because experiments are generally not perfect.

Nobody believes a theory, except the person who made it. Everybody believes an experiment, except the person who made it. — Albert Einstein

From http://XKCD.com/552

The “Directed Acyclic Graph”

[source: Introduction to the Practice of Statistics (Moore, McCabe and Craig)]

We need a new source of inspiration and guidance

• Suited for working with data that are dirty, limited, and ambiguous
• Oriented toward decision making and action rather than proof

Originated as the study of the spread of disease: epidemics.

Much more broadly construed now:

• The study of the determinants of health and disease for the purpose of useful intervention

Decisions need to be made

• with data that are limited and dirty
• as quickly as possible
• constructing the best study from available resources
• in the presence of uncontrollable conditions

Succeeded Fisher as president of the Royal Society

“[I]n passing from association to causation I believe in 'real life' we shall have to consider what flows from that decision. … In asking for very strong evidence I would, however, repeat emphatically that this does not imply crossing every t', and swords with every critic, before we act.” AB Hill,

“[T]he glitter of the t table diverts attention from the inadequacies of the fare.” — AB Hill

“The environment and disease: association or causation?” (1965) Proceedings of the Royal Society of Medicine 58:295-300 link

A Daily Habit Of Green Tea Or Coffee Cuts Stroke Risk

by Allison Aubrey, NPR - March 15, 2013

A daily habit of coffee or tea drinking cuts the risk of stroke by 20%, according to a report in the American Heart Association journal Stroke.

… recent studies have linked a regular coffee habit to a range of benefits — from a reduced risk of Type 2 diabetes to a protective effect against Parkinson's disease.

It's interesting to note how much the thinking about caffeine and coffee has changed.

In the 1980s, surveys found that many Americans were trying to avoid it; caffeine was thought to be harmful, even at moderate doses.

One reason? Meir Stampfer of the Harvard School of Public Health says back then, coffee drinkers also tended to be heavy smokers. And in early studies, it was very tough to disentangle the two habits.

“So it made coffee look bad in terms of health outcomes,” says Stampfer.

But as newer studies began to separate out the effects of coffee and tea, a new picture emerged suggesting benefits, not risks.

Researchers say there's still a lot to learn here — they haven't nailed down all the mechanisms by which coffee and tea influence our health. Nor have they ruled out that it may be other lifestyle habits among coffee and tea drinkers that's leading to the reduced risk of disease.

• How to take covariates into account
• How to choose covariates
• How to discount previous studies without discounting statistical method

[Source: 2011 Form B AP Statistics exam]

• What's the purpose of collecting the data?
• What covariates are there?

Is this a meaningful alternative?

There is one letter different between H\( _0 \) and H\( _a \)

Possible role of other factors?

> summary(lm(sat ~ salary, data=SAT))

Coefficients:
            Estimate  Std. Error  p-value   
(Intercept)  1158.86      57.66   <0.0001 
salary         -5.54       1.63    0.0014 

What should the advisory board conclude?

• Increasing sophisticated of statistical methods in the New England Journal of Medicine (NEJM, 2005; CHANCE, 2007)
• Categorized papers using hierarchy due to Emerson and Colditz
• Multiple regression use increased from 5% (1979) to 14% (1989) to more than 50% (2004)

Coefficients:
            Estimate  Std. Error  p-value   
(Intercept)  987.900     31.877   <0.0001
salary         2.180      1.029    0.039   
frac          -2.779      0.228   <0.0001

Conclusion: A somewhat positive relationship of SAT scores with teacher salary (after controlling for fraction taking the SAT)

• Allows students to delve into more complex causal relationships
• Prepares them to ask leading questions when analyzing observational data
• Doesn't paralyze them (a la XKCD)

Use Stratification Rather than Regression

Use Stratification Rather than Regression

Simple multiple regression

• Y ~ X + Z (parallel slopes)
• Y ~ X * Z (interaction)

• introduce simple linear regression early (week 1, purely descriptive)
• introduce multiple regression immediately afterwards (week 2, purely descriptive)
• build concepts of sampling distributions and inference for an interval
• jump to inference for simple linear regression (week 9)
• extend to inference for multiple regression (week 10)
• projects or extended case studies to close out the class

Statistical Literacy

Computation/Data Literacy

Mathematics for modeling and description

Statistical modeling

Epidemiology

• Description, e.g. mortality rates
• Study design, e.g. Case-Control
• Statistics, e.g. relative risk, odds-ratios and CI

• Modeling
• Multiple variables, linear algebra/geometry
• Oriented to support follow-up statistics course

Central Question: Which covariates to include in a model?

• Partial versus total change
• Strength of Evidence
• Causality

An NSF supported collaboration to increase connections among

Modeling Statistics Computation & Calculus

http://uscots2013.mosaic-web.org/

Will this become an iconic image of change in statistics education?

An emphasis on decision-making

Confounding is a reality: Deal with it

Make Statistics about Modeling