# Testing

• ### What P-Value Means

May sing to tune of "Row, Row, Row Your Boat"

It is key to know
What p-value means --

It's the chance
(with the null)
you obtain
data that's

At least that extreme!

• ### I Will Find U

may sing to tune of "I Will Find You" (Peter Hammill)

Trapped by your habit when you must compare,
Two-sample t-tests that blind you
I have a better way;
each observation pair,
I'll assess, greater/less; I will find U.

(For) independent samples X and Y,
Mann-Whitney's test, I'll remind you,
Is so simple--add a half for every tie,
Else just sum, naughts and ones, and you'll find U.

Far away in a normal test
You'll find you don't have so much freedom;
Don't run away to a normal test.

Don't be afraid should it be unknown
What distribution's before you.
Don't be afraid (with) symmetry alone,
Lose no more, (than) point-one-four, if you find U.

Far away in a normal test
Things cannot be much more efficient;
Don't run away to a normal test...

Trapped by your habit when you must compare,
Two-sample t-tests that blind you
I have a better way;
each observation pair,
I'll assess, greater/less; I will find U.

• ### Chi-Square For Us

may be sung to the tune of "Aquarius" (James Rado, Gerome Ragni, Galt MacDermot)

When you have qualitative data
And you need to test goodness of fit
Or do a test of independence,
Then this fine tool is it!
This is the dawning of the age of chi-square for us,
Age of chi-square for us,
Chi-square for us!
Chi-square for us!

It's the oldest test we use now--
In nineteen-hundred, Pearson showed us how
Reasoning with simple rat'os
yield expected frequencies that go in formula summation
note degrees of libera-ation:
chi-square for us, Chi-square for us!

(repeat first section)

• ### It's .05

Lyric copyright by Marc Coram and Matthew Finkelman
may sing to the tune of "My Boyfriend's Back" (Robert Feldman, Gerald Goldstein and Richard Gottehre)

The p-value's low and I'm gonna get it published
Heh-na, heh-na, it's .05

The p-value's low and the journal isn't rubbish
Heh-na, heh-na, it's .05

Yeah, my paper got accepted
Null hypothesis rejected
Heh-na, heh-na, it's .05

The effect is microscopic but I got four thousand patients
Heh-na, heh-na, it's .05

Now people will forget my bad TA evaluations
Heh-na, heh-na, it's .05

Yeah, my paper got accepted
Null hypothesis rejected
Heh-na, heh-na, it's .05
Heh-na, heh-na, it's .05
Heh-na, heh-na, it's .05

• ### Test on Mu (to the other side)

Lyric copyright by Dennis K. Pearl
may sing to the tune of "Break on Through (to the other side)" (The Doors)

You know the tail's to the right
Root-n divides the s
Tried the left
Tried the right
Test on mu to the other side
Test on mu to the other side
Test on mu to the other side, yeah

We placed our measures here
valid measures there
But can you still recall
Testing just one side
Test on mu to the other side
Test on mu to the other side

Yeah!
C'mon, yeah

Both ways reject my null
Both ways reject my null
mean was low
mean was high
mean was low
mean was high

I landed an alternative
Both ways can give
Evidence to claim
The null's a lie
Test on mu to the other side
Test on mu to the other side
Test on mu, oww!
Oh, yeah!

Peak to Peak
Way to weigh
Power to Power
State and negate
z's the guide
Test on mu to the other side
Test on mu to the other side
Test on mu
Test on mu
Test on mu
Test on mu
Yeah, yeah, yeah, yeah
Yeah, yeah, yeah, yeah, yeah

• ### Oops! ... I Picked The Wrong Test

may sing to tune of "Oops! ... I Did It Again" (Britney Spears)

I think I did it again
Oh, Chuck, I have run a Z-test,
But that doesn't mean I know what's best
But to lo-ose my σ, that is just so typically me
Oh baby baby

Oops I did it again
I picked the wrong test
Got invalid results
Oh, baby baby
Oops I ran a two-tailed
When I wanted just one tail
For my hypothesis

You see my problem is this
I'm running t-tests
Wishing my groups they were independent
I cry watching the curve
'Cause if I reject, I've got nerve
But to lose degrees of freedom
That is just so typically me
Oh, baby, Oh baby baby

Oops I did it again, I misread the graph
Where's the interaction?
Oh, baby, baby
Oops, are those parallel lines
And where are the main effects
Is this even an ANOVA?

[Dream Sequence]
Chuck: Britney, Britney, I got these for you.
Britney: But I thought the old lady archived these down in the basement of the library
Chuck: I went down and got them for you.
Britney: Awww, you shouldn't have...
Chuck: No, you really need these

Oops I ran a Pearson's for my test
I tried to correlate categories
Oops you think that I needed a Χ²
For my nominal data

Oops I did it again, rejected the null
Didn't understand p
I thought bigger's better
On my SPSS output
I'm not that innocent

Oops I did it again
I picked the wrong test, got invalid results
Oh, baby baby
Oops I ran a two-way
When I wanted a three-way
For my hypothesis.

Watch the video

• ### Scientific Jam

Lyric and music copyright by Jeff Hale

If you guess at the answer, that's a hypothesis.
The scientific method can make you very smart.
If you want to solve a problem, that's a good place to start.

chorus :
Is your hypothesis wrong or right?
Do a test that will shed some light.
If your experiment goes as planned.

You've collected all the data. Now your searching for a clue.
You just might want to graph it so you'll have a picture too.
With careful observations you've really thought it through.

chorus :
Is your hypothesis wrong or right?
Do a test that will shed some light.
If your experiment goes as planned.

bridge
Make a guess, hypothesize, do a test and experiment.
Think it through, analyze, then conclude with a great surprise

Now just in case you made a major breakthrough,
do the test again just to make sure it's cool.
Some other science dudes just may want to do your gig
A repeatable experiment is something they can dig.

repeat chorus

• ### If Only P were greater

may sing to the tune of "Drops of Jupiter (Tell Me)" (Patrick Monahan, Robert Hotchkiss, James Stafford, Scott Underwood, and Charlie Colin)

Backed by evidence, I've made a new guess
My alternative hypothesis, hey--
I thought you said you still loved me too
But I've rejected this null, it just ain't true, hey--
Why do I care for you when I've
Found the p-value is less than point-oh-five, hey-- hey--

But tell me, is there practical relevance
To stop reaching out to you?
Though statistical significance,
A circumstance unlikely due to chance

Tells me, I don't mean much to you
And there's nothing left to do
Sometimes the difference seems small at best
Should I run another test?

Well if p is low, I guess null must go
Although I'd much prefer Ho to stay--
But the chances you feel the way I do
Are under five percent (wish it wasn't true), hey--
If my results are right, I'm not quite sure
Perhaps I made a Type I error? hey-- hey--

But tell me, to stop reaching out to you,
Is there practical relevance?
Though statistical significance,
A circumstance unlikely due to chance

Tells me, I don't mean that much to you
And there's nothing left to do
I feel the difference is small at best
Maybe I'll run another test?

Could you imagine no Type I error,
Saying things are out of place and the feelings are gone
Even when they're really not

Could you imagine if p was really high
And I did not have to question
The significance of the results

Tell me, does significance mean a thing?
If there exists another possibility
I'll need some new hypotheses

And tell me, is there practical relevance?
Between results I'd expect from null
And those I actually got
Is the difference smaller than I thought?

Tell me, 'cause I'm not so sure
If I made a Type I error
Feelin' so lonely, guess,
I'll need to make another test

Maybe I made a Type I error
Should I still reject Ho?
Still reject Ho?
Or maybe I should set my alpha at less than point-oh-five

• ### ANOVA Table's Coming to Town

May sing to the tune of "Santa Claus is Coming to Town" (John Frederick Coots and Haven Gillespie)

You better watch error;
they better be tight.
Better be normal,
or it don't work right.
ANOVA Table's coming to town!

It's summing those squares,
And checking 'em twice;
Variance 'tween or within a slice.
ANOVA Table's coming to town.
ANOVA Table's coming to town.
ANOVA Table's coming to town!

It sees the data you've been keeping;
...but not if they are fake.
It knows if they're within or between;
So between for goodness sake!

Oh - You better watch error;
they better be tight.
Better be normal,
or it don't work right.

ANOVA Table's coming to town.
ANOVA Table's coming to town.
ANOVA Table's coming to town!

• ### The P-Value Song

By Michael Greenacre & Gurdeep Stephens
Barcelona, Catalonia & Victoria, Canada, March 2015

Statistics, logistics, cladistics seem to me
To have a common theme scientifically,
Economists, biologists, with PhD degrees,
They all need some proof of their theories.
A letter is the key, you'll see clearly,
Not B nor G nor V -- but it's the P !

There's no values like P-values
Like no values I know
Think of something that is not worth proving,
An hypothesis that everyone calls null,
If your P is too large to reject it
Then your experiment is rather dull.

There's no values like P-values,
Especially when they are low,
Just try again with samples twice the size,
Everything is possible, just trust in me:
Put your faith in the P.

The F test, the Z test, the chi-square and the T
And other cryptic terminology
Anova, regression, tests distribution-free,
They all need some sort of guarantee.
So if you find a tiny effect size
The P-value will be a good disguise.

There's no values like P-values,
The frequentist's hero,
When you get that data modeling feeling
But results you have are not a lot,
You will need some stats that are appealing
To show the journals your work is hot!

There's no values like P-values
Especially when they're low