{"id":521,"date":"2014-12-22T17:05:35","date_gmt":"2014-12-22T22:05:35","guid":{"rendered":"https:\/\/www.causeweb.org\/sbi\/?p=521"},"modified":"2014-12-23T13:59:22","modified_gmt":"2014-12-23T18:59:22","slug":"what-teachers-should-know-about-the-bootstrap-resampling-in-the-undergraduate-statistics-curriculum","status":"publish","type":"post","link":"https:\/\/www.causeweb.org\/sbi\/?p=521","title":{"rendered":"What teachers should know about the Bootstrap: Resampling in the undergraduate statistics curriculum"},"content":{"rendered":"<p><strong>Tim Hesterberg &#8211; Google<\/strong><\/p>\n<p><strong><a href=\"https:\/\/www.causeweb.org\/sbi\/wp-content\/uploads\/2015\/01\/hesterberg2.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignleft wp-image-552 size-full\" src=\"https:\/\/www.causeweb.org\/sbi\/wp-content\/uploads\/2015\/01\/hesterberg2.jpg\" alt=\"hesterberg2\" width=\"245\" height=\"252\" \/><\/a><\/strong><\/p>\n<p>Here are some arguments for why we\u00a0should not use bootstrap methods and permutation tests in\u00a0teaching Stat 101:<\/p>\n<ul>\n<li>Our usual cookbooks of formulas is such a resounding success,\u00a0inspiring generations of students to further study\u00a0(and rewarding their instructors with stellar reviews),<\/li>\n<\/ul>\n<p>[pullquote]Bootstrapping and permutation\u00a0tests make hard abstract concepts like sampling distributions, p-values, standard errors, and confidence intervals more concrete;[\/pullquote] <!--more--><\/p>\n<ul>\n<li>Students find concepts like sampling distributions, p-values,\u00a0standard errors, and confidence intervals to be intuitive,<\/li>\n<li>Graphs\u00a0that demonstrate these concepts would just confuse them,<\/li>\n<li>Students welcome the mental gymnastics involved in working\u00a0with a pivotal statistic like <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.causeweb.org\/sbi\/wp-content\/uploads\/2014\/12\/image001.png\" alt=\"image001\" width=\"91\" height=\"20\" \/>,\u00a0rather than working directly with\u00a0<img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-543\" src=\"https:\/\/www.causeweb.org\/sbi\/wp-content\/uploads\/2014\/12\/image002.png\" alt=\"image002\" width=\"17\" height=\"19\" \/>.<\/li>\n<li>Students do so well with formulas that they don&#8217;t need a\u00a0way to check their work.<\/li>\n<\/ul>\n<p>I&#8217;m being facetious, of course. Bootstrapping and permutation\u00a0tests make hard abstract concepts like sampling distributions, p-values, standard errors, and confidence intervals more concrete; students can visualize them using histograms of bootstrap or\u00a0permutation distributions. They can use tools they earlier applied\u00a0to data &#8211; histograms, normal quantile plots, and numerical summaries -to these resampling distributions. They can work directly with the statistics of interest, like a mean, or difference of means.<\/p>\n<p>These methods also let us do better statistics. We talk early in\u00a0Stat 101 about the advantages of a median. But when it is time\u00a0for confidence intervals and hypothesis tests, we try to ignore it,\u00a0like a crazy uncle in a closet, because we don&#8217;t have easy\u00a0formulas. With resampling we can use the statistics we want to,\u00a0without developing new formulas for inferences for every statistic.\u00a0So we can use the mean, or median, or heck, even a midmean\u00a0(aka 25% trimmed mean, the mean of the middle 50% of the observations).\u00a0We can use robust regression.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignright wp-image-548 size-medium\" src=\"https:\/\/www.causeweb.org\/sbi\/wp-content\/uploads\/2015\/01\/bootstrapGoodFit-300x300.png\" alt=\"bootstrapGoodFit\" width=\"300\" height=\"300\" srcset=\"https:\/\/www.causeweb.org\/sbi\/wp-content\/uploads\/2015\/01\/bootstrapGoodFit-300x300.png 300w, https:\/\/www.causeweb.org\/sbi\/wp-content\/uploads\/2015\/01\/bootstrapGoodFit-150x150.png 150w, https:\/\/www.causeweb.org\/sbi\/wp-content\/uploads\/2015\/01\/bootstrapGoodFit.png 480w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/>Speaking of regression, bootstrapping regression lines gives a nice\u00a0picture of how accurate\u00a0predictions are, and how the more you extrapolate\u00a0the less accurate the predictions are. Doing these lines superimposed\u00a0over the original data also helps demonstrate another idea,\u00a0the difference between a confidence interval and a prediction interval&#8212;students can see that the range of predictions from the regression\u00a0lines (a confidence interval) captures only a small fraction of the\u00a0observations.<\/p>\n<p>The very process of resampling, bootstrapping in particular, reinforces\u00a0statistical ideas. In bootstrapping we draw samples from a population\u00a0to estimate how the statistic varies due to random sampling &#8211; this\u00a0reinforces the central role that random sampling plays in obtaining\u00a0data.<\/p>\n<p>The methods are important in practice, too. I work at a little\u00a0internet search company called Google. In an <a href=\"http:\/\/arxiv.org\/abs\/1411.5279\" target=\"_blank\">article\u00a0<\/a>&#8220;What Teachers Should Know about the Bootstrap: Resampling Within the Undergraduate Statistics Curriculum&#8221; that I recently placed on ArXiv, I give a number of examples where we use resampling, because\u00a0it is the only practical way to get answers, or to avoid bias.<\/p>\n<p>Check out that article; I talk about<\/p>\n<ul>\n<li>Pedagogical advantages of resampling,<\/li>\n<li>The idea behind the bootstrap,<\/li>\n<li>What you can and cannot do with it,<\/li>\n<li>Things to watch out for (like the sample standard deviation),<\/li>\n<li>Accuracy.<\/li>\n<\/ul>\n<p>What do I mean by accuracy? You&#8217;re probably familiar with the old\u00a0rule &#8220;if <em>n<\/em> <span style=\"text-decoration: underline;\">&gt;<\/span> 30 and the sample is not too skewed,&#8221; then it is OK\u00a0to rely on the CLT and use <em>t<\/em>-test and intervals. Tell me now &#8211; what\u00a0does &#8220;not too skewed&#8221; mean? What if my sample is size 50, is it OK then?\u00a0How accurate is it? What if the sample is skewed, then how\u00a0big does n have to be? Can&#8217;t answer, huh? Not too useful a rule, is it?<\/p>\n<p>In the article, I describe how to use either bootstrap methods, or\u00a0some relatively easy formulas, to estimate how accurate the standard\u00a0<em>t<\/em> methods are.<\/p>\n<p>So take a guess &#8211; how large does <em>n<\/em> have to be, before <em>t<\/em> methods\u00a0are reasonably accurate, if the population has the skewness\u00a0of an exponential distribution? You&#8217;re too low, guess again.<\/p>\n<p>Nope, still too low \ud83d\ude42<\/p>\n<p>You need <em>n<\/em> <span style=\"text-decoration: underline;\">&gt;<\/span>\u00a05000 before <em>t<\/em> methods are reasonably accurate\u00a0(probabilities are within 10% of 0.025 on each side). \u00a0[pullquote]The CLT operates on glacial time scales, when there is skewness.[\/pullquote]<\/p>\n<p>I also talk about which bootstrap methods are more accurate.\u00a0Some things will surprise you. You may think about using\u00a0the bootstrap in small samples, and <em>t<\/em> methods for larger samples.\u00a0But that is backward, at least for the bootstrap percentile interval&#8212;\u00a0it is horrible\u00a0for small samples (it is too narrow), but better than <em>t\u00a0<\/em>methods for large samples (<em>n<\/em> &gt; 34, for an exponential population).<\/p>\n<table>\n<tbody>\n<tr>\n<td bgcolor=\"#FAEBD7\">\n<p style=\"text-align: justify;\"><a href=\"https:\/\/www.causeweb.org\/sbi\/wp-content\/uploads\/2015\/01\/coverage.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-549 size-medium\" src=\"https:\/\/www.causeweb.org\/sbi\/wp-content\/uploads\/2015\/01\/coverage-300x284.jpg\" alt=\"coverage\" width=\"300\" height=\"284\" srcset=\"https:\/\/www.causeweb.org\/sbi\/wp-content\/uploads\/2015\/01\/coverage-300x284.jpg 300w, https:\/\/www.causeweb.org\/sbi\/wp-content\/uploads\/2015\/01\/coverage.jpg 604w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Figure 21 from Hesterberg(2014):\u00a0The\u00a0lines with codes are non-coverage probabilities on the right,\u00a0where the interval\u00a0is below the parameter. The lines without codes correspond to the left side.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Another thing &#8211; when the sample is skewed, the bootstrap percentile\u00a0interval is asymmetrical &#8211; but not asymmetrical enough by a factor\u00a0of 3. For positively skewed data, a good interval needs to reach\u00a0way to the right.<\/p>\n<p>So check out the article &#8211; <a href=\"http:\/\/arxiv.org\/abs\/1411.5279\">http:\/\/arxiv.org\/abs\/1411.5279<\/a> .<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Tim Hesterberg &#8211; Google Here are some arguments for why we\u00a0should not use bootstrap methods and permutation tests in\u00a0teaching Stat 101: Our usual cookbooks of formulas is such a resounding success,\u00a0inspiring generations of students to further study\u00a0(and rewarding their instructors with stellar reviews), [pullquote]Bootstrapping and permutation\u00a0tests make hard abstract concepts like sampling distributions, p-values, standard [&hellip;]<\/p>\n","protected":false},"author":21,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[14],"tags":[],"class_list":["post-521","post","type-post","status-publish","format-standard","hentry","category-5-should-i-teach-bootstrapping"],"_links":{"self":[{"href":"https:\/\/www.causeweb.org\/sbi\/index.php?rest_route=\/wp\/v2\/posts\/521","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.causeweb.org\/sbi\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.causeweb.org\/sbi\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.causeweb.org\/sbi\/index.php?rest_route=\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/www.causeweb.org\/sbi\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=521"}],"version-history":[{"count":13,"href":"https:\/\/www.causeweb.org\/sbi\/index.php?rest_route=\/wp\/v2\/posts\/521\/revisions"}],"predecessor-version":[{"id":553,"href":"https:\/\/www.causeweb.org\/sbi\/index.php?rest_route=\/wp\/v2\/posts\/521\/revisions\/553"}],"wp:attachment":[{"href":"https:\/\/www.causeweb.org\/sbi\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=521"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.causeweb.org\/sbi\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=521"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.causeweb.org\/sbi\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=521"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}