Hello All, As person who spends most of her summer working with high school teachers on stats and probability content and creating lesson plans, which are used in the next school year, I've followed this discussion eagerly. High school teachers are relatively easily convinced that a large enough, random sample is usually representative of the population. Convincing teachers that one of these samples could be used to mimic the entire population and then be utilized to generate more random samples is quite a different thing. I am convinced of the bootstrapping process, but to leap there immediately with teachers versus the more cumbersome routes discussed in this chain of responses might cause serious distress. Are there resources to help educate high school teachers (and myself further) in regard to bootstrapping? Research and experience shows that teachers with either omit or superficially enact contact that they feel is beyond their current knowledge base. Simulation, in general, has been daunting for high school teachers. Of 23 we worked with last summer, only 25% took the plunge with re-randomization. However, the ones that did, thoroughly enjoyed the experience, as did their students. Best, Maryann ---------------------------------------- Maryann E. Huey Mathematics and Computer Science Drake University 515/271-2839 [cid:8508355A-3452-4BC5-9222-B812C0568C3F] From: Kari Lock Morgan <klm47@psu.edu<mailto:klm47@psu.edu>> Reply-To: Simulation-Based Inference <sbi@causeweb.org<mailto:sbi@causeweb.org>> Date: Tuesday, May 12, 2015 9:00 PM To: Simulation-Based Inference <sbi@causeweb.org<mailto:sbi@causeweb.org>> Subject: Re: [SBI] How to estimate parameters: To bootstrap or not? Hi all, Thanks for posing such a discussion-provoking question, Daren, and for the comments of all those who have chimed in so far. I wanted to add a couple more points: 1) In order to really do inference of any kind (intervals or testing) for a single mean via simulation, you have to do some form of bootstrapping. Even if you prefer to invert lots of tests or use a null SE, the only way I know of to do this (using simulation) involves bootstrapping, in which case, why not directly use this for the margin of error? I think Beth's approach of sampling repeatedly from a population is a great starting point for building conceptual understanding (this is how I start as well), but this has to be followed up with something students can actually use when faced with a single sample; either traditional methods (s/sqrt(n)) or bootstrapping. To estimate a margin of error for a mean using just simulation (as required in the Common Core, which doesn’t cover traditional methods), bootstrapping seems necessary. 2) For a proportion, using the SE from a null distribution works in situations where a hypothesis test makes sense (in this case phat and p0 are likely to be somewhat close), but in many estimation settings there is no natural null hypothesis. For example, suppose you want to estimate the proportion of people who have some disease. Here there is no natural null, and using a SE from a distribution based off of p = 0.5 will drastically overestimate the SE if the disease is rare. 3) My favorite aspect of bootstrapping is that once you learn how to do it in one case (resample with replacement from the sample, compute statistic, repeat many times), the exact same method can be applied to any parameter! It would be nice to highlight the universality of this very simple approach. Just my three cents. Happy end of the semester, everyone! Kari On Mon, May 11, 2015 at 12:27 PM Nathan Tintle <nathan.tintle@dordt.edu<mailto:nathan.tintle@dordt.edu>> wrote: Beth, Robin, et al. While the method of finding plausible values (i.e. a confidence interval) for a single proportion by testing many different null hypothesis values is inefficient, I personally find it valuable, at least as a starting point, because (a) It reinforces the idea of how to do tests of significance and (b) It reinforces the language of 'null is plausible' vs. the common student mistake of 'null is true' when the p-value is large. While I don't spend a lot of time with this approach (as others have mentioned it is time consuming from the students perspective and limited in terms of cases when it is applicable) it seems to act as a nice 'bridge' to other techniques, like estimating the SE from the simulated null distribution and taking 2*SE as a rough 95% CI, and/or theory-based approaches, without needing to introduce another type of simulation (e.g., bootstrap). Nathan On Tue, May 5, 2015 at 6:01 PM, Beth Chance <bchance@calpoly.edu<mailto:bchance@calpoly.edu>> wrote: Hi, I of course have to argue with Robin :) But not on all points In the one sample, quantitative variable case, instead of bootstrapping, I have students sample from a made-up population. So this is still a bit ad hoc, but I think helps them better see the sampling from population connection we are emphasizing at this point in the course. With proportions, I agree that you have to decide whether you want to use a hypothesized value or the sample proportion to estimate the SD. In my class I want students to think about both methods, partly to see it often doesn’t make a difference, especially with a large sample size, which I assume is what the CCSSM will focus on. And of course, “traditional” methods make the same “arbitrary” decision – use hypothesized if you have one, use sample if you don’t. We do have students try lots of different null values the first time we are creating a confidence interval of plausible values (and that’s when we introduce the idea of level of significance), but we have added a feature to the technology to make this a little more efficient once they get the idea (think slider). My hope, though I don’t have a lot of data and what I do have isn’t “great,” is that students will be better able to focus on the interval being for the *parameter* rather than the common misconception that it’s an interval for sample proportions that I worry bootstrapping might reinforce. Basically I want students to think about a confidence interval as estimate +- 2SD, which they seem to get pretty easily, and then we can worry about the details of how to estimate the (right) SD in different cases/use technology. This is what we carry over to other statistics later in the course. I think CCSSM is focused on having students understand sampling variability and the idea of margin-of-error, of the proportion being close to the parameter and that the “plus or minus part” depends on sample size. Lots of good ways to get those ideas across. Like them, I’ve been starting with proportion, I guess they thought mean would be too tough as they didn’t want to have them get into bootstrapping. There is more discussion on exactly this issue on the SBI blog: https://www.causeweb.org/sbi/?cat=14 Beth From:sbi-bounces@causeweb.org<mailto:sbi-bounces@causeweb.org> [mailto:sbi-bounces@causeweb.org<mailto:sbi-bounces@causeweb.org>] On Behalf Of Robin Lock Sent: Tuesday, May 05, 2015 1:17 PM To: Simulation-Based Inference Subject: Re: [SBI] How to estimate parameters: To bootstrap or not? Daren - I'm a firm believer in using the bootstrap as the way to get a margin of error via simulation. The ideal way would be to form a sampling distribution but, in the real world, taking 1000's of new samples from the actual population is not a feasible way to assess the accuracy of the one estimate you have from an original sample! I think that the logic of hypothesis testing is already tricky for students to get a handle on, to have intervals depend on inverting that logic seems even trickier. The example you provided in the CCSS Progressions document is even more confusing. Here's what I gather is its "logic" Start with a sample of size 50 with a sample proportion of 0.40. You want to estimate its "margin of error". 1. Suppose the population proportion is really p=0.50. Simulate a sampling distribution using that p. 2. Observe that the sample phat=0.40 is not far in the tail of that distribution, so 0.50 is a "plausible" value for the population proportion. - Not bad so far. 3. Estimate the standard error by finding the std. dev. of the sample proportions in the distribution generated around p=0.50 (SE=0.07). 4. Use 0.4 +/- 2*(0.07) = 0.4 +/-0.14 = 0.26 to 0.54 to get a CI of similar "plausible values. Of course the SE for p=0.5 and p=0.4 are not a lot different, but I don't see the logic of picking some random "other" proportion, when you can do the simulation just as well aound p=0.40 (which is what the bootstrap would do in the first place!). I wonder what advice the document would give for finding a CI when phat=0.12? I think it is possible to do an interval more coherently by doing lots of tests for lots of null parameters and seeing which would be rejected for the sample data, but (a) That sort of guess/check process is not very efficient. (b) I'd like to downplay the hard 5% reject Ho decision, and would rather have a test p-value be interpreted as "strength of evidence" (c) Creating the simulations to test lots of nulls is more problematic (especially via simulation) for other parameter situations like a difference in proportions, difference in means, or correlation. The bootstrap procedure is pretty straightforward: take lots of samples (with replacement) from the original sample, calculate the statistics of interest, estimate SE as the std. dev. of all those bootstrap statistics. A rough margin of error as 2*SE is easy to find and the same process works for lots of different parameters. Robin On 5/3/2015 4:03 PM, Daren Starnes wrote: Happy May, everyone. There is an interesting thread on the AP Statistics Teacher Community about two distinct views on estimating parameters via simulation. This came up because the Common Core State Standards includes this Statistics and Probability standard S-IC.4 Use data from a sample survey to estimate a population mean or proportion; develop a margin of error through the use of simulation models for random sampling. View #1: Use bootstrapping. View #2: Determine whether "nearby" values of the parameter are plausible by simulating a "null distribution" with that parameter value and seeing if the observed statistic is a believable outcome from such a null distribution. Keep doing this for other nearby values until you have an interval of plausible values for the parameter. The attached CCSS Progression document seems to suggest View #2, at least as far as estimating a proportion is concerned. There is no discussion of how to estimate the margin of error for a mean in this way (I wonder why!). This seems like an issue that this experienced group of SBI folks would already have grappled with--both philosophically and pedagogically. So I thought I would ask what the prevailing wisdom is. Daren Starnes _______________________________________________ SBI mailing list SBI@causeweb.org<mailto:SBI@causeweb.org> https://www.causeweb.org/mailman/listinfo/sbi -- Robin Lock Burry Professor of Statistics St. Lawrence University -- Nathan Tintle, Ph.D. Associate Professor of Statistics and Dept. Chair Director for Research and Scholarship Dordt College Sioux Center, IA 51250 nathan.tintle@dordt.edu<mailto:nathan.tintle@dordt.edu> Phone: (712) 722-6264 Office: SB1612 _______________________________________________ SBI mailing list SBI@causeweb.org<mailto:SBI@causeweb.org> https://www.causeweb.org/mailman/listinfo/sbi

Hi, It seems like part of our question is still trying to interpret the intent of the Common Core Standard, so I also picked Roxy Peck’s brain. Here is her take: The Common Core Standards just attempt to introduce students to inferential thinking in a very informal way in high school and there really isn't any intent to link estimation and testing at this point in students' development. In fact, the standards specifically limit the idea of margin of error to situations involving sampling from a population and limit informal testing via randomization tests to situations involving simple randomized experiments with two treatments. The main goal of the standards is not to provide students with general methods for doing calculations, but rather to develop an informal understanding of what a margin of error is by acknowledging sampling variability via simulation and to develop an informal understanding of statistical significance as meaning not likely to have occurred by chance alone. The standard about margin of error mentions both proportions and means and the one about randomization tests is silent on the issue, so the intent isn't clear there. I asked Dick about it once since advised the writing teams and wrote the progressions documents and he said margin of error was also meant to include means, although he agreed that was the more difficult case! Thanks, Beth From: sbi-bounces(a)causeweb.org [mailto:sbi-bounces@causeweb.org] On Behalf Of Kari Lock Morgan Sent: Tuesday, May 12, 2015 7:00 PM To: Simulation-Based Inference Subject: Re: [SBI] How to estimate parameters: To bootstrap or not? Hi all, Thanks for posing such a discussion-provoking question, Daren, and for the comments of all those who have chimed in so far. I wanted to add a couple more points: 1) In order to really do inference of any kind (intervals or testing) for a single mean via simulation, you have to do some form of bootstrapping. Even if you prefer to invert lots of tests or use a null SE, the only way I know of to do this (using simulation) involves bootstrapping, in which case, why not directly use this for the margin of error? I think Beth's approach of sampling repeatedly from a population is a great starting point for building conceptual understanding (this is how I start as well), but this has to be followed up with something students can actually use when faced with a single sample; either traditional methods (s/sqrt(n)) or bootstrapping. To estimate a margin of error for a mean using just simulation (as required in the Common Core, which doesn’t cover traditional methods), bootstrapping seems necessary. 2) For a proportion, using the SE from a null distribution works in situations where a hypothesis test makes sense (in this case phat and p0 are likely to be somewhat close), but in many estimation settings there is no natural null hypothesis. For example, suppose you want to estimate the proportion of people who have some disease. Here there is no natural null, and using a SE from a distribution based off of p = 0.5 will drastically overestimate the SE if the disease is rare. 3) My favorite aspect of bootstrapping is that once you learn how to do it in one case (resample with replacement from the sample, compute statistic, repeat many times), the exact same method can be applied to any parameter! It would be nice to highlight the universality of this very simple approach. Just my three cents. Happy end of the semester, everyone! Kari On Mon, May 11, 2015 at 12:27 PM Nathan Tintle &lt;nathan.tintle(a)dordt.edu&gt; wrote: Beth, Robin, et al. While the method of finding plausible values (i.e. a confidence interval) for a single proportion by testing many different null hypothesis values is inefficient, I personally find it valuable, at least as a starting point, because (a) It reinforces the idea of how to do tests of significance and (b) It reinforces the language of 'null is plausible' vs. the common student mistake of 'null is true' when the p-value is large. While I don't spend a lot of time with this approach (as others have mentioned it is time consuming from the students perspective and limited in terms of cases when it is applicable) it seems to act as a nice 'bridge' to other techniques, like estimating the SE from the simulated null distribution and taking 2*SE as a rough 95% CI, and/or theory-based approaches, without needing to introduce another type of simulation (e.g., bootstrap). Nathan On Tue, May 5, 2015 at 6:01 PM, Beth Chance &lt;bchance(a)calpoly.edu&gt; wrote: Hi, I of course have to argue with Robin J But not on all points In the one sample, quantitative variable case, instead of bootstrapping, I have students sample from a made-up population. So this is still a bit ad hoc, but I think helps them better see the sampling from population connection we are emphasizing at this point in the course. With proportions, I agree that you have to decide whether you want to use a hypothesized value or the sample proportion to estimate the SD. In my class I want students to think about both methods, partly to see it often doesn’t make a difference, especially with a large sample size, which I assume is what the CCSSM will focus on. And of course, “traditional” methods make the same “arbitrary” decision – use hypothesized if you have one, use sample if you don’t. We do have students try lots of different null values the first time we are creating a confidence interval of plausible values (and that’s when we introduce the idea of level of significance), but we have added a feature to the technology to make this a little more efficient once they get the idea (think slider). My hope, though I don’t have a lot of data and what I do have isn’t “great,” is that students will be better able to focus on the interval being for the *parameter* rather than the common misconception that it’s an interval for sample proportions that I worry bootstrapping might reinforce. Basically I want students to think about a confidence interval as estimate +- 2SD, which they seem to get pretty easily, and then we can worry about the details of how to estimate the (right) SD in different cases/use technology. This is what we carry over to other statistics later in the course. I think CCSSM is focused on having students understand sampling variability and the idea of margin-of-error, of the proportion being close to the parameter and that the “plus or minus part” depends on sample size. Lots of good ways to get those ideas across. Like them, I’ve been starting with proportion, I guess they thought mean would be too tough as they didn’t want to have them get into bootstrapping. There is more discussion on exactly this issue on the SBI blog: https://www.causeweb.org/sbi/?cat=14 Beth From: sbi-bounces(a)causeweb.org [mailto:sbi-bounces@causeweb.org] On Behalf Of Robin Lock Sent: Tuesday, May 05, 2015 1:17 PM To: Simulation-Based Inference Subject: Re: [SBI] How to estimate parameters: To bootstrap or not? Daren - I'm a firm believer in using the bootstrap as the way to get a margin of error via simulation. The ideal way would be to form a sampling distribution but, in the real world, taking 1000's of new samples from the actual population is not a feasible way to assess the accuracy of the one estimate you have from an original sample! I think that the logic of hypothesis testing is already tricky for students to get a handle on, to have intervals depend on inverting that logic seems even trickier. The example you provided in the CCSS Progressions document is even more confusing. Here's what I gather is its "logic" Start with a sample of size 50 with a sample proportion of 0.40. You want to estimate its "margin of error". 1. Suppose the population proportion is really p=0.50. Simulate a sampling distribution using that p. 2. Observe that the sample phat=0.40 is not far in the tail of that distribution, so 0.50 is a "plausible" value for the population proportion. - Not bad so far. 3. Estimate the standard error by finding the std. dev. of the sample proportions in the distribution generated around p=0.50 (SE=0.07). 4. Use 0.4 +/- 2*(0.07) = 0.4 +/-0.14 = 0.26 to 0.54 to get a CI of similar "plausible values. Of course the SE for p=0.5 and p=0.4 are not a lot different, but I don't see the logic of picking some random "other" proportion, when you can do the simulation just as well aound p=0.40 (which is what the bootstrap would do in the first place!). I wonder what advice the document would give for finding a CI when phat=0.12? I think it is possible to do an interval more coherently by doing lots of tests for lots of null parameters and seeing which would be rejected for the sample data, but (a) That sort of guess/check process is not very efficient. (b) I'd like to downplay the hard 5% reject Ho decision, and would rather have a test p-value be interpreted as "strength of evidence" (c) Creating the simulations to test lots of nulls is more problematic (especially via simulation) for other parameter situations like a difference in proportions, difference in means, or correlation. The bootstrap procedure is pretty straightforward: take lots of samples (with replacement) from the original sample, calculate the statistics of interest, estimate SE as the std. dev. of all those bootstrap statistics. A rough margin of error as 2*SE is easy to find and the same process works for lots of different parameters. Robin On 5/3/2015 4:03 PM, Daren Starnes wrote: Happy May, everyone. There is an interesting thread on the AP Statistics Teacher Community about two distinct views on estimating parameters via simulation. This came up because the Common Core State Standards includes this Statistics and Probability standard S-IC.4 Use data from a sample survey to estimate a population mean or proportion; develop a margin of error through the use of simulation models for random sampling. View #1: Use bootstrapping. View #2: Determine whether "nearby" values of the parameter are plausible by simulating a "null distribution" with that parameter value and seeing if the observed statistic is a believable outcome from such a null distribution. Keep doing this for other nearby values until you have an interval of plausible values for the parameter. The attached CCSS Progression document seems to suggest View #2, at least as far as estimating a proportion is concerned. There is no discussion of how to estimate the margin of error for a mean in this way (I wonder why!). This seems like an issue that this experienced group of SBI folks would already have grappled with--both philosophically and pedagogically. So I thought I would ask what the prevailing wisdom is. Daren Starnes _______________________________________________ SBI mailing list SBI(a)causeweb.org https://www.causeweb.org/mailman/listinfo/sbi -- Robin Lock Burry Professor of Statistics St. Lawrence University -- Nathan Tintle, Ph.D. Associate Professor of Statistics and Dept. Chair Director for Research and Scholarship Dordt College Sioux Center, IA 51250 nathan.tintle(a)dordt.edu Phone: (712) 722-6264 Office: SB1612 _______________________________________________ SBI mailing list SBI(a)causeweb.org https://www.causeweb.org/mailman/listinfo/sbi