Paper on using math link cubes to introduce data wrangling with examples in order, which describes a classroom activity that includes hands-on tasks paralleling commonly, data wrangling processes such as filtering, selecting, and mutating, followed by their coding equivalents using RSDPlier package. Next, a couple of announcements. Us COTS is happening this year. Registration is available. And birds of a feather proposals are due on June 20th. I think that's the last deadline coming up there. And the next JSDSE cause webinar will be on Tuesday, May 27th at 4 p.m. Eastern Time, in which actually today's co-host, Kelly Finley and his colleague Anne Justice will present their work on career interests and aspirations for incoming statistics majors. So stay tuned for that webinar coming up next month and registration for that will be available shortly. All right. On to our wonderful speakers. Dr. Amanda Ellis is an associate professor in the Department of Biostatistics at the University of Kentucky. Her focus is on graduate education, course and curriculum development, and she currently serves as the assistant dean for the Public Health Sciences programs. And Dr. Emily Slade is an assistant professor in the Department of Biostatistics at the University of Kentucky. She primarily focuses on developing infrastructure and training to promote effective team science with biostatisticians. She teaches statistical consulting and collaboration courses, and she's the PI of an NIH-funded R25 grant focused on developing a national training and mentorship program and collaborative skills for biostatisticians. She was telling me about some of the really cool classes that she teaches in collaboration before we hopped onto the webinar today, and I'm Really excited to learn more about it in future. Amanda and Emily, welcome and thank you so much for sharing your work I'll stop sharing now and turn it over to y'all. Thank you so much. Just a moment, I'll get my slides up. And then just to double check, can you see this? Perfect. Well, thank you to everyone for joining us today. Emily and I wrote this paper a couple of years ago and we are constantly learning more about this topic. And so we're really excited not only to get to talk to you all, but to also hear the things that you all have to say. So just to kind of start out, this is kind of a broad outline of what we're going to talk about today. We want to talk first just motivating the use of generative AI and statistics and data science education. And then we really want to go into some practical examples. And these examples, we'll talk about ways that instructors, students both can use generative AI And then give you some example assignments that we have used. In our courses. These examples by no means are going to be exhaustive. So I do want to point out really what we're trying to do here is to generate ideas And hopefully when you leave, there may be some things that you think about while we're talking that we haven't even suggested. And then finally, we'll mention some challenges of generative AI and statistics and data science education. So just to start out here with a disclaimer, there are lots of generative AI tools out there. We are going to primarily talk about ChatGPT. But it is by no means the only generative AI tools that educators or students are using. And in fact, many of the programs or software that you've probably used in the past are now incorporating generative AI. So for example, if you've ever used Grammarly, I know Grammarly is now incorporating generative AI. So even products that we don't typically think of as generative AI are now incorporating it. So why should we teach students to responsibly interact with generative AI tools? Well, I think there are several reasons that we should talk about this. One of them is workforce readiness. When our students go out into the workforce. More than likely, they're going to be using generative AI. So if we can go ahead and start getting them to think about using generative AI in the classroom and how to ethically and effectively use it, we're just going to do a better job of getting them ready for the workforce. We can improve their study techniques. Obviously, they're students, they're in class. We want them to study the best that they can. And I think, you know, especially as classroom sizes grow. Any way that we can help our students improve their study techniques is going to be really important. Efficiency you know as an instructor, you know, any way that I can improve my workflow and be more efficient, I want to do that. And that's one of the things that generative AI has really helped me with personally is I find that it just it saves me a lot of time with the things that I do. And so I wouldn't want that for my students as well, I should say. And then fourth here, if they're going to use it anyways, we should teach them how to use it responsibly. And so first what I'm going to talk a little bit about is just some student perspectives on ChatGPT. And what you're going to see actually comes from a course that I taught last spring. I taught, it was their second data analysis course for our Bachelor of Public Health students. So these were all students that had already had what I consider your typical intro statistic biostatistics course. And in the second course, that's where we take really more of a deep dive and we start looking at things that they may interact with professionally for those that have a bachelor of public health degree. And when I had redesigned this course and when I was teaching it, I really wanted to not only talk about generative AI, but I wanted to figure out how I could actually embed it in that class and use it within the parameters of the classroom. But before doing that, what I decided I really needed to do was try to get a gauge of what students were already using ChatGPT for and what their feelings were. And so I did a survey kind of close to the beginning of the semester. And one of the questions I asked on that survey were, give me three words that you associate with generative AI. And this is kind of the word cloud that the students came up with. And really quickly, as you can see here, the largest word that came up with was cheating. But then we've got all these smaller world words too, like quick. Helpful. Brainstorming, easy help. And so I could tell from this, you know, my students knew that it could be helpful But at the same time, the largest word that was popping up was cheating. And kind of my personal takeaway from this was I think some of that trepidation that we've had as educators Our students are hearing that. And I think they're using it, but they're also a little hesitant to admit that they're using it. And I don't have this on the slides here, but I asked the students, are you using this? How are you using it? And some of them said no, but some of them said yes. And I was amazed for those that said yes They told me, you know, we're not just using it to help study, we're using it in our everyday lives. I had one student tell me they used it to build D&D characters for when they play D&D. And I've had other ones tell me they use it to help them think of recipes when they're cooking dinner. So it's not just in the classroom. They're using it in all facets of their life. And then some of the other questions that I asked students was, would you like for your classes to incorporate chat GPT? And only 6% of them said no. We had 55% maybe and 39% yes. And then would you like more training on using ChatGPT? As you can see, majority of here suggests. Only a little less than 20% said no. So kind of my takeaway from what I found out in the classroom was students are interested in generative AI. They are using it And they would like to see how we're using it and they would like for it to be incorporated in the classroom. So I'm going to hand it over to Emily next. Thank you. So as Amanda mentioned before, the middle section of this talk is going to really dive into just giving lots of examples on ways that you may consider incorporating ChatGPT into your educational experiences. So I'll start first with talking about How can we as instructors use ChatGPT? Next slide, please, Amanda. So a few applications to consider, and I'm going to give some examples in each of these areas. The first being curriculum development. So thinking about using generative AI to assist in creating and refining course materials. Syllabi, curriculum outlines. It can be very helpful for this. For the actual content development, ChatGPT can assist in generating examples or data sets. That are tailored to a particular domain area or to demonstrate a particular concept. And that's one that in the past, I can think of times where I've spent wasted countless hours searching for data sets that are really serving a niche that I need it for, whether that's a really specific content area or to demonstrate a really specific concept. So I'll show you an example in a moment. Or chat GPT, for me at least, has really revolutionized my efficiency in that arena. And creating assessments. For me personally, I don't always love using ChatGPT to actually create quiz questions themselves, but I use it a lot to help me generate grading rubrics from the assessments that I write myself. So there are many ways you could think of incorporating generative AI into the assessment arena. And then even in terms of writing. Something we do a lot as instructors is respond to emails, sometimes particularly tricky responses that have to be carefully worded and ChatGPT can be useful for helping get that wording right. So I'll show an example of that as well. Next slide. Thanks. So an example of curriculum development for a syllabus, a prompt that could be used something like. Write language for a syllabus to prevent the use of chat GPT or other generative AI tools on exams specifically. And so this is one, you know, I like to use ChatGPT in this method where it gives me what I would consider to be a starting point, just some text to work off of. So I'm not staring at a blank screen. And then I'll But it gives me like this example here and really tailor it to my own needs and edit it to suit What I really wanted to say. So here you can see ChatGPT's response. They give six different areas discussing everything from what you can and can't do to penalties for violations. Resources, things like that. So again, you can put this whole thing in your syllabus. You can pick and choose parts and pieces, edit it to really suit your own needs. Next slide. And so shifting gears, here's an example in the content development arena. So this is what I was alluding to before. The prompt given here is I'm teaching a graduate level statistics class. You could do undergraduate level as well. It really doesn't matter. On the topic is effect modification. And can you give me a data set and illustrative example related to public health? So you can see it takes that response and this was actually quite a lot or it takes that prompt This is actually quite a long response that ChatGPT gave, so I'll show you a couple of excerpts from it. The example that it's suggesting here is looking at the relationship between air pollution and hospital admissions for asthma and assuming that the effective air pollution might be modified by age group. So they start with a hypothetical data set. I think they gave maybe 10 rows here, a small data set. You can go to the next slide, Amanda. And then it's offered suggestions for how to teach this. So talking about the stratification with descriptive statistics Looking at fitting a model with an interaction term and really what could be discussed here. If you go to the next slide, though, what I really liked about this one is that it then asked, do you want me to generate a larger data set that you can actually export as a CSV for this example? And I said, yes, please do that. And lo and behold, it gave me a download link to download this simulated data that was in the public health domain, like I requested and I knew was going to be able to demonstrate the concept of effect modification that I was trying to To teach here. So like I alluded to before This has saved me countless hours. I mean, you can even be very specific, you know, telling it, use this variable and that variable or the more general context such as this one, general public health I'm telling you, you can request it a certain number of variables, number of rows. You can be quite specific with these requests and really refine them iteratively to get just the data set you need to teach the concept that you're trying to teach. I find this one very useful. And then, of course, this is simulated data. So there's no issues with data sharing here if this was a data set that needed to be shared with students. Next slide, please. Thanks. So thinking more about assessment, I had mentioned before how I like to use ChatGBT to generate grading rubrics. So here's one example of how I've used it before. This was for a statistical consulting class that I was teaching. So I fed ChatGPT the entire assignment prompt. So the prompt for the students was to read the American Statistical Association's ethical guidelines for Statistical Practice. And then ask the students to write a short response about what they found important, what content they found challenging, and any other general thoughts or impressions. So kind of a fairly vague prompt. You can go to the next slide, Amanda. I asked it to generate a grading rubric for me, a simple grading rubric. And this is what it came up with. So it suggested looking at five different criteria. The identification of key concepts. Identifying challenging or unclear content, which I specifically asked for in the prompt. The overall reflections and impressions, and then looking at the writing style and professional tone. And the formatting and completeness. And so this is really nice that it not only suggested these five different criteria to look at, but also summarizing what would constitute full credit, partial credit, or no credit in each of these areas. So in my classes, I really like to share grading rubrics with the students in advance. And so this can be a very quick way to generate an initial grading rubric And then again, tailor it, edit it to match exactly what you need it to be. But I find that these generative AI tools give a very great Jumping off point to do that. Next one, Amanda. Thanks. And then the last example I'll show you is an example here about writing emails. So you can see the prompt, write a short email to a student declining a request to submit homework late in a firm but kind tone. And so this is one where you really can specifically request the tone that you would like. I, again, use this iteratively too where if the initial response from ChatGBT wasn't quite hitting exactly the tone I wanted. I asked it to revise it and make it Nicer or make it friendlier, make it firmer, whatever it needs to be. So I won't read you the entire response here. But actually, is it on this side? Could you go back to the last slide, Amanda Thanks. So look in the second paragraph of ChatGPT's response here. Second sentence you see. However, in the interest of maintaining fairness and consistency with our firm's policies, we're unable to grant an extension for this particular assignment. So you can see areas where it's not great, right? So I asked for a firm but kind tone and it obviously interpreted the word firm as referring to like a firm that I work for, like a company or something like that. So saying that it's not maintaining consistency within our firm's policies. Is not really what I wanted it to say since I was referring to an academic institution. You know, not trying to say that ChatGPT is perfect by any means. You still need to read what it wrote and edit it as necessary. But just getting the bones of an email like this written can be very, very helpful. All right, I'll pass it back over to you, Amanda. Thank you. So Emily talked about some ways that we can use chat GPT as instructors I'm going to kind of switch and talk about ways that students can use ChatGPT. And I'm going to go through several different applications here, but I really want to point out that asterisk that's at the bottom. There are potential concerns with some of these. So I think before we think about the concerns, we have to think about, okay, well, how are they using it? And then we can think about, well, are there problems with those? And then tackle those concerns. So here, I'm not necessarily advocating students use these this way. I'm just saying these are potential ways they could use it. Working with code. So ChatGPT can be used to generate code. Help with debugging code and translate between coding languages. So I teach, I've taught both undergraduate and graduate, especially in the graduate space. I find that the translating between languages can be really useful and we'll see an example of that. It can be used to summarize text. So for example, students could give it their class notes You could also give it journal articles, papers, and it will summarize for them. It can be used to help generate ideas. So for example, I know a common project that's often given to undergraduate STAT students is to go and to implement a survey and the students have to write their own survey questions. It can be used to help generate ideas. It can help with studying. So to create studying guides, additional questions. It can help with writing skills to check grammar and spelling, revise tone, and get personalized feedback on its writing. Time management, it can give suggestions on time management. So students can say things like, hey, I have two weeks before this big exam is coming up. Can you give me a recommendation on how much I should study each day and what I should study on each of those days? And then it can also act as a tailored search engine. And so just like we did with the instructors, we're going to look at just some examples of a couple of these applications. So the first one we're going to look at here is working with code and translating between languages. And so as you'll see up here at the top, it says, here's my logistic regression model. Can you translate this over to SAS code. And I will say I've used it for this in the past. And for the most part, it does a pretty good job. Now, I wouldn't I wouldn't recommend this to someone that has never written SAS code before. But if R is the primary language and you're comfortable with SAS, but maybe it's not one that you're just going to write on the fly. I think this typically works pretty well. And you'll notice here, even as it goes through and it makes those changes. It will go through. And so, for example, it has health data here and data health data here. So it is smart enough that it will go through and make sure that it's either changing those data set names or maybe variable names as well. So you can see smoking status, for example, is the same in each of them. And so I find this really, really useful, especially if a student is really comfortable with one language but not as comfortable with the other. And I will say if this works not just with languages. So even if you were going to use something like SPSS, for example, you can also ask it, how do I do this thing in SPSS? And it will walk through what you need to click to do that. As I mentioned earlier, it's great at generating ideas. So for example, if I were asking my students to go and create a survey related to public health that I could ask college students, like if they were going to go out and collect some actual data. What it'll do is it'll generate lots of different ideas that are related to that overall topic. So in this case, public health. I'll kind of scroll through these so you can see some of the options here. And when I talk to students about using it in this way, I tell them. You know a lot of times ask it to give you lots and lots of options. Really do thinking of it as a way to help you brainstorm if you don't have someone else to brainstorm with. Instead of just saying, hey, create a survey for me, because we really don't want it to where it's just creating the survey for the students. But more as a way to just help generate ideas. Here's another example with studying help. So for example, here are my notes on linear regression. Can you give me 10 multiple choice questions to help me study? Don't show me the answers right away. So one of the things that's really nice if you're Going to get ChatGPT to help with studying and acting as kind of that study buddy is it will give you the answers and they can give it to you right away or you can ask for them later. So in this case, it produced those 20 questions. And then it says, hey, let me know when you want to see the answer key. And so that way the students can go through and answer it themselves and then look at the answer key. Now again. Chatgpt is never going to be perfect. This is one of the things that I tell students to use it for studying help to be really careful with. I will say as time goes by, it's getting better and better, but they do have to have a pretty good base knowledge to start with so that it's not you know telling them something that's completely incorrect or they're not studying the wrong material. So there's always kind of that risk there when they are using this for study help. But I do think it can be a really good study tool. And I will say, so I teach an inference course And even for that inference course, I can give it probability questions. And what's really interesting is it will give me really detailed solutions. So if a student is stuck and they're like, hey, I don't know how to approach this type of problem. You can give it that problem and it will give a really detailed solution that not only walks through the steps. But we'll actually say, this is why you're going to move from this step to this step, or this is why you want to use this distribution. So even for more complex topics, it can do a really good job. And I'll pass it back over to Emily. So one thing that Amanda and I have talked a lot about over the last couple of years is rather than just sort of blanket recommending that our students engage with ChatGPT, how can we really directly bake it into assignments that we have in our courses? And so we'll show you a few examples that we've used here. Next slide, please. So one that Amanda's used in her undergraduate class At the end of the class, the students develop a research poster. And so she specifically directs the students to engage with chat GPT essentially as a tailored search engine. And part of the prompt for this is just understanding differences in chat GPT responses and what may be good or bad about those. So she instructs the students to first ask ChatGPT what's the purpose of a research poster. And then discuss the response that they got with the response that the person next to them got in class. Second question tells the students to ask ChatGPT what are the important parts of a research poster, kind of repeat that process. So there's a few questions sort of getting them to engage with ChatGBT like a search engine. You can go to the next one, Amanda. And then the students are prompted to create their own question to ask ChatGPT. And then after they go through this series of questions that they're asking, the students write a response on. Number one, what they actually learned about creating a research poster, which was the point of the assignment, but also to really reflect on what they liked or didn't like about the responses that ChatGBT gave them. Think about if there was any information that chat GBT was unable to provide. And then go out and find two other resources to learn about research posters and list those resources. So this is just trying to think of a way to, you know, we want the students to learn about research posters, but along the way, can we also get them to think about engagement with generative AI tools in a responsible way? You can go to the next one, Amanda. So in this one, this is an example that I've used in a graduate level class, but I think it would work just as well in an undergraduate level. So the students first are prompted to create a data visualization. For the purposes of this example, it doesn't really matter what it's for. They create their data visualization. In my class, it was an R, but could really be in any coding language. And then ask ChatGPT for suggestions on how to improve the plot. And so I then asked the students to submit the plot that they created with the enhancements and also a brief description of the insights that can be drawn from the plot and a reflection on Chad GPT's feedback and how it actually helped them improve the plot. So getting them to think through, were there any particular suggestions, suggestions they found particularly useful? And how it really affected their understanding of visualizing these relationships. And then a third example that I'll mention is more in the writing space. So kind of similar flavor to the last one, but the first step is asking the students to, they needed to write the results section from a regression analysis they had done previously in the class. And the first step asks them to use their own words without relying on generative AI tools in the first step. And to save that version as version one. And then rather than telling them to just generally interact with ChatGPT, I gave them very specific prompts to ask. So first, ask ChatGPT to review their results section for any spelling or grammar errors. Second, asking if the results section includes all the relevant statistical details and if there are any important elements missing. And then third, asking if they have any suggestions for improving the flow and readability. And then asking about the level of technical language that's used and whether it's appropriate for their intended audience. So they're given a few specific questions to ask. And then, um. Encouraged to also ask ChatGPT any other questions that they may want to help refine their writing of their results section. So then based on these suggestions, the students are then prompted to revise the results section to improve clarity, precision, and readability. And then the final, there's a part three on the next slide. They're asked to submit the original results section that they wrote themselves. The prompts they used in ChatGPT, the revived version of the results section, and a brief reflection on how interacting with ChatGBT either helped improve their writing or not. So the students are welcome to say that they felt like the chat GPT's responses did not improve their writing. And really discuss what suggestions they found useful or not and how they contributed. To the results section itself. So again, the purpose of this talk is not to suggest that you use these specific examples in your classes. It's really just to get you thinking about different ways You may be able to incorporate Generative AI tools into assignments and really get students to think about what's coming out of them and whether they like the responses, whether the responses are helpful That type of thing. So these are just a few examples of ways that Amanda and I have incorporated it into our past classes. Back over to you, Amanda. And then I'm going to finish talking about just some challenges with generative AI. So we've presented ways that both instructors and students can use it. But there are going to be some challenges. And so what I've presented here, certainly not an exhaustive list, and I have two slides on this, so we're going through some of these. Lack of personalization. So although AI tools are constantly evolving, they are going to lack that human emotion that's sometimes necessary in effective teaching and learning. I think we all know that if you can be one-on-one with a student. The better that interaction is going to be. But I do think there is a place for generative AI when it comes to education. Bias and discrimination. Bias can occur in AI systems because the data used to train the algorithms. So an example that I give here, it's not in this talk, but I gave another talk. Where in the beginning slides, I asked ChatGPT to generate an image of myself and I gave it some parameters. And you can't tell because my hair is up, but I have very, very curly hair. And so I would ask ChatGPT, well, yeah, it was ChatGPT at the time. I had said, hey, I want you to create this image of me and I want to have curly hair. And it just would not do that. And every time it would change my hair to curly, it would change my skin tone. And so that was a place where there was bias and discrimination in how it was generating those images. And so you can see this is writing as well. I think the easiest place to spot it is certainly, like I said, in images, but we do see it in writing as well. An over dependence on technology. So we definitely don't want to become, you know, we want to use technology, but we don't want to become too dependent on it. And I think one place that I really see this is in the coding arena. So we want to be in a place where we can use generative AI to help us code and improve efficiency. But we don't want to end up in a place where we are blindly using it to generate the code and not actually understanding what that code is doing, especially if you're generating large amounts of code. We don't actually understand what's happening there. Then that's not going to work well in the long run. So we don't want to have this over dependence on technology. There's problems with data privacy and security. So often collect vast amounts of personal data. And I think this is really important when we think about if we're asking students to use ChatGPT or another generative AI tool in the classroom, we want to make sure that they have the agency to choose whether or not they're interacting with those tools and how they're interacting. So, for example, I've heard recommendations of if you're using ChatGPT in the classroom, for example, consider having a classroom account for ChatGPT instead of asking the students to individually sign up for ChatGBT. And to give their information. Equity is another one. This is one that I'm really concerned about when I think about the future of generative AI. When it first came out. And even now, a lot of the tools are free. But as people start using those tools more, of course, they're going to start charging money for these. And so I really worry about our students that can afford the tools versus those that can't. And we even see this in ChatGPT right now. There's a big difference between what ChatGP can chat GPT can do with the free version versus the paid version. And so those students that have the paid version They're going to get much better responses than those that have the free version. Of course, academic integrity. We want to make sure that our students understand the material. And if they're just using generative AI to produce solutions and not having any understanding. Then we're going to have issues with academic integrity. The tools are constantly evolving. So if I integrate generative AI tools in my classroom this semester. How I integrate them a year from now is likely to change because those tools are evolving. So it does take time to build these things into the classroom. And then it takes time to keep up with everything that's happening. And then the last one I'll put here is environmental impact. So the data centers powering AI model training and operations require high energy consumption and carbon emissions. So the more that generative AI gets integrated into both education and our personal lives and people are using this more. There is definitely going to be an environmental impact. So I have just a couple general recommendations here. So consider how you can use AI tools to improve your own efficiency as an instructor. Teach students to use AI tools responsibly rather than avoiding them. And I think this is really important because I think they're going to use them anyways. Or what's going to happen is you're going to have some of your students use them and they're going to end up getting an advantage compared to those students that don't. But as we use these generative AI tools, make sure we're encouraging critical thinking. We're using the generative AI tools in a way that aids the learning. And isn't taking the place of learning. And then staying up to date because the landscape is constantly evolving. And Elena and I first wrote this paper two years ago, and even the things that I see that ChatGPT can do has drastically changed the availability of other generative AI tools has drastically changed. So it is constantly evolving. And then if we had a crystal ball and we could ask ourselves, where will we be five years from now? I definitely think there's going to be more tools. There's going to be tools five years from now that we haven't even thought about. Better performance. But I would also guess there's going to be fewer free options or the options that are free are going to be nearly as good as the ones that you pay for. And my hope is as educators, we're going to have a better understanding of how students are interacting with those tools And as we get that better understanding, I think we're going to be able to utilize it even better in our classrooms and to better serve our students. So I just want to say thank you all for listening. Again, we're really thankful that you all have us here today and we are happy to answer any and all questions. All right. Thank you so much, Amanda and Emily, for that. So I will open up with the question that I have, and then I will be looking through the Q&A for questions that y'all are plugging in there. So feel free to use the Q&A now if you have anything on your mind. So something that I was wondering about, and I think Amanda hit on it with the challenges, and that's that I teach an entry-level course that does have a coding component. And one thing I do notice is that ChatGPT code is often more complicated than the code that I directly teach and share. And it leads me to wonder that students may not really understand the elements of their code. And are bypassing kind of the tutorials or the more carefully constructed materials that I'm providing. So I was wondering, do you have any thoughts on this practice by students who are first time coders using ChatGPT like that And do you have suggestions for how to appropriately scaffold the use of generative AI tools for first-time coders Or do you perhaps think it's best for students to have some fundamentals in coding and only use ChatGPT for more higher level skills for coding? Yeah, so a lot of thoughts here. So I will, before I answer this, I will say that I teach a wide range of classes. So I actually teach in our, pardon me. And Corset is just on our programming, but I also teach several intro level courses where just like you said, that first time coders And it's just embedded. And then I teach higher level courses where the coding is more important. And so for me, for any of those courses that I'm teaching, the first thing I do is I kind of step back And I ask myself, what is the most important or what are the most important takeaways from this course. Is it the content that I'm teaching in terms of the statistics or biostatistics? Or is it the coding? So for example, currently right now I'm teaching an introductory level MPH course. And in that course, the most important pieces are really the interpretations of the output. The student's ability to interpret really mainly descriptive statistics and some inferential, but I do have them code because I think it's important they work with some real data and they see those outputs. I recognize in that course in particular that when I think of the coding that students are doing. The coding for me is not the most important takeaway. And so for that reason, I actually give the students code that I have written that they just have to make minor tweaks to because I don't consider the code to be one of the more important learning pieces in that course. And so I've scaffold that course in a way that majority of the code is given to them and they have to be able to read the code and make minor tweaks. And I will say, surprisingly, at least not that I've noticed, I don't think the students are really using much generative AI for the code in that course because I've provided that scaffolding already for them and I've taken away, I think, some of that fear when it comes to coding. Now, in my R programming course, on the other hand. I do see some of the generative AI use. And now in that course, again, I just have some very careful scaffolding to kind of build them up to the code. But I do see there are places where some of them are using generative AI and that's a place where I would really rather them I want to make sure that they understand it because it's a course on our programming. And so one of the things that I've started doing across my courses is, again, I'm really evaluating What are the most important things I want my students to learn? And then I start thinking about my assessments for that course. And so wherever the most important key learning pieces are, I've started incorporating assessment pieces that are proctored, for example. So that I know for those key elements, I know they're not using generative AI. And then for myself, like I said, you know, the other piece is I'm okay if they're using generative AI there and I talk about it. So I think it's really this balance between thinking about What are the most important pieces How are you scaffolding the different pieces in your course and then tying that to your assessment and saying, okay. Baked into your assessments, what are those key pieces that I want to ensure that they really understand and I know it just isn't generative AI and building assessments so that you can check that. And I don't think all assessments should be proctored, but I think those key elements should be. And I'll hand it over to Emily to see if she has other thoughts. I'll just add a very short comment to that, which is I spend a lot of time with my students talking about or sort of comparing the use of generative AI tools for code development to techniques like Googling stuff and taking code from Stack Overflow, for instance. I can think of many times where I've been teaching students who are new to coding and they don't realize that you can Google and they just get stumped and you have to really teach them like, no, that's how you code basically. And so we spend a lot of time talking about How do you know that the code you're writing is working correctly? And so making sure that they're using those practices of validation for any code they're using, whether it's coming from something they wrote, whether it's coming from Stack Overflow, whether it's coming from ChatGPT, et cetera. It's really no different. All right. And then looking through the chat, I'm going to try to identify questions that I think maybe haven't come up explicitly in the talk yet, but you might have thoughts about. We have a question about, have you noticed any differences between different types of generative AI tools? So ChatGPT is just one of several tools. Microsoft Copilot is another one that folks might be familiar with. But do you have any experiences comparing the use of different generative AI tools in the context of teaching statistics? I have not really dug into Copilot as much as most. So I'm primarily a chat GPT user. One thing I do is talking with my students about forms of generative AI they may have already been seeing or using for a long time without realizing it. Like for instance, Grammarly. And so talking to them about how This is not necessarily like a fully novel concept. And you can see some differences there between, for instance, Grammarly and ChatGPT. Of course, they're doing different things. But Amanda, have you used Copilot much or other similar tools? I haven't. I do know there are differences depending on what it is you want to do and you all will have to forgive me. I was just having a conversation with someone the other day and we were talking about the issue of having ChatGPT tweak tone with emails. And they had actually recommended another tool to me that I haven't had a chance to explore yet that's even better or they thought was better than ChatGPT at changing tone. So I think if you have specific needs. Then there probably are Other tools that might do that specific need better. For me personally, much like Emily, I've primarily been using ChatGPT because it meets most of my needs. So I haven't kind of branched off into some of the other tools as much. But I do know for sure there are other tools out there. And I think some of them probably do specific things better. All right. And then we have a question related to using GPT-0 or other AI detection tools. Do you have any experience using tools like this to see how much students are using AI or do you have thoughts about whether these tools are productive for instructors to use? Yeah, this is a really, people are going to have really different feelings about this topic for sure. For me personally, I find that those tools are poor at detecting generative AI generated content I don't use them. I prefer to actually encourage my students to use generative AI tools in specific ways and then talk about the way that they use them. I find that if you just bake it directly into an assignment like that. The students are less likely to go outside of those parameters, if that makes sense. But Amanda, I'd love to hear your thoughts on this one also. My thoughts are similar to yours. I don't use them either. And I know there are those that do use them. I think one thing to consider, especially when we think of writing for ChatGPT, the way that the models are written, they're going to kind of write to the average when it comes to the English language. And I think for students where English is their second language, that's also how they are going to tend to speak. And so I think we want to be careful when we look at flagging things as generative AI, especially for those students whose English is their second language. They are more likely to get flagged. Their writing style is. Than those students who English is their first language. So yeah, I tend not to use them. Because again, I don't know how good they either were at detecting. And I'm sure they have gotten better As time has gone on, I know there was a lot of controversy about using them a year and a half, a year ago. I don't really use them personally. All right. I think we might have one more question. And I think I actually want to pull this one from the chat just because I find this something I'm really curious about. And that is how much y'all have experienced Experimented with making your own tutor and whether you teach students how to use ChatGPT. As a tutor or is it just kind of an open suggestion to them? Go take this one first, Amanda. So open suggestion for me and part of the issue with this is the paid version, at least in my experience, the paid version of ChatGTP is much better at this. Because it's going to remember what you said. It's funny. So I use the paid version and it's gotten almost creepy how good it is when I ask it for something and it knows that I work in biostatistics and knows I work in a college of public health. I don't even have to add that context anymore. And it will just add it to some of the things that I do. And so I think to be a really effective tutor. If it has that ability to have that back and forth, it's really useful. But I find the paid versions are better than that. So I feel really reluctant to have my you know to have to have my Tell my students and to work through them knowing that some of my students aren't going to be able to afford those paid versions. So I kind of do it more as a general suggestion. And some of the assignments that I have do talk about that tailoring I talk with the students about what is your style? How do you like to get information? So for example. I don't think it was in the one that we showed here. One of the conversations that I'll have with students is I'll ask students. How do you want information? Do you like stuff in bullet points? Do you like stuff in more paragraph form? How do you like to read information? And then I'll talk with them about, okay, well, when ChatGPT is giving you information. Think about even the format. How do you better digest information? And I'll talk through those pieces with them. Kind of again in this more general arena than really sitting down with them and saying, okay, you're going to have this private tutor, if that makes sense. All right. I think we can kind of wrap up the questions there. There are a few more questions slash comments in the chat that I think are also just getting at some of these ethical questions, both the ethics of using ChatGPT and perhaps the ethics of avoiding it. So I'll just kind of highlight that that's a theme that A lot of us are probably thinking about, but I don't know that a lot of us have a lot of good answers about. But I think that's something that you kind of acknowledge in the talk as well. I'll just add really briefly also talk to your students about that. So that's kind of one of the main points Amanda and I were trying to make is that if we avoid it. Chatgpt in general, the students are still going to use it. So we need to talk to them about the issues that exist. We need to talk to them about how the field is evolving in ways that we as a common public don't really hear about. We need to talk about how to avoid some of these issues and be upfront about it. So I hope that that's a main take A waypoint of this talk as well. And I'll add to that just very quickly. Another reason I think this is so important is our students are going to be entering the workforce. They're going to be the ones making these decisions when it comes to how generative AI is used in the workforce. So if we can get them critically thinking about these issues while they're in our classrooms or while they're in our programs. I think that's going to make it even easier for them to have those conversations once they enter the workforce and they're the ones that are making those decisions. Thank you both so much for presenting this work and for doing such important work on thinking about how to use ChatGPT in the classroom and for instruction. It's been really wonderful having you here and sharing all of your cool results. So thank you so much for coming. Audience, thank you so much for participation and for all of your questions in the chat. And we will see you all next month for the next webinar with Kelly and Justice. HaveThank youThank you