Why are CS students so hard to nudge? A theory for why it’s so hard to promote a growth mindset in CS1
May 7, 2018 at 7:00 am 19 comments
Pearson took a lot of heat recently for trying to improve students’ mindset in My Programming Lab. I’m slightly worried about the ethics of their “embedded experiment.” I’m more worried that it didn’t work.
Titled “Embedding Research-Inspired Innovations in EdTech: An RCT of Social-Psychological Interventions, at Scale,” the study placed 9,000 students using MyLab Programming into three groups, each receiving different messages from the software as they attempted to solve questions. Some students received “growth-mindset messages,” while others received “anchoring of effect” messages. (A third control group received no messaging at all.) The intent was to see if such messages encouraged students to solve more problems. Neither the students nor the professors were ever informed of the experiment, raising concerns of consent.
The “growth mindset messages” emphasized that learning a skill is a lengthy process, cautioning students offering wrong answers not to expect immediate success. One example: “No one is born a great programmer. Success takes hours and hours of practice.” “Anchoring of effect” messages told students how much effort is required to solve problems, such as: “Some students tried this question 26 times! Don’t worry if it takes you a few tries to get it right.”
As Education Week reports, the interventions offered seemingly no benefit to the students. Students who received no special messages attempted to solve more problems (212) than students in either the growth-mindset (174) or anchoring groups (156). The researchers emphasized this could have been due any of a variety of factors, as the software is used differently in different schools.
Source: Pearson Embedded a ‘Social-Psychological’ Experiment in Students’ Educational Software [Updated]
Beth Simon and her colleagues tried a similar experiment, reported at ICER 2008. They did get informed consent. They tried a similar kind of “nudge” to get students to adopt a growth mindset. It didn’t work for Beth et al., either.
I advised Kantwon Rogers’ MS in HCI project, where he tried to nudge CS1 students (both on-line and off-line) to have a greater sense of “belongingness” in CS. Similar to these previous studies, he sent email prompts to students — some just encouraged study skills, and others promoted a sense that they belongs and could succeed in CS. In almost all of his conditions, belongingness dropped.
What’s going on here? Why are CS students so impervious to these prompts that have been successful in other settings?
I have a theory. There’s a notion in the behavioral sciences literature that you get more success changing behavior or promoting attitudes by reducing barriers than by prompting for desired behavior or attitudes. The analogy is to a large boulder that you want to move: You can push it and push it, or you can just dig away the dirt from the bottom. The latter is likely to get the boulder rolling without as much effort.
Here’s my theory: Introductory CS classes have systemic issues that encourage a fixed mindset and discourage a sense of belonging. There are too many signals to students that they can’t succeed, that they can’t get better, and that they don’t belong — perhaps especially in times of rising enrollment. Mere nudges are not going to move the boulder. We’re going to have to remove the barriers to belonging, self-efficacy, and the sense that students can succeed at CS.
Entry filed under: Uncategorized. Tags: behavioral economics, computing education research.
19 Comments Add your own
Leave a comment
Trackback this post | Subscribe to the comments via RSS Feed
Computing Ed Research - Guzdial's Take 
1.
kirkpams | May 7, 2018 at 10:09 am
As I read through Beth Simon’s paper, I began thinking about whether and how the language choice in CS1 could be one of those barriers. Two of the three institutions were using Java (the third wasn’t reported). The demand to have correct syntax throughout the program just to achieve compilation seems like it would be a pretty strong anti-nudge.
As an undergrad, I did just okay in my first two CS courses (in Scheme and Java), but building CGI web pages with Perl and Javascript was what really hooked me. The incremental nature of these languages, where they would produce output all the way up to a syntax error, was very forgiving and fun. I’d be interested to see someone repeat Beth’s work with some of the newer CS1 courses using Python or your media comp work.
My other thought was that these structural problems persist beyond just CS1. I think the way we do projects throughout the curriculum (pre-defined outcomes that students must achieve, with little or no room for students to personalize or bring their own experiences) is also problematic. By providing too much structure (presumably to reduce some of their cognitive load by pointing them in a single direction), we strip them of abilities for self-determination.
2.
Camille Moody McCue | May 7, 2018 at 2:28 pm
“I think the way we do projects throughout the curriculum (pre-defined outcomes that students must achieve, with little or no room for students to personalize or bring their own experiences) is also problematic. By providing too much structure (presumably to reduce some of their cognitive load by pointing them in a single direction), we strip them of abilities for self-determination.”
This is an on-point reflection, and really speaks to the nature of programming (and school) itself. So much of school is scaffolding, building up, guiding students — without teaching them how to tinker and forage for information themselves — something every programming must master. Because students have engaged with tech so much of their lives, they know that there are “things the tech can do,” and they want their programs to do those things, but when starting out, they simply don’t have the skills to get there. Programming courses may need to cultivate self-reliance and investigation/research skills as much as they do content skills.
3.
Mark Guzdial | May 7, 2018 at 2:31 pm
Really interesting insight! One of the studies that *has* shown impact on mindset in CS1 was in Python, and like Andy’s, included explicit instruction on problem-solving strategies: https://doi.org/10.1145/1734263.1734409
4.
Briana Morrison | May 7, 2018 at 10:09 am
I agree and I believe that some of this stems from what I call the “compiler effect.” In general if the program doesn’t compile the student gets a 0 on the assignment – no partial credit and no way to figure out how to fix it. Or a last minute change that broke something that they didn’t realize. In other subjects students can figure out ways to bluff to get some partial credit. When they can’t figure it out in CS they think they don’t belong.
5.
Austin Cory Bart | May 7, 2018 at 10:04 pm
I was just thinking about this yesterday – I do partial credit on programming assignments in my course: https://sites.google.com/a/vt.edu/acbart-eportfolio/blog/partialcreditonprogrammingexercises
I rolled out this feature fairly casually, but I would be curious to see what impact it would have on students.
6.
alfredtwo | May 7, 2018 at 10:10 am
I’m more surprised that these sorts of things work in other areas than I am that they don’t work in CS. Aside from that though I agree with you that students don’t have a sense of belonging. Observation suggests that some students do feel like they belong. These are the typical “computer geeks” who seem to live for this stuff. It may be that other students have the idea that CS is for “those people” and not for “people like me.”
I don’t think that is just a factor of CS classes. There is some societal messaging along the same lines. But we make it worse when CS is only an elective in schools. Students in required courses seem to have a sense that they are all in this together. Sure some students love it and some hate it but they are all taking it. They all belong there.
7.
Andy J. Ko | May 7, 2018 at 11:31 am
Some more evidence for your theory, Mark:
Loksa, D., Ko, A. J., Jernigan, W., Oleson, A., Mendez, C. J., & Burnett, M. M. (2016, May). Programming, problem solving, and self-awareness: effects of explicit guidance. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 1449-1461). ACM. https://dl.acm.org/citation.cfm?id=2858252
This wasn’t in a CS1 class, but was a 2 week high school summer course, where some of the cultural issues were removed. We nudged on mindset in both conditions and still saw no effect in the control group. However, in the experimental group, we nudge *and* provided explicit guidance on problem solving and saw huge effects on mindset, self-efficacy, autonomy, and productivity.
This evidence, combined with the other papers you cite, is starting to suggest that nudging itself is not enough to overcome the inherent difficulties of programming. Learners need guidance on process.
8.
Rodrigo Silva Duran (@Rodrigosduran) | May 8, 2018 at 6:23 am
This meta-review presents the correlation of growth mindset with academic achievement and impact of growth mindset interventions on academic achievements. TL;DR: Both tiny ( r = .01, d = .08)
http://journals.sagepub.com/doi/pdf/10.1177/0956797617739704
9.
Lex Nederbragt | May 7, 2018 at 12:18 pm
>Introductory CS classes have systemic issues that encourage a fixed mindset and discourage a sense of belonging
There are some examples in the other comments, but I would be interested in what other systemic issues you see? I’m not teaching a ‘true’ CS1 course, but (Python) programming as part of a first semester ‘computational modelling for Biosciences’ course and I’m interested in encouraging a growth mindset in my students.
10.
Mark Guzdial | May 7, 2018 at 1:26 pm
“Unlocking the Clubhouse” is a good analysis of systemic barriers, as are Lecia Barker’s defensive climate studies.
11.
gasstationwithoutpumps | May 7, 2018 at 12:40 pm
I wonder also if the nudges were poorly chosen. If I got a message that said “No one is born a great programmer. Success takes hours and hours of practice” when I made a mistake, I would not be thinking “I need to change to a growth mindset” but “If it takes hours and hours of practice to do these simple exercises that no one respects, what is the point of even starting?”.
Similarly with ” “Some students tried this question 26 times! Don’t worry if it takes you a few tries to get it right.” Telling a student to go beat their head against the wall does not lead to “belongingness” but to despair.
12.
alanone1 | May 7, 2018 at 12:54 pm
Yes, I quite agree with your comment here.
Also, I wonder about “Why are CS students so impervious to these prompts that have been successful in other settings?” — what other settings? If so, are they comparable?
My larger feeling is that so many of the current choices in trying to teach programming are just quite wrong …
13.
Mark Guzdial | May 7, 2018 at 1:19 pm
Kantwon’s prompts were based on this study from Science. Andy, your paper mentions a “growth mindset exercise” — where did it come from, and in what fields was it tried? In Simon et al., they used prompts from reading and from a general study of Stanford undergrads.
14.
Mark Guzdial | May 7, 2018 at 12:57 pm
I get that, but that’s only one of the four (counting Andy’s study, too) sets of prompts.
15.
Sylvia martinez | May 8, 2018 at 12:06 am
Maybe the students just recognize cheesy generic cheerleading when they see it. It’s an insult and a turn off.
16.
Beth Simon | May 8, 2018 at 3:18 pm
I’d love to get into a collaborative effort to both enumerate and categorize these barriers. Which do we guess might be highest impact to address? Which are easiest (by some definition) to address? Group googlesheet?
17.
Becky Roehrs (@beckyroe8400) | May 10, 2018 at 12:36 pm
Depending on what you’re teaching/learning, computer language textbooks can be dreadful. This doesn’t help! If instructors are forced to use publisher content (either by managers or due to time constraints), some of that content is suspect as well (and does not promote “community”).
I finally chose an HTML/CSS intro book that was popular with new programmers on Amazon! My students love it, but I had to create my own assignments, tests, discussions, projects and create my own videos to introduce the assignments (I used Kahn Academy and others to supplement). This is an online course with 25 students enrolled each semester. I also made students create web pages for a personal web site-no canned web pages were allowed, as they seemed to learn little when they did that.
I provide a pre- and post- course surveys, and ask students to agree to be patient with themselves while learning; and ask questions about their tech skills and provide resources to help them “get up to speed”. At course end, I ask what they like best and what they would like to see improved. My videos are primitive, but that’s what they say they like the best. It took hours and hours of my personal time to create the course, but it works.
But guess what? I’m not teaching it after this semester, after many years, because it takes too long to update, maintain, and provide effective feedback.
18.
Growth mindset matters for individual human performance, with a more indirect connection to academic success | Computing Education Research Blog | September 7, 2018 at 7:00 am
[…] replicated (see post here). The efforts to intervene on mindset in CS have had mixed success (see my blog post here on that). But it’s one thing to say that fixed vs growth mindset does not seem to matter much (the […]
19.
Social studies teachers programming, when high schools choose to teach CS, and new models of cognition and intelligence in programming: An ICER 2019 Preview | Computing Education Research Blog | August 12, 2019 at 7:00 am
[…] in computing education research is why growth mindset interventions are not always effective (see blog post here). We get hard-to-interpret results. I met Jamie and Nell at the Northwestern Symposium on Computer […]