Computer science researcher and lecturer Chris Piech ’10 M.A. ’11 Ph.D. ’16, who currently teaches the introductory computer science course CS 106A: Programming Methodologies, has implemented a new assignment submission system in a personal research project to better understand students’ learning processes. The project is a joint work between Piech, Lisa Yan Ph.D. ’18 and Annie Hu B.S. ’18 M.S. ’19.
According to Piech, many computer science students struggle to complete programming homework assignments — but this challenge is not reflected in their working submissions. Since computer science instructors have historically only seen students’ final code, educators don’t have a strong sense of students’ learning processes and the concepts they struggle with as they progress through assignments.
To understand students’ individual routes to their final code submissions, Piech created a tool called Pensieve, which “helps students visualize how they approach problems so that they and the TAs can review their approaches.”
In order to gain information about students’ progress, the tool periodically takes screenshots of the code created by students in Eclipse, a computer programming development environment.
Early in his research, Piech found that exam scores, rather than at-home assignments, are better predictors of students’ course material comprehension. Unlike the grade distributions of coding assignments, Piech said, the grade distributions of exams tend to vary much more widely, making it easier to see which students understand the concepts at a deeper level and which need more assistance. However, Piech said it would be ideal for instructors to be able to identify students who have difficulty understanding the material before they take their exams.
Piech believes that the process through which a student writes their code is crucial to understanding their grasp of the material.
“The process that a student goes through to get to their final code submission is highly predictive of midterm scores,” Peich said.
“In CS 106A, we’re not just teaching students how to program,” Piech said. “We’re also teaching students how to approach problems.”
For example, the picture accompanying this article is a visualization created by Pensieve that shows six major stages many students progress through when instructed to program a graphical pyramid in one of their homework assignments for CS 106A. Stages include creating a single brick, creating a loop to create multiple bricks in a single line, creating a loop nested inside another loop to create a two-dimensional grid of bricks and adjusting the parameters on the loops to generate a pyramid shape.
With Piech’s guidance, Yan inputted 130,000 pictures of graphical pyramids in various stages of completion into a deep-learning algorithm, along with information about the stages that high-performing students went through as they progressed through the problem at hand. Having learned to pinpoint different stages of graphical pyramids and the “model” route to take based on those of high-performing students, the algorithm is able to identify which students are going through unnecessary stages before arriving at a final working answer.
“[The student who goes through many unnecessary steps in their code] is the student who I think we [instructors] need to work with,” Piech said. “Somebody who puts in the time and effort but for some reason is not learning how to approach problems. We have all these great resources, and if we can identify this student, then we can spend some extra TA time to see if we can teach them these concepts.”
When asked whether this presented a possible privacy concern, one student in the class, who asked to remain anonymous, said they were comfortable being monitored periodically.
“I don’t find anything wrong with it because they’re only using the data to make sure people aren’t cheating and to help people see where they make mistakes and how they can improve,” the student said. “I’m fine with them monitoring my Eclipse every 10 minutes because they let us know beforehand, and it’s only recording our code.”
In addition to helping individual students, Piech also uses the tool to identify stages at which large numbers of students encounter difficulties. For example, when trying to create a graphical pyramid, many students get stuck when trying to adjust the parameters on the loops. Once he identifies stages where students tend to get stuck, Piech modifies his teaching to help students better understand the relevant concepts.
So far, Piech said this method has been successful. According to Piech, students in CS 106A this quarter have spent significantly less time on later, more difficult assignments than in previous quarters, which Piech attributes to a better understanding of core concepts. In addition, students performed significantly better on the midterm exam this quarter than previous quarters.
“I’ve found it pretty helpful,” said Avi Khemani ’21, reflecting on the program. “The TA can check my progression through each project and tell me what approach was good and what I can work on. For example, commenting while I code instead of waiting until the end or starting with a general idea of the program before coding instead of just coding right away.”
Other students feel that the system is not as effective.
“Personally, in my own interactive grading sessions, it’s never really come up,” said Jonah Wu ’21, referring to Pensieve’s tracking. “But I think [for] someone who has [received feedback through the model] it’s probably useful because going over how you solved the problem is always really helpful.”
In the future, Piech plans to continue using the tool to optimize his teaching.
“Originally, I was just conducting research on the learning process,” said Piech. “But now it’s less experimentation and more, ‘How can we best help these students?’”
This article has been updated to reflect Lisa Yan and Annie Hu’s contributions to the project. The article has also been updated to reflect the correct spelling of ‘Pensieve.’ The Daily regrets these errors.
Contact Ricky Grannis-Vu at rickygv ‘at’ stanford.edu.