As a growing number of issues plague an aging campus infrastructure, Stanford officials from a range of academic and IT departments sought to assuage faculty members’ concerns over problematic course enrollment systems at the first Faculty Senate meeting of the year Thursday.
At the same meeting, senators were also briefed on the University’s response to the increasingly frequent power outages to the main campus. In late June, a three-day power outage caused by a nearby wildfire affected most of the main campus.
Students faced a nearly three hour long delay in enrolling in fall quarter courses when Axess, the legacy course enrollment system powered by decades-old software, crashed due to excessive load.
“You are almost certain to have a highly stressful, frustrating night in which you do not sleep well” during course enrollment night, a sophomore student wrote to University administrators, as Chief Information Officer Steve Gallagher shared to the Senate.
The root cause of the botched course enrollment, according to Gallagher, was an overload on Axess’ underlying legacy student information system. The overload triggered a bug that had been erroneously introduced when the University revamped the student-facing web portal component of Axess in August.
As students began to log on to Axess in the minutes before enrollment opened at 9 p.m. PT, Gallagher said, the bug was triggered and overwhelmed the underlying legacy system. The bug was not caught in load testing that University IT had conducted in advance of unveiling the new system, he added.
While plans to replace the legacy system are being discussed, Gallagher said that the current range of available systems is not able to support Stanford’s complex technological infrastructure. In the future, he added, he hopes that the entire course management system will be hosted on a modern cloud environment.
Both Gallagher and University Registrar Johanna Metzgar apologized for the crash and the resulting stress on students. The Sept. 1 outage was not an isolated event, however. Over the years, students have regularly reported system slowdowns during peak enrollment periods.
Gallagher and Metzgar stated that efforts on multiple fronts are underway to prevent the issue from recurring in winter quarter. A team of external consultants will be deployed to address bugs identified in Axess, enhance peak registration load simulation and reconcile load balancing against legacy system bottlenecks, they said.
“These sorts of issues are entirely inconsistent with the service levels you should expect,” Gallagher said. “The number one priority is to get this right for the upcoming Dec. 1 registration.”
Metzgar also said that the technical problems students face in enrolling in courses has caused some students to lose trust in the University’s academic services.
“We are really committed to stabilizing the technical environment,” Metzgar said. “Once that’s done, I think the work of rebuilding trust among students comes into place.”
The lack of trust, she added, has even led some students to build bots in an attempt to optimize their chances of gaining access to Axess during peak load times. In order to repair the trust, the registrar pointed to a multi-year project called the ConnectED curriculum to modernize enrollment infrastructure by implementing a new class scheduling system and a degree audit system that allows students to track their degree progress.
Power outage issues
The University has not only had to contend with course enrollment issues, but also increasingly frequent power outages to the main campus as a result of downed power lines caused by wildfires.
The most significant of these outages last three days, starting June 21. A wildfire near the Edgewood Substation impacted the main power line to the campus substation, the Jefferson Line. Though power to the main campus was not restored for more than three days, limited back-up power was provided to the campus via a secondary line, the Cooley Landing. Nearly all of the secondary power went to powering Stanford Hospital’s chilled water system, according to Jack Cleary, Associate Vice Provost for Land, Buildings and Real Estate.
Cleary said that although the University’s emergency operations centers were well coordinated during the outage, communication to research staff “was not structured.” Improving that line of communication is an important learning lesson, he added.
There have also been shorter power outages this year to blackout the main campus. For nearly 8 hours on July 12, Green Library and surrounding buildings were impacted due to a failed cable. On Sept. 25, another issue impacting the Jefferson Line caused a nearly three hour outage for the entire campus.
Cleary said that a new committee, the Faculty Committee on Electrical Resilience in the Research Enterprise, has been established by the provost to advise and recommend measures to reduce risk to research posed by power outages. In addition, the University will conduct a study to modernize its electrical infrastructure, Cleary said.