Augmented reality (AR) is about enhancing the real-world environment with computer-generated perceptual information. Though it was experienced by many during the Pokemon Go craze, AR can be applied to more than just the mobile gaming realm. Google Maps, for instance, has AR street views to help users follow direction in real time. UC Berkeley computer science Ph.D. student Anastasio Angelopoulos ’19 is using AR to help the blind navigate through different environments. 

During Angelopoulos’ freshmen year at Stanford, he lost a significant part of his visual field due to an autoimmune disorder. His vision loss has shifted the way he perceives the world. Because of this, he said he can better relate to blind and partially sighted community members who struggle with more severe disabilities. 

In the summer after his freshman year, Angelopoulos reached out to work with Mark Humayun, a professor of ophthalmology and director of the University of Southern California (USC) Roski Eye Institute, where he built imaging processing software for the retina.

Angelopoulos found his inspiration to build adapted AR glasses for the blind when he came across ads on Facebook and Instagram for low-vision aids. These low-vision aids, however, such as eSight, were avoided by some in the blind community, Angelopoulos said, as they cover up the user’s eyes and thus hinder one’s interpersonal relationships with others. Angelopoulos proposed to Humayun his idea to build AR glasses using Microsoft HoloLens for those with low vision, and Humayun agreed not only to help fund Angelopoulos’s research but also to mentor him. 

“There’s something really special about having the privilege to do work that has the potential to really help people on a day to day [basis], and where you can see that happen,” Angelopoulos said. “It’s incredibly motivating because that’s ultimately what I want my life to be about: to be able to make people healthier, more free, safer and happier, with higher quality of life.” 

For the past two years, Angelopoulos, supported by Humayun at the USC Institute of Biomedical Therapeutics, has been developing a special pair of glasses that helps people with retinitis pigmentosa (RP) — a group of genetic disorders involving breakdown and loss of cells in the retina — to navigate through their surroundings more safely.

Though RP affects only 1 in 4,000 people, patients in an advanced stage of the disease may have severely restricted peripheral vision, causing poor mobility and decline in quality of life. According to the National Eye Institute, RP causes a progressive loss of the visual field that makes it difficult with those who have the condition to perform daily tasks. 

As part of an Aug. 2 paper he wrote for Scientific Reports, Angelopoulos cited results in which he found that the AR glasses he developed can improve patients’ mobility by 50% and grasp performance by 70%. 

Pseudocolor wireframe

Selecting Microsoft HoloLens as the headset for his AR glasses, Angelopoulos developed a new technique called the pseudocolor wireframe, which uses simultaneous location and mapping to fully render the 3D structure of a room in real time. It does so by displaying a 4-color depth-encoded wireframe over potential obstacles, with each color indicating a distance between the user and the object. This encoding system is not exclusive to Microsoft HoloLens, and it is compatible with other commercially available AR devices. 

“Some groups have looked at using AR augmented reality to help with certain training and certain object recognition, but nobody has really done it this setting with subjects with visual impairment to kind of help them,” Humayun said. “And just the approach and the implementation of using colors for depth perception is also very novel.” 

Takeaways from research 

Angelopoulos’ paper describes his research testing the mobility performance of subjects on an obstacle course based on an FDA-validated functional test and a custom-made grasping setup with the AR glasses he developed in comparison to without the glasses.

His research team’s project is the first of its kind to the first to show a statistically significant mobility improvement when RP patients use a visual AR low vision aid in a test validated by the FDA to correspond with a real-world mobility improvement, and also the first to do the same in a grasp experiment. 

“[My project] taught me that research is harder than I could have ever imagined — going into it, it seems insurmountable,” Angelopoulos said. “And then afterwards, you’ve looked at all the things that had to happen for the project to succeed, and you’re like, ‘Wow, how did we get so lucky?’ or ‘How did everybody come together to put in that much work?’ It taught me as a real researcher, not to settle for an easy methodology that will just get to a publication, and try to hold yourself to a higher standard of doing something that you truly believe is going to help the world before you think about whether or not this is the highest return on investment in terms of publication.” 

Next steps

Angelopoulos expressed gratitude toward his research subjects and mentors: Humayun, USC Retinal Degeneration Center Director Hossein Ameri and Roski Eye Institute research-administrative liaison Debbie Mitra.

“The subjects that we had in our study, were amazing, because they donated their time hours of their time to participate in an extensive study,” Angelopoulos said. “I feel that they have experienced a huge loss. And I think that anything that we can do to help them would be really meaningful.”

Moving forward, Angelopoulos is looking toward the testing of his AR glasses to other types of visual disabilities and the development of more compact hardware for the AR glasses with a high battery life. 

“We tested on RP patients, but we also recruited a few macular degeneration patients, and I really think the device would work for them,” Angelopoulos said. “I really think that we or somebody should take this work and try to apply it to the broader blind community because there’s a much broader population [than RP patients] that could still benefit. Once we get a system that can get people through a workday, wearing an augmented reality headset, and that looks a little bit more visually pleasing and isn’t so heavy to wear on your head, then I think the potential for impacting the lives of blind people is enormous.”

MHR Vision

In 2018, Angelopoulos and Humayun co-founded the prescription-eyeglass diagnostics company MHR Vision, which aims to provide prescription glasses and eye exams in third world countries, with Angelopoulos as the lead engineer. In terms of incorporating Angelopoulos’s AR glasses into the project, Humayun said it is something the two have considered, but MHR Vision is focusing on providing basic prescription glasses to those in need. 

“There are a lot of underrepresented and underprivileged people out there that have difficulty getting eyeglasses,” Humayun said. “The idea [of MHR Vision] is to see if we can help people who can’t get [glasses] or maybe even can’t afford to get eye exams and get eyeglasses by helping them out with inexpensive ways to get some eyeglasses. It’s a humanitarian mission that I’ve had that was important to me.”

Contact Rachel Wu at rwu2579 ‘at’ gmail.com.