Research being conducted at Colorado State University is bringing technology once thought to be science fiction one step closer to reality.

Augmented reality technology allows three-dimensional objects to be placed in real-world surroundings as if they were physically present. It is most commonly found in mixed reality headsets such as the Apple Vision Pro or Meta Quest 3, which combine both augmented reality and virtual reality. While these are consumer products, often used for media consumption or gaming, the AR technology being studied at CSU could have key life-saving applications in search-and-rescue operations, emergency services and medical 

One example is helping firefighters navigate a smoke-filled building, said Matthew Sturgeon, a former CSU graduate research assistant who completed his master’s degree last December.

“If we’re able to give them information and pinpoint exactly (where) there is a person in there, … (and) they’re able to see where that is in space, they can possibly navigate more efficiently and help them faster,” Sturgeon said. 

Sturgeon remained onboard with the research after completing his bachelor’s degree in computer science at CSU in 2023 and moving into his graduate program.

“Originally it kind of started as a game design thing,” Sturgeon said. “I always found interest in that, and then I heard about human factors in human-computer interaction, and it was kind of cool to me because it scratched that itch of game design and fun little puzzles … It does have a purpose for helping people, and within (Human-Computer Interaction) as a broader topic, there’s a lot of accessibility. The whole point is designing computers to be for people.”

Another potential application discussed in the research literature is providing assistance to surgeons. 

“Think of using an AR device and providing information behind the skin of the patient,” said Mohammed Arefin, a CSU assistant professor of computer science who led the research. “It could be a tumor visualization, which is actually behind the occluded surface.” 

As shown in these examples, AR technology has the potential to give medical personnel and first responders information about objects behind occluding surfaces, or in other words, surfaces blocking their vision.

“Superman has X-ray vision,” Arefin said. “He can actually see through the wall. … With augmented reality, we can provide additional information to enhance our understanding. … We are mimicking and saying that this is augmented reality X-ray vision, which is this very unique capability of augmented reality technology.”

By using available data, such as video feeds from security cameras, visual information can be superimposed onto the image provided in an AR headset; in doing this, any objects behind walls can be displayed to the user. However, human vision is not necessarily adapted to such a task, so perception of depth and distance of virtual objects is not always in line with the actual surroundings.

“The main thing that we did was we were gauging depth perception,” Sturgeon said.

One of the main procedures of the experiments involved participants wearing head-mounted AR devices and first providing verbal estimates of how far they believed 3D objects in front of and behind occluding surfaces were, and then again after the objects and occluding surfaces were removed.

The researchers investigated several aspects of AR technology. They tested occluding surfaces of various patterns, such as a solid black surface, a black-and-white checkerboard and a realistic brick wall texture. They also compared the two main AR techniques: optical see-through, in which the user looks through a transparent lens at the surroundings, and video see-through, in which video feed from external cameras is used instead.

They found the texture of the occluding surface had very little impact on users’ ability to estimate how far a virtual object was, but there was a difference in the AR technique, as optical see-through users had more accurate estimates.

“The potential reason could be that it’s much more natural,” Arefin said. “Because when you are seeing the real world and the virtual world as a video feed, you may not get the true perception of the environment.”

This will not be the last of this research, according to Arefin. This study surveyed depth perception in what is known as “action space,” which could be from one end of a room to the other. What still needs more research, he said, is depth perception in what is called “near space,” or the area within the immediate arm’s length of the user. The findings of such a study would be useful for the surgical applications of this burgeoning technology.