How do we measure realism in VR anyway?
Mmmm... virtual steamed hams
Early attempts at VR gaming, such as the ill-fated Nintendo Virtual Boy and the never-released Sega VR, failed because they were unable to produce a convincing psychological effect that the player was actually in the game. That effect, called presence, has finally been achieved after decades of research.
Presence, also known as immersion, is a state of mind created when the virtual world around the user appears to be so real that they temporarily forget they are in a simulation. In order to maximize presence, the technology has to be developed enough to mimic sensory data that the brain would otherwise get naturally.
Seeing Is Believing
Visual systems were the first components of VR to be developed because if the simulated environment doesn't look real then everything else becomes moot. It wasn't until the last few years that displays and GPUs became good enough to provide a level of realism necessary to make a player forget that they are in a game. For visual presence, displays had to have a high pixel density or the dreaded screen-door effect happens. That is when a person can make out the pixel grid in front of their face which can diminish the effect. They also had to have high refresh rates, vivid colors, and provide a wide field of view. Recent advances in OLED displays made this possible. Having a great display isn't enough however.
To bring the imagery to life a powerful graphics processor is required. As with displays, it wasn't until the last few years that consumer graphics cards became powerful enough to achieve that at an affordable price. It is generally agreed that in order to achieve and maintain presence the graphics have to be displayed at a minimum of around 90 frames per second. Anything lower than that causes the illusion to break and stomachs to turn. Even with these headsets making it to the market there are still visual issues to iron out.
The biggest challenge is replicating how light reflects off of objects and into the eyes. There are companies working on a new generation of display technology called light field displays. These promise to make imagery look ultra-realistic as light is guided into the user's eyes creating the same light field that would exist if they were looking at a real version of that same scene. That technology is still in its infancy but is poised to revolutionize the display market.
Binaural Audio
The next sensory challenge is creating sounds that match the realism of the visual world. High fidelity 3D audio has been utilized in other non-VR applications. Players with surround sound know the feeling of hearing a sound behind them in a game and they have to quickly react. There are audio tricks that can be invoked to achieve the same effect with headphones.
By manipulating the slight difference in time that the right and left headphone channel plays the same sound, along with minute volume and frequency differences, the brain can be fooled into thinking it came from a particular direction. Variances in how the right and left ears hear sound allow people to determine distance, height, openness, and where the sound is coming from. Technology is getting closer to the goal of replicating a complete soundscape with precision.
Can You Feel Me Now?
The final piece of the sensory puzzle for now is touch. Smell and taste, while undeniably important for everyday life, isn't as necessary for presence even though at some point in the future those senses will be replicated as well. The ability to convince a VR user that they are holding a sword, with every detail like weight, temperature, and texture being simulated, is the holy grail for companies working on a class of technologies called haptics. Crude haptic systems have been around for years. Gamers are familiar with the rumble of their controller as they take damage in a game. Haptics have come a long way since those early attempts. Gloves and other wearable components that apply very precise levels of force and vibration to very specific areas of the body to match what is going on in VR are finally starting to come to the market. It may be a while before it is accurate enough to completely fool the brain, but these new generation of haptic devices are surprisingly realistic.
The human brain and the sensory organs that feed it data make up a near-infinitely complex system. While current technology doesn't quite create total presence and the maximum suspension of disbelief possible, it does seem likely that it will be a reality in the not-to-distant future.
