This article was initially published on August 10th, 2015 by Mark Ristich
How much of what we see is real? How much is merely interpreted?
The above question has captured my fascination for years, particularly when we discuss questions surrounding colour, light, and sound.
The story started back when I was a young child. In Grade 6, in a small school in Surrey, British Columbia, our teacher handed us a book. It was called The Giver by Lois Lowry. It described a young boy, Jonas, caught in a world where sedatives and selective breeding prevented people from hearing music, seeing colour, and smelling fragrances. They all had the physical capacity to do so, however, and Jonas's sense organs were abberant, allowing him to see flickers of colour in the world around them.
The curiosity of whether others are perceiving in the same way as I am, or if there is something different, something communicable between our perspectives that reveals some underlying truth is what drew me to the next book, Plato's Republic. Deep in his Allegory of the Cave, normal people are trapped, their heads locked in place staring at a black wall, while shadows flicker in front of them. Yet these shadows weren't real, merely mimics of real things. In Plato's story, he suggests that philosophy is the only way to break the bindings and turn around to see what the world really looks like behind the scenes.
After years of studying philosophy and seeking answers to this question, all the satisfying answers are unreasonable, and the <a href=http://plato.stanford.edu/entries/monism/">unsatisfactory answers are practical. Nevertheless, throughout all my formal education the question continued to nag at me, and finally, with the advent of modern Virtual Reality, I was able to put some of these questions to semi-empirical testing. How far must we go before our minds are tricked into believing that what we see, hear, or feel is real?
More specifically:
How deep does our conviction go?
After using an Oculus Rift for the first time in August 2014, my mind went wild. The first thing I did was tell myself that I would have to get one. The second thing I did was tell myself that I would have to find someone who has one, because there was no way I would able to get my hands on one before my brain exploded. A week later, I grabbed a pen and paper and started writing down the things I wanted to know. I came up with a series of experimental experiences. Some of them were violent, all of them were highly manipulative, so I promised that I would test them on myself first before any others.
Here is the list of what became known as my Meditations.
Meditation 1: Into the Lion's Mouth
First, predators. When we look at a picture of a lion's mouth compared to being in an HMD looking at a virtual lion's mouth, compared to looking at an actual lion , our emotional states should differ. A picture, for instance, should pose no threat. By nature it is flat, and lacks depth. A skilled artist may use perspective effectively, but moving your body or head will break this effect, due to the lack of parallax.
Virtual Reality, inside an HMD, however, can provide this parallax effect. I de-constructed this problem into its components - what about the lion makes it frightening? Is it the size, an aggressive expression, or just a tied evolutionary fear to its physical structure (sharp teeth and claws inspire painful memories). I discovered that, above all, evolutionary fear plays the largest part. The sharpness of the teeth, when seen in VR, is enough to trigger the emotional fear response, and this effect can be transferred to other pointed objects. As you get closer to the pointy tip, the sensation becomes more visceral. When the mouth is closed, the sensation fades. After much testing removing the virtual flesh and modelling the skeleton, changing proportions, moving towards and away, it seems that the pure visual sensation of "sharpness" is what we find discomforting.
Those pointy ends are terrifying.
Meditation 2: Charybdis
Next, drowning. I love swimming, and can be seen several times a week swimming laps or dipping in lakes. The sensation of drowning will never go away, however. It's a claustrophobic entrapment, a suffocating type of fear, entirely unique from that of predation or attack. I could not produce this extreme sense of vulnerability in a virtual scenario, particularly because without a preceding sense of claustrophobia, my brain just knew I wasn't in that much danger. It seems that the sensation of drowning is more learned, and it requires the added resistance that submerging oneself in viscous liquid supplies. As soon as my arms moved, the illusion broke. As soon as my head went underwater, the illusion broke. (I tried carefully to give the other sensations that being underwater induces: limited visual range, deflection and diffusion of light, muffling of sound, etc.) It just won't sell.
Meditation 3: The Ark of the Covenant
Next, I moved to our visual cortex, asking how would intense light sourced in a virtual space be handled by our sensory system?
With conventional light, powerful beams of photons fit one (or several) light wavelengths. After staring at a bright colour for a prolonged period of time, the constant barrage of photons hitting your retina becomes too much. Your visual cortex gives the cones a break, turning them off, but freezing the colour signature in our visual cortex. In effect, while we are still looking at a fixed colour, our eyes have stopped firing the signal for that colour and instead the brain just loops feedback. This speeds up processing, but has a nasty side-effect. The standard name for these afterglows are [insert here], but you know it as the "blackout" effect you feel after looking at the sun briefly. It is also one of the reasons that a red ball stands out so brightly on a desert landscape.
I recreated this experience by creating a virtual world where a glowing coloured orb hovers in a forest clearing. Staring at the orb is comfortable, albeit bright. After roughly 10 seconds of burn-in, looking away had a noticeable effect, but not in the way that I had predicted.
I thought that the eye would have an afterglow of the inversion of the colour, assuming that the cones had been turned off, they would only respond to colours without the colour signature of the orb. If I stared at an orange ball, I would see a blue afterglow, since any orange receptors would be deactivated, thus my sensitivity to orange would be minimal or removed. Fascinatingly, the opposite was true. After looking at the orange ball, looking away caused an orange afterglow to tint the space. It was as if my brain had set the loop state to "orange", and left it running in that section of my visual field, even after my eyes moved. After much reading, I did manage to get the predicted blue effect, but it was only by looking at pure white light. Apparently the wide range spectrum has the opposite effect on your brain when looking at very light surfaces, as the strong signals from adjacent photoreceptors on the retina are strong enough to override the afterglow effect.
Meditation 4: The Steeple of Pisa
Feel free to test yourself here.
The last meditation that I was able to test is only partially complete, but under construction. This was testing the ability to replicate the feeling of falling, purely by putting the user in a virtual space with no floor, but walls that slowly accelerate a pattern upwards. To replicate motion, rather than moving the camera/user, I am determined to use layered stripes with a consistent width to mimic movement. This process is inspired by Galileo, who famously reprimanded scholars criticising his heliocentric data, saying "What is more reasonable, for you to sit at the steeple of the Chapel and have the entire city of Pisa spin around you, or for you to turn your head?"
Clearly, I want to go about this the hard way, spinning the 'world', instead of the player. What I want to know is whether it feels as though the world is moving, the user is moving, or if it is indistinguishable.
By accelerating the stripes, I hope to imply movement to the secondary visual field, located in extrastriate visual areas, such as the lateral geniculate nucleus (LGN). The case has yet to be proven, however, that the visual cortex (V1) is so easily tricked, or if centuries of heliocentrism has persuaded us that the Sun ought not to revolve around us.