Can Augmented Reality's Biggest Blind Spot Be Fixed?

Augmented,virtual and mixed reality are trying to create more enticing experiences.

The next - probably transitional effort - comes from Disney, which is creating a virtual space in which the 'reality' can be focused, not unlike Instagram tables at restaurants which provide the 'perfect' environment to encourage customers to take pictures. That said, such staging is proving popular and may meet consumers needs until a more comprehensive experience is available. JL

Brian Barrett reports in Wired:

Dependence on devices (with) augmented reality tends to favor isolated moments. Even shared experiences, where everyone in a physical space sees the same digital apparition, happen at the remove of whatever devices enable the effect. “This is somewhat like the ‘take your picture here’ cutouts," the kind where a painting of a giraffe leaves a hole for you to poke your head through. (But) these experiences put the user into an augmented world, as opposed to augmenting the world that the user is already in.

You sit on the bench, as one does. Across from you hangs what appears to be a mirror. Before you can fix your cowlick, a pygmy elephant saunters over, plops down next to you, and hands you a glowing, golden orb.

You’re not dreaming, hallucinating, or having a bad (good?) trip. You happened to take a seat on Magic Bench, a Disney Research project that overcomes one of the biggest limitations of mixed reality. Forget the clunky headset. All this brain-bending experience requires is that you take a seat.

Bench Player

When you talk about mixed or augmented reality, you can’t separate the experience from the hardware delivering it. Microsoft’s Hololens pulls off impressive feats of world-creation, if you don’t mind the narrow viewing angle or the headgear. Apple’s ARKit will make Pokémon Go’s AR feel like a stencil drawing—as long as you don’t need your iPhone for anything else at the moment.

Because of this dependence on devices, augmented reality tends to favor isolated moments. Even shared experiences, where everyone in a physical space sees the same digital apparition, happen at the remove of whatever devices enable the effect.

What if instead of expensive equipment and a narrow aperture, all mixed reality required was a place to sit, a screen, and a healthy dose of whimsy? You’d get something like Magic Bench. “We want technology to be as seamless as possible,” says Moshe Mahler, principal digital artist at Disney Research. “We don't want to instrument the user with hardware.” The result is an experience in which visitors interact with elephants and toads, where they can see, hear, and feel indoor rain. Crucially, that’s visitors, plural. Everyone who sits on the bench at the same time sees the same layered-on reality.

Magic Bench isn’t the first “walk up and play” mixed-reality experience. But it surpasses previous efforts creatively and technologically. (Never before, to my knowledge, has an AR experience included a cranky talking toad.) And as with most magic acts, the real fun starts when you peer behind the curtain.

Kinect the Dots

This isn't about the bench. Not entirely. While the Disney Research team outfitted the bench in its mixed reality demo with haptic actuators aplenty—when that elephant snores, you feel the vibrations—the secret lies in its use of cameras. The Magic Bench system uses the depth and RGB color sensors of a Kinect (as in the Xbox accessory) to reconstruct a scene over which 3-D, animated vignettes play out.

Of course, it's not as simple as pointing a Kinect at a bench. A depth camera struggles to faithfully represent a 3-D space for reasons that become quickly obvious. “Like a normal camera, it cannot record what it cannot see,” says Mahler. “For example, if I stand in front of a desk, blocking line of sight to the desk, we cannot see what is on the desk.” Compounding the problem, the Kinect’s RGB camera is offset from its depth camera, meaning they have slightly different viewing angles. The set-up conspires to create “depth shadows,” or patches where the 3-D effect doesn’t work. Nothing magic about that! The Disney Research team figured out how to turn the 3-D reconstruction into a 2-D space, and use that “flat” as a sort of redundant backdrop that fills in any gaps. With a stable 3-D image to play with, the animations that can appear on the bench-facing screen are limited only by the imagination. “The platform runs on a common game engine,” says Mahler. “Anything we can typically program in a game engine we can accomplish here.”

Presto

Of course, there are only so many fantastical experiences you can have on a bench. But those restrictions help inform the results, like having a writing prompt in a creative writing class. “The bench allows us to constrain space,” says Mahler. “People typically want to touch the CG characters. Of course, they aren't really there, so they can't 'feel' the characters directly. We play on these expectations.” Try to touch an animated toad sitting next to you, for instance, and he’ll spurn your advances.

“We want technology to be as seamless as possible. We don't want to instrument the user with hardware.” - Moshe Mahler, principal digital artist at Disney Research

It’s that limited-space, third-person perspective that allows Magic Bench to be a shared experience—and what sets it apart from the mixed and augmented reality experiences you might be accustomed to.

“Think of it as the difference between tourist attraction photo opportunities,” says Todd Richmond, director of the Mixed Reality Lab and Studio at the University of Southern California’s Institute for Creative Technologies. Headgear and phone-based AR, he says, are more like selfies, where you choose the camera angle and perspective. “The Magic Bench is somewhat like the ‘take your picture here’ cutouts,” says Richmond, the kind where a painting of a giraffe leaves a hole for you to poke your head through.

That comes with downsides, though. “Experiencing augmented content through a screen or projection in front of the user can take them one step back from the world,” says David Nelson, who specializes in virtual storytelling at ICT. “These experiences put the user into an augmented world, as opposed to augmenting the world that the user is already in.”

Still, in the hands of a company like Disney, you can imagine some fairly special augmented worlds. Just think of what the Mouse House could do with a bench and a near-limitless CGI palette. The company hasn’t mentioned any plans to introduce the technology to its theme parks, but it seems like a natural fit.

“You may want to rest on a bench or check your email for a second, and all of a sudden it could be raining indoors,” says Mahler.

It seems inevitable that you'll eventually take in AR experiences through Warby Parker iGlasses or Facebook-brand corneal implants. But experiences like the Magic Bench offer a welcome stopgap. After all, mixing reality shouldn’t mean isolating yourself from it.