Microsoft's HoloLens is no joke: My reality augmented with Skype, Minecraft

With the HoloLens, Microsoft promises it can deliver a true next-gen computing experience while other tech giants struggle to do the same.Microsoft

REDMOND, Wash. -- In the bowels of Building 92, hidden underneath the company's public visitor center in a secret series of labs, Microsoft let a few people try out what may be the most ambitious Windows device ever made: a holographic headset that aims to rival the most advanced virtual reality devices out there.

Microsoft's HoloLens is expected to run Windows 10 and apps -- holographic ones that will float in front of your line of vision and apps that can be run on phones, tablets, PCs and the Xbox One game console. With the holographic programs, Microsoft is trying to transform how we think about computing, productivity and communication. Just as VR rivals Oculus (owned by Facebook) and Google are trying to reimagine virtual experiences with their head-worn devices, Microsoft wants us to imagine a world without screens, where information merely floats in front of you.

"We're not talking about putting you into virtual worlds," HoloLens leader Alex Kipman said Wednesday during an event at Microsoft's headquarters here. "We're dreaming beyond virtual worlds, beyond screens, beyond pixels."

Kipman started working at Microsoft seven years ago, when he pitched the idea for the Kinect motion camera, a video game device that tracked a player's body movements. The Kinect went on to become one of the fastest-selling devices in history.

For the last five years, Kipman has been focused on taking the innovations inside the Kinect -- cheap and powerful motion-sensing cameras, voice control -- and packing them into a pair of transparent goggles.

Microsoft appears far along in realizing this augmented reality vision. With HoloLens today, the company has designed a convincing prototype that floats 3D images in front of you and that can change the look of real-world objects all around. But it's unclear how Microsoft expects to deliver on CEO Satya Nadella's commitment that such a device will be for both consumers and businesses.

Also unsaid: How much it will cost. Microsoft said it expects to release a finished HoloLens within the same time frame as Windows 10, which should arrive sometime this year. The Oculus Rift's various developer kits, on the other hand, have cost upward of $300 in the past, with its consumer model expected to come in between $200 and $400. Samsung's Gear VR headset runs around $350.

Microsoft's glasses are different from Oculus Rift goggles, which promise to transport you to a different world and open up numerous possibilities for film, TV, sports and other entertainment. HoloLens uses a technology called augmented reality, which overlays images onto real life and lets you interact with them. In theory, this is easy, but the biggest struggles competitors have had so far have been to design a headset that can stand alone, untethered from a computer or power source, and travel into various environments. Overcoming those challenges is necessary before mainstream consumers will buy into such a bold vision for next-generation computing.

From Mars to Minecraft

As we're led down the stairs into the basement, we're told that we can't try the more polished, all-in-one prototype Microsoft just showed onstage. Instead, we'll be using an earlier, uglier prototype. The company doesn't allow smartphones or cameras into the room.

The device's holographic processing unit, the special processor Microsoft designed to basically help the HoloLens interpret movement and sound, is cased in a separate, chunky box intended to be worn around your neck. The glasses aren't the sleek, space gray model Microsoft unveiled this morning, but a mass of metal. A long chord tethers me to a pair of PCs that are helping feed the goggles their images.

 14 Photos

Microsoft HoloLens (pictures)

The demo was unique in that it showcased a feasible and realistic use of augmented reality that wasn't bombastic or meant to marvel. I was able to do something I can say I haven't done in quite a long time with help from a total stranger who was seeing through my eyes and drawing on my reality to direct me through a task.

After checking the voltage on the wires, wrapping coils around the respective screws and capping the loose wires, I picked up a remote control and tapped a button. The light came to life. I had only to look up so the person on the other end of my eyes could see the end result of our successful collaboration.

Microsoft's HoloLens is no joke: My reality augmented with Skype, Minecraft

9:51 PM augmented reality, HoloLens, Microsoft HoloLens

Project HoloLens: Our Exclusive Hands-On With Microsoft’s Holographic Goggles

Microsoft HoloLens

It’s the end of October, when the days have already grown short in Redmond, Washington, and gray sheets of rain are just beginning to let up. In several months, Microsoft will unveil its most ambitious undertaking in years, a head-mounted holographic computer called Project HoloLens. But at this point, even most people at Microsoft have never heard of it. I walk through the large atrium of Microsoft’s Studio C to meet its chief inventor, Alex Kipman.

The headset is still a prototype being developed under the codename Project Baraboo, or sometimes just “B.” Kipman, with shoulder-length hair and severely cropped bangs, is a nervous inventor, shifting from one red Converse All-Star to the other. Nervous, because he’s been working on this pair of holographic goggles for five years. No, even longer. Seven years, if you go back to the idea he first pitched to Microsoft, which became Kinect. When the motion-sensing Xbox accessory was released, just in time for the 2010 holidays, it became the fastest-selling consumer gaming device of all time. Right from the start, he makes it clear that Baraboo will make Kinect seem minor league.

Kipman leads me into a briefing room with a drop-down screen, plush couches, and a corner bar stocked with wine and soda (we abstain). He sits beside me, then stands, paces a bit, then sits down again. His wind-up is long. He gives me an abbreviated history of computing, speaking in complete paragraphs, with bushy, expressive eyebrows and saucer eyes that expand as he talks. The next era of computing, he explains, won’t be about that original digital universe. “It’s about the analog universe,” he says. “And the analog universe has a fundamentally different rule set.”
Translation: you used to compute on a screen, entering commands on a keyboard. Cyberspace was somewhere else. Computers responded to programs that detailed explicit commands. In the very near future, you’ll compute in the physical world, using voice and gesture to summon data and layer it atop physical objects. Computer programs will be able to digest so much data that they’ll be able to handle far more complex and nuanced situations. Cyberspace will be all around you.

What will this look like? Well, holograms.

First Impressions

That’s when I get my first look at Baraboo. Kipman cues a concept video in which a young woman wearing the slate gray headset moves through a series of scenarios, from collaborating with coworkers on a conference call to soaring, Oculus-style, over the Golden Gate Bridge. I watch the video, while Kipman watches me watch the video, while Microsoft’s public relations executives watch Kipman watch me watch the video. And the video is cool, but I’ve seen too much sci-fi for any of it to feel believable yet. I want to get my hands on the actual device. So Kipman pulls a box onto the couch. Gingerly, he lifts out a headset. “First toy of the day to show you,” he says, passing it to me to hold. “This is the actual industrial design.”

Oh Baraboo! It’s bigger and more substantial than Google Glass, but far less boxy than the Oculus Rift. If I were a betting woman, I’d say it probably looks something like the goggles made by Magic Leap, the mysterious Google-backed augmented reality startup that has $592 million in funding. But Magic Leap is not yet ready to unveil its device. Microsoft, on the other hand, plans to get Project HoloLens into the hands of developers by the spring.

Kipman’s prototype is amazing. It amplifies the special powers that Kinect introduced, using a small fraction of the energy. The depth camera has a field of vision that spans 120 by 120 degrees—far more than the original Kinect—so it can sense what your hands are doing even when they are nearly outstretched. Sensors flood the device with terabytes of data every second, all managed with an onboard CPU, GPU and first-of-its-kind HPU (holographic processing unit). Yet, Kipman points out, the computer doesn’t grow hot on your head, because the warm air is vented out through the sides. On the right side, buttons allow you to adjust the volume and to control the contrast of the hologram.



Microsoft’s Lorraine Bardeen demonstrates HoloLens at the Windows 10 event at the company’s headquarters in Redmond, Washington on Wednesday, Jan. 21, 2015.

Tricking Your Brain

Project HoloLens’ key achievement—realistic holograms—works by tricking your brain into seeing light as matter. “Ultimately, you know, you perceive the world because of light,” Kipman explains. “If I could magically turn the debugger on, we’d see photons bouncing throughout this world. Eventually they hit the back of your eyes, and through that, you reason about what the world is. You essentially hallucinate the world, or you see what your mind wants you to see.”

To create Project HoloLens’ images, light particles bounce around millions of times in the so-called light engine of the device. Then the photons enter the goggles’ two lenses, where they ricochet between layers of blue, green and red glass before they reach the back of your eye. “When you get the light to be at the exact angle,” Kipman tells me, “that’s where all the magic comes in.”

Thirty minutes later, after we’ve looked at another prototype and some more concept videos and talked about the importance of developers (you always have to talk about the importance of developers when launching a new product these days), I get to sample that magic. Kipman walks me across a courtyard and through the side door of a building that houses a secret basement lab. Each of the rooms has been outfitted as a scenario to test Project HoloLens.

A Quick Trip to Mars

The first is deceptively simple. I enter a makeshift living room, where wires jut from a hole in the wall where there should be a lightswitch. Tools are strewn on the West Elm sideboard just below it. Kipman hands me a HoloLens prototype and tells me to install the switch. After I put on the headset, an electrician pops up on a screen that floats directly in front of me. With a quick hand gesture I’m able to anchor the screen just to the left of the wires. The electrician is able to see exactly what I’m seeing. He draws a holographic circle around the voltage tester on the sideboard and instructs me to use it to check whether the wires are live. Once we establish that they aren’t, he walks me through the process of installing the switch, coaching me by sketching holographic arrows and diagrams on the wall in front of me. Five minutes later, I flip a switch, and the living room light turns on.

Another scenario lands me on a virtual Mars-scape. Kipman developed it in close collaboration with NASA rocket scientist Jeff Norris, who spent much of the first half of 2014 flying back and forth between Seattle and his Southern California home to help develop the scenario. With a quick upward gesture, I toggle from computer screens that monitor the Curiosity rover’s progress across the planet’s surface to the virtual experience of being on the planet. The ground is a parched, dusty sandstone, and so realistic that as I take a step, my legs begin to quiver. They don’t trust what my eyes are showing them. Behind me, the rover towers seven feet tall, its metal arm reaching out from its body like a tentacle. The sun shines brightly over the rover, creating short black shadows on the ground beneath its legs.



Microsoft

Norris joins me virtually, appearing as a three-dimensional human-shaped golden orb in the Mars-scape. (In reality, he’s in the room next door.) A dotted line extends from his eyes toward what he is looking at. “Check that out,” he says, and I squat down to see a rock shard up close. With an upward right-hand gesture, I bring up a series of controls. I choose the middle of three options, which drops a flag there, theoretically a signal to the rover to collect sediment.

After exploring Mars, I don’t want to remove the headset, which has provided a glimpse of a combination of computing tools that make the unimaginable feel real. NASA felt the same way. Norris will roll out Project HoloLens this summer so that agency scientists can use it to collaborate on a mission.

A Long Way Yet

Kipman’s voice eventually brings me back to Redmond. As I remove the goggles, he reminds me that it’s still early days for the project. This isn’t the kind of thing that will be, say, a holiday best seller. It’s a new interface, controlled by voice and gesture, and the controls have to work flawlessly before it will be commercially viable. I get that. I love voice controls, and I talk to Siri all the time. But half the time, she doesn’t give me a good answer and I have to pull up my keyboard to find what I’m looking for more quickly. Project HoloLens won’t have a keyboard. If the voice and gesture controls don’t work perfectly the first time, consumers will write it off. Quickly.

That said, there are no misfires during three other demos. I play a game in which a character jumps around a real room, collecting coins sprinkled atop a sofa and bouncing off springs placed on the floor. I sculpt a virtual toy (a fluorescent green snowman) that I can then produce with a 3-D printer. And I collaborate with a motorcycle designer Skyping in from Spain to paint a three-dimensional fender atop a physical prototype.

As I make my way through each, Kipman seems less nervous than when we began, but no less focused. It has been three hours since we met. In each scenario, he watches a screen that shows him what I am seeing, and he watches me trying to use his device for the first time. His eyebrows draw down in deep concentration as he checks to see if every calculation is perfect—noting the touch of my thumb and forefinger as I make an upward gesture, the words I reach for instinctively to instruct the computer. Seven years in, he is trying to see Project HoloLens as if for the first time. To see it through the eyes of a 30-something female New Yorker. But that is one thing his magical head-mounted holographic computer cannot do. At least not yet.

Project HoloLens: Our Exclusive Hands-On With Microsoft’s Holographic Goggles

5:09 PM HoloLens, Microsoft, Microsoft HoloLens, Project HoloLens.