Microsoft HoloLens - Augmented Reality/Holographic Headset

[Cypher3927]

Microsoft's HoloLens augmented reality/holographic headset is disruptive technology that, as we saw firsthand, offers what could become a somewhat practical and significantly useful way to work, communicate and play. We keep learning more about this mysterious tech little by little, but there are still huge unanswered questions. Here is what we know, and don't know, about HoloLens and Windows Holographic.

Overview

The whole "Holo" package consists of the HoloLens headset and Windows Holographic. The latter is the "operating system." We put that last bit in quotes, because it's not technically a whole new OS; Windows Holographic is a version of Windows 10, according to Microsoft's Alex Kipman.

Even so, it uses that same shared code that underpins Windows 10 on all platforms, and because of that, Universal Apps run on it as easily as they do on any other screen or device (technically speaking). Put another way, there are no Windows 10 APIs that are unique to Windows Holographic.

The system creates 3D avatars so you have something besides a disembodied voice to communicate with. As we mentioned in our hands-on article, this does not need be a facsimile of that person; it can be anything. In the demo, it was a 3D, blue, faceless man-shaped avatar.

In terms of a user experience, the HoloLens environment can be shared among multiple parties. The idea is that anyone with a HoloLens can stand around together looking at and interacting with the same things. As in the Skype demo we saw, that could happen even when one party is not wearing a HoloLens, which will no doubt be a big selling point.

We were also delighted to find that HoloLens accommodates glasses, so unlike some VR demos we've experienced, Microsoft's AR offering is relatively kind to the bespectacled.

Development And User Experience

Kipman said that developing for HoloLens is effectively a 3D mobile development experience (with several middleware tools to help).

In terms of Cortana integration, Windows Holographic leverages her speech engine and APIs. Office and related apps will be "flat" experiences for the time being, meaning that although you should be able to move their windows around, there will be no 3D components; however, that can change. In a talk, Kipman said that "can" change, not that it "will" change. Thus, we infer that there isn't anything specific in Microsoft's roadmap to that effect, but he seemed to hope that some clever developers might come up with some ideas.

We don't know what CPU and GPU is on the HoloLens, but judging from the size of the headset as well as various comments from Microsoft (such as Kipman's 3D mobile development experience comment), it's effectively a mobile platform. We presume there's an SoC in there, although we don't know which SoC. The company keeps mentioning "custom silicon," and we're unclear if that refers to the CPU/GPU, the HPU, or some combination.

We know from our demo that the device is, as Microsoft promised it would be, fully untethered. There are no wires of any kind; the headset provides the experience as a 100 percent self-contained unit.

That brings up the question of battery life, though; the HoloLens hardware is certainly graphically and computationally demanding, no matter how juiced up its SoC may be. One saving grace, though, could involve the display. Thinking of a smartphone as a closed system, those bright, hi-res displays are hard on battery life. However, the HoloLens has a projection system, not an actual display. It pushes the visuals through its lenses and visor for your eyes to behold. It's possible that this method demands a great deal less from the battery than a smartphone would.

Indeed, in a blog post, Microsoft said that the HoloLens employs an "advanced optical projection system." It further read, "The see-through lenses generate multi-dimensional full-color images with very low-latency so you can see holographic objects in the physical world."

Microsoft also noted that this projection system stays synced with the multiple "advanced" sensors on the headset.

The visor assembly actually consists of two parts. There's the transparent rounded shield portion that you can clearly see in pictures of the headset, but there are a pair of lenses that are closer to your eyes, too. Microsoft called these "high definition" lenses, but that tells us little. Are they 720p or 1080p? Is that for each eye, or the combined image?

Processing And Sensors

The HoloLens has a CPU and GPU (of some kind), but it has a third processing unit called the Holographic Processing Unit (HPU). Kipman explained the HPU a bit by saying that the third chip handles all of the human understanding, and it frees up the CPU and GPU so that developers can play with them. Physically, it's located within the bar that goes across the top of the headset.

Kipman said that the component of HoloLens that understands human speech, identity, and other such features -- which we take to be the HPU -- is a sort of super-Kinect sensor. It handles all of the HoloLens' spatial and surface mapping.

This is what enables the device to, for example, recognize hand gestures such as air taps. We don't have any specs on this feature, but its range should be impressively wide. For example, in my demo, I found that I could look to highlight something (which is to say, I could move my head to place a cursor on it) and air tap with my hand way down at my side.

He used the term "Kinect" to describe it, but he was clear that this is not the Kinect that's currently available. Whether we will eventually see this new and improved Kinect sensor on the Xbox One (or a future-gen Xbox) remains unclear.

Kipman noted that although devs no doubt would like to tinker with that particular sensor, Microsoft is keeping it locked down and inaccessible to outside devs. Microsoft doesn't want anyone to "break" that experience for any users; however, devs will have access to the "experiences" that sensor offers.

This HPU is custom silicon, designed to be "easily used by developers so they can focus on creating amazing experiences without having to work through complex physics calculations," or so sayeth Microsoft.

It appears that the HPU works in tandem with something called the intertial measurement unit, or "IMU," which has an accelerometer, gyroscope, and magnetometer, and the head-tracking cameras mounted on the headset, to handle head tracking and movement.

Whatever soup of processors and sensors Microsoft is using here, it's working. The HoloLens image remained glitch-free and stable. The only way I could create any issues with the visuals and head tracking was to shake my head rapidly.

Camera And Audio

Kipman was particularly excited about the Mixed Reality Capture feature. Essentially, there's a still/video camera on the HoloLens that can capture holographic images (or at least a reasonable facsimile). What's more, the HoloLens can "intercept" that feed and distribute it in such a way that others can see it.

Mixed Reality capture works more or less the same way as the camera setup Microsoft used to demonstrate HoloLens demos onstage and behind closed doors at Build. They basically mounted stereoscopic "eyes" over the camera lens to "see" in 3D. The biggest difference between these cameras and the HoloLens is quality; the onstage camera was a 4K/2K device, while the HoloLens camera is lower-res. (No, we do not know the resolution. See "Lingering Questions" below.)

He also noted that the human understanding APIs the sensor uses are the same as those in all of Windows 10.

There are two tiny speakers mounted on the HoloLens, one each on the left and right sides of the earpieces. Microsoft said that these provide spatial sound by trying to mimic how the human ear works. "Using a scientific model that characterizes how the human ear receives sound from a specific physical location, HoloLens can synthesize a binaural sound so that you perceive it to come from a specific location in the physical space around you," read a Microsoft post.

There's also a built-in mic. In our demo, we left a voice message for the avatar, who played it back. In my case, the mic picked up both my voice and my tour guide's. Hers was clear enough, but a little faint -- she was standing a couple of feet away and to my left -- but my words were clear and loud.

Lingering Questions

Unfortunately, there's a huge pile of questions that remain unanswered, and most of those center around actual hardware specifications. We did our best to pump the HoloLens folks for more, but they were tight-lipped.

We don't know when HoloLens will be available nor what it might cost. All we were told was that it will be available "within the Windows 10 time frame," which is another way of saying, "Sometime before Windows 11 comes out." The optimist would hope for a July launch, as that's when supposedly Windows 10 will be available. (No, Microsoft did not confirm that either.)

We want to know if there will be a consumer version of the device and an industrial one, and we're curious how Windows Holographic differs, specifically, from Windows 10.

We would also like to know how the specs on the SoC (we're assuming it's an SoC), and how it and the HPU communicate, and exactly which sensors are on the device. Further, what kind of onboard storage does HoloLens have, if any? What about battery life -- what kind of battery is inside this thing, and what is its capacity?

There is obviously wireless Internet connectivity on board -- Skype, remember -- but we don't know what kind of wireless chip it is, nor any specs on it. And what about Bluetooth, NFC and the like?

And so on. We'll put it to you as well, dear readers: What remaining questions do you have about HoloLens?

Source: Tom's Hardware

[nIGHT]

I want this but the device is still big to be worn on the head. :mellow:

They should concentrate to make this as a replacement for monitors to display OS desktop that may be attached, hopefully wireless, to a more powerful workstation.

[windowsvistas]

its interesting technology but with a microsoft brand on it i doubt its bug free