Innovation or hype? We examine Nokia's 41 megapixel smartphone camera

[nsane.forums]

The Nokia 808 PureView capture 41MP digital images—a resolution higher than the top-end Nikon D800 DSLR.

Nokia ignited a bit of a controversy on Monday when it unveiled a smartphone with a 41 megapixel camera sensor dubbed the 808 PureView. Yes, you read that right—41 megapixels, not 14, or 4.1. It will soon be possible to buy a smartphone with as many megapixels as some low-end, medium-format digital SLRs.

Has Nokia completely succumbed to the megapixel myth? Well, no. Instead of positioning the PureView as the smartphone equivalent of a professional digital camera, Nokia is turning the conventional thinking that originally drove the increasing pixel counts of digital cameras on its head, and developing novel ways to exploit an overabundance of pixel data.

While we think there is room to criticize Nokia's specific implementation, there are plenty of good ideas wrapped in the "PureView technology" rubric that we think other smartphone makers—or for that matter, digital camera manufacturers—should consider.

Ridiculous megapixel counts

Forty-one megapixels is, on its face, a seemingly ridiculous number. Many current smartphones struggle to produce decent images even at a more common 8MP. Current compact cameras max out at about 16MP, and even top-end digital SLRs rarely reach pixel counts beyond 20 million or so.

The highest megapixel count on a conventional DSLR to date is Nikon's recently announced D800, which tops out at 36MP. And even that camera, which contains a full-frame 35mm sensor (24 x 36mm), has been criticized for having an overabundance of pixels crammed into the sensor.

The reason for the criticism is fairly simple. With all other factors being the same, capturing a higher number of pixels with a given sensor size means the individual photodiodes at each pixel location are smaller. Generally speaking, the smaller the photodiode, the less sensitive it is to light. Smaller sizes also mean that a smaller number of photons can saturate the diode, reducing dynamic range. And smaller sensors also mean decreased sharpness due to diffraction.

So, if the relatively large sensor on the D800 can suffer these effects with full-size professional lenses at 36MP, how can we honestly expect great performance from a much smaller, smartphone-sized sensor, its corresponding tiny lens, and a seemingly astronomical 41MP count?

If the 808 PureView were intended to be a 41MP camera, you would be justified in criticizing Nokia engineers' sanity. However, Nokia made some design decisions, such as utilizing a relatively large 1/1.2" sensor, to improve performance. Also, it's exploiting the high megapixel count to do interesting things such as oversampling to reduce noise, and creating one of the first usable "digital zoom" features to grace a compact camera.

Nokia says that its 808 PureView is merely the first smartphone to launch with its PureView technology, so it stands to reason that the company will exploit the technology in future models. We also think the ideas behind the technology are smart ones—more on that shortly—so don't be utterly surprised to see the megapixel counts of smartphones suddenly skyrocket if other vendors decide to implement similar features.

The bad news first

The 808 PureView runs Nokia's ancient Symbian operating system. Not only does this mean you'll be stuck in the relative smartphone dark ages if you opt for its impressive photography specs, it also means this particular model will probably never officially make it to US shores.

Though the iPhone 4 and 4S have gained significant popularity among casual shooters for its image quality, the iPhone is still a smartphone first, with camera capability added on—and the same can be said of most Android and Windows Phone 7 handsets. Many serious photographers would still prefer to have something more like a camera first, with smartphone features grafted on.

The Nokia N8 was a step in this direction with its 12MP sensor, Carl Zeiss lens, and proper xenon flash. The 808 PureView is another step in this direction, with the smartphone's size largely dictated by its internal hardware. To maintain comparable noise response to today's smartphones, Nokia had to use a relatively gargantuan sensor. Most smartphones today use a 1/3.2" sensor, roughly measuring about 4.5 x 3.4mm, and with 8MP that makes each pixel about 1.4 microns apart. To keep the same pixel pitch with 41MP, the 808 PureView has a 1/1.2" sensor that's roughly 10.8 x 7.5mm in size. That's over 5 times larger than typical smartphone sensors, and larger than the sensors used in most compact cameras sold today.

A larger sensor also requires a larger lens relative to smaller sensors. And, the 808's xenon flash tube and its power capacitors likewise require far more physical space than the meager white LEDs that pass for flashes on most smartphones. With such beefy hardware, it's no surprise that the 808 PureView is a bulky 18mm thick where the camera is positioned. That makes other smartphones seem wafer-thin in comparison, with the Samsung Galaxy SII measuring 9.9mm at its thickest point, and the iPhone 4S just 9.3mm.

So, if you want one of the most advanced smartphone cameras on the market, you'll have to settle for a brick in your pocket running yesteryear's operating system.

But boy, what an upside

On the other hand, Nokia has made, judging from specs and early samples alone, one of the best smartphone cameras on the market. For one thing, the 808 PureView is actually, by default, a 5MP camera. If the camera is making images with just 5MP, though, why bother with the huge 41MP sensor?

The answer is oversampling. At its default 5MP setting, every pixel in the finished image corresponds to about eight pixels on the sensor. This oversampling helps reduce noise, increase color accuracy, and increase sharpness.

Digital photo sensors are typically only sensitive to brightness, not color, which would result in a monochromatic, black and white image. To record color, sensors are covered with a patterned array of red, green, and blue filters known as a Bayer filter. By filtering the light that hits each pixel, some pixels record the intensity of red light, some blue, and some green. To determine a red, green, and blue value for each pixel location, the missing values for each pixel are interpolated from surrounding pixels.

The Bayer pattern on a small video CCD at 600x magnification.

Since each pixel in a finished 5MP image on the 808 PureView gets data from eight different pixel locations on the sensor, a value for red, green, and blue can be determined without extra interpolation. Furthermore, noise caused by some pixels returning random or inaccurate values is effectively averaged out by combining data from more than one pixel location. This results in more accurate color and reduced noise overall.

Additionally, oversampling reduces the softening effects of the Bayer filter, the anti-aliasing filter, and diffraction of the tiny lens, resulting in sharper images.

This type of oversampling is actually something you have probably seen quite often without even realizing it. If you have ever viewed an image in Photoshop at less than 100 percent view, you have seen the same effect. It's also the same kind of oversampling that is done when you reduce an image to a smaller size. Anyone with enough experience knows that if an image isn't super sharp or if it looks a little noisy at full resolution, that image will look much better printed at a small size or displayed on a screen at, say, 600-800 pixels wide. (It's also the same reason that images that may be noisy or soft can look sharp on the screen of your camera or smartphone.)

A sample 41MP image captured in bright daylight at ISO 55. Detail is quite sharp.

Cropped at 100 percent, detail is still quite sharp. However, even at ISO 55 there is visible noise. Oversampling would make the noise virtually disappear, though, as seen above.

In fact, the 808 PureView isn't even the first camera to implement such oversampling—it's just more commonly called "pixel-binning." Some compact cameras use this technique to offer special "high ISO" modes for low light, as long as you're willing to settle for reduced pixel resolution. The difference is that by default the 808 PureView uses pixel binning all the time.

Who's zooming who?

The other benefit of building a 5MP camera with a 41MP sensor is zoom. The lens on the 808 PureView is an 8mm f/2.4 lens, a fairly wide angle lens with a field of view equivalent to a 28mm on a 35mm camera. To get closer images, you would need a zoom lens, but the 808 PureView, like most smartphones, doesn't have one.

Many smartphones—as well as digital cameras that lack a zoom lens—offer a "digital zoom" feature. Anyone that has used digital zoom knows it often results in blurry, pixelated images. The reason is that "digital zoom" is really a fancy way of saying "crop and resize." Digital zoom essentially crops the middle part of the image coming into the lens, and resizes it up to the camera's full resolution. Again, interpolation is used to fill in the missing pixels, but the higher the zoom factor, the more interpolation must be done.

The 808 PureView doesn't have to do any interpolation, resulting in what Nokia calls "lossless zoom." At the default 5MP setting, zoom can be cranked up to 3x. At that point, the camera is capturing exactly 3072 x 1728 pixels from the middle of the sensor, essentially with no oversampling. In between, the camera simply applies a variable amount of oversampling to achieve a finished 5MP image, from 8:1 down to 1:1.

Another nice benefit here is that at maximum zoom, the image is being captured from the center of the lens—the sharpest part. And since there's no interpolation, a 3x zoomed image is comparably as sharp as a non-zoomed image. This can all be done without the extra bulk of an optical zoom lens and at a full f/2.4 aperture.

The 808 PureView additionally offers selectable resolution in 3MP, 5MP, 8MP, and full 41MP size, in either 4:3 or 16:9 aspect ratios. 3MP images have as much as 14:1 oversampling, while 8MP have more like 4:1 oversampling. Shooting at a higher resolution reduces the zoom range down to about 2x, while shooting at the lower resolution increases it to nearly 4x.

The 808 PureView can perform the same pixel oversampling and lossless zoom when shooting video as well. 1080p video can be zoomed 4x, 720p at 6x, and 360p at an impressive 12x.

Even this lossless zoom idea isn't entirely new, either. For instance, Nikon's professional, full frame "FX" format DSLRs can shoot in a cropped "DX format mode. This gives shooters the equivalent of a 1.6x teleconverter effect, but it could just as easily be called "lossless zoom."

With all the available options, why did Nokia settle on 5MP as the default? The answer is that it's a great compromise between storage space and image quality. 5MP images stored in JPEG format are typically around 1MB in size, and have enough resolution to print respectable 8x10" type prints. Likewise, they can be displayed on nearly any monitor or HD display and look sharp. Hundreds of these images can be stored on the 16 gigabytes of available flash storage, and the size makes it relatively fast and affordable to transmit even over 3G wireless networks.

The selectable resolutions offer something for pretty much every one. And if you want a full 41MP image—for whatever reason—you can have that, too.

Room for improvement

Unfortunately, not everything with the 808 PureView is 8x oversampled double rainbows. The lossless zoom, for instance, trades oversampling to achieve increased magnification. So the benefits of the oversampling, particularly the decreased noise and increased color accuracy, decrease as you zoom in. Images won't look any worse than those from a lower-resolution camera would, but they won't always look as good as they do without the zoom applied.

We don't know many specifics about the 808 PureView's sensor itself—for instance, does it use backside illumination like the iPhone 4S?—but the samples offered by Nokia show that it performs quite well in bright light. Even full resolution 41MP images look sharp and highly detailed. Keep in mind that 41MP images, even in JPEG format, are about 10MB each in size.

A 3MP image captured in dim natural lighting at ISO 800.

On the other hand, the 808 PureView doesn't seem to do all that well in low light. The only low-light sample Nokia provides is at the lowest resolution setting of 3MP. Looking closely at the image, there still seems to be a lot of noise even at 800 ISO. We would imagine that a 41MP image in such low light would look pretty terrible.

UPDATE: Nokia says the sensor, which it co-developed with an unnamed partner, is front-side illuminated—that helps explain the high level of noise at high ISO. "We don't know of any BSI sensors available that are this large," Nokia's Damian Dinning told Ars. "BSI technology is still maturing; we will adopt it when the benefits between FSI and BSI are more pronounced," he said.

Even at 14x oversampling, noise is quite apparent in this image captured at ISO 800.

To be fair, the benefit of 14x oversampling is apparent in the 3MP image provided by Nokia, which looks rather nice. Still, we did find a couple hot pixels and a fair amount of noise in the image, which suggests that the sensor really struggles in low light. On the upside, it appears that Nokia wisely shunned aggressive noise reduction, which often results in loss of detail. And worse comes to worse, the 808 PureView does have a bright xenon flash that will result in much better images than you would see from an LED "flash."

Notice the bright "hot" pixel near the eye. While noise is also present, Nokia opted to keep it instead of losing fine detail and sharpness.

We were left wondering, however, if a better design trade-off could be made. What if a smaller, lower pixel count sensor were used, with a pixel pitch of 1.75 microns? The larger pixel size would result in better low light performance, that, combined with something like 4x oversampling, could offer a good compromise between size and performance.

Oversampling future

Despite our quibbles, the 808 PureView represents some impressive imaging technology for a smartphone. As we mentioned, we think it's rather unfortunate that the device is built around the deprecated Symbian OS, which practically assures that it won't sell in huge numbers. Nokia has said that it is considering incorporating PureView technology into its Windows Phone devices, so we hope that means that the tech stands a chance of reaching more than a tiny niche of users.

Digital sensors continue to be improved, but we think Nokia's idea of oversampling and lossless zoom are good ones. Cramming huge amounts of pixels into the tiny sensors typically used in smartphones probably isn't the answer, but it seems like there is a lot of middle ground between 1/3.2" sensors used in the most compact smartphones and the 1/1.2" in the 808 PureView. We expect other smartphone makers to adopt similar techniques as sensors with larger pixel counts continue to be pumped out from the likes of Aptina or OmniVision.

Beyond smartphones, however, we believe all camera makers should consider oversampling as an integral part of future camera designs. The current crop of compact digital cameras are pushing 16MP, though very few of these images are printed. Fewer still are printed at a size larger than about 4x6", which doesn't require resolution much higher than 3MP.

According to Nokia, the fixation on megapixels stems from the early days of digital photography, when even small increases in pixel count resulted in dramatically improved image quality. But for most shooters, increases beyond 6MP didn't really translate into real-world benefits.

"But by then," Nokia explains in its PureView whitepaper, "the market had made a direct correlation between number of pixels and quality of image. The more pixels the better, was the received wisdom. And this thinking has stuck. Though today manufacturers would happily reduce the number of pixels in their cameras, and instead concentrate on their lenses and sensors, they’re not so sure the market would accept this."

Perhaps the 808 PureView will help consumers and camera makers change their minds.

Further reading

• Nokia PureView imaging technology (PDF) (europe.nokia.com)
  
• Nokia 808 PureView Specifications (europe.nokia.com)
  
• Nokia 808 PureView sample images (press.nokia.com)
  
• Nokia 808 PureView ushers in a revolution in smartphone imaging (press.nokia.com)
  

View: Original Article