Image capture and processing challenges--and solutions--in portable designs--Part IV

The number of images and length of video footage captured by consumers has increased exponentially since the advent of digital photography. Almost all cell phones have at least one camera--a high-resolution device pointing outwards for photography and a lower-resolution device aimed at the user for video conferencing. Camera phones are compact, easy to use and usually readily available through being permanently turned on and quite literally "to-hand."

The adoption of wafer-level packaging and new manufacturing techniques has reduced the price of camera modules to the point where they are now considered a standard item on most phones, like a color screen. Innovation in die packaging and optics has reduced the height of camera modules so they can be incorporated in the thinnest of cell phones. Software-enhanced optics improves the picture quality of these compact and low-cost camera modules to the point where they can compete with the digital still cameras of comparable resolution.

The vast majority of consumers are not natural or trained photographers. They do not understand the principles of exposure and through failure to read manuals, frequently don't know how to adjust camera settings. Consequently the majority of digital images are (or should be) discarded. This is an unsatisfactory situation for all parties: the camera designer who has invested heavily to deliver good optical performance, the network operator looking to boost revenue from data transmission, and the user due to mediocre pictures.

A look through most photograph albums will confirm that people are the subject matter of the majority of photographs. Even when capturing a great scenic view it is traditional to position a family member or friend to one side of the frame. Because our subconscious latches onto faces, the one region of a photograph that must be exact is the principal face. Given that the average user is unable or unwilling to change the camera settings for each photograph, software that can locate faces and ensure they are without optical aberration can greatly enhance the perception that a camera takes really great photographs. Having a face correctly exposed, in focus and properly color balanced can make the difference between user satisfaction and the photograph being discarded. The goal is to endow camera phones with features that enhance the picture-taking experience and gratify users with their photographic skills. The tool available to accomplish this is software using the process of numerical image enhancement.

Numerical Image Enhancement
Features made possible by numerical image enhancement--or "smart imaging"--can be broadly grouped into two categories: those that improve image quality and that are often transparent to the user; and those that gratify the user by providing an experience.

The camera lens designer is often faced with the challenge of managing chromatic distortion. The trade-off is between performance and cost, as more expensive lens materials generally possessing better optical properties. An example is "purple fringing," a distortion that occurs in areas of high contrast shot with inexpensive lenses. Purple fringes can show up along a roofline set against a bright sky, or around bare tree branches. An example is shown in Figure 9. Purple fringe generally only becomes visible when a picture is enlarged on a video monitor, so will not be apparent when the customer tests the camera phone in the shop prior to purchase. Fortunately, chromatic aberration is constant for a given optical design. Therefore, rather than tolerate a higher-cost lens solution, the camera module manufacturer can use an inexpensive lens train and apply numerical image enhancement to correct the appearance of the images with software.

Another example of numerical image enhancement to improve image quality is red-eye reduction. Red-eye is a common complaint of low-light photography. For people of Asian descent the effect often manifests itself as golden-eye. The effect has several causes, including, most commonly, light from a flash source being reflected by the retina back into the camera. The problem is most severe when the flash source is located close to the optical axis of the camera, as it is on a camera phone. While the trend today is toward better low-light performance without flash, customers expect high-end camera phones to have flash and continue to have the need for red-eye and golden-eye removal. There is also the unwritten presumption that having paid additional money for the flash option, the customer is more dissatisfied by a good quality picture spoiled by red-eye than a mediocre picture without it. While various image enhancements like red-eye removal can be done on pictures after downloading them to a computer, this fix requires time, a software package and some computing knowledge to achieve a good end result. Because most consumers simply want pictures without red- or golden-eye, the preferred approach is to embed the necessary algorithms on the cell phone and correct the affected photographs without any user intervention or even knowledge that it has been done (see Figure 10).

Face-based Imaging
Because faces are so important to the perception of photographic quality and whether or not a consumer retains or discards a photograph, many numerical image enhancement solutions are based around face-based imaging.

The first challenge of face-based imaging is to identify the faces in the scene. Mathematically this is not a trivial exercise. Face identification is challenging because of the diversity of face types, and is further complicated when photograph subjects wear glasses, hats, earrings and other accessories. The solution commonly adopted is one of statistics. One commercial software program has approximately 200 rules of what constitutes a face, looking for artifacts like hairlines and eyes-nose and ears-mouth triangulations. The software decides it has found a face when approximately 10% of the rules flag as valid. Because face tracking is done in real time, once the software locks on to a face it can continue to track its location in the image until the shutter is pressed.