Camera Tuning : Understanding the Image Signal Processor and ISP Tuning

Date: March 2, 2020

Author: Nazneen A

Every day, 1.8 billion images are posted on the web, pausing life and turning moments into digital information pixels. Have you ever thought about how a camera captures an image? How can those images turn out so crisp and clear in varied lighting conditions? This action looks so easy that a lot of us take the tech behind it for granted. Get ready to learn how the technology behind the camera works and how we can improve the image quality with ISP tuning (Image signal processing) methods designed to enable advanced imaging experiences.

With camera performance becoming increasingly important for artificial intelligence and other use-cases, manufacturers have been working hard to add functionality and enhance image quality.

Taking the light from your camera sensor and transforming that into a beautiful image is a complicated process involving a significant amount of processing. Fortunately, digital cameras handle this in an efficient way through the use of a dedicated ISP as there are several noise factors that degrade the quality of an image.

This blog is meant to acquaint you with the basics regarding ISP tuning - how your camera works, making it appropriate for a wide range of camera systems, including high-quality digital cameras, sports cameras, surveillance, smartphones, drive recorders, mobile devices and video-image-based industrial equipment.

Basic Terminology

Raw image: An uncompressed version of the image file. Raw files are named so because they are not yet processed while containing all information for image processing.

YUV image: YUV is a color encoding system used as part of a color image pipeline. It encodes a color image, allowing reduced bandwidth for chrominance components.

A CMOS sensor is an image sensor type that consists of an array of pixels, each containing a photo detector and active amplifier widely used in digital camera technologies such as cell phones, CCTV, web cameras, etc.

A Bayer pattern is an array for arranging RGB color filters on a square grid of photo sensors. Its particular arrangement of red, blue, and green color filters live above the millions of light-sensitive photosite on the surface of a sensor chip.

Image sensor: A device that captures light when it strikes the lens of a camera and converts it into an electronic signal and then transmits it to an imaging device processor, which transforms the electronic signal into a digital image.

Image signal processing (ISP) pipeline: a method to convert an image into digital form by performing operations like demosaicing, noise reduction, auto exposure, autofocus, auto white balance and image sharpening designed for digital processing and image quality enhancement.

Auto-exposure (AE) and auto-white balance (AWB): detect the constantly changing ambient light qualities of the scene you’re photographing and adjusts the camera to maintain just the right brightness and color.

How does a digital camera sensor work?

The most basic way to understand how a sensor works is when the camera’s shutter opens and captures the photons that hit it and converts it to an electrical signal that the processor in the camera reads and interprets as colors. Once the exposure finishes, the photons that reach each photosite are perceived as an electrical signal that varies in strength depending on how many photons were captured in the cavity. This information is then stitched together to form an image.

Figure 1 - A digital camera without sensor would not be able to capture light and produce the images that we see.

How do we get colored images by what’s known as a “Bayer transformation”?

A Bayer filter is an integral part of a digital camera image sensor. Bayer sensors use a simple strategy: capture alternating red, green and blue colors at each photosite and do so in a way that twice as many green photosites are recorded as either of the other two colors, reason being a human eye is more sensitive to the color green. Values from these photosites are then intelligently combined to produce full-color pixels using a process called "demosaicing" and then further processed through an ISP pipeline method to achieve high quality.

Figure 2 - Top-view of Bayer array (on left) and side-view of color photosites (on right)

Figure 3 -Bayer filter pattern

What is an Image Signal processor (ISP) and what does ISP tuning mean?

Significant processing is necessary to transform a RAW image from sensor and convert it into a high-quality image. It is here that an ISP comes to use. An Image signal processor is a dedicated processor that takes the raw data from a camera sensor and converts it into a workable image. The ISP performs many of the following steps to deliver a high-quality image for a particular camera sensor and use-case:

Noise Reduction
Digital images are prone to various types of noises during the image acquisition process that results in an abrupt change in pixel values that do not reflect the true intensities of the real scene. Denoising techniques are applied to image data that erase noise created depending on behavior & type of data and provides clear images
Auto white-balance & Color correction
Processing operations performed to ensure proper color fidelity in a captured digital camera image which applies color correction matrix (CCM) that transforms to adjust the colors to fit a particular output color space
Colour interpolation
Receiving Bayer inputs from the image sensor converts raw image, typically captured using a Bayer color filter array (CFA) into a color RGB image. This process is also known as demosaicing.
Lens shading correction
Is applied to improve brightness and color non-uniformity towards the image periphery
Defect pixel correction
Corrects defective pixels on the image sensor
Gamma correction
Compensates for the nonlinearity of relative intensity as the frame buffer value changes in output displays
Local tone mapping
Combines different exposures together in order to increase the local contrast within disparate regions of an HDR scene
Auto Exposure
Performs automatic fine tuning of the image brightness according to the amount of light that reaches the camera sensor
Auto Focus
Auto focus automatically adjusts the sharpness of the image, which improves the image definition. All types of actuator, lens position tuning, AF stats engine tuning etc

Bottom Line

So, what purpose does ISP have? Well, pixels are sensitive to light between some set of wavelengths. Getting a color image out is usually done by applying a filter and then interpolating the color of the adjacent pixels. The ISP then performs demosaicing i.e. it guesses the red, green & blue for each pixel based on what’s next to it. In addition, ISP does all other parameter settings like auto focus, exposure, and white balance for the camera system along with noise reduction, lens shade correction, pixel correction, other filtering, and conversion between color spaces. Ultimately it gets the Bayer data into an image than can be stored, streamed or processed upon.

Every camera sensor works differently and has different properties. Camera designs and output image requirements also differ based on use-cases and applications. The image signal processor needs to be tuned to these variances to derive the perfect output image for a given setup.

PathPartner provide image quality and ISP tuning services for a wide array of sensors, processors and use-cases. Reach out to us to know more or for quick consultation. marcom@pathpartner.com

Further reading:

You might also like these blogs

post
In the Spotlight: Drone Delivery, COVID -19 and Artificial Intelligence

The golden age of drone delivery has begun. Did you…

Read More
post
Evolution of In-cabin Sensing Solution

With the ease of getting all the information in a…

Read More
post
Audio Signal Processing- Understanding Digital & Analog Audio Signal Processing

Modern communication systems have completely changed the outlook of the…

Read More
Back to Top