Basic Terminology
Raw image: An uncompressed version of the image file. Raw files are named so because they are not yet processed while containing all information for image processing. YUV image: YUV is a color encoding system used as part of a color image pipeline. It encodes a color image, allowing reduced bandwidth for chrominance components. A CMOS sensor is an image sensor type that consists of an array of pixels, each containing a photo detector and active amplifier widely used in digital camera technologies such as cell phones, CCTV, web cameras, etc. A Bayer pattern is an array for arranging RGB color filters on a square grid of photo sensors. Its particular arrangement of red, blue, and green color filters live above the millions of light-sensitive photosite on the surface of a sensor chip. Image sensor: A device that captures light when it strikes the lens of a camera and converts it into an electronic signal and then transmits it to an imaging device processor, which transforms the electronic signal into a digital image. Image signal processing (ISP) pipeline: a method to convert an image into digital form by performing operations like demosaicing, noise reduction, auto exposure, autofocus, auto white balance and image sharpening designed for digital processing and image quality enhancement. Auto-exposure (AE) and auto-white balance (AWB): detect the constantly changing ambient light qualities of the scene you’re photographing and adjusts the camera to maintain just the right brightness and color.How does a digital camera sensor work?
The most basic way to understand how a sensor works is when the camera’s shutter opens and captures the photons that hit it and converts it to an electrical signal that the processor in the camera reads and interprets as colors. Once the exposure finishes, the photons that reach each photosite are perceived as an electrical signal that varies in strength depending on how many photons were captured in the cavity. This information is then stitched together to form an image.
Figure 1 - A digital camera without sensor would not be able to capture light and produce the images that we see.
How do we get colored images by what’s known as a “Bayer transformation”?
A Bayer filter is an integral part of a digital camera image sensor. Bayer sensors use a simple strategy: capture alternating red, green and blue colors at each photosite and do so in a way that twice as many green photosites are recorded as either of the other two colors, reason being a human eye is more sensitive to the color green. Values from these photosites are then intelligently combined to produce full-color pixels using a process called "demosaicing" and then further processed through an ISP pipeline method to achieve high quality.
Figure 2 - Top-view of Bayer array (on left) and side-view of color photosites (on right)
Figure 3 -Bayer filter pattern
What is an Image Signal processor (ISP) and what does ISP tuning mean?
Significant processing is necessary to transform a RAW image from sensor and convert it into a high-quality image. It is here that an ISP comes to use. An Image signal processor is a dedicated processor that takes the raw data from a camera sensor and converts it into a workable image. The ISP performs many of the following steps to deliver a high-quality image for a particular camera sensor and use-case:
Noise Reduction
Digital images are prone to various types of noises during the image acquisition process that results in an abrupt
change in pixel values that do not reflect the true intensities of the real scene. Denoising techniques are applied to
image data that erase noise created depending on behavior & type of data and provides clear images
Auto white-balance & Color correction
Processing operations performed to ensure proper color fidelity in a captured digital camera image which applies color correction matrix (CCM) that transforms to adjust the colors to fit a particular output color space
Colour interpolation
Receiving Bayer inputs from the image sensor converts raw image, typically captured using a Bayer color filter array (CFA) into a color RGB image. This process is also known as demosaicing.
Lens shading correction
Is applied to improve brightness and color non-uniformity towards the image periphery
Defect pixel correction
Corrects defective pixels on the image sensor
Gamma correction
Compensates for the nonlinearity of relative intensity as the frame buffer value changes in output displays
Local tone mapping
Combines different exposures together in order to increase the local contrast within disparate regions of an HDR scene
Auto Exposure
Performs automatic fine tuning of the image brightness according to the amount of light that reaches the camera sensor
Auto Focus
Auto focus automatically adjusts the sharpness of the image, which improves the image definition. All types of actuator, lens position tuning, AF stats engine tuning etc
Bottom Line
So, what purpose does ISP have? Well, pixels are sensitive to light between some set of wavelengths. Getting a color image out is usually done by applying a filter and then interpolating the color of the adjacent pixels. The ISP then performs demosaicing i.e. it guesses the red, green & blue for each pixel based on what’s next to it. In addition, ISP does all other parameter settings like auto focus, exposure, and white balance for the camera system along with noise reduction, lens shade correction, pixel correction, other filtering, and conversion between color spaces. Ultimately it gets the Bayer data into an image than can be stored, streamed or processed upon. Every camera sensor works differently and has different properties. Camera designs and output image requirements also differ based on use-cases and applications. The image signal processor needs to be tuned to these variances to derive the perfect output image for a given setup. PathPartner provide image quality and ISP tuning services for a wide array of sensors, processors and use-cases. Reach out to us to know more or for quick consultation. marcom@pathpartner.comFurther reading:
- Role of Machine Vision technology in Packaging Industry
- 10 Amazing AI-Based Video Analytics Use Cases in Retail
- How to Integrate Industrial Robots and Machine Vision in Manufacturing: Beyond Industry 4.0
- Case study - Development of a Wireless Smart Home Monitoring Camera
- Image quality tuning
- Connected cameras
- Imaging algorithms
- Plug and play camera modules
