RGB-IR Sensor for Automotive Cabin Application

Date: March 17, 2021

Author: Akshay Srinivasa

Vehicles are mandated with cabin sensing for either driver monitoring or left out child monitoring. This has made use of camera inside the vehicle cabin mainstream. Having a camera inside the cabin gives an opportunity to include more features when the car is on the go when child left behind feature is not in use. RGB-IR type of sensors are the ones predominantly used for this. However, separating the RGB feed and IR feed with existing ISP is a new problem put forth, read on to understand how RGB-IR filtering is done and how it she road ahead for automotive cabin application.


Driver monitoring system is mandated by various governing bodies, making OEMs to incorporate a system that tracks various facial features to determine the sleep deprivation and ability to drive. Camera sensor is widely used for this application. In addition to driver monitoring, another safety feature- cabin monitoring/ left out child monitoring. One of the preferred camera sensors for this application is RGB-IR, which combines RGB (for human vision) as well as IR (for machine vision), where in IR is used for face landmark tracking and in turn for driver state monitoring. RGB channels opens up the opportunity for the OEMs to integrate new features such as video calling, video conferencing and for large transport companies to keep an eye on its fleet drivers.

Figure 1: Cabin sensing and Driver state monitoring

RGB-IR sensor combines the capabilities of RGB sensor and IR sensor in one single sensor. However, the additional IR pixel in the RGBIR sensor reduces the effective number of pixels allocated to visible region introducing aliasing artifacts. Also, the presence of IR content in R, G and B channels poses new challenges in accurate colour reproduction. Sharpness and colour reproduction are very important image quality factors for visual aesthetic as well as computer vision algorithms. The image processing pipeline has not been fully explored for RGB-IR sensors.

Colour image sensors based on RGB colour filter arrays (CFAs) are now a robust, cost−effective and widely available solution for visible−light imaging. Bayer sensors with IR cut filters restricts only a certain portion of visible spectrum to pass through it and capture either of the three primary channels (R, G or B). Single sensor RGB-IR cameras provides the power of both Bayer and NIR sensors without the need of two separate cameras. In case of RGB-IR sensor depending on the pattern (2x2 or 4x4), for every 2x2 block either B, R or G is replaced by IR(Infrared) pixel.

Traditional Image processing Pipeline

Figure 2: Cabin sensing and Driver state monitoring

Video processing Pipeline for RGB-IR

Figure 3: Video Processing Pipeline for RGB-IR

Figure 4: RGB-IR Output

Challenges in RGB-IR

BGB-IR puts forth numerous challenges in getting the RGB channel and IR channel output from a single sensor, some of which are listed below

  • Most of existing ISPs do not support processing 4x4 RGB-IR directly. Requires pre-processing algorithm to extract RGB and IR to form 2 streams (RGB & IR).
  • Recovering accurate colors for RGB due to presence of NIR (crosstalk between IR and RGB because of dual band filter allowing visible and NIR)
  • Recovering resolution or sharpness (lost due to reduced R and B pixels in place of NIR)
  • A/B mode (proprietary to OV sensor) and main challenge it poses at framework level to switch ISP context (set of ISP parameters to be loaded for RGB path and NIR path) for every frame.
  • Achieving KPI thresholds and tuning for machine vision applications (ISP-IR) and for human vision applications (ISP-RGB).

PathPartner RGB-IR Solution/IP

PathPartner with extensive experience in IQ tuning and automotive imaging for ADAS recently developed RGB-IR solution for commonly used ISP platforms from TI, Qualcomm and more. PathPartner solution was developed for extended DMS application for video conferencing in car.

You might also like these blogs

Understanding Traffic Sign Recognition system used in ADAS (Advance driver assistance system)

Autonomous machines/ vehicles are all set to drive along the…

Read More
Bayesian Grouping of Multi Sensor Radar Fusion for effective Pedestrian Classification in Automotive Surround View

From long time camera is used as the sensor for…

Read More
DBSCAN for High Resolution Automotive Radar Application

Cars with SAE Level 2 and beyond are already on…

Read More
Back to Top