USB Webcam market size is estimated to be around USD 6.8Bn this year 2020. This market is projected to grow at 8.2% CAGR for the next 6-7 years. The plug-n-play nature of USB Webcams makes them highly popular in the consumer market. However, in the embedded world, other camera interfaces like MIPI-CSI2 are highly used. MIPI-CSI is a high-speed data transmission protocol capable of transmitting raw pixel images directly into the memory.
Most of the embedded platforms use MIPI as the last mile bridge to get camera data to the processor. Embedded engineers would agree that “embedded platform evaluations,” however, would either need a USB interface to the PC or a wireless interface to a remote display/ tablet. This article explains how an Android embedded platform with built-in camera support is turned into a USB webcam. More specifically, we demonstrate this transformation on our in-house developed NXP IMX8x based Android-8 platform. USB has a well-defined USB Video Class specification (UVC) exclusively for standardizing the video streaming over the USB. Using the UVC support in kernel along with a user-space application, we can turn our Android device to a USB camera, i.e. when the device is plugged to host via USB, the host will detect it as a normal webcam, and a capture device node will be created in host side (/dev/videox) through which host can run any v4l2 compatible capture application to get the video.
UVC Software stack
The uvc software stack involves different layers such as the usb stack, libcomposite layer (drivers/usb/gadget/composite.c), webcam gadget driver (usb/gadget/legacy/webcam.c), uvc gadget function driver (usb/gadget/function/f_uvc.c) and there is one user space application uvc-gadget application involved to complete the usb enumeration process. Each layer of the uvc software stack and the necessary changes to be done at each layer will be explained in detail in the following sections.
USB Stack
In USB, all communication is performed through endpoints. At the device side, the endpoints are (hardware) FIFO queues associated with the UDC. UDC or USB Device controller is the lower layer that talks to the hardware. UDC accepts streams of IN/OUT buffers through the endpoint and interacts with the gadget driver through callbacks. There are 4 types of USB endpoints: control, bulk, iso, and interrupt. Control endpoint is endpoint 0 and all USB devices shall have at least endpoint 0. All endpoints except the endpoint 0 are unidirectional. Bulk endpoints are meant for transferring large amounts of data on a "best-effort" but there are no timing guarantees. For the multimedia application, the Iso(chronous) endpoints will be used mainly, since they are meant for time-sensitive data with guaranteed bandwidth and bounded latency. Interrupt endpoints are for non-periodic data transfers "initiated" by the device. The controller driver can support any number of different gadget drivers, but only one of them can be used at a time. So when we enable the uvc functionality we lose the adb access.
Gadget Function drivers
The Linux kernel has added an implementation of several USB gadget drivers (drivers/usb/gadget/function/*) for different functionalities such as mass storage, serial, ethernet, and similarly for webcam as well, which is the uvc gadget driver. There is one layer in between the UDC and the gadget driver which is the USB composite layer. It contains the codes common for all such gadget functionalities and it talks with the UDC (USB Device controller) driver. Since the USB device can provide more than one functionality over a single USB, it needs to be configured for each functionality and can be one active at a time in most cases.
USB descriptors and configfs
USB standard uses some descriptors for each device to keep the necessary data structures. The device descriptors store general device information, such as product ID and device ID; configuration descriptors store device configuration modes, such as whether the device is bus-powered or self-powered; interface descriptors make USB can support more functions; endpoint descriptors store the final endpoint information of the device
There are two ways we can keep these data structures: one is the configfs method and the other is g_webcam kernel module incase of uvc.
Configfs is a ram-based filesystem (/sys/kernel/config/) that provides user access to kernel objects and instantiates the USB functions by creating the respective directories and associates the functions to configurations with a symbolic link. In the Configfs method, we need to be aware of what all files/directories need to be created and their values and nearly need some 15 shell commands to fully configure the functionalities and create the symlinks. In this way, the method of the g_webcam module is pretty straightforward and the configurations are defined within the driver.
Kernel g_webcam module
The module g_webcam provides the userspace API to process UVC control requests and stream video data to the host. And can build the module with below changes in kernel config file- kernel_imx/arch/arm64/configs/product_defconfig
CONFIG_USB_G_WEBCAM=m
Enable the UVC gadget function driver support as a part of the kernel, as below:-
CONFIG_USB_CONFIGFS_F_UVC=y
We can also choose to enable the g_webcam as a part of the kernel instead of a module.
CONFIG_USB_G_WEBCAM=y
When the module is loaded, it populates all the gadget configurations and the gadget function pointers and binds with the uvc gadget function driver(f_uvc.c). Also a new v4l2 video device node is created /dev/videox and registered, through which the gadget application subscribes to the events and feeds the video data to uvc domain.
Before loading the module we should unbind any current device configuration in UDC and then load the module.
$ echo "" >/config/usb_gadget/g1/UDC
Install module g_webcam.ko.
$ modprobe g_webcam
We can also configure the max packet size while loading the module.
$ modprobe g_webcam streaming_maxpacket=1024
Once the gadget then becomes bind to this particular UDC and will lose other functionalities such as adb . Available UDC names are in /sys/class/udc. Only after a gadget is bind to a UDC can it be successfully enumerated. One thing is to be noted that, the USB is not configured in the host mode while enabling this device functionality.
Once the module is loaded and the device is connected to a USB host, we can see the dmesg on the host side as below.
usb 1-1.4: new high-speed USB device number 76 using xhci_hcd usb 1-1.4: New USB device found, idVendor=1d6b, idProduct=0102 usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0 usb 1-1.4: Product: Webcam gadget usb 1-1.4: Manufacturer: Linux Foundation uvcvideo: Found UVC 1.00 device Webcam gadget (1d6b:0102)
And the "lsusb -v " shows the list of the descriptors and values
As seen in the dmesg, the uvc device supports the UVC 1.0 version. UVC 1.0 is the initial version of the UVC driver (driver/media/usb/uvc/uvc_driver.c) which supports the uncompressed format YUYV and compressed format - MJPEG formats. Whereas the latest UVC 1.5 version supports uncompressed format - YUYV, NV12, and Video stream formats - MPEG-2 TS, H.264, MPEG-4 SL, SMPTE VC1, VP8 and MJPEG.
The webcam gadget module (webcam.c) and gadget function driver(gadget/function/uvc*) supports the YUYV and MJPEG formats at 360p and 720p resolution as we see below.
drivers/usb/gadget/function/uvc_v4l2.c
static struct uvc_format uvc_formats[] = { { 16, V4L2_PIX_FMT_YUYV }, { 0, V4L2_PIX_FMT_MJPEG }, };
we can find the corresponding guid in usb/uvc/uvc_driver.c ,
static struct uvc_format_desc uvc_fmts[] = { { .name = "YUV 4:2:2 (YUYV)", .guid = UVC_GUID_FORMAT_YUY2, .fcc = V4L2_PIX_FMT_YUYV, }, .... }
Corresponding guidformat is defined in usb/uvc/uvcvideo.h.
#define UVC_GUID_FORMAT_YUY2 \ { 'Y', 'U', 'Y', '2', 0x00, 0x00, 0x10, 0x00, \ 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}
If the supported format needs to be modified, need to modify the uvc_formats in uvc_v4l2.c and set the corresponding guid, guid-formats and other corresponding values such as bBitsPerPixel , ratecontrol n the webcam.c.
static const struct uvc_format_uncompressed uvc_format_yuv = { .bLength = UVC_DT_FORMAT_UNCOMPRESSED_SIZE, .bDescriptorType = USB_DT_CS_INTERFACE, .bDescriptorSubType = UVC_VS_FORMAT_UNCOMPRESSED, .bFormatIndex = 1, .bNumFrameDescriptors = 2, .guidFormat = { 'Y', 'U', 'Y', '2', 0x00, 0x00, 0x10, 0x00, 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}, .bBitsPerPixel = 16, .bDefaultFrameIndex = 1, .bAspectRatioX = 0, .bAspectRatioY = 0, .bmInterfaceFlags = 0, .bCopyProtect = 0, }
And the g_webcam module defines the control and streaming descriptors for different usb speed - full-speed(fs), high speed(hs), and super speed(ss).
Below is the descriptor for streaming control for full speed.
static const struct uvc_descriptor_header * const uvc_fs_streaming_cls[] = { (const struct uvc_descriptor_header *) &uvc_input_header, (const struct uvc_descriptor_header *) &uvc_format_yuv, (const struct uvc_descriptor_header *) &uvc_frame_yuv_360p, (const struct uvc_descriptor_header *) &uvc_frame_yuv_720p, (const struct uvc_descriptor_header *) &uvc_format_mjpg, (const struct uvc_descriptor_header *) &uvc_frame_mjpg_360p, (const struct uvc_descriptor_header *) &uvc_frame_mjpg_720p, (const struct uvc_descriptor_header *) &uvc_color_matching, NULL, };
For each uvc_descriptor, default values for the resolution, bitrate control, framerate controls are defined as below.
static const struct UVC_FRAME_UNCOMPRESSED(3) uvc_frame_yuv_360p = { .bLength = UVC_DT_FRAME_UNCOMPRESSED_SIZE(3), .bDescriptorType = USB_DT_CS_INTERFACE, .bDescriptorSubType = UVC_VS_FRAME_UNCOMPRESSED, .bFrameIndex = 1, .bmCapabilities = 0, .wWidth = cpu_to_le16(640), .wHeight = cpu_to_le16(360), .dwMinBitRate = cpu_to_le32(18432000), .dwMaxBitRate = cpu_to_le32(55296000), .dwMaxVideoFrameBufferSize = cpu_to_le32(460800), .dwDefaultFrameInterval = cpu_to_le32(666666), .bFrameIntervalType = 3, .dwFrameInterval[0] = cpu_to_le32(666666), .dwFrameInterval[1] = cpu_to_le32(1000000), .dwFrameInterval[2] = cpu_to_le32(5000000), };
If the supported format needs to be added/modified, we need to update these areas.
UVC Gadget Application
The UVC gadget driver does not implement all the UVC specifications and therefore we need a userspace application (uvc-gadget application) to complete the enumeration process and start video streaming. It uses the v4l2events interface to communicate with the driver below through the node created during the gadget driver installation.
Below are the three significant functions in the uvc gadget application.
- uvc_events_init
- uvc_events_process
- uvc_video_process
uvc_events_init fills the probe and commit of uvc_device, and then sets the setup, data, streamon, streamoff , connect, disconnect of UVC events to the driver through VIDIOC_SUBSCRIBE_EVENT.
uvc_events_process function invoked on the occurrence of the subscribed events and handles the events.
The uvc_video_process function will dequeue the buffers from the uvc domain, fill buffer from capture application and queue the buffer back to the uvc domain and pump.
Uvc gadget application supports different types of input feeding. If we have some real video capture hardware, the application provides the methods to complete the path from the V4L2 based video-capture driver domain to UVC based video-output domain. It can be run as below,
uvc-gadget -v /dev/video<capture node #> -u dev/video<uvc video node #>
It also supports the standalone mode where it generates a dummy pattern and can be streamed over the standard applications like CHEESE or vlc.
This can be run as below.
uvc-gadget -d -u /dev/videox.
Once the user application is run, it will create a video node at the host side. We can see the supported formats on running the below command.
v4l2-ctl --list-formats
Now the video can be captured at the host using any v4l2 capture application or with any standard webcam applications or vlc. Another good tool is the guvcview which offers dynamic control on the format, resolution etc.Build the uvc gadget Application
Follow the below steps to build the gadget application.
1. Copy the gadget application to the Android/device/fsl/imx8q/
2. Create the Android.mk file for the application.
LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS) LOCAL_MODULE := uvc_gadget LOCAL_SRC_FILES := uvc-gadget.c LOCAL_MODULE_PATH := $(TARGET_OUT_VENDOR_EXECUTABLES) LOCAL_MODULE_TAGS := optional #LOCAL_MODULE_CLASS := APPS include $(BUILD_EXECUTABLE)
# uvc_camera service type uvc_camera, domain; type uvc_camera_exec, exec_type, vendor_file_type, file_type;
init_daemon_domain(uvc_camera)
3. Add the application in Android/device/fsl/imx8q/product.mk in to make it as part of the main build.
# UVC Gadget Application PRODUCT_PACKAGES += \ uvc_gadget
Now the uvc_gadet application is part of the main build and binary will be available in /vendor/bin partition.
Modifying the uvc gadget application
In case our capture from the hardware doesn't compromise with the supported formats, bandwidth/speed, or we need to do some post-processing before feeding to uvc we can add the custom implementation and feed the final output to uvc . We can either capture and encode using the native APIs or with the v4l2 capture, and configuring the hardware encoder to convert to the desired format and apply the post processing. Gadget applications can be run in the standalone mode and instead of the test pattern can pass this processed data buffer in the uvc_video_fill_buffer function.
Like the gadget driver keeps the info of supported formats, framerate, and resolution, the uvc gadget application also keeps the corresponding mapping as below.
static const struct uvc_frame_info uvc_frames_yuyv[] = { { 640, 360, {666666, 10000000, 50000000, 0}, }, { 1280, 720, {50000000, 0},}, { 0, 0, {0, },}, };
static const struct uvc_frame_info uvc_frames_mjpeg[] = { {640,360,{666666, 10000000, 50000000, 0},}, { 1280,720,{50000000, 0}, }, { 0,0,{0,}, }, }
static const struct uvc_format_info uvc_formats[] = { {V4L2_PIX_FMT_YUYV, uvc_frames_yuyv}, {V4L2_PIX_FMT_MJPEG, uvc_frames_mjpeg}, }
In case the video-capture device supports other resolutions and frame formats, we need to replace/ add new values used in the UVC gadget test application with the same and also update the UVC webcam gadget kernel driver.
One thing to note is that the frame rate at which the gadget application sends the video data is based on the framerate at which the capture /custom processing application feeds input to the gadget application. So, in case of running the gadget application in the standalone mode with test pattern, it will run at the full potential speed depending on the usb type
Launch the uvc gadget application on boot.
To boot the device as a USB camera in Android, we can add the user space application in the init.device.rc and run as a service. Here the service is named as uvc_camera.
1. Create the service and start on boot complete
on property:sys.boot_completed=1 start uvc_camera
service uvc_camera /vendor/bin/uvc_gadget class main user root group root disabled
2. Create a new domain "uvc_camera"
Create the file device/manufacturer/device-name/sepolicy/uvc_camera.te with the following contents:
# uvc_camera service type uvc_camera, domain; type uvc_camera_exec, exec_type, vendor_file_type, file_type;
init_daemon_domain(uvc_camera)
3. Label /vendor/bin/uvc_gadget
Add the following to device/manufacturer/device-name/sepolicy/file_contexts:
# UVC camera service /vendor/bin/uvc_gadget u:object_r:uvc_camera_exec:s0
4. Use denials to determine the required permissions and refine the SELInux rules or append the 'androidboot.selinux=permissive' in bootcmd.
BOARD_KERNEL_CMDLINE += androidboot.selinux=permissive
