1、 Why does autonomous driving require fisheye cameras
In autonomous driving and advanced driver assistance systems (ADAS), the 360 ° Around View Monitor (AVM) is one of the most fundamental and important functions. It uses four cameras installed on the front, rear, left, and right sides of the vehicle to stitch together a panoramic bird’s-eye view of the surrounding area in real time, helping drivers to clearly perceive the surrounding environment of the vehicle in parking, narrow road traffic, and complex road conditions.
The AVM system has a special requirement for the cameras: each camera must cover a horizontal field of view of at least 180 ° in order to achieve 360 ° no dead angle coverage under the scheme of 4 cameras. This is exactly where fisheye cameras come in handy.
Traditional narrow angle cameras (60 °~90 °) require 6-8 cameras to achieve 360 ° coverage, which not only increases hardware costs and wiring complexity, but also increases the algorithm difficulty of multi camera synchronization and splicing. A fisheye camera with an angle of 180 °~210 ° can cover the complete field of view on one side of the vehicle with a single camera, and achieve a 360 ° panoramic view of the entire vehicle with four cameras.
2、 Technical Architecture of AVM System
A typical AVM surround view system consists of the following components:
| Components | 功能 | 关键参数 |
| Fisheye camera x 4 | Collect vehicle front, rear, left, and right images | 180°~210° FOV，1080P@30fps |
| Image processing unit | Distortion correction, viewpoint transformation, image stitching, fusion | SoC or FPGA, supporting multi-channel real-time processing |
| Display unit | Output bird’s-eye view+zoomed in view | Central control screen or dashboard screen |
| Calibration Data | Calibration data/internal parameters, external parameters, distortion coefficients of each camera | Factory pre calibration, supports online recalibration |
The core algorithm process is: original fisheye image → distortion correction → perspective transformation (bird’s-eye projection) → image stitching and fusion → overlaying vehicle models → outputting panoramic bird’s-eye view.
3、 Key Selection Parameters for Fisheye Cameras
In AVM systems, the selection of fisheye cameras directly affects the quality of the final panoramic image. The following are the parameters that need to be focused on:
3.1 Field of View (FOV)
AVM systems typically require a horizontal FOV of no less than 180 ° for each camera to ensure sufficient overlap between the four cameras (typically 20 °~30 °) for image stitching. A 210 ° fisheye camera can provide greater overlap margin, reduce the difficulty of stitching algorithms, and ensure coverage even when there is a deviation in the installation angle of the camera.
3.2 Resolution and frame rate
The display resolution of AVM system is usually 720P or 1080P. Considering the reduction of effective pixels after distortion correction and cropping, it is recommended to choose cameras with 1080P (1920 × 1080) or above. In terms of frame rate, 30fps is the basic requirement for smooth display, and some high-end systems use 60fps.
3.3 Sensors and Low Light Performance
The lighting conditions in car scenes vary greatly, from strong noon light to underground garages at night, with a dynamic range exceeding 100dB. Therefore, it is necessary to choose sensors that support HDR (high dynamic range) or WDR (wide dynamic range). At the same time, the target size of the sensor is also important – targets with a size of 1/2.7 “or larger have better signal-to-noise ratio in low light environments.
3.4 Distortion Characteristics
The distortion of fisheye lenses is inevitable, but different projection models (isometric projection, isometric projection, orthogonal projection, stereoscopic projection) have different distortion characteristics. AVM systems typically use equidistant projection because its angle radius mapping is linear, facilitating subsequent bird’s-eye view transformation calculations.
3.5 Vehicle specification level requirements
For pre installed mass-produced models, the camera needs to meet AEC-Q100 vehicle specification certification, a working temperature range of -40 ° C~85 ° C, an IP67 or higher protection level (external camera), and ESD anti-static requirements. For aftermarket and commercial vehicle scenarios, the requirements can be appropriately reduced.
4、 Distortion correction and bird’s-eye view generation principle
The core algorithm of the AVM system is to convert the distorted images of four fisheye cameras into a unified bird’s-eye view. This process is divided into three steps:
4.1 Fish eye distortion correction
The distortion correction of fisheye images is based on the fisheye camera model. Taking OpenCV’s cv2. fisheye module as an example, calibration requires four parameters: the internal parameter matrix K (fx, fy, cx, cy) and the distortion coefficients D (k1, k2, p1, p2). Pre calculate the mapping table using initUndistrortRectifyMap, and then perform real-time correction using the remap function.
The corrected image is still a perspective view, but the lines have been restored to straight lines and the edges are no longer curved.
4.2 Perspective Transformation (Aerial Projection)
The corrected perspective view needs to be converted to a bird’s-eye view. This is achieved through a 3×3 homography matrix, which describes the projection mapping relationship from the camera perspective to the ground overhead perspective.
The calculation of the homography matrix requires knowledge of the installation parameters of the camera (height, pitch angle, yaw angle) and the equation of the ground plane. In AVM systems, these parameters are usually obtained through calibration at the factory.
4.3 Image stitching and fusion
The bird’s-eye view from four cameras needs to be stitched together to create a complete 360 ° panoramic image. The key to splicing lies in the fusion of overlapping areas:
Brightness Fusion: Adjust the exposure and gain of 4 cameras to ensure consistent brightness in overlapping areas
Color Fusion: Correct the white balance and color deviation of 4 cameras
Feather Fusion: Use gradually fading weight fusion in overlapping areas to eliminate seam seams
5.Summary and Outlook
Fisheye camera is the core sensor of AVM surround view system, and its field of view angle, resolution, low light performance, and distortion characteristics directly determine the quality of panoramic images. With the development of autonomous driving technology, AVM systems are evolving from simple panoramic display to panoramic perception – adding obstacle detection, parking space recognition, pedestrian warning and other functions on the basis of bird’s-eye view, which puts higher requirements on the resolution and frame rate of fisheye cameras.
For teams developing AVM systems, it is recommended to use a 1080P 210 ° fisheye camera with a USB interface during the prototype validation phase, in conjunction with an open-source algorithm framework for rapid validation; Customize a dedicated camera solution according to vehicle specifications during the mass production stage.