TWI860908B - Method for constructing panoramic view model, vehicle-mounted device, and storage medium - Google Patents

Abstract

The present application relates to a field of smart driving and provides a method for constructing a panoramic view model, a vehicle-mounted device and a storage medium. The method includes: identifying a target object in an environment image and determining a first coordinate of the target object in the environment image; determining a second coordinate of the target object in an overhead image of the environment image according to the first coordinate; determining a third coordinate of the target object in a third coordinate system corresponding to a camera according to the second coordinate; determining a fourth coordinate of the target object in a fourth coordinate system corresponding to a vehicle according to the third coordinate, and determining an initial distance between the target object and the vehicle according to the fourth coordinate; determining a target distance based on a preset correction parameter and the initial distance, and constructing a target panoramic view model of the vehicle based on the target distance and the environment image. This application can improve the safety of users when driving a car.

Description

Panoramic view model construction method, vehicle-mounted device and storage medium

The present invention belongs to the field of intelligent driving and relates to image processing technology, and specifically relates to a method for constructing a panoramic view model, a vehicle-mounted device and a storage medium.

The panoramic view system installed in the vehicle can obtain images around the vehicle through the cameras installed in the front, back, left and right directions of the vehicle, and build a panoramic view three-dimensional model based on these images to simulate the environment around the vehicle, thereby helping users to better understand the distance between the vehicle and surrounding objects. However, the objects in the three-dimensional model constructed in the relevant technology are prone to distortion or distortion, making it difficult for users to accurately identify objects around the vehicle and judge the distance between the vehicle and the objects based on the three-dimensional model, resulting in increased driving risks for users.

In view of the above, it is necessary to propose a panoramic view model construction method, a vehicle-mounted device and a storage medium, which can solve the problem of increased driving risk for users caused by distortion or distortion of objects in the constructed vehicle panoramic view model.

The embodiment of the present application provides a method for constructing a panoramic view model, the method comprising: identifying a target object in an environment image acquired from a camera of a vehicle, and determining a first coordinate of the target object in a first coordinate system corresponding to the environment image; converting the environment image into a bird's-eye view image, and determining a second coordinate of the target object in a second coordinate system corresponding to the bird's-eye view image according to the first coordinate; and determining a second coordinate of the target object in a second coordinate system corresponding to the bird's-eye view image according to the second coordinate. Determine a third coordinate of the target object in a third coordinate system corresponding to the camera device; determine a fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle according to the third coordinate, and determine an initial distance between the target object and the vehicle according to the fourth coordinate; determine a target distance based on a preset correction parameter and the initial distance, and construct a target panoramic surround model of the vehicle according to the target distance and the environmental image.

In one embodiment, the target object in the environment image taken by the camera of the vehicle is identified, and the first coordinates of the target object in the first coordinate system corresponding to the environment image are determined, including: using a preset feature recognition algorithm to identify the target object in the environment image, and generating a rectangular enclosing frame of the target object in the environment image; determining the coordinates of the target corner points of the rectangular enclosing frame in the first coordinate system, and using the coordinates of the target corner points as the first coordinates.

In one embodiment, the preset feature recognition algorithm includes: one or more of a feature recognition algorithm based on a machine learning model and a feature recognition algorithm based on a deep learning model.

In one embodiment, the target corner points include the lower left corner point and the lower right corner point of the rectangular enclosing frame; the second coordinates include: the coordinates of the first projection point of the lower left corner point in the overhead image in the second coordinate system, and the coordinates of the second projection point of the lower right corner point in the overhead image in the second coordinate system.

In one embodiment, determining the third coordinates of the target object in the third coordinate system corresponding to the photographic device based on the second coordinates includes: determining the midpoint coordinates between the coordinates of the first projection point in the second coordinate system and the coordinates of the second projection point in the second coordinate system; and determining the coordinates of the midpoint coordinates in the third coordinate system as the third coordinates based on a first conversion relationship between the second coordinate system and the third coordinate system.

In one embodiment, determining the fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle based on the third coordinate includes: determining the installation distance between the camera device and the center of the vehicle in the bird's-eye view, the installation distance including a horizontal distance and a vertical distance; determining a second conversion relationship between the third coordinate system and the fourth coordinate system based on the installation distance, and converting the third coordinate to the fourth coordinate based on the second conversion relationship, wherein the origin of the fourth coordinate system is located at the center of the vehicle.

In one embodiment, determining the initial distance between the target object and the vehicle based on the fourth coordinate includes: determining the European distance between the fourth coordinate and the center of the vehicle as the initial distance.

In one embodiment, the target distance is determined based on a preset correction parameter and the initial distance, and a target panoramic surround view model of the vehicle is constructed according to the target distance and the environmental image, including: determining the target distance according to the difference between the initial distance and the correction parameter; constructing a three-dimensional bowl-shaped grid model with the target distance as the length of the bottom radius, and projecting the environmental images of the vehicle in four directions onto the three-dimensional bowl-shaped grid model to obtain the target panoramic surround view model.

The embodiment of the present application provides a panoramic view model construction device, the device comprising: an identification module, used to identify a target object in an environment image obtained from a camera of a vehicle, and determine a first coordinate of the target object in a first coordinate system corresponding to the environment image; a determination module, used to convert the environment image into a bird's-eye view image, and determine a second coordinate of the target object in a second coordinate system corresponding to the bird's-eye view image according to the first coordinate; The second coordinate determines the third coordinate of the target object in the third coordinate system corresponding to the camera device; the fourth coordinate of the target object in the fourth coordinate system corresponding to the vehicle is determined according to the third coordinate, and the initial distance between the target object and the vehicle is determined according to the fourth coordinate; a modeling group is constructed to determine the target distance based on a preset correction parameter and the initial distance, and a target panoramic surround model of the vehicle is constructed according to the target distance and the environmental image.

An embodiment of the present application provides a vehicle-mounted device, which includes: a memory and at least one processor, wherein the processor is used to implement the panoramic view model construction method when executing a computer program stored in the memory.

An embodiment of the present application provides a vehicle, which includes at least one camera device and the vehicle-mounted device.

An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the panoramic view model construction method is implemented.

In summary, the panoramic view model construction method described in the present application determines the first coordinates of the target object in the first coordinate system corresponding to the environmental image by identifying the target object in the environmental image obtained by the camera device of the vehicle; converts the environmental image into a bird's-eye view image based on a preset perspective projection matrix, and determines the second coordinates of the target object in the second coordinate system corresponding to the bird's-eye view image according to the first coordinates; and determines the second coordinates of the target object in the second coordinate system corresponding to the bird's-eye view image according to the second coordinates. The method comprises the following steps: determining the third coordinate of the target object in the third coordinate system corresponding to the camera device; determining the fourth coordinate of the target object in the fourth coordinate system corresponding to the vehicle according to the third coordinate, and determining the initial distance between the target object and the vehicle according to the fourth coordinate; determining the target distance based on the preset correction parameter and the initial distance, and constructing the target panoramic surround model of the vehicle according to the target distance and the environmental image. The method can avoid the distortion or distortion of objects in the constructed panoramic surround model of the vehicle, obtain a panoramic surround model of the vehicle that is closer to the real environment, and ensure the driving safety of users when driving according to the panoramic surround model of the vehicle.

In order to more clearly understand the above-mentioned purpose, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the application. The terms used herein in the specification of the application are only for the purpose of describing an embodiment in an embodiment and are not intended to limit the application.

It should be noted that in this application, "at least one" means one or more, and "plurality" means two or more than two. "And/or" describes the relationship between related objects, indicating that three relationships may exist. For example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claim and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or precedence.

In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. The following embodiments and features in the embodiments may be combined with each other without conflict.

In one embodiment, the panoramic view system installed in the vehicle can obtain images of the surroundings of the vehicle through cameras installed in the front, rear, left, and right directions of the vehicle, and construct a panoramic view three-dimensional model based on these images to simulate the environment around the vehicle, thereby helping users to better understand the distance between the vehicle and surrounding objects. However, the objects in the three-dimensional model constructed in the related technology are prone to distortion or other problems, making it difficult for users to accurately identify objects around the vehicle and judge the distance between the vehicle and the objects based on the three-dimensional model, resulting in increased driving risks for users.

To solve the above problems, the present application provides a method for constructing a panoramic model, which determines the first coordinates of the target object in a first coordinate system corresponding to the environment image by identifying the target object in the environment image obtained by the camera device of the vehicle; converts the environment image into a bird's-eye view image based on a preset perspective projection matrix, and determines the second coordinates of the target object in a second coordinate system corresponding to the bird's-eye view image according to the first coordinates; and determines the second coordinates of the target object in a second coordinate system corresponding to the bird's-eye view image according to the first coordinates. The third coordinate of the target object in the third coordinate system corresponding to the camera device is determined based on the second coordinate; the fourth coordinate of the target object in the fourth coordinate system corresponding to the vehicle is determined based on the third coordinate, and the initial distance between the target object and the vehicle is determined based on the fourth coordinate; the target distance is determined based on the preset correction parameter and the initial distance, and the target panoramic surround model of the vehicle is constructed based on the target distance and the environmental image. It can avoid the distortion or distortion of objects in the constructed vehicle panoramic surround model, obtain a vehicle panoramic surround model that is closer to the real environment, and ensure the driving safety of users when driving according to the vehicle panoramic surround model.

FIG1 is a schematic diagram of the structure of a vehicle-mounted device provided in an embodiment of the present application. The embodiment of the present application does not impose any restrictions on the specific type of the vehicle-mounted device.

As shown in FIG1 , the vehicle-mounted device 10 may be installed in a vehicle 1, and the vehicle-mounted device 10 may include a communication module 101, a memory 102, a processor 103, an input/output (I/O) interface 104, and a bus 105. The processor 103 is coupled to the communication interface 101, the memory 102, and the I/O interface 104 through the bus 105.

The communication module 101 may include a wired communication module and/or a wireless communication module. The wired communication module may provide one or more of the wired communication solutions such as universal serial bus (USB), controller area network bus (CAN), local interconnect network (LIN), FlexRay, etc. The wireless communication module may provide one or more of the wireless communication solutions such as wireless fidelity (Wi-Fi), bluetooth (BT), mobile communication network, frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memories can be directly read and written by the processor 103, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

The non-volatile memory can also store executable programs and user and application data, etc., and can be loaded into the random access memory in advance for direct reading and writing by the processor 110. The non-volatile memory can include a disk storage element and a flash memory.

The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 103, the panoramic view model construction method executed on the vehicle-mounted device 10 can be implemented.

In other embodiments, the vehicle-mounted device 10 further includes an external storage interface for connecting to an external storage to expand the storage capacity of the vehicle-mounted device 10.

The processor 103 may include one or more processing units, for example, the processor 103 may include an application processor (AP), a modulation and demodulation processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural-network processing unit (NPU), etc. Different processing units may be independent components or integrated into one or more processors.

The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned panoramic view model construction method.

The I/O interface 104 is used to provide a channel for user input or output. For example, the I/O interface 104 can be used to connect various input and output devices, such as a mouse, a keyboard, a touch device, a display screen, etc., so that the user can enter information or visualize the information.

The I/O interface 104 may also be used to provide a channel for data transmission with the camera 106. For example, the I/O interface 104 may be used to obtain environmental images of the vehicle from the camera 106.

The camera device 106 includes at least one camera device installed in the vehicle 1, which is used to capture the environmental image of the environment in which the vehicle 1 is located. The camera device 106 can be a fisheye camera, an infrared camera, etc. For example, as shown in FIG. 2, it is an example diagram of the installation position of the camera device in the vehicle provided in an embodiment of the present application. The camera device includes four cameras installed in the front, rear, left, and right directions of the vehicle. The union of the four fields of view of the four cameras can cover a range of 360 degrees around the vehicle.

The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the I/O interface 104 in the vehicle-mounted device 10 .

It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the vehicle-mounted device 10. In other embodiments of the present application, the vehicle-mounted device 10 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

FIG3 is a flow chart of a panoramic view model construction method provided by an embodiment of the present application. The panoramic view model construction method is applied to a vehicle-mounted device, such as the vehicle-mounted device 10 in FIG1 , and specifically includes the following steps. According to different requirements, the order of the steps in the flow chart can be changed, and some can be omitted.

Step S31, identifying a target object in an environment image acquired from a camera of a vehicle, and determining a first coordinate of the target object in a first coordinate system corresponding to the environment image.

In one embodiment, at least one camera is installed in a vehicle, and the vehicle-mounted device can receive a unique identifier of each camera input by a user and an installation position of each camera relative to the vehicle, thereby distinguishing cameras installed at different positions. The installation position of each camera relative to the vehicle may include an installation distance of each camera from the center of the vehicle, and the installation distance includes a horizontal distance and a vertical distance, wherein the center of the vehicle represents the center of a top view of the vehicle.

For example, as shown in FIG4 , a rectangular coordinate system O1X1Y1 corresponding to the vehicle is established with the center of the top view of the vehicle (i.e., the center of the dotted line enclosed frame) as the origin, and the direction of the front of the vehicle in the top view is defined as the front of the vehicle. A camera is installed in each of the four directions of the vehicle. Among them, the camera installed in the front of the vehicle is marked as cameraF, the camera installed at the rear of the vehicle is marked as cameraB, the camera installed on the left of the vehicle is marked as cameraL, and the camera installed on the right of the vehicle is marked as cameraR. The horizontal distance of any camera relative to the center of the vehicle includes the distance between the camera and the Y1 axis, and the vertical distance of any camera relative to the center of the vehicle includes the distance between the camera and the X1 axis. In the subsequent embodiments, cameraF and cameraB as shown in FIG. 4 are mounted on the Y1 axis for example. In other embodiments, if cameraF or cameraB is not mounted on the Y1 axis, a certain translation compensation can be performed during the coordinate transformation to implement the subsequent process.

In one embodiment, before identifying the target object in the environment image obtained from the camera of the vehicle, the method further includes preprocessing the environment image, and the preprocessing includes but is not limited to: resizing, for example, resizing the environment image to the size required for input of the feature recognition algorithm; image optimization, for example, improving the texture clarity of the environment image based on an interpolation algorithm (such as a bilinear interpolation algorithm); grayscale processing, for example, using a weighted average method to transform the environment image into a grayscale image; Convert to grayscale image; filter processing, for example, use preset filter (such as mean filter, median filter, etc.) to smooth the environment image to remove noise; distortion correction, for example, when the camera device is a fisheye camera, the objects in the environment image may be distorted, and the environment image can be distorted based on the camera parameters of the fisheye camera (such as focal length, principal point coordinates, distortion coefficient, etc.) and pre-selected correction model (such as Pinhole model, fisheye model, etc.). By pre-processing the environment image, the accuracy of identifying the target object in the environment image can be improved.

In one embodiment, after obtaining the distortion-corrected environment image corresponding to any camera in the vehicle, in order to facilitate the subsequent step of obtaining a panoramic surround view of the vehicle from a bird's-eye view angle to construct a panoramic surround model of the vehicle, all cameras of the vehicle can be jointly calibrated based on the checkerboard calibration method, thereby converting all environment images taken by all cameras of the vehicle into the same coordinate system.

Specifically, the process of jointly calibrating all the camera devices of the vehicle based on the chessboard calibration method includes: obtaining the chessboard image of the ground in the corresponding direction taken by each camera device; performing intrinsic and extrinsic calibration on each camera device to obtain the initial camera parameters such as the intrinsic and extrinsic parameters of each camera device; extracting the target feature points in the chessboard image in each direction, and matching the feature points between the camera devices based on the target feature points; using the method of minimizing ghosting errors, etc. The method optimizes and updates the initial camera parameters of each camera device based on feature point matching information, so that the updated camera parameters can align the chessboard image taken by the corresponding camera device more accurately; the accuracy of the updated camera parameters is evaluated, and if the evaluation result indicates that the accuracy of the updated camera parameters is less than the preset accuracy threshold, the above steps are repeated until the camera parameters with an accuracy greater than or equal to the accuracy threshold are obtained, thereby completing the joint calibration of all camera devices.

In an embodiment, the world distance corresponding to the length of each primitive point in the environment image can also be determined in the above joint calibration process. For example, it can be determined that the world distance corresponding to the length of each primitive point in the environment image is 1 cm.

In one embodiment, for any camera device of a vehicle (e.g., camera F), the target object in the environment image taken by the camera device of the vehicle is identified, and the first coordinate of the target object in the first coordinate system corresponding to the environment image is determined, including: using a preset feature recognition algorithm to identify the target object in the environment image, and generating a rectangular enclosing frame of the target object in the environment image; determining the coordinates of the target corner points of the rectangular enclosing frame in the first coordinate system, and using the coordinates of the target corner points as the first coordinates. The target object includes objects with height, such as vehicles, walls, columns, steps, human bodies, and other objects.

In one embodiment, the preset feature recognition algorithm includes: one or more of a feature recognition algorithm based on a machine learning model and a feature recognition algorithm based on a deep learning model.

In one embodiment, the feature recognition algorithm based on the machine learning model may use algorithms such as linear regression algorithm, support vector regression algorithm, ridge regression algorithm, decision tree algorithm, etc. The feature recognition algorithm based on the machine learning model can learn how to classify the pixel points in the environment image through a supervised learning method, and can realize feature recognition of the environment image without relying on display programming, which can improve the efficiency of feature recognition.

In one embodiment, the feature recognition algorithm based on the deep learning model can use a variety of neural network structures, such as convolutional neural networks, recurrent neural networks, long short-term memory networks, etc. The feature recognition algorithm based on the deep learning model can extract low-level features of the environment image based on the deep learning model, obtain a more abstract high-level representation of the environment image based on the low-level features, and determine the distributed features in the environment image based on the high-level representation to achieve feature recognition of the environment image. It can be applied to feature recognition of environment images containing complex objects, and improve the accuracy and efficiency of feature recognition.

In one example, using a fully convolutional neural network model to identify a target object in an environment image includes: using multiple convolutional layers to filter the environment image through convolution operations to obtain feature images of multiple scales of the environment image; using an activation layer to perform a nonlinear transformation on the output of the convolutional layer using an activation function (such as ReLU, sigmoid, tanh, etc.) to increase the expression ability of the network; using a pooling layer to reduce the size of the feature image to reduce parameters. The number of features and the computational complexity of the network are reduced; the output of the pooling layer is interpolated or deconvolved using the upper sampling layer to restore the size of the feature image and obtain a high-resolution feature representation of the environment image; the different layers in the network are connected using a skip connection mechanism to better capture feature information of different scales in the environment image; the output layer is used to perform feature classification based on the semantic segmentation algorithm to obtain the object category to which the pixel points in the environment image belong, thereby determining the target object in the environment image.

In one embodiment, in order to determine the specific position of the target object in the environment image, a rectangular enclosing frame of the target object may be first generated in the environment image, and then the coordinates of the target corner points of the rectangular enclosing frame of the target object may be determined in the first coordinate system corresponding to the environment image, and finally the coordinates of the target corner points may be used as the first coordinates of the target object in the first coordinate system corresponding to the environment image. The target corner points include the lower left corner point and the lower right corner point of the rectangular enclosing frame.

For example, as shown in FIG5, it is an example diagram of the first coordinates of the target object provided in the embodiment of the present application. In it, the first coordinate system O2X2Y2 is established with the position of the upper left corner of the environment image captured by camera F as the origin, and a rectangular enclosing frame shown by a dotted line of the target object (such as the rear view of the vehicle shown in FIG5) is generated in the environment image, and the coordinates (x1, y1) of the lower left corner point and the coordinates (x2, y2) of the lower right corner point of the rectangular enclosing frame shown by the dotted line in the first coordinate system are determined, and the coordinates (x1, y1) and the coordinates (x2, y2) are used as the first coordinates.

Step S32: convert the environment image into a bird's-eye view image, and determine the second coordinates of the target object in a second coordinate system corresponding to the bird's-eye view image according to the first coordinates.

In one embodiment, when the environment image is converted into a bird's-eye view image, the projection transformation of the environment image into the bird's-eye view image can be realized by using a preset perspective projection matrix based on the principle of constructing a two-dimensional panoramic surround view monitor (AVM).

Specifically, taking cameraF as an example, the method for obtaining the perspective projection matrix may include: based on the chessboard calibration method, using cameraF to shoot a first image of a chessboard on the ground within the field of view of cameraF, and using an auxiliary camera to shoot a second image of the chessboard, wherein the auxiliary camera is located above the center of the vehicle and the shooting angle of the auxiliary camera is parallel to the center of the vehicle. At the ground level; determining the environmental coordinates of a plurality of calibrated feature points in the first image in the coordinate system of the first image, and determining in the second image the bird's-eye view coordinates of points corresponding to the plurality of calibrated feature points in the coordinate system of the second image; calculating a homography matrix for transforming the environmental coordinates corresponding to the plurality of calibrated feature points into corresponding bird's-eye view coordinates, and using the homography matrix as the projection perspective matrix.

In one embodiment, the perspective projection matrix is used to perform projection transformation on each primitive point in the environment image to obtain a bird's-eye view image corresponding to the environment image (for example, as shown in FIG. 6 ), and the coordinates of the projection points corresponding to each primitive point in the environment image in the bird's-eye view image in the second coordinate system where the bird's-eye view image is located, wherein the second coordinate system may be O3X3Y3 as shown in FIG. 7 . In addition, since the world distance corresponding to the length of each primitive point in the environment image is a known parameter, the perspective projection matrix is used to perform corresponding transformation on the known parameter to obtain the world distance corresponding to the length of each primitive point in the bird's-eye view image. For example, as shown in FIG6 , when the environment image is converted into a bird's-eye view image, due to the change in viewing angle, the image closer to the top in the environment image is stretched (including lengthening and widening) to a greater extent, and the image closer to the bottom in the environment image is stretched to a lesser extent.

In one embodiment, when the perspective projection matrix is used to perform a projection transformation on each primitive point in the environment image, the rectangular enclosing frame of the target object can be projected and transformed to obtain the enclosing frame of the target object at the bird's-eye view angle in the bird's-eye view image (e.g., the dotted frame shown in FIG6 ); the projection point corresponding to the lower left corner point of the rectangular enclosing frame of the target object and the projection point corresponding to the lower right corner point of the rectangular enclosing frame of the target object can also be obtained. The second coordinates of the target object in the second coordinate system corresponding to the bird's-eye view image include: the coordinates of the first projection point of the lower left corner point in the bird's-eye view image (e.g., as shown in FIG6 ) in the second coordinate system, and the coordinates of the second projection point of the lower right corner point in the bird's-eye view image (e.g., as shown in FIG6 ) in the second coordinate system.

In one embodiment, since all the camera devices in the vehicle have been jointly calibrated in step S31, a panoramic view of the vehicle from a bird's-eye view angle can be obtained based on the joint calibration according to the projection perspective matrix between the environment image and the bird's-eye view image. For example, FIG8 is an example of a panoramic view of the vehicle from a bird's-eye view angle provided in an embodiment of the present application.

Step S33, determining the third coordinate of the target object in a third coordinate system corresponding to the camera device according to the second coordinate.

In one embodiment, determining the third coordinates of the target object in the third coordinate system corresponding to the photographic device based on the second coordinates includes: determining the midpoint coordinates (e.g., (x3, y3)) between the coordinates of the first projection point in the second coordinate system and the coordinates of the second projection point in the second coordinate system; and determining the coordinates of the midpoint coordinates in the third coordinate system as the third coordinates based on a first conversion relationship between the second coordinate system and the third coordinate system.

In one embodiment, for the convenience of calculation, when the distance between the target object and the corresponding camera device is calculated using the midpoint coordinates, the second coordinate of the target object in the bird's-eye view image can be converted into a third coordinate in a third coordinate system corresponding to the camera device.

Specifically, taking camera F as an example, as shown in FIG7 , the center of the field of view of the camera device is located on the central vertical line of the environment image, and the lower boundary of the bird's-eye view image is the boundary of the front of the vehicle, which is also the installation position of camera F. Therefore, the center position of the lower boundary of each bird's-eye view image is the location of each camera device, and the third coordinate system O4X4Y4 where the camera device is located can be established with the center position of the lower boundary of the bird's-eye view image as the origin.

In one embodiment, since the second coordinate system and the third coordinate system are both rectangular coordinate systems, the first conversion relationship between the second coordinate system and the third coordinate system can be determined by determining the positional relationship between the two origins of the second coordinate system and the third coordinate system. Specifically, the length W and width H of the bird's-eye view image can be determined according to the length and width of the environment image using a projection transformation matrix, wherein the length of the bird's-eye view image can be the length of the side where the origin of the second coordinate system is located (e.g., the upper boundary of the bird's-eye view image in FIG7 ), and the width of the bird's-eye view image can be the distance between two parallel boundaries of the bird's-eye view image (e.g., the upper boundary and the lower boundary of the bird's-eye view image in FIG7 ); the length and width of the environment image can be determined according to the length and width of the environment image. The positional relationship between the two origins of the second coordinate system and the third coordinate system includes: the coordinates of the origin of the third coordinate system in the second coordinate system are (W/2, -H); the first transformation relationship is determined according to the positional relationship between the two origins of the second coordinate system and the third coordinate system, and the first transformation relationship includes: if the coordinates of any point in the second coordinate system are (x4, y4), then the coordinates of any point in the third coordinate system are (x5, y5) = (-x4+W/2, H-|y4|).

In one embodiment, the coordinates (-x3+W/2, H-|y3|) of the midpoint coordinate (x3, y3) in the third coordinate system can be determined as the third coordinate according to the first conversion relationship between the second coordinate system and the third coordinate system.

In one embodiment, since the world distance corresponding to the length of each pixel point in the environment image and the bird's-eye view image are known parameters, the world distance between the target object and the camera device can be determined based on the third coordinate. Specifically, the world distance between the target object and the camera device can be the Euclidean distance between the third coordinate and the origin of the third coordinate system.

In one embodiment, for any camera device, since the environment image may contain multiple target objects, in order to avoid distortion or distortion of the target panoramic surround model constructed in the subsequent steps, the method further includes: determining the distance between each target object in the environment image and the corresponding camera device, and constructing the target panoramic surround model based on the target object corresponding to the minimum distance.

Step S34, determining a fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle according to the third coordinate, and determining an initial distance between the target object and the vehicle according to the fourth coordinate.

In one embodiment, determining the fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle based on the third coordinate includes: determining the installation distance between the camera device and the center of the vehicle in the bird's-eye view, the installation distance including a horizontal distance and a vertical distance; determining a second conversion relationship between the third coordinate system and the fourth coordinate system based on the installation distance, and converting the third coordinate to the fourth coordinate based on the second conversion relationship, wherein the origin of the fourth coordinate system is located at the center of the vehicle.

In one embodiment, cameraL is used as an example, as shown in FIG9 , which is an example diagram of the third coordinate system and the fourth coordinate system provided by the embodiment of the present application. Among them, O1X1Y1 represents the fourth coordinate system with the center of the vehicle as the origin, and O4X4Y4 represents the third coordinate system corresponding to cameraL.

In one embodiment, the method for determining the second transformation relationship is similar to the method for determining the first transformation relationship. Since the horizontal distance and vertical distance between the camera device (e.g., camera L) and the center of the vehicle in the bird's-eye view are known parameters (refer to step S31), the second transformation relationship may include a translation transformation of the third coordinate based on the horizontal distance and the vertical distance. Specifically, reference may be made to the method for determining the first transformation relationship.

In one embodiment, determining the initial distance between the target object and the vehicle based on the fourth coordinate includes: determining the European distance between the fourth coordinate and the center of the vehicle as the initial distance.

Step S35, determining the target distance based on the preset correction parameters and the initial distance, and constructing a target panoramic surround model of the vehicle according to the target distance and the environmental image.

In one embodiment, the target distance is determined based on the preset correction parameter and the initial distance, and the target panoramic surround model of the vehicle is constructed according to the target distance and the environmental image, including: determining the target distance according to the difference between the initial distance and the correction parameter; constructing a three-dimensional bowl-shaped grid model with the target distance as the length of the bottom radius, and projecting the environmental images of the vehicle in four directions onto the three-dimensional bowl-shaped grid model to obtain the target panoramic surround model. The bottom of the three-dimensional bowl-shaped grid model is centered at the center of the vehicle.

For example, as shown in FIG10, it is an example diagram of a panoramic view model provided in an embodiment of the present application. Among them, the panoramic view model uses a three-dimensional bowl-shaped grid model. The panoramic view model of the vehicle can be obtained by projecting the environmental images in four directions of the vehicle onto the three-dimensional bowl-shaped grid model. The left model in FIG10 is a model constructed by directly using the initial distance as the length of the bottom radius of the three-dimensional bowl-shaped grid model. Compared with the right model, it can be seen that the target object of the column in the left model is distorted at the bottom of the bowl. In order to avoid the above problems, the initial distance can be corrected so that the target object can be projected as a whole to the inner side of the bowl wall during projection, such as shown in the right model of FIG10.

In one embodiment, the method for determining the correction parameter includes: determining the category of the target object; determining the width of the target object based on the category, and using the width of the target object as the correction parameter. For example, when the category of the target object is a human body, the average width of the human body can be used as the correction parameter. In another embodiment, a parameter input by a user can also be used as the correction parameter.

In one embodiment, the difference between the initial distance and the correction parameter is used as the target distance, so that the target object can be projected to the outside of the three-dimensional bowl-shaped grid model with the target distance as the base radius, thereby projecting the target object onto the bowl wall of the three-dimensional bowl-shaped grid model to avoid distortion of the target object during projection. During the driving process of the vehicle, by continuously determining the target object and the correction parameter corresponding to the target object, it can be ensured that the panoramic view model of the vehicle at every moment is not distorted, thereby ensuring the driving safety of the user.

In one embodiment, the method may further include: if the target panoramic view model is distorted, updating the target panoramic view model, including: using the updated correction parameters to reduce the bottom radius of the three-dimensional bowl-shaped grid model. The determination result of whether the target panoramic view model is distorted by the user driving the vehicle may be received.

In one embodiment, the panoramic view model construction method provided by the embodiment of the present application determines the first coordinates of the target object in the first coordinate system corresponding to the environmental image by identifying the target object in the environmental image obtained from the camera device of the vehicle; converts the environmental image into a bird's-eye view image based on a preset perspective projection matrix, and determines the second coordinates of the target object in the second coordinate system corresponding to the bird's-eye view image according to the first coordinates; and determines the second coordinates of the target object in the second coordinate system corresponding to the bird's-eye view image according to the first coordinates. The second coordinate determines the third coordinate of the target object in the third coordinate system corresponding to the camera device; the fourth coordinate of the target object in the fourth coordinate system corresponding to the vehicle is determined according to the third coordinate, and the initial distance between the target object and the vehicle is determined according to the fourth coordinate; the target distance is determined based on the preset correction parameter and the initial distance, and a target panoramic surround model of the vehicle is constructed according to the target distance and the environmental image. It can avoid the distortion or distortion of objects in the constructed vehicle panoramic view model, obtain a vehicle panoramic view model that is closer to the real environment, ensure the driving safety of users when driving according to the vehicle panoramic view model, and reduce the property loss of users caused by vehicle collisions.

FIG. 11 is a structural diagram of a panoramic view model construction device provided in an embodiment of the present application.

In some embodiments, the panoramic view model construction device 40 may include a plurality of functional modules composed of computer program segments. The computer programs of the various program segments in the panoramic view model construction device 40 may be stored in a memory of the vehicle-mounted device and executed by at least one processor to execute (see FIG. 3 for details) the function of building a panoramic view model.

In this embodiment, the panoramic view model construction device 40 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: an identification module 401, a determination module 402, and a construction module 403. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functional implementation method of each module in the panoramic view model construction device 40 can refer to the above definition of the panoramic view model construction method, and will not be repeated here. The identification module 401 is used to identify a target object in an environment image acquired from a camera of a vehicle, and determine a first coordinate of the target object in a first coordinate system corresponding to the environment image. The determination module 402 is used to convert the environment image into a bird's-eye view image, determine a second coordinate of the target object in a second coordinate system corresponding to the bird's-eye view image according to the first coordinate; determine a third coordinate of the target object in a third coordinate system corresponding to the camera according to the second coordinate; determine a fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle according to the third coordinate, and determine an initial distance between the target object and the vehicle according to the fourth coordinate. The constructed model group 403 determines the target distance based on the preset correction parameters and the initial distance, and constructs the target panoramic surround model of the vehicle according to the target distance and the environmental image.

The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein the computer program includes program instructions, and the method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application. Among them, the computer-readable storage medium can be the internal storage of the vehicle-mounted device described in the above-mentioned embodiment, such as the hard disk or storage of the vehicle-mounted device. The computer-readable storage medium can also be an external storage device of the vehicle-mounted device, such as a plug-in hard disk equipped on the vehicle-mounted device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card), etc.

In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the vehicle-mounted device, etc.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in a certain embodiment, please refer to the relevant description of other embodiments. A person of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

In the embodiments provided in the present application, it should be understood that the disclosed devices/terminal equipment and methods can be implemented in other ways. For example, the device/terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical functional division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technical personnel in this field should understand that they can still modify the technical solutions described in the above-mentioned embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

1: Vehicle 10: On-board device 101: Communication module 102: Memory 103: Processor 104: I/O interface 105: Bus 106: Camera device O1X1Y1: Fourth coordinate system O2X2Y2: First coordinate system O3X3Y3: Second coordinate system O4X4Y4: Third coordinate system 40: Panoramic view model building device 401: Identification module 402: Determination module 403: Model building group S31~S35: Steps

FIG1 is a structural diagram of a vehicle-mounted device provided in an embodiment of the present application.

FIG. 2 is an example diagram of the installation position of a camera device in a vehicle provided in an embodiment of the present application.

FIG3 is a flow chart of a method for constructing a panoramic view model provided in an embodiment of the present application.

FIG. 4 is an example diagram of a top view of a vehicle provided in an embodiment of the present application.

FIG. 5 is an example diagram of the first coordinates of a target object provided in an embodiment of the present application.

FIG. 6 is an example diagram of an environmental image and a corresponding bird's-eye view image provided by an embodiment of the present application.

FIG. 7 is an example diagram of the second coordinate system and the third coordinate system provided in an embodiment of the present application.

FIG8 is an example diagram of a panoramic view from a bird's-eye view of a vehicle provided in an embodiment of the present application.

FIG. 9 is an example diagram of a third coordinate system and a fourth coordinate system provided in an embodiment of the present application.

FIG. 10 is an example diagram of a panoramic view model provided in an embodiment of the present application.

FIG. 11 is a structural diagram of a panoramic view model construction device provided in an embodiment of the present application.

S31~S35: Steps

Claims (10)

A method for constructing a panoramic view model, wherein the method comprises: Identifying a target object in an environment image obtained from a camera of a vehicle, and determining a first coordinate of the target object in a first coordinate system corresponding to the environment image; Converting the environment image to a bird's-eye view image, and determining a second coordinate of the target object in a second coordinate system corresponding to the bird's-eye view image according to the first coordinate; Determining a third coordinate of the target object in a third coordinate system corresponding to the camera according to the second coordinate; Determining a fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle according to the third coordinate, and determining an initial distance between the target object and the vehicle according to the fourth coordinate; The target distance is determined based on the preset correction parameters and the initial distance, and a target panoramic view model of the vehicle is constructed based on the target distance and the environmental image. The method for constructing a panoramic view model as described in claim 1, wherein the step of identifying a target object in an environment image captured by a camera of the vehicle and determining a first coordinate of the target object in a first coordinate system corresponding to the environment image comprises: Identifying the target object in the environment image using a preset feature recognition algorithm and generating a rectangular bounding box of the target object in the environment image; Determining the coordinates of the target corner points of the rectangular bounding box in the first coordinate system, and using the coordinates of the target corner points as the first coordinates. A method for constructing a panoramic surround model as described in claim 2, wherein the default feature recognition algorithm includes: one or more of a feature recognition algorithm based on a machine learning model and a feature recognition algorithm based on a deep learning model. A method for constructing a panoramic model as described in claim 2, wherein the target corner points include the lower left corner point and the lower right corner point of the rectangular enclosing frame; and the second coordinates include: the coordinates of the first projection point of the lower left corner point in the bird's-eye view image in the second coordinate system, and the coordinates of the second projection point of the lower right corner point in the bird's-eye view image in the second coordinate system. The method for constructing a panoramic model as described in claim 4, wherein determining the third coordinate of the target object in the third coordinate system corresponding to the camera device according to the second coordinate comprises: Determining the midpoint coordinates between the coordinates of the first projection point in the second coordinate system and the coordinates of the second projection point in the second coordinate system; Determining the coordinates of the midpoint coordinates in the third coordinate system as the third coordinates according to the first conversion relationship between the second coordinate system and the third coordinate system. The method for constructing a panoramic model as described in claim 1, wherein the determining the fourth coordinate of the target object in the fourth coordinate system corresponding to the vehicle according to the third coordinate comprises: Determining the installation distance between the camera device and the center of the vehicle in the bird's-eye view angle, wherein the installation distance comprises a horizontal distance and a vertical distance; Determining a second conversion relationship between the third coordinate system and the fourth coordinate system according to the installation distance, and converting the third coordinate to the fourth coordinate according to the second conversion relationship, wherein the origin of the fourth coordinate system is located at the center of the vehicle. The method for constructing a panoramic view model as described in claim 1, wherein determining the initial distance between the target object and the vehicle based on the fourth coordinate includes: Determining the European distance between the fourth coordinate and the center of the vehicle as the initial distance. The method for constructing a panoramic view model as described in claim 1, wherein the target distance is determined based on the preset correction parameter and the initial distance, and the target panoramic view model of the vehicle is constructed according to the target distance and the environmental image, including: Determining the target distance according to the difference between the initial distance and the correction parameter; Constructing a three-dimensional bowl-shaped grid model with the target distance as the length of the bottom radius, and projecting the environmental images of the vehicle in four directions onto the three-dimensional bowl-shaped grid model to obtain the target panoramic view model. A vehicle-mounted device, wherein the vehicle-mounted device includes a memory and at least one processor, wherein the memory stores at least one instruction, and when the at least one instruction is executed by the at least one processor, the panoramic view model construction method as described in any one of claim 1 to claim 8 is implemented. A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements a method for constructing a panoramic view model as described in any one of claim 1 to claim 8.