CN114862658A

CN114862658A - Image processing system, method, intelligent terminal and computer readable storage medium

Info

Publication number: CN114862658A
Application number: CN202210338866.8A
Authority: CN
Inventors: 王刚; 余洪涛; 谷涛
Original assignee: Orbbec Inc
Current assignee: Orbbec Inc
Priority date: 2022-04-01
Filing date: 2022-04-01
Publication date: 2022-08-05
Anticipated expiration: 2042-04-01
Also published as: WO2023184740A1; CN114862658B

Abstract

The invention discloses an image processing system, a method, an intelligent terminal and a computer readable storage medium, wherein the image processing system comprises: a depth camera configured to acquire a depth image of a target scene; a two-dimensional camera configured to acquire a two-dimensional image of a target scene; a memory configured to store the depth image, a two-dimensional image, calibration information between the depth camera and the two-dimensional camera, and a depth measurement range of the depth camera; and the processor is configured to determine a corresponding line index range of each line of the depth image in the two-dimensional image according to the calibration information and the depth measurement range, and extract at least one line of depth data in the depth image and at least one corresponding line of two-dimensional data in the two-dimensional image according to the line index range to perform alignment processing to determine two-dimensional data corresponding to the depth data. Compared with the prior art, the method and the device are beneficial to improving the image alignment processing efficiency.

Description

Image processing system, method, intelligent terminal and computer readable storage medium

Technical Field

The present invention relates to the field of image information computing technology, and in particular, to an image processing system, method, intelligent terminal, and computer-readable storage medium.

Background

With the development of science and technology, the acquisition and application of image information become more and more extensive. In some application scenarios, it is necessary to perform alignment processing on the acquired multiple images so as to acquire corresponding data information in the multiple images. For example, in an application scene of 3D vision, in order to better apply depth data in a depth image, it is necessary to perform corresponding image processing on a depth image corresponding to one scene and another two-dimensional image (for example, an RGB image, a grayscale image, and the like) corresponding to the scene, so that the depth image is aligned with the two-dimensional image, that is, depth information in the depth image is mapped onto pixels of the corresponding two-dimensional image, thereby facilitating operations such as 3D recognition using the depth information.

When the existing image processing system carries out alignment processing on two (or more) images, all data in the two images need to be processed at one time, and batch calculation and processing of the data cannot be carried out. The problem of the prior art is that a large amount of data needs to be processed simultaneously when image alignment processing is performed, which is not beneficial to improving the processing efficiency and the accuracy of image alignment. Meanwhile, the huge calculation amount requires high performance of hardware equipment for processing, which is not beneficial to image alignment processing under the scene of low performance of hardware equipment, namely, the applicability of image alignment processing is influenced.

Disclosure of Invention

The invention mainly aims to provide an image processing system, an image processing method, an intelligent terminal and a computer readable storage medium, and aims to solve the problems that in the prior art, a large amount of data needs to be processed simultaneously when image alignment processing is carried out, the processing efficiency is not favorably improved, the image alignment processing is not favorably carried out in a scene with low hardware equipment performance, and the applicability of the image alignment processing is influenced.

In order to achieve the above object, a first aspect of the present invention provides an image processing system comprising:

a depth camera configured to acquire a depth image of a target scene;

a two-dimensional camera configured to acquire a two-dimensional image of a target scene;

a memory configured to store the depth image, a two-dimensional image, calibration information between the depth camera and the two-dimensional camera, and a depth measurement range of the depth camera;

and the processor is configured to determine a corresponding line index range of each line of the depth image in the two-dimensional image according to the calibration information and the depth measurement range, and extract at least one line of depth data in the depth image and at least one corresponding line of two-dimensional data in the two-dimensional image according to the line index range to perform alignment processing to determine two-dimensional data corresponding to the depth data.

Optionally, the extracting, according to the line index range, at least one line of depth data in the depth image and at least one line of two-dimensional data corresponding to the two-dimensional image, and performing alignment processing to determine two-dimensional data corresponding to the depth data includes:

and dividing the depth image and the two-dimensional image according to the memory size of the processor and the line index range, and extracting multiple groups of data to be processed, wherein each group of data to be processed comprises at least one line of depth data and multiple corresponding lines of two-dimensional data, and adjacent lines of depth data share part of two-dimensional data.

Optionally, the determining the two-dimensional data corresponding to the depth data by performing alignment processing on the to-be-processed data includes:

carrying one group of the data to be processed into a memory of the processor, carrying out alignment processing on the group of the data to be processed through the processor, determining two-dimensional data corresponding to the depth data in the group of the data to be processed, and carrying the next group of the data to be processed into the memory for alignment processing until each group of the data to be processed is aligned;

the sum of the data volume of a group of memory storage data is not larger than the memory size of the processor, and the group of memory storage data comprises a group of to-be-processed data and processed data obtained after the group of to-be-processed data is aligned.

Optionally, the image processing system is further configured to:

when the ith group of data to be processed is conveyed, acquiring the row index overlapping range of the ith group of data to be processed and the (i-1) th group of data to be processed, reserving the overlapping data in the memory, and conveying the data to be conveyed to the memory;

the overlapped data is data corresponding to the overlapped range of the line index in the i-1 th group of the data to be processed, and the data to be transported is data outside the overlapped range of the line index in the i-th group of the data to be processed.

Optionally, the calibration information stored in the memory includes: an internal reference of the depth camera, an internal reference of the two-dimensional camera, a rotation matrix between the depth camera and the two-dimensional camera, and a translation matrix between the depth camera and the two-dimensional camera.

Optionally, the depth measurement range stored in the memory includes: the maximum detection value and the minimum detection value of the depth camera.

Optionally, the determining, according to the calibration information and the depth measurement range, a corresponding row index range of each row of the depth image in the two-dimensional image includes:

respectively acquiring a maximum row index and a minimum row index corresponding to each row of the depth image according to the internal reference of the depth camera, the internal reference of the two-dimensional camera, the rotation matrix, the translation matrix, the maximum detection value and the minimum detection value;

acquiring the line index range corresponding to each line in the depth image according to the maximum line index and the minimum line index;

the line index range corresponding to one to-be-aligned line in the depth image is used for indicating the range of a target aligned line in the two-dimensional image, target aligned pixel points in the two-dimensional image belong to the target aligned line, and the target aligned pixel points correspond to any to-be-aligned pixel point in the to-be-aligned line.

A second aspect of the present invention provides an image processing method, wherein the image processing method is applied to any one of the image processing systems, and the method includes:

acquiring a depth image of a target scene through a depth camera and storing the depth image into a memory;

acquiring a two-dimensional image of a target scene through a two-dimensional camera and storing the two-dimensional image into a memory;

acquiring calibration information between the depth camera and the two-dimensional camera and a depth measurement range of the depth camera, which are pre-stored in the memory;

and determining a corresponding row index range of each row of the depth image in the two-dimensional image according to the calibration information and the depth measurement range through a processor, and extracting at least one row of depth data in the depth image and at least one corresponding row of two-dimensional data in the two-dimensional image according to the row index range to perform alignment processing to determine two-dimensional data corresponding to the depth data.

A third aspect of the present invention provides an intelligent terminal, where the intelligent terminal includes a memory, a processor, and an image processing program stored in the memory and executable on the processor, and the image processing program implements the steps of the image processing method when executed by the processor.

A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon an image processing program which, when executed by a processor, implements the steps of the image processing method.

As can be seen from the above, the image processing system provided in the aspect of the present invention includes: a depth camera configured to acquire a depth image of a target scene; a two-dimensional camera configured to acquire a two-dimensional image of a target scene; a memory configured to store the depth image, a two-dimensional image, calibration information between the depth camera and the two-dimensional camera, and a depth measurement range of the depth camera; and the processor is configured to determine a corresponding line index range of each line of the depth image in the two-dimensional image according to the calibration information and the depth measurement range, and extract at least one line of depth data in the depth image and at least one corresponding line of two-dimensional data in the two-dimensional image according to the line index range to perform alignment processing to determine two-dimensional data corresponding to the depth data. Compared with the scheme that all data in two images need to be processed at one time and the data cannot be subjected to batch calculation and processing in the prior art, the image processing system can acquire the corresponding row index range of each row in the depth image in the two-dimensional image according to the calibration information and the depth measurement range, so that the data needing to be processed are grouped according to the corresponding row index range, and a group of data to be processed is aligned each time. Therefore, the processor does not need to process all data at one time, and the data calculation amount is small each time, so that the processing efficiency is improved. Meanwhile, data grouping can be reasonably carried out according to the memory size of the processor, so that the computing capacity and the storage capacity of the processor are reasonably utilized, the method is favorable for being suitable for processors with different performances, and the applicability of the image alignment processing process is improved. And further, the operation flow (such as face recognition) after the image alignment is conveniently carried out, and the user experience is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.

Fig. 1 is a schematic structural diagram of an image processing system according to an embodiment of the present invention;

fig. 2 is a schematic diagram of a mapping relationship between a depth image and an RGB image according to an embodiment of the present invention;

fig. 3 is a schematic diagram of a mapping relationship between a depth image and an RGB image according to an embodiment of the present invention;

FIG. 4 is a flowchart illustrating an image processing method according to an embodiment of the present invention;

fig. 5 is a schematic block diagram of an internal structure of an intelligent terminal according to an embodiment of the present invention.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be further understood that the term "and/or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items and includes such combinations.

As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when …" or "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted depending on the context to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".

The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings of the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those specifically described and will be readily apparent to those of ordinary skill in the art without departing from the spirit of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.

With the development of science and technology, the acquisition and application of image information become more and more extensive. In some application scenarios, it is necessary to perform alignment processing on the acquired multiple images so as to acquire corresponding data information in the multiple images. For example, in an application scene of 3D vision, after a depth image is acquired from a 3D camera module, in order to better apply depth data in the depth image, it is necessary to perform corresponding image processing on a depth image corresponding to one scene and another two-dimensional image (for example, an RGB image, a grayscale image, an IR image, etc.) corresponding to the scene so that the depth image is aligned with the two-dimensional image, that is, depth information in the depth image is mapped onto pixels of the corresponding two-dimensional image, thereby facilitating operations such as 3D recognition using depth information. For example, the Depth image is aligned with an RGB image acquired from a two-dimensional camera module (e.g., an RGB camera module), and the acquired Depth information is mapped onto pixels of the Color image (i.e., D2C, Depth to Color) for better information utilization.

In the prior art, when two (or more) images are aligned, all data in the two images need to be processed at one time, and the data cannot be calculated and processed in batches. The problem of the prior art is that a large amount of data needs to be processed simultaneously when image alignment processing is performed, which is not beneficial to improving the processing efficiency and the accuracy of image alignment. Meanwhile, the huge calculation amount requires high performance of hardware equipment for processing, which is not beneficial to image alignment processing under the scene of low performance of hardware equipment, namely, the applicability of image alignment processing is influenced.

Specifically, limited by the amount of calculation, the image alignment processing calculation in the currently common 3D device needs to be calculated on an additionally arranged host or a CPU of the mobile terminal, and cannot be directly calculated on an image processor (such as a digital signal processor) corresponding to the 3D device. If the host is used, the use scene of the 3D equipment is limited, and if the CPU of the mobile terminal is used, the load of the CPU is increased, and the progress of other tasks is influenced. Meanwhile, the D2C algorithm in the prior art can only calculate according to the data collected by a specific 3D camera module, and different 3D camera modules have individual differences and installation differences, so that the existing D2C algorithm cannot adaptively calculate the data collected by different 3D camera modules.

In order to solve at least one of the problems of the prior art, the present invention provides an image processing system comprising: a depth camera configured to acquire a depth image of a target scene; a two-dimensional camera configured to acquire a two-dimensional image of a target scene; a memory configured to store the depth image, a two-dimensional image, calibration information between the depth camera and the two-dimensional camera, and a depth measurement range of the depth camera; and the processor is configured to determine a corresponding line index range of each line of the depth image in the two-dimensional image according to the calibration information and the depth measurement range, and extract at least one line of depth data in the depth image and at least one corresponding line of two-dimensional data in the two-dimensional image according to the line index range to perform alignment processing to determine two-dimensional data corresponding to the depth data. Compared with the scheme that all data in two images need to be processed at one time and the data cannot be subjected to batch calculation and processing in the prior art, the image processing system can acquire the corresponding row index range of each row in the depth image in the two-dimensional image according to the calibration information and the depth measurement range, so that the data needing to be processed are grouped according to the corresponding row index range, and a group of data to be processed is aligned each time. Therefore, the processor does not need to process all data at one time, and the data calculation amount is small each time, so that the processing efficiency is improved. Meanwhile, data grouping can be reasonably carried out according to the memory size of the processor, so that the computing capacity and the storage capacity of the processor are reasonably utilized, the method is favorable for being suitable for processors with different performances, and the applicability of the image alignment processing process is improved. And further, the operation flow (such as face recognition) after the image alignment is conveniently carried out, and the user experience is improved.

As shown in fig. 1, an embodiment of the present invention provides an image processing system, including:

a depth camera 10 configured to acquire a depth image of a target scene;

a two-dimensional camera 20 configured to acquire a two-dimensional image of a target scene;

a memory 30 configured to store the depth image, a two-dimensional image, calibration information between the depth camera and the two-dimensional camera, and a depth measurement range of the depth camera;

and the processor 40 is configured to determine a line index range corresponding to each line of the depth image in the two-dimensional image according to the calibration information and the depth measurement range, and extract at least one line of depth data in the depth image and at least one line of two-dimensional data corresponding to the two-dimensional image according to the line index range to perform alignment processing to determine two-dimensional data corresponding to the depth data.

The depth image and the two-dimensional image are images which need to be aligned, and the target scene is a scene for image shooting. In this embodiment, an example of performing alignment processing on two images is described, and in an actual use process, the alignment processing may be performed on a plurality of images based on the image processing system, that is, more than one two-dimensional image, for example, a first alignment image is obtained after aligning a depth image and a first two-dimensional image, and a second alignment image is obtained by performing alignment processing on a second two-dimensional image and the first alignment image, so that alignment processing between three images is realized.

In this embodiment, the depth image is aligned with the two-dimensional image (e.g., an RGB image), so that depth information in the depth image can be mapped to corresponding pixels of the two-dimensional image, which is beneficial to assist the two-dimensional image with depth information to perform three-dimensional information recognition (e.g., 3D face recognition).

In this embodiment, the depth image and the two-dimensional image are captured by different cameras, the depth image is captured by the depth camera 10, and the two-dimensional image is captured by the two-dimensional camera 20. Specifically, the two-dimensional image is an RGB image to be aligned and is obtained by shooting with an RGB camera. Based on the image processing system in the embodiment, the depth image to be aligned and the RGB image to be aligned are subjected to image alignment processing, and the depth information is mapped onto pixels of a color image (RGB image to be aligned), so that image recognition is further performed based on the color image, information of a corresponding environment or a photographed object is acquired, and utilization of the depth information is favorably realized.

Specifically, the calibration information includes an internal reference of the depth camera 10, an internal reference of the two-dimensional camera 20, a rotation matrix between the depth camera 10 and the two-dimensional camera 20, and a translation matrix between the depth camera 10 and the two-dimensional camera 20, and the depth measurement range includes a maximum detection value and a minimum detection value of the depth camera 10.

In this embodiment, the alignment process may be performed on the depth image to be aligned and the RGB image to be aligned based on the calibration information. Meanwhile, the line index range of each line in the depth image, which may correspond to the two-dimensional image, may be determined based on the maximum detection value and the minimum detection value corresponding to the depth camera 10, so that the depth image and the two-dimensional image may be aligned in rows, that is, batch processing of data may be implemented.

The line index range corresponding to one line in the depth image is the line index range of each pixel point corresponding to the pixel point in the line in the two-dimensional image, wherein the line index range is possibly existed in each pixel point. For example, pixel points corresponding to each pixel point in the 1 st row of the depth image may be distributed between the 1 st row and the 3 rd row of the two-dimensional image, and the row index range corresponding to the 1 st row of the depth image is 1 to 3 rows.

It should be noted that the depth measurement range may be determined according to the maximum working distance and the minimum working distance of the depth camera 10.

In this embodiment, the determining, according to the calibration information and the depth measurement range, a corresponding row index range of each row of the depth image in the two-dimensional image includes: respectively acquiring a maximum row index and a minimum row index corresponding to each row of the depth image according to the internal reference of the depth camera 10, the internal reference of the two-dimensional camera 20, the rotation matrix, the translation matrix, the maximum detection value and the minimum detection value; and acquiring a row index range corresponding to each row in the depth image based on the maximum row index and the minimum row index.

The line index range corresponding to a line to be aligned (i.e., any line in the depth image) in the depth image is used for indicating the range of a target alignment line (i.e., a line corresponding to the line to be aligned) in the two-dimensional image, target alignment pixel points in the two-dimensional image belong to the target alignment line, and the target alignment pixel points correspond to any one pixel point to be aligned in the line to be aligned.

Specifically, for a row to be aligned, all pixel points in the row are pixel points to be aligned, each pixel point to be aligned has a target alignment pixel point in the two-dimensional image, and all target alignment pixel points corresponding to the row to be aligned belong to the row index range in the two-dimensional image. For example, there are 640 pixel points in line 1 in the depth image, and the 640 pixel points have corresponding 640 target alignment pixel points in the two-dimensional image. The row index range (e.g., rows 1 to 3) corresponding to the 640 target alignment pixel points in the two-dimensional image is used as the row index range of the row to be aligned. Wherein, the maximum row index corresponding to the row to be aligned is 3, and the minimum row index is 1.

Specifically, there are deviations in distance and rotation angle between the optical centers of the depth camera (e.g., 3D camera module) and the two-dimensional (e.g., RGB) camera (module), so when mapping the depth information onto the RGB image, the depth information needs to be converted into a world coordinate system first, and then converted into an RGB image pixel coordinate system from the world coordinate system, so that D2C is implemented.

In this embodiment, a coordinate conversion relationship between a depth image pixel coordinate system and an RGB image pixel coordinate system is constructed, as shown in the following formula (1):

wherein u is _rgb And v _rgb Respectively representing the abscissa and ordinate, u, of the image in the RGB image pixel coordinate system _ir And v _ir Respectively representing the abscissa and the ordinate of the image in the pixel coordinate system of the depth image, R _ir2rgb Representing a rotation matrix from a depth (IR) camera to an RGB camera, T _ir2rgb Representing a translation matrix from a depth camera to an RGB camera, K _ir And K _rgb Representing the internal parameters of a depth camera and an RGB camera respectively,

is K _ir The depth represents the coordinate of (u) in the depth image _ir ，v _ir ) Depth value of the position of (1), Z _rgb Then (u) in the depth image _ir ，v _ir ) Coordinate points corresponding to (u) in RGB image _rgb ，v _rgb ) A depth value of the coordinate location.

It should be noted that, in the above formula (1), the coordinate of one pixel point to be aligned in the depth image is (u) _ir ，v _ir ) And the coordinates of the target alignment pixel point corresponding to the pixel point to be aligned in the RGB image are (u) _rgb ，v _rgb ). The D2C algorithm maps depth values of a depth imageThe RGB image is shot to the corresponding pixel point (target alignment pixel point) on the RGB image, so that the RGB image can be changed into RGBD data (namely the data combining RGB color and depth information), and the value of the RGB pixel does not need to be considered in the whole algorithm. In an application scenario, the RGB camera and the depth camera respectively take images, the pixel coordinates of the same object in the two images are different, that is, there is a deviation when the two images are overlapped, D2C needs to overlap the depth image with RGB to make the same object in the two images overlap, Z is mentioned above _rgb Then is (u) _ir ，v _ir ) Depth mapping of coordinate points to (u) _rgb ，v _rgb ) The depth at a point, i.e. the depth value of a pixel point on the aligned RGB image, i.e. (u) _ir ，v _ir ) And (4) processing data corresponding to the coordinate points after the alignment processing (the processed data comprises RGB color values and depth values of each point in the aligned image).

In this embodiment, each pixel point in the depth image is aligned with each pixel point in the two-dimensional image, and the corresponding depth information is mapped. Thus (u) _ir ，v _ir ) Is determined (determined according to the coordinates of the corresponding pixel points to be aligned), and calibration information between the depth image and the two-dimensional image and the depth measurement range corresponding to the depth image are obtained in advance, so that in the formula (1), only u is used _rgb 、v _rgb And Z _rgb Is an unknown quantity that needs to be calculated. In this embodiment, pixel points in the depth image are aligned line by line, so v does not need to be considered _rgb The corresponding column index range. In the actual use process, the pixel points in the depth image can be aligned column by column, and only v needs to be considered _rgb Corresponding column index range without considering u _rgb The corresponding range of the row index may also refer to the embodiment, and is not described herein again.

In this embodiment, the line-by-line processing is taken as an example for explanation, and the unknown quantity Z in the formula (1) may be set _rgb Move to the left of the equation, thus aligning for a certain line to be aligned in the depth image (u (th) th _ir Line), respectively obtaining u according to the maximum value and the minimum value of depth _rgb And taking the corresponding maximum value and minimum value as the maximum line index and minimum line index of the line to be aligned.

In an application scenario, taking a common depth image as an example, the resolution of the common depth image is 640 × 480, and then u is _ir Is in the range of 0 to 639 (counting from line 0), v _ir Is in the range of 0 to 479 (counting from column 0). The maximum and minimum values of the depth values depth (i.e. the maximum and minimum detection values in the depth image described above) may be determined according to the working distance of the depth camera.

Specifically, based on the principle of D2C shown in the above formula (1), the corresponding line index u in the RGB image _rgb And column index v _rgb Are all related to depth. Fig. 2 is a schematic diagram of a mapping relationship between a depth image and an RGB image according to an embodiment of the present invention, as shown in fig. 2, when depth is a minimum detection value, a corresponding minimum line index may be obtained according to the above formula (1), and when depth is a maximum detection value, a corresponding maximum line index may be obtained according to the above formula (1), so that a line index range in a two-dimensional image corresponding to a line to be aligned in the depth image may be obtained, and data that needs to be transported each time may be determined.

In this embodiment, for each line in the depth image, a corresponding line index range may be calculated in advance according to a working distance of the depth camera (i.e., a detection distance range of the depth camera, which is determined according to a specific depth camera and an application scene setting) and the camera calibration information. And a corresponding index table can be established according to the minimum row index (color _ y _ min) and the maximum row index (color _ y _ max) corresponding to each row, so as to determine the row index range corresponding to each row in the depth image. It should be noted that fig. 2 shows a corresponding mapping relationship between a depth image and an RGB image, and in an actual use process, an aligned image may be considered to be overlapped with the RGB image, that is, fig. 2 may also be used as a corresponding mapping relationship between a depth image and an aligned image.

In this embodiment, an algorithm (or program) for calculating the row index range may be preset, and the rotation matrix R between the cameras _ir2rgb And move in the horizontal directionMatrix T _ir2rgb Depth camera internal reference K _ir And RGB Camera internal reference K _rgb Inputting corresponding algorithm, and simultaneously acquiring maximum detection value depth based on working distance of depth camera _max And minimum probe value depth _min And inputs the above algorithm.

In an application scenario, the maximum detection value and the minimum detection value may also be determined in advance by setting the depth camera and the application scenario thereof, for example, in a corresponding shooting area, a closest distance and a farthest distance of a shot object (such as a user needing face detection) from the depth camera are predetermined. In another application scenario, depth value analysis can be performed on the depth image, and specific maximum detection values and minimum detection values in the shot depth image are obtained for calculation, so that calculation accuracy is improved, better data transfer is performed, and image alignment efficiency is improved.

Specifically, the maximum row index and the minimum row index corresponding to each row in the depth image are sequentially calculated, so that a row index range corresponding to each row is obtained. For a line to be aligned in the depth image, u _ir Is definite, traverse v _ir Each value (e.g., 0 to 639), depth will be individually set _max And depth _min And substituting the minimum indexes into the formula (1) to calculate 640 candidate minimum indexes and 640 candidate maximum indexes of the row to be aligned, taking the minimum value in all the candidate minimum indexes as the minimum row index corresponding to the row to be aligned, and taking the maximum value in all the candidate maximum indexes as the maximum row index corresponding to the row to be aligned. In this way, the maximum row index and the minimum row index corresponding to each row in the depth image can be obtained, so that the row index range corresponding to each row is obtained. It should be noted that, each row and the row index range corresponding to each row may be further made into a lookup table, which facilitates storage and lookup, and what rows a row in the depth image may correspond to in the two-dimensional image may be quickly obtained according to the lookup table, which is beneficial to data grouping and carrying.

It should be noted that, for convenience of calculation, in the present embodiment, each row to be aligned is calculated by usingIs the same depth _max And depth _min . In one embodiment, for more precise mapping, the depth measurement range may include the maximum detection value and the minimum detection value of each line in the depth image, so that the maximum line index, the minimum line index and the line index range corresponding to each line are obtained by calculation according to the maximum detection value and the minimum detection value of each line.

In the present embodiment, the processor 40 is an image processor, and is specifically an image processor for performing image alignment processing, and the image processor may be a digital signal processor. Specifically, by using the system in this embodiment, all data (including all data in the depth image and the two-dimensional image) that needs to be subjected to image alignment processing may be subjected to grouping (batch) processing, so that the hardware performance of the corresponding image processor is not required to be very high, and thus the processing may be directly performed based on the digital information processor in the depth camera (or the 3D camera module), and no additional host or mobile terminal is required, which may not only improve the computational efficiency of the D2C algorithm, but also reduce the system consumption resources, reduce the hardware power consumption, and improve the applicability. Therefore, unlike the prior art in which the host and the CPU of the mobile terminal are used, the calculation process of the image alignment processing is performed based on the digital signal processor in this embodiment, so that the load of the CPU of the mobile terminal can be reduced and the power consumption of the entire system can be reduced.

In this embodiment, the extracting at least one line of depth data in the depth image according to the line index range and performing an alignment process on at least one line of two-dimensional data corresponding to the two-dimensional image to determine two-dimensional data corresponding to the depth data includes: simultaneously extracting multiple lines of depth data in the depth image and multiple lines of two-dimensional data corresponding to the two-dimensional image according to the memory size and the line index range of the processor 40, and forming multiple groups of data to be processed; and respectively aligning each group of data to be processed, and determining two-dimensional data corresponding to each line of depth data.

Specifically, in the process of performing image alignment processing, there are a large number of memory access operations, the D2C calculation process performs calculation on the DSP, and the depth image and the two-dimensional image are not stored in the on-chip memory of the DSP, that is, the data to be accessed is not in the on-chip memory of the DSP, and then the corresponding data needs to be carried to the on-chip memory of the DSP. Meanwhile, the on-chip memory of the DSP is limited in size and generally cannot accommodate all data in the depth image and the two-dimensional image, so that batch transportation and calculation of data are required, and it is required to ensure that data in a target alignment line corresponding to a line to be aligned is also successfully transported in a batch of data transported each time. In this embodiment, data needs to be transferred from a Synchronous Dynamic Random Access Memory (SDRAM) to the on-chip Memory of the DSP.

In an application scenario, a user is required to judge that several rows of data are carried at each time, and manually input a corresponding instruction to realize the back-and-forth movement of data in the on-chip memory and the SDARM, and different instructions are required to be input to acquire data of different sizes and types, so that the operation is inconvenient, and the improvement of the image processing efficiency is not facilitated. Meanwhile, the user is required to manually calculate the data volume of each transportation, so that the on-chip memory is difficult to be fully utilized, the number of times of data transportation is increased, and the calculation time is consumed.

Therefore, in this embodiment, preferably, a set of to-be-processed data is formed by extracting a line of depth data in the depth image and at least one line of corresponding two-dimensional data in the two-dimensional image according to the line index range, a set of to-be-processed data is transported to a memory of the processor 40, the processor 40 performs alignment processing on the set of to-be-processed data, two-dimensional data corresponding to the depth data in the set of to-be-processed data is determined, and then the next set of processed data is transported to the memory for alignment processing until each set of to-be-processed data completes alignment processing. The sum of the data amounts of a group of memory storage data is not greater than the memory size of the processor 40, and the group of memory storage data includes a group of to-be-processed data and processed data obtained by aligning the group of to-be-processed data.

Furthermore, it is also possible to simultaneously transfer multiple sets of data to be processed to the processor 40 for alignment. It should be noted that, limited by the memory size of the processor 40, only a few lines to a few tens of lines of data can be transferred to the on-chip memory for processing each time, so that the depth image and the two-dimensional image can be divided in advance according to actual requirements to determine the data to be transferred each time. Specifically, in this embodiment, the number of rows in the two-dimensional image corresponding to one row in the depth image may be determined according to a predetermined row index range, so as to determine the amount of data that needs to be processed corresponding to one row in the depth image, and thus, data grouping may be performed (for example, determining several rows in the depth image as a group). Thus, it can be determined that several rows need to be calculated each time and several rows of data can be transferred to the on-chip memory based on the predetermined row index range. Thus, the amount of calculation and the number of times of moving data can be reduced. Meanwhile, the image processing system can also be suitable for image data of different image data (for example, different resolutions) acquired by different camera modules, and different image data are appropriately grouped and transported, so that the D2C algorithm is suitable for the difference between different modules, and the hardware performance of the DSP can be fully utilized without excessive manual intervention.

In this embodiment, the depth image and the two-dimensional image are divided according to the memory size and the line index range to obtain a plurality of sets of data to be processed. It should be noted that each set of the to-be-processed data includes at least one row of depth data and multiple corresponding rows of two-dimensional data, and adjacent rows of the to-be-processed data share part of the two-dimensional data. The shared part two-dimensional data refers to that the two-dimensional data corresponding to the to-be-processed data of the adjacent rows possibly have an overlapping part, and the overlapping part can be retained and reused in the alignment process and does not need to be repeatedly carried.

For example, the group of images to be processed includes the 1 st line and the 2 nd line in the depth image, where the line index range corresponding to the 1 st line in the depth image is 1 to 3 lines of the two-dimensional image, and the line index range corresponding to the 2 nd line in the depth image is 2 to 5 lines of the two-dimensional image, then the data to be processed includes all data of the 1 st to 5 th lines in the two-dimensional image, and the first line depth data and the second line depth data may share the 2 nd to 3 rd line two-dimensional data when performing the alignment processing. Therefore, the data required to be used in each image alignment processing can be guaranteed to be carried into the on-chip memory, and the waste of data volume caused by repeated carrying is avoided.

Meanwhile, the storage space of the on-chip memory is also considered, that is, the sum of the data amount of a group of memory storage data is not larger than the size of the on-chip memory, and the group of memory storage data comprises a group of to-be-processed data and processed data corresponding to the group of to-be-processed data. It should be noted that, not only input data (a set of data to be processed) but also output data (Aligned image data, Aligned _ Depth) need to be carried on the DSP, that is, processed data corresponding to the set of data to be processed, the two pieces of data used in calculation need to be stored in an on-chip memory at the same time, and the size of the on-chip memory is fixed, so that the sum of data amounts of a set of memory stored data needs to be limited to be not larger than the size of the on-chip memory.

In an application scenario, each set of the to-be-processed data includes first to-be-processed data and second to-be-processed data, the first to-be-processed data is data corresponding to a to-be-processed line in the depth image, the second to-be-processed data is all data in a line index range corresponding to the to-be-processed line in the two-dimensional image, and the to-be-processed line includes at least one line in the depth image. Correspondingly, at least one line of depth data in the depth image and at least one line of corresponding two-dimensional data in the two-dimensional image can be extracted according to the line index range to form at least one group of data to be processed, which specifically comprises: the depth image and the two-dimensional image are divided and a plurality of sets of data to be processed are extracted according to the memory size and the line index range of the processor 40, wherein each set of data to be processed comprises first data to be processed and second data to be processed, the first data to be processed is data corresponding to a line to be processed in the depth image, the second data to be processed is all data in the line index range corresponding to the line to be processed in the two-dimensional image, and the line to be processed comprises at least one line in the depth image.

The data amount of the corresponding output data can be acquired based on the data amount of the input data, and the data per conveyance is limited. For example, in an application scenario, a line-to-line data amount of a depth image and a line-to-line data amount of an RGB image may be preset, and output data obtained after alignment is data including RGB image information and depth information, that is, a data type is determined, and one pixel point in the depth image may correspond to one output pixel point data, and a line number and a data amount of the output data may be determined according to input data to be processed. In this embodiment, it is only necessary to ensure that the sum of the size of the input data and the size of the output data does not exceed the size of the on-chip memory.

For different camera modules, the internal parameters of the camera are different, so that the corresponding row index ranges of each row in the acquired depth image are also different, and the number of rows of the data to be processed, which can be carried by different camera modules at one time, is also different. For example, data generated by a camera module can be moved 8 lines at a time, and 480 lines of data need to be calculated 60 times. Another camera module can transport 10 lines of data at a time, and 480 lines of data need only be transported 48 times. The smaller the number of times of conveyance, the smaller the overhead of the system. Based on the scheme in the embodiment, the performance of the DSP can be fully utilized, data grouping can be reasonably performed, multiple sets of data to be processed can be obtained, and each set of data to be processed can be sequentially processed, so that the efficiency of image alignment processing can be improved while the memory and the processing capability of the DSP are reasonably utilized.

It should be noted that different image resolutions may also result in different numbers of transfers, i.e. the larger the number of pixels in a row, the fewer the number of rows that can be transferred at a time, e.g. a 1280 x 960 image may only be able to transfer 4 rows of data at a time. Based on the image processing system in the embodiment, images with different resolutions acquired by different modules can be adapted in a self-adaptive manner, so that the utilization rate of corresponding algorithm codes is improved, a user does not need to modify corresponding codes when switching the modules and the resolutions, the improvement of the applicability of the image processing process is facilitated, and the image alignment processing in different scenes is facilitated.

Meanwhile, the line index range calculated in this embodiment is an approximate range calculated based on the maximum detection value and the minimum detection value, and is not an accurate range calculated based on the specific depth value of each pixel point in the depth image, so that overlapping line index ranges may occur in several consecutive lines to be aligned (there may be no overlapping line index ranges), that is, corresponding adjacent line depth data may share two-dimensional data in the overlapping line index ranges. However, if the corresponding data in the overlapping row index ranges are repeatedly transported, time is wasted. Therefore, in this embodiment, the system is further configured to:

when the ith group of data to be processed is carried, acquiring the row index overlapping range of the ith group of data to be processed and the (i-1) th group of data to be processed, reserving overlapping data in the on-chip memory, and carrying the data to be carried to the on-chip memory;

the overlapped data is data corresponding to the overlapped range of the line indexes in the i-1 th group of data to be processed, and the data to be carried is data outside the overlapped range of the line indexes in the i-th group of data to be processed.

Fig. 3 is a schematic diagram of a mapping relationship between a depth image and an RGB image according to an embodiment of the present invention, and in fig. 3, it is assumed that a set of data to be processed in the depth image includes one line of data in the depth image and multiple lines of data in the RGB image. As shown in fig. 3, the line index ranges of the i-1 th line of data in the depth image and the i-th line of data in the depth image overlap, and the corresponding overlapping data is data between the dotted lines in the RGB image in fig. 3. When the i-1 th group of data to be processed is processed, the overlapped data is carried to the on-chip memory, and when the i-th group of data to be processed is processed, the overlapped data is also used. Therefore, based on the scheme in this embodiment, the overlapped data in the on-chip memory may be retained, or the overlapped data may be carried on the on-chip memory according to actual requirements (the carrying speed inside the on-chip memory is much faster than the carrying speed between the SDRAM and the on-chip memory), so that redundant carrying operations in the data carrying process may be reduced, which is beneficial to reducing the data carrying time, and increasing the operating speed of the whole system, thereby improving the efficiency of image alignment processing.

In an application scenario, the image processing process can be performed in combination with a Single Instruction Multiple Data (SIMD) technology, so as to further increase the speed of image Data alignment processing.

It should be noted that in this embodiment, image alignment processing is performed in batches on the digital signal processor, and in the actual use process, image alignment processing may also be performed in batches by a CPU, or image alignment processing may be performed in batches by using an FPGA, a digital chip, or the like, so as to reasonably and sufficiently use the image processor.

As can be seen from the above, in this embodiment, prior information (including a row index range corresponding to each row in a depth image) of a 3D camera module is obtained, data is grouped according to the prior information, multiple groups of data to be processed are obtained, and the groups of data to be processed are sequentially carried and processed, so that the performance of a digital signal processor is fully utilized under the condition that it is ensured that data to be stored at each time does not exceed an on-chip internal memory of the digital signal processor, and the data carrying times are reduced, thereby reducing the power consumption of a hardware system, and enabling the 3D camera module to be applicable to various scenes. Meanwhile, the system can be adaptive to different 3D camera modules, the difference of calibration parameters of different modules can be considered, configuration is carried out according to specific calibration parameters, the hardware performance of the DSP is fully utilized, and parallel processing is carried out on data based on the DSP.

Meanwhile, in the embodiment, overlapped data in the data to be carried is also considered, repeated carrying of the overlapped data is avoided, the processing time is further reduced, the efficiency of image alignment processing is improved, an operation flow (such as face recognition) after image alignment is conveniently carried out, and the use experience of a user is favorably improved.

As shown in fig. 4, in correspondence with the image processing system, an embodiment of the present invention further provides an image processing method, which is applied to any one of the image processing systems, and the method includes:

s100, acquiring a depth image of a target scene through a depth camera and storing the depth image into a memory;

s200, acquiring a two-dimensional image of a target scene through a two-dimensional camera and storing the two-dimensional image into a memory;

step S300, obtaining calibration information between the depth camera and the two-dimensional camera and a depth measurement range of the depth camera, which are pre-stored in the memory;

step S400, determining, by the processor, a corresponding line index range of each line of the depth image in the two-dimensional image according to the calibration information and the depth measurement range, and extracting at least one line of depth data in the depth image and at least one line of two-dimensional data corresponding to the two-dimensional image according to the line index range to perform an alignment process to determine two-dimensional data corresponding to the depth data.

The depth image and the two-dimensional image are images which need to be aligned, and the target scene is a scene for image shooting. In this embodiment, an example of performing alignment processing on two images is described, and in an actual use process, alignment processing may be performed on a plurality of images based on an image processing method, that is, more than one two-dimensional image, for example, a first alignment image is obtained after aligning a depth image and a first two-dimensional image, and a second alignment image is obtained by performing alignment processing on a second two-dimensional image and the first alignment image, so that alignment processing between three images is realized.

In this embodiment, the depth image and the two-dimensional image are obtained by different camera shooting, the depth image is obtained by the depth camera shooting, and the two-dimensional image is obtained by the two-dimensional camera shooting. Specifically, the two-dimensional image is an RGB image to be aligned and is obtained by shooting with an RGB camera. Based on the image processing method in the embodiment, the depth image to be aligned and the RGB image to be aligned are subjected to image alignment processing, and the depth information is mapped onto pixels of a color image (RGB image to be aligned), so that image recognition is further performed based on the color image, information of a corresponding environment or a photographed object is acquired, and utilization of the depth information is favorably realized.

Specifically, the calibration information includes internal parameters of the depth camera, internal parameters of the two-dimensional camera, a rotation matrix between the depth camera and the two-dimensional camera, and a translation matrix between the depth camera and the two-dimensional camera, and the depth measurement range includes a maximum detection value and a minimum detection value of the depth camera.

In this embodiment, the alignment process may be performed on the depth image to be aligned and the RGB image to be aligned based on the calibration information. Meanwhile, the line index range of each line in the depth image, which may correspond to the two-dimensional image, can be determined based on the maximum detection value and the minimum detection value in the depth image, so that the depth image and the two-dimensional image are aligned in a line-by-line manner conveniently, that is, batch processing of data is realized.

The line index range corresponding to one line in the depth image is the line index range of each pixel point corresponding to the pixel point in the line in the two-dimensional image, wherein the line index range is possibly existed in each pixel point.

In this embodiment, the processor is an image processor, and particularly an image processor for performing image alignment processing, and the image processor may be a digital signal processor. Specifically, based on the method in this embodiment, all data (including all data in the depth image and the two-dimensional image) that needs to be subjected to image alignment processing may be subjected to grouping (batch) processing, so that the hardware performance of the corresponding image processor is not required to be very high, and thus the data may be directly processed based on the digital information processor in the depth camera (or the 3D camera module), and no additional host or mobile terminal is required, which may not only improve the computational efficiency of the D2C algorithm, but also reduce the system consumption resources, reduce the power consumption of the hardware, and improve the applicability. Therefore, unlike the prior art in which the host and the CPU of the mobile terminal are used, the calculation process of the image alignment processing is performed based on the digital signal processor in this embodiment, so that the load of the CPU of the mobile terminal can be reduced and the power consumption of the entire system can be reduced. Based on the above embodiment, the present invention further provides an intelligent terminal, and a schematic block diagram thereof may be as shown in fig. 5. The intelligent terminal comprises a processor and a memory. The memory of the intelligent terminal comprises an image processing program, and the memory provides an environment for the running of the image processing program. The image processing program, when executed by a processor, implements the steps of any of the image processing methods described above. It should be noted that the intelligent terminal may further include other functional modules or units, which are not specifically limited herein.

It will be understood by those skilled in the art that the block diagram shown in fig. 5 is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation of the intelligent terminal to which the solution of the present invention is applied, and in particular, the intelligent terminal may include more or less components than those shown in the figure, or combine some components, or have a different arrangement of components.

In one embodiment, the image processing program when executed by the processor performs the following operations:

respectively acquiring a depth image and a two-dimensional image of a target scene by using a depth camera and a two-dimensional camera;

acquiring calibration information between the depth camera and the two-dimensional camera and a depth measurement range measured by the depth camera, wherein the calibration information is pose calibration information between the depth camera and the two-dimensional camera;

acquiring a corresponding row index range of each row of the depth image in the two-dimensional image according to the calibration information and the depth measurement range;

and extracting at least one line of depth data in the depth image and at least one line of corresponding two-dimensional data in the two-dimensional image according to the line index range to form at least one group of data to be processed, and aligning the data to be processed to determine the two-dimensional data corresponding to the depth data.

The embodiment of the present invention further provides a computer-readable storage medium, where an image processing program is stored on the computer-readable storage medium, and when the image processing program is executed by a processor, the image processing program implements the steps of any one of the image processing methods provided in the embodiment of the present invention.

It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned functions may be distributed as different functional units and modules according to needs, that is, the internal structure of the apparatus may be divided into different functional units or modules to implement all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.

Those of ordinary skill in the art would appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

In the embodiments provided in the present invention, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the above modules or units is only one logical division, and the actual implementation may be implemented by another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed.

The integrated modules/units described above, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium and can implement the steps of the embodiments of the method when the computer program is executed by a processor. The computer program includes computer program code, and the computer program code may be in a source code form, an object code form, an executable file or some intermediate form. The computer readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signal, telecommunication signal, software distribution medium, etc. It should be noted that the contents of the computer-readable storage medium can be increased or decreased as required by the legislation and patent practice in the jurisdiction.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art; the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein.

Claims

1. An image processing system, comprising:

a depth camera configured to acquire a depth image of a target scene;

and the processor is configured to determine a corresponding row index range of each row of the depth image in the two-dimensional image according to the calibration information and the depth measurement range, and extract at least one row of depth data in the depth image and at least one corresponding row of two-dimensional data in the two-dimensional image according to the row index range to perform alignment processing to determine two-dimensional data corresponding to the depth data.

2. The image processing system of claim 1, wherein the extracting at least one row of depth data in the depth image according to the row index range and at least one corresponding row of two-dimensional data in the two-dimensional image for performing an alignment process to determine two-dimensional data corresponding to the depth data comprises:

and dividing the depth image and the two-dimensional image according to the memory size of the processor and the line index range and extracting multiple groups of data to be processed, wherein each group of data to be processed comprises at least one line of depth data and multiple corresponding lines of two-dimensional data, and adjacent lines of depth data share part of the two-dimensional data.

3. The image processing system of claim 2, wherein the determining the two-dimensional data corresponding to the depth data by performing the alignment process on the data to be processed comprises:

the method comprises the steps that a group of data to be processed is transported to a memory of a processor, the processor aligns the group of data to be processed, two-dimensional data corresponding to depth data in the group of data to be processed is determined, and then the next group of data to be processed is transported to the memory to be aligned until each group of data to be processed is aligned;

4. The image processing system of claim 3, wherein the image processing system is further configured to:

when the ith group of data to be processed is carried, acquiring the row index overlapping range of the ith group of data to be processed and the ith-1 group of data to be processed, reserving the overlapping data in the memory, and carrying the data to be carried to the memory;

the overlapped data is data corresponding to the overlapped range of the line indexes in the i-1 th group of the data to be processed, and the data to be carried is data outside the overlapped range of the line indexes in the i-th group of the data to be processed.

5. The image processing system of claim 1, wherein the calibration information stored in the memory comprises: an internal reference of the depth camera, an internal reference of the two-dimensional camera, a rotation matrix between the depth camera and the two-dimensional camera, and a translation matrix between the depth camera and the two-dimensional camera.

6. The image processing system of claim 5, wherein the depth measurement ranges stored in the memory comprise: a maximum detection value and a minimum detection value of the depth camera.

7. The image processing system of claim 6, wherein the determining a corresponding row index range of each row of the depth image in the two-dimensional image according to the calibration information and the depth measurement range comprises:

8. An image processing method applied to the image processing system according to any one of claims 1 to 7, the method comprising:

and determining a corresponding row index range of each row of the depth image in the two-dimensional image according to the calibration information and the depth measurement range through a processor, and extracting at least one row of depth data in the depth image and at least one row of two-dimensional data corresponding to the two-dimensional image according to the row index range to perform alignment processing to determine two-dimensional data corresponding to the depth data.

9. An intelligent terminal comprising a memory, a processor, and an image processing program stored on the memory and executable on the processor, the image processing program when executed by the processor implementing the steps of the image processing method as claimed in claim 8.

10. A computer-readable storage medium, having stored thereon an image processing program which, when executed by a processor, implements the steps of the image processing method of claim 8.