CN116665245A - Multi-view gesture recognition method, device, computer equipment and storage medium - Google Patents

Abstract

The application discloses a multi-view gesture recognition method, which comprises the following steps: at least two TOF sensors for acquiring 3D information of the hand are arranged in the gesture interaction space; acquiring infrared IR images and depth images under each view angle in a gesture interaction space in real time through each TOF sensor; detecting the acquired infrared IR image; correcting and calculating the hand key point coordinates under each view angle to obtain a hand key point 3D coordinate set under all view angles; performing calibration mapping processing on the 3D coordinate sets of the hand key points under all view angles; and performing calculation processing to obtain a hand feature map, and performing gesture recognition processing on the obtained hand feature map to obtain a gesture recognition result. An apparatus, a computer device and a storage medium for implementing the multi-view gesture recognition method are also disclosed. The application improves the recognition accuracy and can recognize more complex gestures.

Description

Multi-view gesture recognition method, device, computer equipment and storage medium

technical field

The present invention relates to the technical field of gesture recognition methods, in particular to a multi-view gesture recognition method, device, computer equipment and storage medium.

Background technique

In the field of human-computer interaction, gesture interaction, as the most widely used and efficient interaction method, is widely used in many fields such as robot control, industrial production, sign language recognition, and multimedia audio-visual control. The accuracy and efficiency of gesture interaction depend on reliable static and dynamic gesture recognition.

The current mainstream gesture recognition relies on 2D image recognition results from a single perspective, using RGB cameras to obtain video streams, and then performing gesture recognition on hands by detecting hands. There are also some ways to model the video frame to obtain 3D information of the hand and improve the accuracy of gesture recognition. However, relying on information from a single viewing angle, it is difficult to accurately identify due to factors such as the limitation of the field of view and the size of the environmental space. At the same time, it cannot be solved for some complex gestures and partial occlusions that lead to incomplete hands, which eventually leads to gesture recognition. reduced accuracy and the inability to recognize more complex gestures.

For this reason, the applicant has found a solution to the above-mentioned problems through beneficial exploration and research, and the technical solutions to be introduced below are generated under this background.

Contents of the invention

One of the technical problems to be solved by the present invention is to provide a multi-view gesture recognition method that improves the recognition accuracy and can recognize more complex gestures in view of the deficiencies in the prior art.

The second technical problem to be solved by the present invention is to provide a multi-view gesture recognition device for implementing the above multi-view gesture recognition method.

The third technical problem to be solved by the present invention is to provide a computer device for realizing the above multi-view gesture recognition method.

The fourth technical problem to be solved by the present invention is to provide a computer-readable storage medium for realizing the above multi-view gesture recognition method.

A multi-view gesture recognition method as the first aspect of the present invention includes:

Install at least two TOF sensors for collecting hand 3D information in the gesture interaction space, and perform calibration processing on each TOF sensor to obtain the internal reference matrix and lens distortion parameters of each TOF sensor and multiple TOF sensors The relative position transformation matrix between;

When working, each TOF sensor collects infrared IR images and depth images at each angle of view in the gesture interaction space in real time;

Detect and process the collected infrared IR images to obtain the coordinates of key points of the hand at each viewing angle;

According to the internal reference matrix and lens distortion parameters obtained from the calibration and combined with the collected depth images, the coordinates of the key points of the hand under each viewing angle are corrected and calculated, and the 3D coordinates of the key points of the hand under all viewing angles are obtained;

According to the relative position transformation matrix between multiple TOF sensors obtained through calibration, the 3D coordinate set of key points of the hand under all viewing angles is calibrated and mapped, and the key points of the hand in the three-dimensional coordinate system with the key point of the wrist as the coordinate origin are obtained. A collection of 3D coordinates; and

Calculate and process the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin to obtain the hand feature map, and perform gesture recognition processing on the obtained hand feature map to obtain the gesture recognition result.

In a preferred embodiment of the present invention, the detection and processing of the collected infrared IR images to obtain the key point coordinates of the hand under each viewing angle includes:

The collected infrared IR image is detected and processed in real time through the hand detection model based on the IR image;

When it is detected that a human hand appears in the infrared IR image, the hand area in the collected infrared IR image is cut and processed to obtain a partial IR image of the human hand; and

Send the cropped local IR image of the human hand to the hand key point detection model based on the IR image for detection, and obtain the coordinates of the key points of the hand under each viewing angle;

In a preferred embodiment of the present invention, the coordinates of the key points of the hand at each viewing angle are corrected and calculated according to the internal reference matrix and lens distortion parameters obtained through calibration and combined with the collected depth images, to obtain the coordinates of key points of the hand at each viewing angle. A collection of 3D coordinates of hand key points, including:

According to the internal reference matrix and lens distortion matrix obtained by calibration, the coordinates of the key points of the hand and the collected depth images in each viewing angle are corrected;

Remap the corrected hand keypoint coordinates to the corrected depth image to obtain the two-dimensional coordinates and depth data of each hand keypoint at each viewing angle; and

Calculate and process the two-dimensional coordinates and depth data of each hand key point at each viewing angle according to the internal reference matrix obtained through calibration, and obtain the 3D coordinates of the hand key point at each viewing angle; and

The 3D coordinates of the key points of the hand in each viewing angle are collected and processed to obtain the 3D coordinates of the key points of the hand in all viewing angles.

In a preferred embodiment of the present invention, the relative position transformation matrix between the plurality of TOF sensors obtained according to the calibration is used to calibrate and map the 3D coordinate sets of key points of the hand under all viewing angles, and obtain the key points of the wrist as The set of 3D coordinates of key points of the hand under the 3D coordinate system of the coordinate origin, including:

According to the relative position transformation matrix between multiple TOF sensors obtained through calibration, the 3D coordinate set of key points of the hand under all viewing angles is fused and calibrated; and

Remap the 3D coordinate set of key points of the hand under all viewing angles after the fusion calibration process to the three-dimensional coordinate system with the key point of the wrist as the coordinate origin, and obtain the key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin A collection of point 3D coordinates.

In a preferred embodiment of the present invention, the calculation process is performed on the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the origin of the coordinates, and the hand feature map is obtained, including:

Normalize the hand key point 3D coordinate set under the three-dimensional coordinate system with the wrist key point as the coordinate origin, and obtain the normalized hand key point 3D coordinate set;

Calculate the normalized Euclidean distance between each hand key point, and generate a normalized distance map between multiple hand key points;

According to each key point, select and connect with the adjacent point farthest from the wrist key to generate a node connection graph with multiple hand key points farther away from the wrist;

According to each key point, select its adjacent points closest to the wrist key point to connect to generate a node connection graph with multiple hand key points closer to the wrist;

The normalized distance map between multiple hand key points, the node connection graph with multiple hand key points far away from the wrist, and the node connection graph with multiple hand key points close to the wrist are synthesized to obtain the hand Partial feature map.

In a preferred embodiment of the present invention, gesture recognition is performed on the hand feature map obtained by using a gesture recognition model based on the 3D coordinate information fusion feature map of the hand key points, and a gesture recognition result is generated.

As a second aspect of the present invention, a multi-view gesture recognition device implementing the above multi-view gesture recognition method includes:

At least two TOF sensors installed in the gesture interaction space for collecting 3D information of the hand;

A calibration processing module, the calibration processing module is used to perform calibration processing on each TOF sensor, so as to obtain the internal reference matrix and lens distortion parameters of each TOF sensor and the relative position transformation matrix between multiple TOF sensors;

An image acquisition module, the image acquisition module is used to collect infrared IR images and depth images under each angle of view in the gesture interaction space in real time through each TOF sensor during work;

A hand key point coordinate detection and acquisition module, the hand key point coordinate detection and acquisition module is used to detect and process the collected infrared IR image, to obtain the hand key point coordinates under each viewing angle;

Correction calculation processing module, the correction calculation processing module is used to correct and calculate the coordinates of key points of the hand at each angle of view according to the internal reference matrix and lens distortion parameters obtained through calibration and combined with the collected depth images, to obtain 3D coordinate collection of hand key points;

A calibration mapping processing module, the calibration mapping processing module is used to perform calibration mapping processing on the 3D coordinate sets of key points of the hand under all viewing angles according to the relative position transformation matrix between multiple TOF sensors obtained through calibration, and obtain the key points of the wrist A set of 3D coordinates of key points of the hand under the 3D coordinate system of the coordinate origin; and

Gesture recognition calculation and processing module, the gesture recognition calculation and processing module is used to calculate and process the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the origin of the coordinates, to obtain the hand feature map, and to obtain the obtained Gesture recognition processing is performed on the hand feature map to obtain gesture recognition results.

As a third aspect of the present invention, a computer device for implementing a multi-view gesture recognition method includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Install at least two TOF sensors for collecting hand 3D information in the gesture interaction space, and perform calibration processing on each TOF sensor to obtain the internal reference matrix and lens distortion parameters of each TOF sensor and multiple TOF sensors The relative position transformation matrix between;

When working, each TOF sensor collects infrared IR images and depth images at each angle of view in the gesture interaction space in real time;

Detect and process the collected infrared IR images to obtain the coordinates of key points of the hand at each viewing angle;

According to the internal reference matrix and lens distortion parameters obtained from the calibration and combined with the collected depth images, the coordinates of the key points of the hand under each viewing angle are corrected and calculated, and the 3D coordinates of the key points of the hand under all viewing angles are obtained;

According to the relative position transformation matrix between multiple TOF sensors obtained through calibration, the 3D coordinate set of key points of the hand under all viewing angles is calibrated and mapped, and the key points of the hand in the three-dimensional coordinate system with the key point of the wrist as the coordinate origin are obtained. A collection of 3D coordinates; and

Calculate and process the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin to obtain the hand feature map, and perform gesture recognition processing on the obtained hand feature map to obtain the gesture recognition result.

As a fourth aspect of the present invention, a computer-readable storage medium for implementing the above multi-view gesture recognition method has a computer program stored thereon, and the computer program implements the following steps when executed by a processor:

Install at least two TOF sensors for collecting hand 3D information in the gesture interaction space, and perform calibration processing on each TOF sensor to obtain the internal reference matrix and lens distortion parameters of each TOF sensor and multiple TOF sensors The relative position transformation matrix between;

When working, each TOF sensor collects infrared IR images and depth images at each angle of view in the gesture interaction space in real time;

Detect and process the collected infrared IR images to obtain the coordinates of key points of the hand at each viewing angle;

According to the internal reference matrix and lens distortion parameters obtained from the calibration and combined with the collected depth images, the coordinates of the key points of the hand under each viewing angle are corrected and calculated, and the 3D coordinates of the key points of the hand under all viewing angles are obtained;

According to the relative position transformation matrix between multiple TOF sensors obtained through calibration, the 3D coordinate set of key points of the hand under all viewing angles is calibrated and mapped, and the key points of the hand in the three-dimensional coordinate system with the key point of the wrist as the coordinate origin are obtained. A collection of 3D coordinates; and

Calculate and process the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin to obtain the hand feature map, and perform gesture recognition processing on the obtained hand feature map to obtain the gesture recognition result.

Owing to adopting above technical scheme, the beneficial effect of the present invention is:

1. The present invention takes the key point of the wrist as the origin of 3D coordinates to reproject the coordinates of the key points of the hand, which can restore the real state of the hand to the greatest extent, and by extracting the feature map of the hand, the extracted gesture features have rotation, Translation invariance, so that gestures under various poses can be recognized;

2. Compared with the traditional single- and binocular RGB sensors, the present invention can obtain the depth map and IR map through one frame of data, with a small amount of calculation and high real-time performance; Less interference can be used under complex ambient light conditions, and has low hardware requirements, stable and reliable;

3. Compared with the single-view solution commonly adopted at present, the present invention adopts a multi-view solution, which can obtain more comprehensive state information of the human hand, and fully utilizes the technical characteristics of TOF to restore the geometric relationship of each finger of the human hand in space, which can More accurate recognition of gestures and better robustness in the face of occlusion and other scenarios.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a flow chart of the multi-view gesture recognition method of the present invention.

FIG. 2 is a schematic structural diagram of the multi-view gesture recognition device of the present invention.

Fig. 3 is an internal structure diagram of the computer device of the present invention.

Detailed ways

In order to make the technical means, creative features, goals and effects achieved by the present invention easy to understand, the present invention will be further described below in conjunction with specific illustrations.

Referring to Fig. 1, provided among the figure is the multi-view gesture recognition method of the present invention, comprises the following steps:

Step S10, install two TOF sensors for collecting 3D information of the hand in the gesture interaction space, and perform calibration processing on each TOF sensor to obtain the internal reference matrix I_M and lens distortion parameter D_M of each TOF sensor and The relative position transformation matrix T_M between multiple TOF sensors. In this embodiment, the number of TOF sensors is not limited to the number in this embodiment. It should be set according to the gesture interaction space and gesture recognition accuracy. To fully cover the gesture interaction space.

Step S20, when working, each TOF sensor collects the infrared IR image IR_M and the depth image DEP_M at each viewing angle in the gesture interaction space in real time.

Step S30 , performing detection processing on the collected infrared IR image IR_M to obtain the key point coordinates of the hand under each viewing angle.

Step S40, according to the internal reference matrix I_M obtained through calibration and the lens distortion parameter D_M and combined with the acquired depth image DEP_M, the coordinates of the key points of the hand under each viewing angle are corrected and calculated to obtain the 3D coordinates of the key points of the hand under all viewing angles gather.

Step S50, according to the relative position transformation matrix T_M between multiple TOF sensors obtained through calibration, the 3D coordinate set of key points of the hand under all viewing angles is calibrated and mapped, and the 3D coordinate system with the key point of the wrist as the coordinate origin is obtained. A collection of 3D coordinates of hand key points.

Step S60, calculate and process the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin, obtain the hand feature map, and perform gesture recognition processing on the obtained hand feature map to obtain gesture recognition result.

In step S30, the collected infrared IR image IR_M is detected and processed to obtain the key point coordinates of the hand under each viewing angle, including the following steps:

In step S31, the collected infrared IR image IR_M is detected and processed in real time through the hand detection model HD_MODEL based on the IR image; wherein, the hand detection model HD_MODEL based on the IR image is a conventional detection model in the field, and a general detector model is used ---yolov5s framework, which first manually labels the hand positions in a large number of IR images under different viewing angles, obtains the label information, and then sends it to the model for training to obtain HD_MODEL.

Step S32, when it is detected that a human hand appears in the infrared IR image IR_M, the hand area in the collected infrared IR image IR_M is cut to obtain a partial IR image HD_ROI of the human hand;

In step S33, the cropped partial IR image HD_ROI of the human hand is sent to the hand key point detection model KPD_MODEL based on the IR image for detection, and the coordinates of the key points of the hand at each viewing angle are obtained. Among them, the hand key point detection model KPD_MODEL based on the IR image is a conventional detection model in this field, using the general detector model --- yolov5s framework, which cuts the local ROI area of the human head detected by HD_MODEL and then processes it , and mark 21 key points of the hand, obtain the key point labeling information, and then send it to the model for training, and get the hand key point detection model KPD_MODEL.

In step S40, according to the internal reference matrix I_M obtained through calibration and the lens distortion parameter D_M combined with the acquired depth image DEP_M, the coordinates of the key points of the hand under each viewing angle are corrected and calculated, and the key points of the hand under all viewing angles are obtained. A collection of 3D coordinates, including the following steps:

Step S41, according to the internal reference matrix I_M obtained by calibration and the lens distortion parameter D_M, the coordinates of the key points of the hand under each viewing angle and the collected depth image DEP_M are corrected;

Step S42, remap the corrected hand key point coordinates to the corrected depth image DEP_M, and obtain the two-dimensional coordinates and depth data of each hand key point at each viewing angle;

Step S43, calculate and process the two-dimensional coordinates and depth data of each key point of the hand under each viewing angle according to the internal reference matrix I_M obtained through calibration, and obtain the 3D coordinates of the key points of the hand under each viewing angle;

In step S44, the 3D coordinates of the key points of the hand under each viewing angle are aggregated to obtain a set of 3D coordinates of the key points of the hand under all viewing angles.

In step S50, according to the relative position transformation matrix T_M between multiple TOF sensors obtained through calibration, the 3D coordinate set of key points of the hand under all viewing angles is calibrated and mapped to obtain a three-dimensional coordinate system with the key point of the wrist as the coordinate origin The set of 3D coordinates of the key points of the hand includes the following steps:

Step S51, according to the relative position transformation matrix T_M between multiple TOF sensors obtained through calibration, perform fusion calibration processing on the 3D coordinate sets of key points of the hand under all viewing angles;

Step S52, remap the 3D coordinate sets of key points of the hand under all viewing angles after fusion calibration to the three-dimensional coordinate system with the key point of the wrist as the coordinate origin, and obtain the 3D coordinate system with the key point of the wrist as the coordinate origin. A collection of 3D coordinates of hand key points.

In step S60, the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin is calculated and processed to obtain the hand feature map, including the following steps:

Step S61, normalize the 3D coordinate set of hand key points under the three-dimensional coordinate system with the wrist key point as the coordinate origin, and obtain the normalized 3D coordinate set of hand key points;

Step S62, calculate the normalized Euclidean distance between each hand key point, and generate a normalized distance map DISTANCE_MAP between multiple hand key points; specifically, the wrist key point obtained in step S61 The coordinates of 21 key points under the coordinate system of the coordinate origin, calculate the coordinates between each key point, and normalize with the farthest distance as the maximum value to obtain a feature map DISTANCE_MAP with a size of 21*21 and taking the distance as the feature value;

Step S63, according to each key point, select and connect with the adjacent point farthest from the wrist key to generate a node connection graph F_DISTANCE_MAP with multiple hand key points farther away from the wrist; specifically, the wrist key points obtained through step S61 For the coordinates of 21 key points in the coordinate system of the coordinate origin, generate a distance matrix of 21*21, and take each key point as the center, select the point farthest from the wrist among its adjacent key nodes, and set the value of the distance matrix The assignment is 1, and the other assignments are 0, generating a node connection map F_DISTANCE_MAP;

Step S64, according to each key point, select its adjacent points closest to the wrist key point to connect to generate a node connection map N_DISTANCE_MAP with multiple hand key points closer to the wrist; specifically, the wrist key points obtained through step S61 The coordinates of 21 key points in the coordinate system of the coordinate origin, generate a 21*21 distance matrix, and take each key point as the center, select the point closest to the wrist among its adjacent key nodes, and assign the value of the distance matrix is 1, other assignments are 0, and the node connection map N_DISTANCE_MAP is generated;

Step S65, the normalized distance map DISTANCE_MAP between multiple hand key points, the node connection map F_DISTANCE_MAP with multiple hand key points far from the wrist, and the node connection map N_DISTANCE_MAP with multiple hand key points close to the wrist Synthesis processing is performed to obtain hand feature maps.

The present invention takes the key point of the wrist as the origin of 3D coordinates to re-project the coordinates of the key points of the hand, which can restore the real state of the hand to the greatest extent, and by extracting the feature map of the hand, the extracted gesture features have the characteristics of rotation and translation. Denaturation, so as to be able to recognize gestures under each posture.

In step S60, gesture recognition is performed on the obtained hand feature map through the gesture recognition model KPGR_MODEL based on the 3D coordinate information of the hand key point fusion feature map, and a gesture recognition result is generated. Among them, the gesture recognition model KPGR_MODEL based on the 3D coordinate information fusion feature map of hand key points is a conventional recognition model in this field. It is realized based on the ResNet18 framework model, and the fused multi-gesture hand feature map is labeled and sent to The network is trained to obtain gesture recognition results.

Referring to Fig. 2, the multi-view gesture recognition device of the present invention is shown in the figure, including two TOF sensors 110, a calibration processing module 120, an image acquisition module 130, a hand key point coordinate detection and acquisition module 140, and a correction calculation processing module 150 , a calibration mapping processing module 160 and a gesture recognition calculation processing module 170 .

Two TOF sensors 110 are installed in the gesture interaction space, which are used to collect 3D information of the hand. In this embodiment, the number of TOF sensors 110 is not limited to the number in this embodiment, it should be set according to the gesture interaction space and gesture recognition accuracy, and it should be installed as much as possible under the condition that part of the field of view overlaps during installation. To fully cover the gesture interaction space.

The calibration processing module 120 is used to perform calibration processing on each TOF sensor 110 to obtain the internal reference matrix and lens distortion parameters of each TOF sensor 110 and the relative position transformation matrix between multiple TOF sensors.

The image acquisition module 130 is used to collect the infrared IR image IR_M and the depth image DEP_M at each viewing angle in the gesture interaction space in real time through each TOF sensor 110 during operation.

The hand key point coordinate detection and acquisition module 140 is used to detect and process the collected infrared IR image IR_M, so as to obtain the hand key point coordinates under each viewing angle.

The correction calculation processing module 150 is used to correct and calculate the coordinates of the key points of the hand under each viewing angle according to the internal reference matrix I_M obtained through calibration and the lens distortion parameter D_M combined with the acquired depth image DEP_M, and obtain the hand coordinates under all viewing angles. A collection of 3D coordinates of key points.

The calibration mapping processing module 160 is used to perform calibration mapping processing on the 3D coordinate sets of key points of the hand under all viewing angles according to the relative position transformation matrix T_M between multiple TOF sensors obtained through calibration, and obtain a three-dimensional coordinate system with the key point of the wrist as the coordinate origin. A collection of 3D coordinates of hand key points in the coordinate system.

Gesture recognition calculation and processing module 170 is used to calculate and process the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin, obtain the hand feature map, and perform gesture recognition on the obtained hand feature map Processing to get the gesture recognition result.

Each module in the multi-view gesture recognition device of the present invention can be fully or partially realized by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.

The present invention also provides a computer device for implementing the above multi-view gesture recognition method. The computer device may be a server, and its internal structure diagram may be shown in FIG. 3 . The computer device includes a processor, memory, network interface and database connected by a system bus. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs and databases. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer equipment is used to store data such as user information, record information and files. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above-mentioned multi-view gesture recognition method can be realized.

Those skilled in the art can understand that the structure shown in FIG. 3 is only a block diagram of a part of the structure related to the technical solution, and does not constitute a limitation to the computer equipment on which the solution of the application is applied. The specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

Specifically, the computer device of the present invention includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Step S10, install two TOF sensors for collecting 3D information of the hand in the gesture interaction space, and perform calibration processing on each TOF sensor to obtain the internal reference matrix I_M and lens distortion parameter D_M of each TOF sensor and Relative position transformation matrix T_M between multiple TOF sensors;

Step S20, when working, each TOF sensor is used to collect the infrared IR image IR_M and the depth image DEP_M at each angle of view in the gesture interaction space in real time;

Step S30, detecting and processing the collected infrared IR image IR_M, so as to obtain the key point coordinates of the hand under each viewing angle;

Step S40, according to the internal reference matrix I_M obtained through calibration and the lens distortion parameter D_M and combined with the acquired depth image DEP_M, the coordinates of the key points of the hand under each viewing angle are corrected and calculated to obtain the 3D coordinates of the key points of the hand under all viewing angles gather;

Step S50, according to the relative position transformation matrix T_M between multiple TOF sensors obtained through calibration, the 3D coordinate set of key points of the hand under all viewing angles is calibrated and mapped, and the 3D coordinate system with the key point of the wrist as the coordinate origin is obtained. 3D coordinate collection of hand key points;

Step S60, calculate and process the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin, obtain the hand feature map, and perform gesture recognition processing on the obtained hand feature map to obtain gesture recognition result.

The present invention also provides a computer-readable storage medium for realizing the above multi-view gesture recognition method, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

Step S10, install two TOF sensors for collecting 3D information of the hand in the gesture interaction space, and perform calibration processing on each TOF sensor to obtain the internal reference matrix I_M and lens distortion parameter D_M of each TOF sensor and Relative position transformation matrix T_M between multiple TOF sensors;

Step S20, when working, each TOF sensor is used to collect the infrared IR image IR_M and the depth image DEP_M at each angle of view in the gesture interaction space in real time;

Step S30, detecting and processing the collected infrared IR image IR_M, so as to obtain the key point coordinates of the hand under each viewing angle;

Step S40, according to the internal reference matrix I_M obtained through calibration and the lens distortion parameter D_M and combined with the acquired depth image DEP_M, the coordinates of the key points of the hand under each viewing angle are corrected and calculated to obtain the 3D coordinates of the key points of the hand under all viewing angles gather;

Step S50, according to the relative position transformation matrix T_M between multiple TOF sensors obtained through calibration, the 3D coordinate set of key points of the hand under all viewing angles is calibrated and mapped, and the 3D coordinate system with the key point of the wrist as the coordinate origin is obtained. 3D coordinate collection of hand key points;

Step S60, calculate and process the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin, obtain the hand feature map, and perform gesture recognition processing on the obtained hand feature map to obtain gesture recognition result.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the computer programs can be stored in a non-volatile computer-readable memory In the medium, when the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any references to memory, storage, database or other media used in the various embodiments provided in the present application may include non-volatile and/or volatile memory. Nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The basic principles and main features of the present invention and the advantages of the present invention have been shown and described above. Those skilled in the industry should understand that the present invention is not limited by the above-mentioned embodiments. What are described in the above-mentioned embodiments and the description only illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have Variations and improvements are possible, which fall within the scope of the claimed invention. The protection scope of the present invention is defined by the appended claims and their equivalents.

Claims (9)

1. A multi-view gesture recognition method, characterized in that, comprising: Install at least two TOF sensors for collecting hand 3D information in the gesture interaction space, and perform calibration processing on each TOF sensor to obtain the internal reference matrix and lens distortion parameters of each TOF sensor and multiple TOF sensors The relative position transformation matrix between; When working, each TOF sensor collects infrared IR images and depth images at each angle of view in the gesture interaction space in real time; Detect and process the collected infrared IR images to obtain the coordinates of key points of the hand at each viewing angle; According to the internal reference matrix and lens distortion parameters obtained from the calibration and combined with the collected depth images, the coordinates of the key points of the hand under each viewing angle are corrected and calculated, and the 3D coordinates of the key points of the hand under all viewing angles are obtained; According to the relative position transformation matrix between multiple TOF sensors obtained through calibration, the 3D coordinate set of key points of the hand under all viewing angles is calibrated and mapped, and the key points of the hand in the three-dimensional coordinate system with the key point of the wrist as the coordinate origin are obtained. A collection of 3D coordinates; and Calculate and process the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin to obtain the hand feature map, and perform gesture recognition processing on the obtained hand feature map to obtain the gesture recognition result. 2. The multi-view gesture recognition method as claimed in claim 1, wherein said detecting and processing the collected infrared IR image to obtain the hand key point coordinates under each viewing angle comprises: The collected infrared IR image is detected and processed in real time through the hand detection model based on the IR image; When it is detected that a human hand appears in the infrared IR image, the hand area in the collected infrared IR image is cut and processed to obtain a partial IR image of the human hand; and The cropped local IR image of the human hand is sent to the hand key point detection model based on the IR image for detection, and the coordinates of the key point of the hand under each viewing angle are obtained. 3. The multi-view gesture recognition method as claimed in claim 1, wherein the coordinates of the key points of the hand under each viewing angle are carried out according to the internal reference matrix and lens distortion parameters obtained through calibration and in combination with the collected depth images. Correction calculation processing to obtain the 3D coordinate set of key points of the hand under all viewing angles, including: According to the internal reference matrix and lens distortion matrix obtained by calibration, the coordinates of the key points of the hand and the collected depth images in each viewing angle are corrected; Remap the corrected hand keypoint coordinates to the corrected depth image to obtain the two-dimensional coordinates and depth data of each hand keypoint at each viewing angle; and Calculate and process the two-dimensional coordinates and depth data of each hand key point at each viewing angle according to the internal reference matrix obtained through calibration, and obtain the 3D coordinates of the hand key point at each viewing angle; and The 3D coordinates of the key points of the hand in each viewing angle are collected and processed to obtain the 3D coordinates of the key points of the hand in all viewing angles. 4. The multi-view gesture recognition method according to claim 1, wherein the relative position transformation matrix between multiple TOF sensors obtained according to the calibration is used to calibrate the 3D coordinate sets of key points of the hand under all viewing angles Mapping processing to obtain the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin, including: According to the relative position transformation matrix between multiple TOF sensors obtained through calibration, the 3D coordinate set of key points of the hand under all viewing angles is fused and calibrated; and Remap the 3D coordinate set of key points of the hand under all viewing angles after the fusion calibration process to the three-dimensional coordinate system with the key point of the wrist as the coordinate origin, and obtain the key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin A collection of point 3D coordinates. 5. The multi-view gesture recognition method according to claim 1, wherein the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the coordinate origin is calculated and processed to obtain hand features Figures, including: Normalize the hand key point 3D coordinate set under the three-dimensional coordinate system with the wrist key point as the coordinate origin, and obtain the normalized hand key point 3D coordinate set; Calculate the normalized Euclidean distance between each hand key point, and generate a normalized distance map between multiple hand key points; According to each key point, select and connect with the adjacent point farthest from the wrist key to generate a node connection graph with multiple hand key points farther away from the wrist; According to each key point, select its adjacent points closest to the wrist key point to connect to generate a node connection graph with multiple hand key points closer to the wrist; The normalized distance map between multiple hand key points, the node connection graph with multiple hand key points far away from the wrist, and the node connection graph with multiple hand key points close to the wrist are synthesized to obtain the hand Partial feature map. 6. The multi-view gesture recognition method according to claim 1, characterized in that, the obtained hand feature map is subjected to gesture recognition through a gesture recognition model based on hand key point 3D coordinate information fusion feature map, and gesture recognition is generated result. 7. A multi-view gesture recognition device, characterized in that it comprises: At least two TOF sensors installed in the gesture interaction space for collecting 3D information of the hand; A calibration processing module, the calibration processing module is used to perform calibration processing on each TOF sensor, so as to obtain the internal reference matrix and lens distortion parameters of each TOF sensor and the relative position transformation matrix between multiple TOF sensors; An image acquisition module, the image acquisition module is used to collect infrared IR images and depth images under each angle of view in the gesture interaction space in real time through each TOF sensor during work; A hand key point coordinate detection and acquisition module, the hand key point coordinate detection and acquisition module is used to detect and process the collected infrared IR image, to obtain the hand key point coordinates under each viewing angle; Correction calculation processing module, the correction calculation processing module is used to correct and calculate the coordinates of key points of the hand at each angle of view according to the internal reference matrix and lens distortion parameters obtained through calibration and combined with the collected depth images, to obtain 3D coordinate collection of hand key points; A calibration mapping processing module, the calibration mapping processing module is used to perform calibration mapping processing on the 3D coordinate sets of key points of the hand under all viewing angles according to the relative position transformation matrix between multiple TOF sensors obtained through calibration, and obtain the key points of the wrist A set of 3D coordinates of key points of the hand under the 3D coordinate system of the coordinate origin; and Gesture recognition calculation and processing module, the gesture recognition calculation and processing module is used to calculate and process the 3D coordinate set of key points of the hand under the three-dimensional coordinate system with the key point of the wrist as the origin of the coordinates, to obtain the hand feature map, and to obtain the obtained Gesture recognition processing is performed on the hand feature map to obtain gesture recognition results. 8. A computer device, comprising a memory and a processor, the memory stores a computer program, wherein when the processor executes the computer program, the multiple functions described in any one of claims 1 to 6 are realized. The steps of the viewpoint gesture recognition method. 9. A computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the multi-view gesture recognition method according to any one of claims 1 to 6 is realized. step.