CN116935436A - Pedestrian target detection method based on IRD image, electronic equipment and computer readable storage medium - Google Patents

Abstract

The invention discloses a pedestrian target detection method based on an IRD image, electronic equipment and a computer readable storage medium, wherein the pedestrian target detection method comprises the steps of acquiring data of a gray level Image (IR) and a depth image (D) of a pedestrian by utilizing a TOF camera, and fusing the gray level Image (IR) and the depth image (D) into a three-channel Image (IRD); using the trained models, pedestrians in the three-way Image (IRD) are identified inferentially. The method fuses the IR and D images, the resolution of the used image is 320 x 240, the model reasoning speed can be obviously reduced, real-time detection is achieved, and the method can be widely deployed on embedded low-computation platforms; meanwhile, the IRD image is insensitive to information such as color, illumination and the like, the detection effect is less influenced by environmental factors such as clothing color, illumination and the like, the detection precision can be effectively improved, and personal privacy information can be effectively protected.

Description

A pedestrian target detection method, electronic equipment, and computer based on IRD images Read storage media

Technical field

The invention relates to the technical field of infrared and depth image processing and detection, and specifically relates to a pedestrian target detection method, electronic equipment, and computer-readable storage media based on IRD images.

Background technique

Pedestrian target detection uses computer vision technology to determine whether there are pedestrians in images or video sequences and to accurately locate them. Pedestrian detection is widely used in multiple scenarios such as intelligent assisted driving, passenger flow statistics, and intelligent monitoring. It has developed rapidly in recent years, but there are also many problems.

In the field of pedestrian target detection, most of them are scenes with many pedestrians, frequent movements, and complex light and shadow environments; traditional methods are usually based on images from RGB color cameras, and the image resolution is high. In complex scenes of pedestrian target detection, the image resolution A higher rate will seriously affect the inference speed, and the detection accuracy is also reduced by the influence of clothing color, lighting, etc.; and RGB color images contain personal privacy information such as faces, which poses security risks;

As the development of TOF cameras (TOF cameras with infrared depth-of-field sensors and containing grayscale and depth data) becomes increasingly mature, TOF cameras can be used to achieve better pedestrian target detection.

Contents of the invention

The technical problem to be solved by this invention is to solve many problems such as the reasoning speed, low detection accuracy, and hidden dangers of leaking personal privacy when using RGB images for pedestrian target detection. In order to solve the above technical problems, the specific technical solutions adopted by this invention are:

In view of this, the present invention performs pedestrian target detection based on IDR images (three-channel low-resolution images based on the fusion of IR (Infrared Radiation) images and D (Depth) images based on ToF (Time of Flight) technology), which can avoid the above problems; The technology provided by the invention

The case is as follows:

A pedestrian target detection method based on IRD images, which is characterized by including the following steps:

(1) Use a TOF camera to collect data on grayscale images (IR) and depth images (D) of pedestrians;

(2) Fusion of the grayscale image (IR) and the depth image (D) into a three-channel image (IRD);

(3) Use the trained model to infer and identify pedestrians in the three-channel image (IRD).

Among them, a single TOF camera is used to collect the original image of a pedestrian and the grayscale image (IR) and the depth image (D) output by the Sensor are consistent in time and space, so they are aligned in time and pixels. , which can ensure the accuracy of subsequent image fusion;

Preferably, the three-channel image (IRD) fusion method includes:

① Normalize the effective value of the grayscale image (IR) to 0~255;

Wherein, x _IR is the original effective value of the grayscale image (IR), and thresh _IR is the normalized threshold of the grayscale image (IR);

② Normalize the effective value of the depth image (D) to 0~255;

Specifically, the original effective value of the depth image (D) is first subtracted from the camera height, and then normalized to 0 to 255;

x _d =camera_height-x _D

Among them, camera_height is the height of the camera, x _D is the original effective value of the depth image (D), and threshold _D is the normalized threshold of the depth image (D);

③ Merge the grayscale image (IR) and the depth image (D) into the three-channel image (IRD);

Wherein the three-channel image (IRD) includes the two-channel grayscale image (IR) and the one-channel depth image (D);

Combining the camera height to normalize the image data can improve the recognition effect in actual application scenarios.

Further, the training method of the trained model is as follows:

a. Take the three-channel image (IRD) data for data annotation;

b. Divide the labeled data into training set, verification set and test set in proportion;

c. Use the training set and the verification set to train the model;

d. Use the test set to evaluate the model:

If the model evaluation effect meets expectations, the model training is completed;

If the model evaluation effect does not meet expectations, repeat steps a-d.

Preferably, at least two groups of the three-channel image (IRD) data are taken for data annotation, wherein different groups of the three-channel image (IRD) correspond to different mounting heights of TOF cameras;

Collect the data of the grayscale image (IR) and the depth image (D) of pedestrians corresponding to different hanging heights of the TOF camera, and then fuse them into the three-channel image (IRD) data of different hanging heights. The high-hanging three-channel image (IRD) used for data annotation can improve the generalization ability of the model, improve the model's prediction ability for never-before-seen data, and perform well on training data.

Preferably, the labeled data is divided into a training set, a verification set and a test set in proportion, wherein the proportion of the training set, the verification set and the test set is 8:1:1.

Preferably, the model is the yolov5s model.

Further, when doing data annotation for the three-channel image (IRD), the pedestrian's head and upper body are annotated as an annotation box. The annotation format is a txt annotation file, and the format of each line is "category center point, abscissa center point Vertical coordinate labeling frame width labeling frame height", wherein the "center point abscissa", the "center point ordinate", the "labeling frame width", the "labeling frame width", the "labeling "Box height" are divided by the width and height of the image and then normalized; this step can deal with the impact of different size inputs.

Further, for the evaluation of the model, the evaluation indicators include P (Precision), R (Recall), and mAP@0.5 (mean Average Precision); when the P, R, and mAP@0.5 all reach above 96%, the model is considered The evaluation results are in line with expectations.

Further, the data annotation includes manual annotation. When the three-channel image (IRD) data is collected for the first time, the manual annotation method is used to annotate the data;

Further, the data annotation includes model pre-annotation; when the model evaluation effect does not meet expectations, the three-channel image (IRD) data is re-collected, and the three-channel image (IRD) data is made using the model that does not meet expectations. Pre-annotation, and then use manual annotation to fine-tune the annotation data to improve annotation efficiency.

The invention also provides an electronic device for pedestrian target detection based on IRD images, including: a processor, a communication interface, a memory and a communication bus, wherein the processor, communication interface and memory complete communication with each other through the communication bus; the memory, used to store a computer program; a processor used to implement the method steps when executing the computer program.

The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the pedestrian target detection method based on IRD images are implemented.

This method fuses IR and D images, and uses a lower image resolution of 320*240, which can significantly reduce the model inference speed, achieve real-time detection, and can be widely deployed on low-computing power platforms such as embedded platforms; at the same time, IRD images Information such as color and lighting is not sensitive, and the detection effect is less affected by environmental factors such as clothing color and lighting. It can effectively improve detection accuracy and effectively protect personal privacy information.

Description of the drawings

Figure 1 is a flow chart of a pedestrian target detection method based on IRD images provided by an embodiment of the present invention;

Figure 2 is a three-channel image (IRD) provided by an embodiment of the present invention;

Figure 3 is a detection result image provided by an embodiment of the present invention;

Figure 4 is a schematic structural diagram of an electronic device for pedestrian target detection based on IRD images provided by an embodiment of the present invention;

The number in Figure 3 is the probability of being detected as a pedestrian.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The following examples are used to illustrate the present invention. , but shall not be used to limit the scope of the present invention.

Example 1:

As shown in Figure 1-3, this embodiment provides a pedestrian target detection method flow based on IRD images;

(1) Use a TOF camera to collect data on grayscale images (IR) and depth images (D) of pedestrians;

(2) Fuse the grayscale image (IR) and the depth image (D) into a three-channel image (IRD); specifically, take a pixel as an example:

Since the grayscale image (IR) and the depth image (D) are aligned in time and pixels; therefore, taking a pixel position, the grayscale image (IR) and the depth image (D) have a corresponding relationship; assuming that the grayscale The original effective value x _IR of the image (IR) is 604, the original effective value x D of the depth image ( _D ) is 807, the camera height camera_height is 2100, the normalized threshold thresh _IR of the grayscale image (IR) is 3840, and the depth image ( D) The normalized threshold thresh _D is 2100, and the data processing process is as follows:

① Since 604 is greater than or equal to 0 and less than or equal to 3840, f(604)=604÷3840×255≈40;

②x _d = 2100-807 = 1293. Since 1293 is greater than or equal to 0 and less than or equal to 2100, f(1293) = 1293÷2100×255≈157;

③The IRD three-channel image is generated by IR+D+IR splicing and fusion. The final three-channel value of the above pixels is (40,157,40);

After each pixel of the grayscale image (IR) and depth image (D) undergoes the above calculation, it can be spliced and fused into a three-channel image (IRD).

(3) Using the trained model, pedestrians in the three-channel image (IRD) can be inferred and recognized.

Example 2:

This embodiment provides specific model training methods:

The training model can be a target detection model such as yolov5s, yolov7-tiny or SSD. Here we take training the yolov5s model as an example:

a. Before formal pedestrian detection, first take the processed three-channel image (IRD) data for manual data annotation; the relevant data can be multiple sets, of which at least 2 sets of data, the camera height of the TOF camera is different; in the data annotation When the pedestrian's head and upper body are marked as a label box, the label format is a txt label file. The format of each line is "category center point abscissa coordinate center point ordinate label box width label box height", where "center point horizontal coordinate "Coordinates", "center point ordinate", "label box width", "label box width", and "label box height" are all divided by the width and height of the image and then normalized;

b. Divide the labeled data into training set, verification set and test set in a ratio of 8:1:1;

c. Use the training set and verification set to train the yolov5s model; the training parameters include the number of training steps, learning rate and other parameters;

d. Use the test set to evaluate the yolov5s model;

If any of the evaluation indicators P (Precision), R (Recall), mAP@0.5 (mean Average Precision) does not reach 96%, it is considered that the model evaluation effect does not meet expectations; assume that the model is V0;

Further, the processed three-channel image (IRD) data is recollected, the image is pre-annotated using model V0, and then manual annotation is used to fine-tune the annotated data; hsv optimization is turned off and mosaic is enabled during data pre-annotation;

Repeat steps b, c, and d above. If the evaluation index still does not meet the standard, continue the above steps until the evaluation indexes P (Precision), R (Recall), and mAP@0.5 (mean Average Precision) all reach 96%, and the model is considered The evaluation effect is in line with expectations;

Using this model, pedestrians in the three-channel image (IRD) can be inferred and identified in step (3) of Embodiment 1.

As shown in Figure 2-3, since the grayscale image (IR) used in the present invention is a grayscale image and does not contain color information, the corresponding IRD image is not sensitive to color information, and the detection effect is less affected by clothing color factors. ; The TOF camera used in the present invention, the distance measurement principle is to continuously send light pulses to the target, and then use the sensor to receive the light returned from the object, and obtain the target distance by detecting the flight (round trip) time of the light pulse, TOF camera lens There will be a bandpass filter to ensure that only light with the same wavelength as the illumination source can enter. This can suppress incoherent light sources and improve the signal-to-noise ratio, so it is less affected by lighting conditions; the corresponding IRD image is more affected by lighting factors. Small.

Example 3:

As shown in Figure 4, an embodiment of the present invention provides an electronic device for pedestrian target detection based on IRD images, including a processor, a communication interface, a memory and a communication bus, where the processor, communication interface and memory are completed through the communication bus. communication between each other.

Example 4:

Embodiments of the present invention provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the pedestrian target detection method based on IRD images are implemented.

Claims (10)

1. A pedestrian target detection method based on IRD images, characterized by including the following steps: (1) Use a TOF camera to collect data on grayscale images (IR) and depth images (D) of pedestrians; (2) Fusion of the grayscale image (IR) and the depth image (D) into a three-channel image (IRD); (3) Use the trained model to infer and identify pedestrians in the three-channel image (IRD). 2. A pedestrian target detection method based on IRD images according to claim 1, characterized in that the fusion method of the three-channel images (IRD) includes: ① Normalize the effective value of the grayscale image (IR) to 0~255; Wherein, x _IR is the original effective value of the grayscale image (IR), and thresh _IR is the normalized threshold of the grayscale image (IR); ② Normalize the effective value of the depth image (D) to 0~255; Specifically, first use the camera height to subtract the original effective value of the depth image (D), Then normalize to 0~255; x _d =camera_height-x _D Among them, camera_height is the height of the camera, x _D is the original effective value of the depth image (D), and threshold _D is the normalized threshold of the depth image (D); ③ Merge the grayscale image (IR) and the depth image (D) into the three-channel image (IRD). 3. A pedestrian target detection method based on IRD images according to claim 1, characterized in that the training method of the trained model is as follows: a. Take the three-channel image (IRD) data for data annotation; b. Divide the labeled data into training set, verification set and test set in proportion; c. Use the training set and verification set to train the model; d. Use the test set to evaluate the model: If the model evaluation effect meets expectations, the model training is completed; If the model evaluation effect does not meet expectations, repeat steps a-d. 4. A pedestrian target detection method based on IRD images according to claim 3, characterized in that at least 2 groups of the three-channel image (IRD) data are taken for data annotation, wherein different groups of the three-channel images are (IRD) The corresponding TOF cameras have different mounting heights. 5. A pedestrian target detection method based on IRD images according to claim 3, characterized in that the data annotation includes manual annotation, and manual annotation is used when collecting the three-channel image (IRD) data for the first time. Labeling methods label data. 6. A pedestrian target detection method based on IRD images according to claim 5, characterized in that the data annotation includes model pre-annotation; when the model evaluation effect does not meet expectations, the three-channel image is re-acquired ( IRD) data, use a model that does not meet expectations to pre-annotate the three-channel image (IRD) data, and then use manual annotation to fine-tune the annotated data. 7. A pedestrian target detection method based on IRD images according to claim 3, characterized in that the ratio of the training set, the verification set and the test set is 8:1:1. 8. A pedestrian target detection method based on IRD images according to claim 1, characterized in that the model is a yolov5s model. 9. An electronic device for pedestrian target detection based on IRD images, characterized by comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; the memory , used to store the computer program; the processor, used to implement the method steps described in any one of claims 1-8 when executing the computer program. 10. A computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the pedestrian target detection based on IRD images according to any one of claims 1 to 8 is implemented Method steps.