CN109657577B - An animal detection method based on entropy and motion offset - Google Patents

Abstract

The invention discloses an animal detection method based on entropy and motion offset. Based on the existing YOLOv3 model, the current frame image is preliminarily detected and judged. If there is occlusion in the image, the current frame image detection fails, and the minimum entropy is passed. The preceding queue images are used to determine the animal category information of the current frame image, and the animal position information of the current frame image is calculated through multiple preceding queue images. In this way, animal detection for occluded images is realized, and the stability and accuracy of real-time animal detection are improved.

Description

An animal detection method based on entropy and motion offset

technical field

The invention relates to the field of image recognition, in particular to an animal detection method based on entropy and motion offset.

Background technique

Real-time animal detection is an important research direction in the field of machine vision, and its applications cover many fields such as security, industry, and automotive assisted driving. Apply real-time animal detection in the animal field to record the daily behavior and life rules of animals to assist scientific research, so as to better protect precious or endangered animals, better protect complex ecosystems, and avoid the occurrence of accidents caused by a certain animal. Destruction and causing adverse chain reactions. However, in the process of real-time animal detection on animals, the stability and accuracy of animal detection will be reduced due to the influence of motion blur, morphological changes, illumination, complex backgrounds, and occlusion by other objects. Among them, the solution to the occlusion problem in animal detection is the key to improving the stability and accuracy of animal detection.

The detection of animals in the presence of occlusion is mainly divided into traditional occlusion detection methods, classification methods based on artificial feature extraction, and methods based on deep learning. The traditional occlusion detection method detects animals through fusion center weighting, sub-block matching, trajectory prediction, Bayesian theory, etc. The disadvantage is that it cannot be well applied to the situation where the target scale changes, and there is a high false detection rate; based on artificial The feature extraction method obtains the feature description of the target through prior knowledge, and inputs it into a classifier to learn the classification rules, which has shortcomings such as insufficient ability to express the target and poor separability. The product operation allows the computer to automatically extract the target features with high robustness and generality from the image. In recent years, the great advantages of deep learning-based methods in the field of animal detection have made them a research hotspot. Some researchers combine the selective search algorithm and SVM classifier to propose a region-based convolutional neural network R-CNN, which improves the average recognition rate by nearly 20% compared to the animal detection method based on artificial feature extraction, but the time overhead is similar to that of the animal detection method. The space overhead is large; the YOLO model designed by some researchers uses a network to train the image end-to-end, uses the entire image as the input of the network, and returns the position and category of the bounding box at the output layer, and the detection speed is greatly improved. improved, but the detection accuracy rate is reduced; the SSD (Single Shot MultiBox Detector) model designed by some researchers combines the VGG-16 network, sliding window and anchor box to regress the multi-scale regional features in various positions of the whole image, which improves animal performance. The accuracy of detection, but the detection speed is not as fast as YOLO. The YOLOv3 model proposed by some researchers uses the Darknet-53 network and the pyramid network to perform multi-scale detection on images, which achieves the detection accuracy of SSD while maintaining the detection speed of YOLO. Although the above deep learning-based animal detection methods have achieved good research results, when animal detection is performed on videos, the unique time series relationship of videos is not considered, and the detection accuracy is low.

Therefore, how to improve the accuracy of animal detection in the presence of occlusion has become an urgent problem to be solved by those skilled in the art.

SUMMARY OF THE INVENTION

In view of the above deficiencies in the prior art, the problem to be solved by the present invention is: how to improve the accuracy of animal detection in the presence of occlusion.

In order to solve the above-mentioned technical problems, the present invention adopts the following technical solutions:

An animal detection method based on entropy and motion offset, comprising the following steps:

S1, obtain a video sequence image, the video sequence image includes a current frame image and a previous queue image, and the previous queue image includes a plurality of consecutive images before the current frame image, and perform S2;

S2. Detect the current frame image and the previous queue image based on the YOLOv3 model to obtain detection information of the current frame image and the previous queue image, where the detection information includes detection score, animal category information and animal location information, and execute S3;

S3. If the detection score of the current frame image is greater than or equal to the score threshold, execute S6, otherwise, execute S4;

S4. Calculate the entropy of each image in the previous queue, replace the animal category information of the previous queue image with the lowest entropy for the original animal category information of the current frame image, and execute S5;

S5, calculate the animal position information of the current frame image based on the animal position information of all the previous queue images, and execute S6;

S6, output the detection information of the current frame image.

Preferably, in S4, the method for calculating the entropy of any preceding queue image is as follows:

计算在前队列图像不同尺度下单个区域对应的类别分数和，S为类别分数和，c_i1为单个区域对应的类别i1的识别率，C为类别集合，N1为动物类别集合，N1∈C；S401, based on

Calculate the sum of the category scores corresponding to a single region in the front cohort image at different scales, S is the category score sum, c _i1 is the recognition rate of the category i1 corresponding to a single region, C is the category set, N1 is the animal category set, N1∈C;

计算单个区域对应的类别i1的识别率占类别分数和的比值p(c_i1)；S402, based on formula

Calculate the ratio p(c _i1 ) of the recognition rate of the category i1 corresponding to a single area to the sum of the category scores;

计算单个区域的熵，E_j1为第j1个单个区域的熵；S403, based on formula

Calculate the entropy of a single area, E _j1 is the entropy of the j1th single area;

计算在前队列图像单个尺度的熵，E_K为尺度K的熵，m为在前队列图像尺度K的单个区域的总个数，N2表示YOLOv3模型中尺度K对应的单个区域尺寸参数；S404, based on formula

Calculate the entropy of a single scale of the image in the previous queue, E _K is the entropy of the scale K, m is the total number of single regions of the scale K of the previous queue image, and N2 represents the size parameter of a single region corresponding to the scale K in the YOLOv3 model;

计算在前队列图像的熵E。S405, formula based

Calculate the entropy E of the image in the previous queue.

Preferably, S5 includes the following steps:

S501. Obtain the animal position information of the image in the previous queue with the lowest entropy;

计算当前帧图像的动物位置信息，x_i2、y_i2、w_i2及h_i2分别为当前帧图像中动物图像的x轴定位坐标、y轴定位坐标、宽度及高度，x_j2、y_j2、w_j2及h_j2分别为熵最低的在前队列图像中动物图像的x轴定位坐标、y轴定位坐标、宽度及高度，offset_x、offset_y、offset_w及offset_h为当前帧图像中动物图像相对于熵最低的在前队列图像中动物图像的x轴定位坐标变化量、y轴定位坐标变化量、宽度变化量及高度变化量，

S502, based on

Calculate the animal position information of the current frame image, x _i2 , y _i2 , w _i2 and h _i2 are the x-axis positioning coordinates, y-axis positioning coordinates, width and height of the animal image in the current frame image respectively, x _j2 , y _j2 , w _j2 and h _j2 are the x-axis positioning coordinates, y-axis positioning coordinates, width and height of the animal image in the previous queue image with the lowest entropy, respectively, offset_x, offset_y, offset_w and offset_h are the animal image in the current frame image relative to the lowest entropy image In the front queue image, the x-axis positioning coordinate change, the y-axis positioning coordinate change, the width change and the height change of the animal image,

To sum up, the present invention discloses an animal detection method based on entropy and motion offset. The current frame image is preliminarily detected and judged based on the existing YOLOv3 model. If there is occlusion in the image, the current frame image detection fails. , the animal category information of the current frame image is determined by the previous queue image with the smallest entropy, and the animal position information of the current frame image is calculated by using multiple previous queue images. In this way, animal detection for occluded images is realized, and the stability and accuracy of real-time animal detection are improved.

Description of drawings

Fig. 1 is the flow chart of a kind of animal detection method based on entropy and motion offset disclosed by the present invention;

Fig. 2 is the schematic diagram of the entropy of the unoccluded image under the 3rd scale;

Fig. 3 is the schematic diagram of the entropy of the occlusion image under the 3rd scale;

4 is a schematic diagram of the comparison between the entropy of the unoccluded image and the entropy of the occluded image under the third scale;

5 is a schematic diagram of a single-dimensional comparison of the entropy of the unoccluded image and the entropy of the occluded image at the third scale;

FIG. 6 is a schematic diagram of the change curve of the corresponding video sequence image entropy and detection recognition rate in the experiment.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings.

As shown in Figure 1, the present invention discloses an animal detection method based on entropy and motion offset, comprising the following steps:

S1, obtain a video sequence image, the video sequence image includes a current frame image and a previous queue image, and the previous queue image includes a plurality of consecutive images before the current frame image, and perform S2;

S2. Detect the current frame image and the previous queue image based on the YOLOv3 model to obtain detection information of the current frame image and the previous queue image, where the detection information includes detection score, animal category information and animal location information, and execute S3;

S3. If the detection score of the current frame image is greater than or equal to the score threshold, execute S6, otherwise, execute S4;

S4. Calculate the entropy of each image in the previous queue, replace the animal category information of the previous queue image with the lowest entropy for the original animal category information of the current frame image, and execute S5;

S5, calculate the animal position information of the current frame image based on the animal position information of all the previous queue images, and execute S6;

S6, output the detection information of the current frame image.

The YOLOv3 (You Only Look Once) algorithm is a regression-based real-time target detection algorithm proposed by Joseph Redmon and Ali Farhadi in 2018. It is a convolutional neural network that can predict the positions and categories of multiple target boxes at one time. It uses Darknet-53 as the basic network to perform feature extraction on the target; on the basis of Darknet-53, additional convolutional layers are added to predict images at three different scales to obtain higher semantic information.

When predicting the feature map of each scale, three classification predictions and regression prediction of the bounding box position are performed for each region under each feature map, and the feature size T obtained by each prediction task is:

T=N3×N3×[3*(4+1+C)] (1)

In formula (1), N3 is the grid size, and the values of N3 are 13, 26, and 52; 3 is the number of anchor points; 4 is the bounding box offset; 1 is the target prediction value; C is the total number of categories.

After obtaining the corresponding bounding box, target prediction and category prediction, YOLOv3 performs non-maximum suppression (NMS, Non-maximum suppression) to obtain the final prediction result.

Since YOLOv3 splits the video into a series of video sequence images when it performs target detection on the video, it is still detecting pictures in essence, without considering the unique time series relationship of the video. When the video image is affected by a certain degree of occlusion, detection failure will occur and the stability of real-time target detection will be affected. In order to solve the above problem, the present invention expands the YOLOv3 real-time target detection model, and introduces the entropy, motion offset and video-specific time series relationship of the video image into the YOLOv3 model. The detection score of YOLOv3 is judged by the score threshold. If the detection score is lower than the score threshold, the animal category information corresponding to the previous cohort image with the minimum entropy in the previous cohort image is obtained through the time series relationship, and the current frame image and the previous cohort image are calculated. The minimum entropy corresponds to the positional offset between the images in the previous queue, resulting in the final detection output.

In the present invention, the images of the front queue are stored in the front queue, and the front queue is a first-in, first-out queue. In the present invention, the front queue can store 8 images as an example. If the current frame image is occluded, the entropy of the previous queue image with the smallest entropy is taken as the entropy of the current frame queue image. After the detection of the current frame image is completed, the current frame image is also put into the front queue, and the detection of the next frame image is continued, so as to realize the real-time continuous detection of the video image.

The invention discloses an animal detection method based on entropy and motion offset. Based on the existing YOLOv3 model, the current frame image is preliminarily detected and judged. If there is occlusion in the image, the current frame image detection fails, and the minimum entropy is passed. The preceding queue images are used to determine the animal category information of the current frame image, and the animal position information of the current frame image is calculated through multiple preceding queue images. In this way, animal detection for occluded images is realized, and the stability and accuracy of real-time animal detection are improved.

During specific implementation, in S4, the method for calculating the entropy of any preceding queue image is as follows:

计算在前队列图像不同尺度下单个区域对应的类别分数和，S为类别分数和，c_i1为单个区域对应的类别i1的识别率，C为类别集合，N1为动物类别集合，N1∈C；S401, based on

Calculate the sum of the category scores corresponding to a single region in the front cohort image at different scales, S is the category score sum, c _i1 is the recognition rate of the category i1 corresponding to a single region, C is the category set, N1 is the animal category set, N1∈C;

计算单个区域对应的类别i1的识别率占类别分数和的比值p(c_i1)；S402, based on formula

Calculate the ratio p(c _i1 ) of the recognition rate of the category i1 corresponding to a single area to the sum of the category scores;

计算单个区域的熵，E_j1为第j1个单个区域的熵；S403, based on formula

Calculate the entropy of a single area, E _j1 is the entropy of the j1th single area;

计算在前队列图像单个尺度的熵，E_K为尺度K的熵，m为在前队列图像尺度K的单个区域的总个数，N2表示YOLOv3模型中尺度K对应的单个区域尺寸参数；S404, based on formula

Calculate the entropy of a single scale of the image in the previous queue, E _K is the entropy of the scale K, m is the total number of single regions of the scale K of the previous queue image, and N2 represents the size parameter of a single region corresponding to the scale K in the YOLOv3 model;

计算在前队列图像的熵E。S405, formula based

Calculate the entropy E of the image in the previous queue.

To judge whether there is occlusion between video sequence images, the maximum Shannon entropy theory in information theory can be introduced. Entropy represents a measure of the uncertainty of random variables in information theory, and is used to measure the degree of confusion of objects in a plane or an area, reflecting the uncertainty of an information.

Affected by occlusion, when the target enters the occlusion, the target information will be gradually lost, and the number of target feature points will gradually decrease. The reduction of the number of feature points will lead to the instability or even loss of target information, and various recognition results will be produced at this time. When the recognition results are more chaotic, the uncertainty will be higher. According to the definition of information entropy, the greater the degree of confusion, the greater the uncertainty of information, that is, the greater the entropy.

The level of information entropy is inversely proportional to the level of target detection and recognition. The higher the target detection recognition rate, the lower the information entropy; the lower the target detection recognition rate, the higher the information entropy.

Therefore, the present invention uses the animal category information of the previous queue image with the smallest entropy as the animal category information of the current frame image.

During specific implementation, S5 includes the following steps:

S501. Obtain the animal position information of the image in the previous queue with the lowest entropy;

计算当前帧图像的动物位置信息，x_i2、y_i2、w_i2及h_i2分别为当前帧图像中动物图像的x轴定位坐标、y轴定位坐标、宽度及高度，x_j2、y_j2、w_j2及h_j2分别为熵最低的在前队列图像中动物图像的x轴定位坐标、y轴定位坐标、宽度及高度，offset_x、offset_y、offset_w及offset_h为当前帧图像中动物图像相对于熵最低的在前队列图像中动物图像的x轴定位坐标变化量、y轴定位坐标变化量、宽度变化量及高度变化量，

S502, based on

Calculate the animal position information of the current frame image, x _i2 , y _i2 , w _i2 and h _i2 are the x-axis positioning coordinates, y-axis positioning coordinates, width and height of the animal image in the current frame image respectively, x _j2 , y _j2 , w _j2 and h _j2 are the x-axis positioning coordinates, y-axis positioning coordinates, width and height of the animal image in the previous queue image with the lowest entropy, respectively, offset_x, offset_y, offset_w and offset_h are the animal image in the current frame image relative to the lowest entropy image In the front queue image, the x-axis positioning coordinate change, the y-axis positioning coordinate change, the width change and the height change of the animal image,

For occlusion targets, we can predict the target position based on the motion information before occlusion. By using the motion information of the video image before occlusion, the deviation of the predicted position from the actual position of the target caused by the change of the state of the moving target can be avoided, thus the problem of positioning deviation. According to the inertia existing in the movement of the target, its movement speed and acceleration will not change greatly in a short period of time (8 frames are used as an example in the present invention). Therefore, we can assume that the position change relationship between the current frame image and the previous queue image corresponding to the minimum information entropy is a uniform linear change.

The detection and judgment method of the current frame image includes comparing the IoU (Intersection over Union) and the threshold by selecting the intersection between the detection frame and the target real frame. If the real frame of the image is sr and the predicted frame is expressed as sp, then

If the detected target category is represented by '0,1,...,c', and the undetected target is represented by '-1', then determine the category C of the detected target:

The following is the experimental description of the detection by the method disclosed in the present invention:

The experimental environment and configuration of this paper are: Ubuntu 14.04 operating system, Inter Xeon E5-2623 v3 processor, 64GB memory, NVIDIA Tesla K80 GPU, and Keras deep learning framework.

Since the wild animal public dataset AWA2 (animals with attributes) is an image classification dataset, it does not contain the peculiar time series relationship of video datasets. For the video occlusion detection of wild animals, we constructed a wildlife video occlusion detection dataset WVDDS (Wildlife Video Detection Datasets) containing 12 categories. WVDDS manually labels the video data according to the frequency of labeling every 5 frames. The data labeling format For PASCAL VOC, the categories and corresponding quantities contained in the WVDDS dataset are shown in Table 1.

Table 1

In the model training process, we used the EarlyStopping callback function in keras, where val_loss is selected for monitoring data, and training is stopped when val_loss remains stable to a certain extent.

Experiments show that the target detection recognition rate decreases with the increase of the occlusion area of foreign objects; the target detection recognition rate decreases with the increase of the self-occlusion area; the target detection recognition rate decreases with the increase of the occlusion area of foreign objects, and when the occlusion reaches In a certain range, the detection failure will occur; the target detection recognition rate increases with the decrease of the occlusion area of foreign objects; the target detection recognition rate decreases with the increase of the occlusion area of foreign objects, and when the occlusion reaches a certain range, it will A detection error condition occurs.

We obtain a video image with occlusion and a video image without occlusion in the video sequence respectively, and obtain feature maps of three different scales of the image through Darknet-53 and additional convolutional layers. Calculate the information entropy of the image, and obtain the information entropy corresponding to the feature maps of different scales. We select the information entropy corresponding to the third scale feature map for analysis:

If the general form of entropy in three-dimensional coordinate space is

Among them, Φ(N,N,e) represents the change in the image space region, (N,N)∈Ω represents the abscissa and ordinate of the pixel in the image space Ω, W is the value space of Φ, and the function E is the transformation function related to Φ. By comparing the changes of E in the above formula, the change relationship of entropy between different sequences of images can be obtained.

For the image space (N ₁ ,N ₁ )∈Ω1,(N ₂ ,N ₂ )∈Ω2

Figures 2 to 5 show the entropy comparison between the occluded image and the unoccluded image at the third scale. Among them, Figure 2 is the entropy of the unoccluded image; Figure 3 is the entropy of the occluded image, and the raised part represents a sudden increase in entropy, indicating that the content of the corresponding area changes greatly; Figure 4 is the unoccluded image entropy and occluded image entropy The comparison shows that the entropy of the occluded image is significantly larger than the entropy of the unoccluded image in the sudden change of the area; Figure 5 selects the one-dimensional comparison of the entropy of the unoccluded image and the entropy of the occluded image, which shows that the entropy of the occluded image is higher than that of the unoccluded image.

First, we remove the area with a detection score of 0.00 in the feature map of the video sequence image, and use the image entropy calculation step to calculate the entropy data; use the model obtained after training to detect the video sequence image (score threshold = 0.3, Cross-union ratio threshold=0.5), data of detection scores (scores) are obtained, and the entropy of each video image corresponds to the detection score one-to-one. Then, the obtained entropy data are sorted in descending order, and their corresponding detection scores are also arranged in order. Finally, the sorted data is visualized and analyzed, and the analysis result is shown in Figure 6: as the entropy of the video sequence image decreases, it will be affected by other factors (lighting, animal deformation, motion blur) in addition to occlusion during the detection process. etc.), the corresponding test result scores will fluctuate to a certain extent, but on the whole, the curve still shows a clear upward trend.

Figure 6 shows that as the entropy of the video sequence image decreases, the recognition rate of target detection increases; the entropy of the video sequence image is roughly inversely proportional to the recognition rate of target detection.

When the occlusion reaches a certain level, YOLOv3 cannot detect the target; but because the present invention combines information entropy, time series relationship and position offset, it can accurately detect the video image target with occlusion; The design will detect and output each video image in the video, which greatly improves the stability of video target detection.

In order to verify the validity and accuracy of the model in this paper, we use Faster R-CNN, RetinaNet, SSD and YOLOv3 to compare with the model in this paper (ET-YOLO): this paper intends to use mAP (meanAverage Precision) and FPS (Frames Per Second) is used as the performance evaluation index of the algorithm.

Table 4

table 5

Table 4 shows the experimental results of the detection accuracy and detection rate of different models on the WVDDS dataset; Table 5 shows the accuracy rates of different models on the WVDDS dataset for each category. The experimental results show that the detection accuracy of ET-YOLO is higher than that of Faster R-NN, RetinaNet, SSD and YOLOv3; although its detection speed is slightly lower than that of YOLOv3, its detection accuracy is relatively high without affecting real-time target detection. It has increased by 5.5% in YOLOv3.

In summary, the present invention applies the deep learning-based real-time target detection method to the field of wildlife protection, and constructs a wildlife video occlusion detection data set WVDDS containing time-series information, which provides new insights for wildlife target detection research. data resources;

It is proved that the information entropy of video image is inversely proportional to the recognition rate of target detection;

Combining time series information and YOLOv3 model, the problem of occlusion detection is largely solved through the change of information entropy and the relationship between time series, and the stability and recognition rate of real-time target detection are improved;

Calculate the position offset of the detection target over time, which improves the coincidence degree (IoU) between the predicted frame of the occlusion target and the real frame.

The above is only the preferred embodiment of the present invention. It should be pointed out that, for those skilled in the art, without departing from the technical solution, some deformations and improvements can be made, and the technical solutions of the above deformations and improvements should be regarded as falling into the same The scope of protection of the present invention.

Claims (1)

1. An animal detection method based on entropy and motion offset is characterized by comprising the following steps:

s1, acquiring video sequence images, wherein the video sequence images comprise a current frame image and a previous queue image, the previous queue image comprises a plurality of images before the current frame image, and S2 is executed;

S2, detecting the current frame image and the previous queue image based on a YOLOv3 model to obtain detection information of the current frame image and the previous queue image, wherein the detection information comprises detection scores, animal type information and animal position information, and S3 is executed;

s3, if the detection score of the current frame image is larger than or equal to the score threshold value, executing S6, otherwise, executing S4;

s4, calculating the entropy of each previous queue image, replacing the original animal type information of the current queue image with the animal type information of the previous queue image with the lowest entropy, and executing S5; in S4, the entropy of any previous queue image is calculated as follows:

s401, based on

Wherein i1 is equal to N1, the sum of the category scores corresponding to a single region under different scales of the previous queue image is calculated, S is the sum of the category scores, c is the sum of the category scores_i1The identification rate of a category i1 corresponding to a single region, wherein C is a category set, N1 is an animal category set, and N1 belongs to C;

s402, based on formula

Wherein i1 is equal to N1, and the ratio p (c) of the recognition rate of the category i1 corresponding to a single area to the sum of the category scores is calculated_i1)；

S403, based on formula

Where i1 ∈ N1, the entropy of a single region, E, is calculated_j1Entropy of the j1 th individual region;

s404, based on formula

Where m ∈ [0, N2 × N2 × 3)), the entropy of the previous queue image at a single scale is calculated, E _KThe entropy of the scale K is shown, m is the total number of single areas of the scale K of the previous queue image, and N2 represents a single area size parameter corresponding to the scale K in the YOLOv3 model;

s405, based on formula

Calculating the entropy E of the previous queue image;

s5, calculating the animal position information of the current frame image based on the animal position information of all the previous queue images, and executing S6; s5 includes the steps of:

s501, obtaining animal position information of a previous queue image with the lowest entropy;

s502, based on

Calculating animal position information, x, of the current frame image_i2、y_i2、w_i2And h_i2Respectively an x-axis positioning coordinate, a y-axis positioning coordinate, a width and a height of the animal image in the current frame image, x_j2、y_j2、w_j2And h_j2X-axis positioning coordinates, y-axis positioning coordinates, width and height of the animal image in the front queue image having the lowest entropy, respectively, offset _ x, offset _ y, offset _ w and offset _ h are the amount of change in x-axis positioning coordinates, the amount of change in y-axis positioning coordinates, the amount of change in width and the amount of change in height of the animal image in the current frame image with respect to the animal image in the front queue image having the lowest entropy,

and S6, outputting the detection information of the current frame image.

Images