CN116721370A - Capture and identification method of interesting animal behavior clips for long-term video surveillance - Google Patents

Abstract

The invention discloses a capturing and identifying method for interesting animal behavior fragments aiming at long-time video monitoring, which comprises the following steps: acquiring video data of a live video stream; using a motion sensitive detector to find meaningful video clips and materials; outputting the processed motion area list and writing the motion area list into a log; constructing a semantic analysis model of SlowFast; dividing video data and recording time stamp information; acquiring a video frame group corresponding to a single video clip; acquiring the semantics of each video clip; classifying the captured moving pictures into a background and various material categories, outputting different timestamp information of the movement, writing the information into a log, obtaining animal action clips, and submitting the information to a back-end server so as to facilitate later processing and scheduling; and driving the camera to rotate according to the movement type result. The method can help the media staff to acquire animal motion materials, and reduce the workload of the media staff for performing secondary editing on a large number of videos.

Description

Capture and identification method of interesting animal behavior clips for long-term video surveillance

Technical field

The invention belongs to the field of artificial intelligence technology, relates to a video content analysis method, and specifically relates to a method for capturing and identifying interesting animal behavior segments for long-term video monitoring.

Background technique

With the continuous development of video processing technology and the emergence of various live media streams, video content analysis methods have attracted interest from academia and industry. At present, relatively mature products have been successfully used in various monitoring systems to realize functions such as automatic detection and automatic alarm. However, for animal monitoring videos, due to the scarcity of data and the wide variety of data, current mainstream animal media programs or live broadcasts still require manual analysis and editing. , using artificial intelligence algorithms for event detection is still a challenging problem in the field, with the following technical difficulties:

1. For 24-hour live broadcast signals, surveillance videos have redundant segments for a long time, and the collected images are affected by light, shadow, occlusion, etc., and the video image quality is uneven. This limits the deployment and application of the algorithm in complex daily videos.

2. At present, the research goals of motion analysis and understanding at home and abroad are mainly based on humans, and there is still a lack of mature technology and data support for animal motion analysis.

3. The same action has different manifestations in different animal species, such as "tiger running" and "kangaroo running", which results in the animal's actions being divided into a separate category, resulting in a sharp increase in the total number of action categories that need to be detected. This greatly increases the difficulty of classifying actions.

Contents of the invention

In order to overcome the above-mentioned shortcomings of the existing technology, the present invention provides a method for capturing and identifying animal behavior fragments of interest for long-term video surveillance. This method can help media practitioners obtain animal movement materials and reduce the workload of media practitioners in secondary editing of a large number of videos.

The purpose of the present invention is achieved through the following technical solutions:

A method for capturing and identifying animal behavior clips of interest for long-term video surveillance, including the following steps:

Step S1: Obtain live video streaming video data;

Step S2: Propose a motion-sensitive detector based on motion area attention, use the motion-sensitive detector to discover meaningful video clips and materials, and at the same time locate the location of the target with significant semantic expression in the shot;

Step S3: Output the processed motion area list and write it into the log;

Step S4: Construct a SlowFast semantic analysis model. The SlowFast semantic analysis model includes slow channel and fast channel, where:

The slow channel is used to extract spatial semantic information, and its input is a low frame rate sampled video frame;

The fast channel is used to extract temporal semantic information, and its input is high frame rate sampled video frames;

The output results of the fast channel are sent to the slow channel through the lateral connection, realizing information interaction between the two channels and producing more effective video semantic analysis results;

Step S5: Segment the video data in step S1 to obtain one or more video clips, and record the timestamp information of the one or more video clips;

Step S6: Perform frame extraction processing on the one or more video clips to obtain a video frame group corresponding to a single video clip;

Step S7: Input the video frame group corresponding to the one or more video clips into the SlowFast model constructed in step S4 to obtain the semantics of each video clip;

Step S8: Use semantic information to classify the moving pictures captured in step S3 into background and various material categories, output different timestamp information of the movement, write it into the log, obtain animal action clips, and submit the information to the back-end server. , to facilitate post-processing and scheduling;

Step S9: Drive the camera to rotate according to the motion category result of step S8.

Compared with the existing technology, the present invention has the following advantages:

1. The motion-sensitive detector proposed by the present invention can efficiently identify video semantics, filter other redundant information, and has fast computing speed and real-time performance. For captured action clips, the motion-sensitive detector can achieve a recall rate of more than 95%.

2. The Top-5 accuracy of the deep video semantic analysis model based on SlowFast proposed by this invention can reach more than 90%.

3. The present invention can help media practitioners obtain animal motion materials, reduce the workload of media practitioners in secondary editing of a large number of videos, and provide a new idea for subsequent animal motion capture and behavior recognition.

Description of the drawings

Figure 1 is a process thumbnail of the method for capturing and identifying animal behavior clips of interest for long-term video monitoring according to the present invention;

Figure 2 is a structural diagram of a motion-sensitive detector based on motion area attention;

Figure 3 is a schematic diagram of the video semantic analysis network structure based on SlowFast;

Figure 4 is a schematic diagram of the relationship between noise suppression and motion areas;

Figure 5 is a schematic diagram of the capture and recognition results of behavioral fragments of animals of interest.

Detailed ways

The technical solution of the present invention will be further described below in conjunction with the accompanying drawings, but it is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention shall be covered by the present invention. within the scope of protection.

The present invention provides a method for capturing and identifying animal behavior fragments of interest for long-term video monitoring, and completes the research and development of key artificial intelligence technologies in the collection, transmission and distribution, production and production of video information. Through image processing algorithms and Deep learning algorithms build intelligent video semantic analysis systems. The system intelligently obtains animal movement elements from massive video data and accurately locates animal targets in the video. The moving target position information and motion semantics obtained by this system can be applied to other downstream tasks, such as driving surveillance camera movement, intelligent clip push, etc., saving media practitioners the workload of secondary editing. As shown in Figure 1, the method includes the following steps:

Step S1: Obtain live video streaming video data.

In this step, obtain the live video stream by calling the provided interface.

Step S2: Propose a motion-sensitive detector based on motion area attention to discover meaningful video clips and materials, and at the same time locate the location of the target with significant semantic expression in the shot. As shown in Figure 2, the specific steps are as follows:

Step S21: Perform grayscale image processing.

In this step, the OpenCV-python library is used for image grayscale processing.

Step S22: Use foreground detection based on the existing KNN model to remove small change noise. Specific steps are as follows:

Step S221: Compare the new pixel value at a certain position of the image with the historical information of the pixel value (including the pixel values of previous frames and the determination of whether the pixel is foreground or background). If the difference between the pixel values is within the set threshold (needs to be debugged according to the scene and resolution), the new pixel value is considered to match the historical information and is a "potential" category; otherwise, it is not a "potential" category;

Step S222: After all historical information in step S221 is compared, if the number of matches with historical information exceeds the set threshold, then the new pixel is classified as a "potential background point". If the matched historical information belongs to the background The number of points exceeds the set threshold, then the new pixel points are classified as background points;

Step S223: Update new pixels into historical information.

Step S23: Use dilation corrosion operator, area drawing, etc. to connect adjacent motion areas together, and save a list of areas where motion is detected.

In this step, after using KNN to find the foreground, use the image boundary extraction algorithm to find the boundary of the foreground image. Due to the influence of light and other disturbances, the extracted boundary not only contains the target object boundary, but also contains the noise boundary, so the noise boundary is suppressed. And correlating the true areas of motion is important. This step mainly uses the expansion corrosion operator to denoise, and uses methods such as area drawing to correlate the moving areas. The visualization effect is shown in Figure 4. In Figure 4, the upper left is the preliminary drawing of the motion area, the upper right is the first denoising, the lower left is the correlation result of the motion area, and the lower right is the result of the detection frame on the picture.

Step S24: In order to prevent abnormal frames caused by drastic changes in illumination, the sensitive motion detector determines the motion gradient between associated frames, the associated frame area change slope, the ratio of the frame motion area to the associated frame average motion area area, and the number of frame areas. Change amplitude to detect abnormally changing frames.

Step S25: If an abnormally changing frame appears in S24, the frame is considered a non-action frame; otherwise, the frame is considered a normal frame. Whether the frame is a non-action frame depends on the detection result obtained in S23.

Step S3: Output the processed motion area list and write it into the log.

In this step, for frames judged to be in normal motion: directly use the KNN detection results; for frames judged to have huge changes in the motion area: use the existing intersection operation in the motion areas of the previous and next frames, and determine whether the frame is affected by the correlation ratio threshold. It is affected by other factors such as occlusion and lighting.

Step S4: Build SlowFast’s semantic analysis model.

As shown in Figure 3, the sparse frame sampling branch above the SlowFast semantic analysis model structure is the "Slow Pathway" in the SlowFast model. The "Slow Pathway" mainly focuses on extracting spatial semantic information, and its input is a low frame rate sampling video frames (in the experiment, it was set to skip 16 frames for each sampling. If calculated based on the video 30fps, that is, only 2 frames are sampled per second). Low frame rate sampling means that this "slow channel" uses a sequence of video frames with a larger temporal span, so the "slow channel" is not sensitive to temporal changes in the video, but is better at capturing the spatial information of video frames. This is due to "Slow channel" has more 3D convolution kernels.

The branch below Figure 3 is the "Fast Pathway" in the Slow-Fast model. The "Fast Pathway" mainly focuses on extracting temporal semantic information, and its input is a high frame rate sampled video frame (the experiment is set to Slow Pathway α times, usually set to 8). When using a higher timing resolution as the input, the same high timing resolution is maintained as the output without using timing downsampling. At the same time, in order to allow this channel to focus on extracting temporal information, the "fast channel" model is designed to reduce the amount of calculations in the video space. Its convolution channel (determined by the number of convolution kernels) is usually set to the slow channel. β times, usually set to 1/8.

In order to realize the information interaction between "fast channel" and "slow channel", the present invention uses lateral connection, which is a common algorithm in the field of computer vision. For example, in the feature pyramid of 2D image object detection, the features of different layers are connected. Fusion; Two Stream, a mainstream method for video detection, also adopts a similar fusion strategy. Here, as shown in the structure diagram 3, the output results of the fast channel are sent to the slow channel through lateral connections, realizing information interaction between the two channels and producing more effective video semantic analysis results.

Step S5: Divide the video data in step S1 to obtain one or more video segments, and record the timestamp information of the one or more video segments.

In this step, the video data is divided into several 70s video clips, and the timestamp information is recorded for subsequent assembly of the video clips.

Step S6: Perform frame extraction processing on the one or more video segments to obtain a video frame group corresponding to a single video segment.

In this step, for animal videos, there are mostly backgrounds, less foregrounds, and fewer changes between adjacent frames of the video. Frame extraction processing can reduce the amount of model calculations and highlight the amount of information contained in the video frames.

Step S7: Input the video frame group corresponding to the one or more video clips into the trained SlowFast model to obtain the semantics of each video clip.

Step S8: Use semantic information to classify the moving pictures captured in step S3 into background and various material categories, output different timestamp information of the movement, write it into the log, and obtain animal action clips. The recognition results are shown in Figure 5. Show. Furthermore, the obtained logs and animal action clips are submitted to the back-end server for post-processing and scheduling.

Step S9: Drive the camera to rotate according to the motion category result in step S8.

In this step, based on the action priority and target position, a loop correction algorithm is used to drive the camera to rotate so that the target animal is in the middle area of the lens to improve the quality of the material. The specific implementation process is as follows:

Step S91: Obtain the json file containing the coordinates of the moving target captured in step S3.

Step S92: According to the coordinates, set the deflection direction of the spherical camera on the x and y axes, and perform a small deflection to perform position correction.

Step S93: Enable the motion-sensitive detector to detect the position of the moving target again, and repeat steps S91 and S912 until the target is in the middle area of the lens.

Claims (10)

1. A method for capturing and identifying animal behavior clips of interest for long-term video surveillance, characterized in that the method includes the following steps: Step S1: Obtain live video streaming video data; Step S2: Propose a motion-sensitive detector based on motion area attention, use the motion-sensitive detector to discover meaningful video clips and materials, and at the same time locate the location of the target with significant semantic expression in the shot; Step S3: Output the processed motion area list and write it into the log; Step S4: Build SlowFast’s semantic analysis model; Step S5: Segment the video data in step S1 to obtain one or more video clips, and record the timestamp information of the one or more video clips; Step S6: Perform frame extraction processing on the one or more video clips to obtain a video frame group corresponding to a single video clip; Step S7: Input the video frame group corresponding to the one or more video clips into the SlowFast model constructed in step S4 to obtain the semantics of each video clip; Step S8: Use semantic information to classify the moving pictures captured in step S3 into background and various material categories, output different timestamp information of the movement, write it into the log, obtain animal action clips, and submit the information to the back-end server. , to facilitate post-processing and scheduling; Step S9: Drive the camera to rotate according to the motion category result of step S8. 2. The method for capturing and identifying animal behavior segments of interest for long-term video monitoring according to claim 1, characterized in that in step S1, the live video stream is obtained by calling a provided interface. 3. The method for capturing and identifying animal behavior fragments of interest for long-term video monitoring according to claim 1, characterized in that the specific steps of step S2 are as follows: Step S21: Perform grayscale image processing; Step S22: Use foreground detection based on the KNN model to remove small change noise; Step S23: Use the expansion erosion operator and area drawing method to connect adjacent motion areas together, and save a list of areas where motion is detected; Step S24: In order to prevent abnormal frames caused by drastic changes in illumination, the sensitive motion detector determines the motion gradient between associated frames, the associated frame area change slope, the ratio of the frame motion area to the associated frame average motion area area, and the number of frame areas. Change amplitude to detect abnormal change frames; Step S25: If an abnormally changing frame appears in S24, the frame is considered a non-action frame; otherwise, the frame is considered a normal frame. Whether the frame is a non-action frame depends on the detection result obtained in S23. 4. The method for capturing and identifying animal behavior segments of interest for long-term video monitoring according to claim 3, characterized in that in step S21, the OpenCV-python library is used to perform image grayscale processing. 5. The method for capturing and identifying animal behavior fragments of interest for long-term video surveillance according to claim 3, characterized in that the specific steps of step S22 are as follows: Step S221: For the new pixel value at a certain position in the image, compare it with the historical information of the pixel value. If the difference between the pixel values is within the set threshold, it is considered that the new pixel value matches the historical information and is a "potential "" category; on the contrary, it is not a "potential" category; Step S222: After all historical information in step S221 is compared, if the number of matches with historical information exceeds the set threshold, then the new pixel is classified as a "potential background point". If the matched historical information belongs to the background The number of points exceeds the set threshold, then the new pixel points are classified as background points; Step S223: Update new pixels into historical information. 6. The method for capturing and identifying animal behavior fragments of interest for long-term video monitoring according to claim 1, characterized in that in step S3, for frames judged to be normal motion: KNN detection results are directly used; for judgment For frames with huge changes in the motion area: use intersection operation on the motion areas of the previous and next frames, and determine whether the frame is affected by occlusion, lighting and other factors based on the associated proportion threshold. 7. The method for capturing and identifying animal behavior fragments of interest for long-term video monitoring according to claim 1, characterized in that in step S4, the semantic analysis model of SlowFast includes slow channel and fast channel, wherein: The slow channel is used to extract spatial semantic information, and its input is a low frame rate sampled video frame; The fast channel is used to extract temporal semantic information, and its input is high frame rate sampled video frames; The output results of the fast channel are sent to the slow channel through the lateral connection, realizing information interaction between the two channels and producing more effective video semantic analysis results. 8. The method for capturing and identifying animal behavior fragments of interest for long-term video monitoring according to claim 7, characterized in that the convolution channel of the fast channel is set to β times that of the slow channel, and β is set to 1/8 . 9. The method for capturing and identifying animal behavior segments of interest for long-term video monitoring according to claim 1, characterized in that in step S5, the video data is divided into several 70s video segments, and the timestamp information is recorded. So that the video clips can be assembled later. 10. The method for capturing and identifying animal behavior fragments of interest for long-term video monitoring according to claim 1, characterized in that in step S9, a loop correction algorithm is used to drive the camera rotation according to the action priority and the target position, Place the target animal in the middle area of the lens to improve the quality of the material. The specific implementation process is as follows: Step S91: Obtain the json file containing the coordinates of the moving target captured in step S3; Step S92: According to the coordinates, set the deflection direction of the spherical camera on the x and y axes, and perform a small deflection to perform position correction; Step S93: Enable the motion-sensitive detector to detect the position of the moving target again, and repeat steps S91 and S912 until the target is in the middle area of the lens.