CN116052047A - Moving object detection method and related equipment - Google Patents

Abstract

The present application provides a moving object detection method and related equipment, which relate to the field of image processing. The method includes: acquiring video code stream data, and extracting compressed domain syntax elements, which are used to indicate variables in the video code stream data Information; according to the compressed domain syntax elements, use the motion detection network to detect and determine the target moving object. The present application uses the network model to detect moving objects by combining the compressed domain syntax elements, so as to achieve the purpose of ensuring real-time performance and adapting to complex scenes.

Description

Moving object detection method and related equipment

Technical Field

The present application relates to the field of image processing, and in particular to a moving object detection method and related equipment.

Background Art

Moving object detection is a research hotspot in computer vision. It can provide support for video analysis, video retrieval, etc., and has increasingly important application prospects in human-computer interaction, medical diagnosis and other fields.

Most of the existing detection methods use algorithms in the pixel domain to calculate the video pixel data and estimate the moving objects in the video. However, as the video resolution becomes higher and higher, the video pixel data that needs to be processed becomes larger and larger. Such operations will consume a lot of computing resources and the computing speed will also become slower. Therefore, a new moving object detection method is urgently needed.

Summary of the invention

The present application provides a moving object detection method and related equipment, which combines compressed domain syntax elements and uses a network model to perform moving object detection, thereby achieving the purpose of ensuring real-time performance and adapting to complex scenarios.

In a first aspect, a moving object detection method is provided, the method comprising:

Acquire video code stream data, and extract compression domain syntax elements, where the compression domain syntax elements are used to indicate variable information in the video code stream data;

According to the compression domain syntax elements, a motion detection network is used to perform detection to determine the target moving object.

The embodiment of the present application can directly utilize the motion information generated in the video encoding process, saving the steps of calculating the motion information; in addition, it is combined with a motion detection network for detection, thereby realizing efficient and fast video motion target detection tasks, and solving the robustness problem of related methods in complex scenarios while ensuring real-time performance.

In combination with the first aspect, in some implementations of the first aspect, detecting using a motion detection network according to the compressed domain syntax element to determine the target moving object includes:

Determining motion features according to the compressed domain syntax elements;

generating a two-dimensional matrix according to the motion characteristics;

The two-dimensional matrix is input into the motion detection network for detection to determine the target moving object.

In the embodiment of the present application, the present application does not need to decode the frame image data, and can directly extract reliable compressed domain syntax elements from the compressed domain for motion analysis, so the processing speed can easily reach real-time performance. It is also combined with a motion detection network for detection, so that efficient and fast video motion target detection tasks can be achieved, and the problems of weak robustness and poor performance in complex scenes in related methods can be solved while ensuring real-time performance.

In combination with the first aspect, in some implementations of the first aspect, determining the motion feature according to the compressed domain syntax element includes:

Determining motion features corresponding to the P frame according to compression domain syntax elements of the P frame, wherein the video code stream data includes an I frame, a P frame, and a B frame;

Determine motion features corresponding to the B frame according to the compression domain syntax elements of the B frame;

The motion features corresponding to the I frame are determined by using an interpolation method according to the motion features corresponding to the P frames adjacent to and before the I frame and/or the motion features corresponding to the B frame.

In an embodiment of the present application, the motion information of the P frame and the B frame is combined, and the motion features are interpolated based on the spatiotemporal coherence of the moving object, so that the motion information of the I frame can be obtained, so that the motion information corresponding to each frame can be determined.

In combination with the first aspect, in some implementations of the first aspect, the method further includes:

The motion features corresponding to the I frame, the P frame and the B frame are smoothed.

In the embodiment of the present application, after smoothing, the transition of motion information between adjacent video frames is more natural, the noise in the false detection area can be removed, and the occurrence of individual data anomalies and large differences can be avoided.

In combination with the first aspect, in certain implementations of the first aspect, the compression domain syntax elements include: coding bit amount, motion vector and residual coefficient.

In the embodiment of the present application, the present application does not need to decode the frame image data, and can directly extract three kinds of motion information, namely, reliable coding bit amount, motion vector and residual coefficient, from the compressed domain for motion analysis, so the processing speed can easily reach real-time performance.

In combination with the first aspect, in some implementations of the first aspect, the motion features include: motion information amount, motion vector intensity, and residual coefficient density;

The amount of motion information corresponds to the amount of coded bits, the motion vector strength corresponds to the motion vector, and the residual coefficient density corresponds to the residual coefficient.

In the embodiment of the present application, since there are three types of compression domain syntax elements, three motion features, namely, motion information amount, motion vector intensity, and residual coefficient density, are designed based on the three compression domain syntax elements to characterize the motion of the video picture.

In combination with the first aspect, in some implementations of the first aspect, the motion detection network includes: a Darknet neural network model and a YOLOv3 target detection model;

Inputting the two-dimensional matrix into the motion detection network for detection to determine the target moving object includes:

Inputting the two-dimensional matrix into the Darknet neural network model to obtain a multi-scale convolutional feature layer;

The multi-scale convolutional feature layer is input into the YOLOv3 target detection model to determine the frame location corresponding to the target moving object.

In an embodiment of the present application, a two-dimensional matrix is input into a Darknet neural network model to fuse different motion features and learn an effective and comprehensive motion semantic representation; motion detection is performed on the multi-scale feature layer output by the neural network model based on the YOLO-v3 target detection model to predict moving targets in the video screen.

In a second aspect, an electronic device is provided, the electronic device comprising: one or more processors, a memory, and a display screen; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute:

Acquire video code stream data, and extract compression domain syntax elements, where the compression domain syntax elements are used to indicate variable information in the video code stream data;

According to the compression domain syntax elements, a motion detection network is used to perform detection to determine the target moving object.

In conjunction with the second aspect, in some implementations of the second aspect, detecting using a motion detection network according to the compressed domain syntax element to determine the target moving object includes:

Determining motion features according to the compressed domain syntax elements;

generating a two-dimensional matrix according to the motion characteristics;

The two-dimensional matrix is input into the motion detection network for detection to determine the target moving object.

In conjunction with the second aspect, in some implementations of the second aspect, determining the motion feature according to the compressed domain syntax element includes:

Determining motion features corresponding to the P frame according to compression domain syntax elements of the P frame, wherein the video code stream data includes an I frame, a P frame, and a B frame;

Determine motion features corresponding to the B frame according to the compression domain syntax elements of the B frame;

The motion features corresponding to the I frame are determined by using an interpolation method according to the motion features corresponding to the P frames adjacent to and before the I frame and/or the motion features corresponding to the B frame.

In conjunction with the second aspect, in some implementations of the second aspect, the method further includes:

The motion features corresponding to the I frame, the P frame and the B frame are smoothed.

In combination with the second aspect, in certain implementations of the second aspect, the compression domain syntax elements include: coding bit amount, motion vector and residual coefficient.

In conjunction with the second aspect, in some implementations of the second aspect, the motion features include: motion information amount, motion vector intensity, and residual coefficient density;

The amount of motion information corresponds to the amount of coded bits, the motion vector strength corresponds to the motion vector, and the residual coefficient density corresponds to the residual coefficient.

In conjunction with the second aspect, in some implementations of the second aspect, the motion detection network includes: a Darknet neural network model and a YOLOv3 target detection model;

Inputting the two-dimensional matrix into the motion detection network for detection to determine the target moving object includes:

Inputting the two-dimensional matrix into the Darknet neural network model to obtain a multi-scale convolutional feature layer;

The multi-scale convolutional feature layer is input into the YOLOv3 target detection model to determine the frame location corresponding to the target moving object.

It should be understood that the expansion, limitation, explanation and description of the relevant contents in the above-mentioned first aspect also apply to the same contents in the second aspect.

In a third aspect, a moving object detection device is provided, comprising a unit for executing any one of the moving object detection methods in the first aspect.

In one possible implementation, when the motion object detection device is an electronic device, the processing unit may be a processor, and the input unit may be a communication interface; the electronic device may also include a memory, which is used to store computer program code, and when the processor executes the computer program code stored in the memory, the electronic device executes any one of the methods in the first aspect.

In a fourth aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes any one of the moving object detection methods in the first aspect.

In a fifth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by an electronic device, the electronic device executes any one of the moving object detection methods in the first aspect.

In a sixth aspect, a computer program product is provided, the computer program product comprising: a computer program code, when the computer program code is executed by an electronic device, the electronic device executes any one of the moving object detection methods in the first aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

FIG2 is a schematic diagram of a video encoding and decoding process provided by an embodiment of the present application;

Figure 3 shows three types of video frames involved in the encoding and decoding process;

FIG4 is a schematic diagram of a sequence involved in the encoding and decoding process;

FIG5 is a schematic diagram of a flow chart of a moving object detection method provided in an embodiment of the present application;

FIG6 is a schematic diagram of a motion vector provided in an embodiment of the present application;

FIG7 is a schematic diagram of dividing a video frame into pixel blocks provided by an embodiment of the present application;

FIG8 is a visualization image of a two-dimensional matrix provided in an embodiment of the present application;

FIG9 is a schematic diagram of a hardware system of an electronic device applicable to the present application;

FIG10 is a schematic diagram of a software system of an electronic device applicable to the present application;

FIG11 is a schematic structural diagram of a moving object detection device provided by the present application;

FIG12 is a schematic diagram of the structure of an electronic device provided by the present application.

DETAILED DESCRIPTION

The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

First, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

1. Encoding and decoding. Encoding is the process of converting information from one form or format to another, and decoding is the reverse of encoding.

2. H264 and H265 are both video codec standards. H265 is an upgraded version of H264. The H265 standard retains some of the original technologies of H264 and improves some related technologies.

3. Resolution, which is the number of pixels in a plane. Usually expressed as width (W) × height (H). For example, a common 1080P image means that there are 1920 × 1080 pixels in a plane.

4. Data rate refers to the data flow rate used by a video file per unit time, also called bit rate or bit rate, and the unit is Kb/s or Mb/s. The larger the bit rate, the more data is transmitted per unit time, and the more information is included.

5. Macroblock: Macroblock is a basic concept in video coding technology. It divides the picture into blocks of different sizes to implement different compression strategies at different locations.

In video coding, a coded image is usually divided into several macroblocks. A macroblock consists of a luminance pixel block and two additional chrominance pixel blocks. For example, the luminance block is a 16x16 pixel block, and the size of the two chrominance image pixel blocks depends on the sampling format of the image. For example, for a YUV420 sampled image, the chrominance block is an 8x8 pixel block. In each image, several macroblocks are arranged in the form of slices. The video coding algorithm encodes each macroblock one by one in units of macroblocks and organizes them into a continuous video code stream.

6. Optical flow algorithm. Optical flow refers to the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane. The optical flow algorithm uses the change of pixels in the time domain in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames.

The above is a brief introduction to the terms involved in the embodiments of the present application, which will not be repeated below.

To illustrate the scenarios, FIG1 shows schematic diagrams of three application scenarios, and the three application scenarios are introduced and explained respectively below.

Scenario 1: Security field

In order to protect personal safety and property safety, monitoring equipment is usually installed in homes, schools, factories, etc., and monitoring images are collected and displayed using the monitoring equipment. The monitoring image refers to the image displayed on the display screen after the monitoring equipment captures the shooting scene.

For example, taking the installation of monitoring equipment around a factory as an example, as shown in (a) of Figure 1, the monitoring screen is displayed. After the monitoring equipment runs the moving object detection method, the motion in the monitoring screen can be analyzed, and the identified moving objects (such as moving human figures) can be marked and highlighted, reminding the security personnel watching the monitoring screen that there are suspicious persons and to guard against thieves breaking into the house and theft.

Scenario 2: Public transportation

On some roads or intersections, in order to monitor and adjust the flow of vehicles, road surface monitoring equipment is usually installed, and the road surface monitoring equipment is used to collect and display monitoring images.

For example, taking the installation of road monitoring equipment on a certain road as an example, as shown in (b) of Figure 1, it is a displayed monitoring screen. After the road monitoring equipment runs the moving object detection method, it can analyze the movement of the vehicles appearing in the monitoring screen, and identify and display the identified moving objects (such as vehicles). When it is detected that there are a lot of vehicles traveling in a certain direction, such as more than the preset vehicle number threshold, in order to avoid traffic congestion and paralysis, relevant personnel can be prompted to control and adjust the traffic lights to speed up the vehicle passage.

Scene 3: Shooting area

Taking a mobile phone as an example of an electronic device, as shown in (c) in FIG1 , when a user uses a mobile phone to take a photo, in order to improve the shooting effect, the mobile phone can run a moving object detection method to analyze the movement in the shot image, and identify and display the identified moving object (such as a moving puppy); the moving object detection method can also be used to more accurately locate the moving area, which can be used to indicate the area where the moving object is located; then, according to the physical area in the shooting scene corresponding to the moving area, the mobile phone can also use the moving object detection method to perform intelligent focusing, such as reducing the focal length when the physical area is close to the lens, and increasing the focal length when the physical area is far from the lens.

In the above three application examples, the moving object detection method run by the monitoring equipment, road monitoring equipment or mobile phone can be the first detection method or the second detection method provided by the relevant technology. The two detection methods are introduced and explained respectively below.

The first detection method is usually performed in the pixel domain. First, an algorithm is used to calculate the pixel data of multiple frames of video, and then the moving objects in the video are estimated based on the calculation results. For example, the optical flow algorithm is first used to calculate the pixel values at the same position in the pixel data of multiple frames of video to determine the change of the pixel values at the same position; then the changes in the video are captured based on the pixel changes at each same position in the pixel data of multiple frames of video. Pixels at the same position refer to pixels located in the same row and column in different video pixel data.

However, this detection method based on pixel domain detection requires more data to be processed when the video frame resolution is larger. When the resolution is large enough, the video pixel data that needs to be processed is very large, and the calculation may take a very long time. In this way, for monitoring equipment with high real-time requirements, it may cause very serious delays, which may lead to the failure to trigger the burglary alarm in time or the failure to switch the traffic light in time, resulting in serious consequences.

The second detection method is a method from the perspective of the compressed domain. First, the motion information is directly extracted from the compressed domain code stream. Then, based on the motion information, the motion area and/or moving object are determined in combination with traditional spatiotemporal analysis algorithms, such as Markov random fields, conditional random fields, etc. Although the second detection method can directly obtain motion information, it saves calculation steps compared to the first detection method and achieves a faster detection speed, but the second method can only consider simple motion situations and motion scenes with sparse objects, and cannot be applied to complex scenes.

For example, when there are multiple overlapping moving objects in the scene captured by the video, such as multiple people standing together in a dense crowd, when the second detection method is used to detect moving objects in the video, the overlapping targets are often merged into one target object, resulting in missed detections.

For example, when there is obvious dynamic background noise in the scene of the video, such as when a sudden wind blows the branches, causing them to shake continuously, at this time, when the second detection method is used to detect moving objects in the video, the noise is often mistakenly identified as a moving object, resulting in false detection.

In addition to the missed detection and false detection cases shown above, the parameters of the second detection method are often fixed and cannot be adjusted adaptively, so the second detection method can only be applied to specific scenarios and cannot make adaptive responses to various scenarios. In short, although the second detection method can solve the real-time problem, due to the weak robustness of the algorithm, it is prone to missed detection, false detection and other problems, which affects the user experience.

In view of this, an embodiment of the present application provides a moving object detection method, which can directly use the motion information generated in the video encoding process to perform motion analysis; combined with the motion information, based on the spatiotemporal coherence of the moving object, the motion features are interpolated and filtered; then reconstructed into a two-dimensional matrix form and input into a Darknet neural network model to fuse different motion features and learn effective and comprehensive motion semantic representation; finally, based on the YOLO-v3 target detection model, motion detection is performed on the multi-scale feature layer output by the neural network model to predict the moving target in the video screen.

Since the method provided in this application can directly utilize the motion information generated in the video encoding process, the calculation steps of the motion information are saved; in addition, it is combined with the Darknet neural network model and the YOLO-v3 target detection model, thereby realizing efficient and fast video motion target detection tasks, and solving the robustness problems of related methods in complex scenarios while ensuring real-time performance.

The moving object detection method provided in the embodiment of the present application can be used in the three scenarios shown in FIG. 1 above, and can also be applied to but not limited to the following scenarios:

Optionally, intelligent human-computer interaction scenarios, such as skeleton tracking, gesture recognition, spatial mapping, 3D reconstruction, map reconstruction, autonomous navigation, augmented reality (AR) scenarios, etc.

Optionally, the video playback scenarios include video calls, video conferencing applications, long and short video applications, video live broadcast applications, video online course applications, and smart cat's eye scenarios.

It can also be applied to remote medical diagnosis, etc. It should be understood that the above is an example of the application scenario and does not impose any limitation on the application scenario of this application.

Since the present application involves a video encoding and decoding process, the video encoding and decoding process is briefly introduced below in conjunction with Figures 2 and 3, and then the moving object detection method provided in an embodiment of the present application is introduced in conjunction with Figures 4 to 5.

Refer to Figure 2, which is a schematic diagram of a video encoding and decoding process provided in an embodiment of the present application.

As shown in Figure 2, with the increasing demand for high-definition videos, the amount of video data is also increasing. If not compressed, these videos are difficult to apply to actual transmission and storage. For example, the three color components RGB of each pixel of an image require one byte each, so each pixel requires at least 3 bytes, and an image with a resolution of 1280×720 requires 2.76M bytes. In order to reduce the amount of video data, related technologies generally compress video data using video encoding technology (such as H264, H265) before transmission and storage.

Subsequently, when the user triggers the electronic device to play a video, the electronic device first obtains the video file and parses the video file into standard corresponding encapsulation format data according to the decoding protocol. These encapsulation format data may include video code streams and audio code streams. The encapsulation format may be, for example, AVI, MKV, or MP4.

After obtaining the packaged format data, the electronic device performs decapsulation processing on the packaged format data, separates the video code stream and the audio code stream, and obtains the audio code stream data and the video code stream data (also called video compression data). Furthermore, for the video code stream data, the electronic device can use video decoding technology to decode the video code stream data to obtain video pixel data; then, the electronic device plays the video pixel data through the display screen.

During the video transmission process, the first detection method provided by the related technology is equivalent to performing moving object detection processing on the decoded video pixel data, and the second detection method is equivalent to performing moving object detection processing on the video bitstream data after decapsulation format processing. The detection method provided by the present application is also equivalent to performing moving object detection processing on the video bitstream data after decapsulation format processing.

Since the detection method provided in this application is in the decapsulation stage and needs to be processed based on the video file generated by encoding, some simple explanations of the encoding and decoding logic are also given here.

Refer to FIG. 3 , which shows three types of video frames involved in the encoding and decoding process.

In video images, it is often found that there are a lot of repeated contents between video frames, such as background. Content such as background may not change in several frames or even several seconds. Therefore, in order to solve these redundant data problems, the next frame can be rendered based on the previous frame to reduce bandwidth. For example, as shown in (a) in Figure 3, the background and the ball have not changed in the three frames, only the big tree and the rectangle have changed; thus, the background and the ball can be separated from the big tree and the rectangle; based on the background and the ball in the previous frame, the background and the ball in the next frame are rendered, thereby achieving the purpose of reducing bandwidth.

According to this logic, when encoding, as shown in (b) in Figure 3, video frames can be divided into three types. The first type: I frame, also known as key frame, is a frame of complete data that does not rely on other data for rendering, similar to a static image, as shown in the left image in (b) in Figure 3. The second type: P frame, also known as predicted frame, is a frame rendered based on I frame or the previous P frame; when decoding, it is necessary to superimpose the difference between the I frame and the P frame defined by the current frame with the previously cached I frame picture, so that the picture of the current P frame can be constructed. The P frame is shown in the right image in (b) in Figure 3. The third type: B frame, also known as bidirectional predicted frame, is a frame rendered based on I frame and P frame; when decoding, the final picture is obtained by superimposing the cached I frame picture, the decoded P frame picture and the current frame data; B frame has a better compression rate, as shown in the middle image in (b) in Figure 3. It should be understood that I frame removes redundant information of video frame in spatial dimension, and P frame and B frame remove redundant information of video frame in time dimension. It should also be understood that when decoding, it is necessary to decode in the order of I frame, P frame, and B frame.

Based on the three types of video frames introduced above, as shown in Figure 4, multiple frames starting from an I frame and ending at the next I frame can be called a sequence, and a sequence is a continuous video code stream data. Among them, each video frame is composed of multiple macroblocks. It should be noted that in addition to pixel data, a macroblock can also include other information, such as the position information of the macroblock in the corresponding video frame, the type of the corresponding video frame (I frame, P frame or B frame type), etc.

Refer to Figure 5, which is a flow chart of a moving object detection method provided in an embodiment of the present application. As shown in Figure 5, the method 10 may include the following S11 to S16, and S11 to S16 are introduced in detail below.

S11. Obtain video code stream data and extract compression domain syntax elements.

The above S11 can also be described as: acquiring video code stream data, parsing the video code stream data according to the video compression format, and extracting compression domain syntax elements corresponding to P frames and compression domain syntax elements corresponding to B frames.

The video code stream data may be data stored in an electronic device or data received from other electronic devices or cloud servers, and this application does not impose any restrictions on this. The video code stream data is data transmitted after encoding and before decoding, or in other words, the video code stream data is lossy video compression data to be decoded. For example, the video code stream data may be surveillance video captured by a monitoring device, a road monitoring device, or a mobile phone in the three application scenarios shown in FIG1.

The compression domain syntax elements are a general term for variable information in video bitstream data. In the process of H264 and H265 video compression encoding, some syntax elements that describe video motion are generated. That is to say, the video bitstream data that is transmitted after encoding itself includes compression domain syntax elements. Therefore, they can be directly extracted from the video bitstream data and used for motion analysis without other calculations and processing. It should be noted that since the I frame is a key frame, it is a frame of complete data and does not contain information about other frames. Therefore, the I frame has no corresponding compression domain syntax elements; while P frames and B frames need to be rendered based on other frames and carry information about their corresponding previous and next frames. Therefore, P frames and B frames have corresponding compression domain syntax elements.

The compression domain syntax elements may include: coding bit amount, motion vector and residual coefficient. These three compression domain syntax elements are generated during the video encoding process and have certain motion representation capabilities. The principles are described below.

1) Encoding bit quantity. From the perspective of video encoding, the larger the bit quantity consumed in the encoding process, the less the amount of predictable redundant information. For the surveillance video captured in the scene shown in Figure 1, the appearance of the target and its complex movement are unpredictable. The moving area in the video frame will consume more bits to encode and generate unpredictable motion information. Therefore, the amount of bits required for encoding can be equivalent to the amount of information required for encoding the target to a certain extent. There is a high probability that there are moving objects in the moving area with information.

2) Motion vector is a syntactic element used for motion estimation in the video encoding process, which indicates the displacement of a video pixel relative to a reference pixel. The displacement data includes a horizontal component and a vertical component. A motion vector can be understood as a motion representation similar to optical flow. Referring to FIG. 6 , FIG. 6 is a schematic diagram of a motion vector provided in an embodiment of the present application. For example, the arrow shown in (b) of FIG. 6 can be used to represent the motion vector corresponding to the car shown in (a) of FIG. 6 .

3) Residual coefficients: Due to the movement of objects, there are obvious differences between the previous and next frames of pixels in the moving area, so the macroblocks in the moving area will carry more residual information than the background area. After the discrete cosine transform (DCT) in the video compression process, the DCT blocks in these areas will have more non-zero residual coefficients. Therefore, the residual coefficients can be used to reflect the motion in the video picture.

DCT is a type of Fourier transform used for lossy data compression of images or videos. DCT divides an image into small blocks of different frequencies, which are called DCT blocks. Then, the DCT blocks are quantized. During the quantization process, high-frequency components are discarded and the remaining low-frequency components are saved for image reconstruction.

Based on the above principle, the motion contained in the P frame can be represented by extracting the three compression domain syntax elements corresponding to the P frame; the motion contained in the B frame can be represented by extracting the three compression domain syntax elements corresponding to the B frame. It should be understood that the three compression domain syntax elements corresponding to each P frame and each B frame included in the video bitstream data can be extracted.

S12. Determine motion features corresponding to the P frame and the B frame according to the compression domain syntax elements.

The above S12 can also be described as: determining the corresponding motion feature according to the compression domain syntax element corresponding to the P frame; determining the corresponding motion feature according to the compression domain syntax element corresponding to the B frame. The motion feature may include: motion information amount b, motion vector strength m, and residual coefficient density d.

It should be understood that there are three compression domain syntax elements corresponding to the P frame, namely, the coding bit amount, motion vector and residual coefficient. Based on this, the corresponding motion information amount b can be determined according to the coding bit amount, the corresponding motion vector strength m can be determined according to the motion vector, and the corresponding residual coefficient density d can be determined according to the residual coefficient.

Similarly, there are three types of compression domain syntax elements corresponding to B frames, namely, coding bit amount, motion vector and residual coefficient. Based on this, the corresponding motion information amount b can be determined according to the coding bit amount, the corresponding motion vector strength m can be determined according to the motion vector, and the corresponding residual coefficient density d can be determined according to the residual density. The following introduces the process of determining motion features.

，每个像素块

可以基于提取的三种压缩域语法元素，计算出对应的三个运动特征。应理解，光栅扫描顺序指的是从左往右、从上往下，先扫描一行再移动至下一行起始位置进行扫描的顺序。For example, refer to FIG. 7 , which is a schematic diagram of dividing a video frame into blocks provided by an embodiment of the present application. As shown in (a) of FIG. 7 , assuming that the t-th video frame U(t) is divided into non-overlapping 4×4 pixel blocks, and the block addresses are allocated in a raster scanning order, then the i-th pixel block can be represented as

, each pixel block

The corresponding three motion features can be calculated based on the three extracted compression domain syntax elements. It should be understood that the raster scanning order refers to the order of scanning from left to right and from top to bottom, first scanning a line and then moving to the starting position of the next line for scanning.

包含了

个4×4的子块

，其中任一个子块

对应的运动信息量

可以基于以下公式（一）确定：1) Coding unit (CU) is the basic unit of the video encoder. As shown in (b) of Figure 7, it is assumed that in the H.264 encoding process, the video frame is divided into 4×4 coding unit blocks (also called macroblocks). Each coding unit block

Included

4×4 sub-blocks

, any of the sub-blocks

The corresponding motion information

It can be determined based on the following formula (I):

为向下取整运算，

指示一个编码单元块所对应的编码比特量，

指示子块的个数，

用于指示编码单元块

所包含的4×4子块的块地址集合。in,

To perform floor operation,

Indicates the amount of coded bits corresponding to a coding unit block,

Indicates the number of sub-blocks,

Used to indicate a coding unit block

The set of block addresses of the included 4×4 sub-blocks.

It should be noted that the block address can be represented by a digital sequence such as 1, 2, 3, etc., of course, other methods can also be used to represent it, and the embodiments of the present application do not impose any restrictions on this. In addition, the above-mentioned division of the coding unit block is based on H.264 as an example. In H.265, the coding unit block is a block of different sizes, and its size range is 16x16 blocks to 128x128 blocks. The calculation method is similar and will not be repeated here.

包含了

个4×4子块，

是一个运动矢量块

所包含的4×4子块的块地址集合，运动矢量块

提取的横竖两个方向的运动矢量值为

和

，则其中的任一个子块

的运动矢量强度

可以基于以下公式（二）确定：2) As shown in (c) of Figure 7, assume that the displacement from one video frame to the next frame is divided into multiple motion vector blocks, each of which is

Included

4×4 sub-blocks,

is a motion vector block

The block address set of the contained 4×4 sub-blocks, motion vector block

The extracted motion vector values in both horizontal and vertical directions are

and

, then any of the sub-blocks

The motion vector strength

It can be determined based on the following formula (II):

为向下取整运算。in,

This is a floor operation.

，其中，DCT块

包含了

个4×4子块，这些4×4子块的块地址集合表示为

，那么对于其中的任一个子块

的残差系数密度

可以基于以下公式（三）确定：3) In order to unify the block size, that is, the accuracy of the calculation, a DCT block can be given

, where DCT block

Included

4×4 sub-blocks, the block address set of these 4×4 sub-blocks is expressed as

, then for any of the sub-blocks

The residual coefficient density of

It can be determined based on the following formula (III):

为向下取整运算，

表示计算一个DCT块中的非零残差系数密度。in,

To perform floor operation,

It means calculating the density of non-zero residual coefficients in a DCT block.

It should be noted that in H.264, DCT transform has 4×4 and 8×8 sizes, and the number of non-overlapping pixel blocks into which the video frame is divided must match the size of the DCT transform.

Based on the above three formulas, three motion features corresponding to each block included in the P frame and three motion features corresponding to each block included in the B frame can be determined respectively.

It should be understood that since the statistical characteristics of the compressed domain syntax elements in the background area and the foreground area are completely different, this difference can be used as a rough consideration criterion for estimating the motion area in the video picture.

S13: Determine the motion features corresponding to the I frame according to the motion features corresponding to the adjacent B frames and/or P frames before and after the I frame.

The motion characteristics of the I frame also include: motion information amount b, motion vector strength m, and residual coefficient density d.

Among them, the number of adjacent multiple frames before and after the I frame can be 1 frame, 2 frames, or 3 frames before and after. The specific number can be set and adjusted as needed, and the embodiment of the present application does not impose any restrictions on this.

For example, if the motion features of the I frame are determined based on the motion features of the 1 video frames that are adjacent to and before the I frame, as shown in FIG4 , for an I frame in a sequence, the 1 video frame that is adjacent to the front is a P frame, and the 1 video frame that is adjacent to the back is a B frame, then the motion features of the I frame can be determined based on the motion features of the 1 P frame and the 1 B frame that are adjacent to the front and back.

For another example, if the motion features of the I frame are determined based on the motion features of the two adjacent video frames before and after the I frame. As shown in FIG4 , for an I frame in a sequence, the two adjacent video frames in front are P frames, and the two adjacent video frames in the back are B frames. Then, the motion features of the I frame can be determined based on the motion features of the two adjacent P frames and the two B frames. The calculation methods of other quantities are similar and will not be described in detail here.

It should be understood that the above are only two examples, and the types of adjacent video frames before and after the I frame can be other combinations, but the adjacent frames before and after the I frame can only be P frames or B frames. Therefore, when determining the motion features of the I frame, the determination can be made based on the motion features of the B frame and the P frame that have been determined.

It should be noted that since the I frame is a complete frame, no corresponding compression domain syntax elements are generated during the encoding process, and therefore the corresponding motion features cannot be determined. However, considering that the moving objects in the video have spatiotemporal continuity and the motion is continuous, the time domain interpolation method can be used to determine the motion features corresponding to the I frame.

Optionally, the motion feature of the I frame can be determined using the following formula (IV):

、运动矢量强度为

、残差系数密度为

，t时间对应的I帧视频帧中第i个子块对应的运动信息量为

、运动矢量强度为

、残差系数密度为

。Among them, t-φ is the start time, t+φ is the end time, τ is any time between the start time and the end time, and the amount of motion information corresponding to the i-th sub-block in the video frame U(τ) corresponding to the τ time is

, the motion vector strength is

, the residual coefficient density is

, the amount of motion information corresponding to the i-th sub-block in the I-frame video frame corresponding to time t is

, the motion vector strength is

, the residual coefficient density is

.

It should be noted that, according to the motion features of the adjacent video frames before and after the I frame, the above formula (IV) can be used to interpolate and determine the motion features of the I frame, so that the missed detection area can be filled. The interpolation window can be changed and can be set according to needs.

S14, performing smoothing processing on the motion features corresponding to the I frame, the P frame and the B frame respectively.

Optionally, a time domain median filtering algorithm may be used for smoothing. Of course, other filtering algorithms or a combination of multiple algorithms may also be used for smoothing, and the embodiments of the present application do not impose any limitation on this.

It should be understood that after smoothing, the transition of motion information between adjacent video frames is more natural, the noise in the false detection area can be removed, and the occurrence of individual data anomalies and large differences can be avoided.

S15. Reconstruct and generate a two-dimensional matrix according to the motion features corresponding to each video frame.

Optionally, the two-dimensional matrix may be reconstructed and generated according to the motion features that have not been smoothed in S13, or the two-dimensional matrix may be reconstructed and generated according to the motion features that have been smoothed in S14.

Since the motion features corresponding to each video frame include three motion features, namely, the amount of motion information, the intensity of motion vectors, and the density of residual coefficients, three two-dimensional matrices can be obtained after reconstructing and generating a two-dimensional matrix.

Refer to Figure 8, which is a visualization image of the two-dimensional matrix generated by reconstructing three types of motion feature data. (a) in Figure 8 is a visualization image of the two-dimensional matrix corresponding to the amount of motion information, (b) in Figure 8 is a visualization image of the two-dimensional matrix corresponding to the motion vector intensity, and (c) in Figure 8 is a visualization image of the two-dimensional matrix corresponding to the residual coefficient density.

It should be noted that since the eigenvalue of each point in the two-dimensional matrix originates from a 4×4 block, that is, the 4×4 block is a 1×1 point when converted to a two-dimensional matrix, the visualization image size of the two-dimensional matrix shown in FIG8 is one quarter of the resolution of the video frame.

S16, after merging the channels of the two-dimensional matrix, input the matrix into a motion detection network for detection to determine the target moving object.

Referring to FIG. 8 , it can be seen that each video frame corresponds to three two-dimensional matrices. Therefore, the three two-dimensional matrices can be channel-merged (concat) and then input into the motion detection network for detection processing.

It should be noted that the input of the deep learning motion detection network is usually in the form of a two-dimensional image. Therefore, in this application, the motion features need to be reconstructed and input into the motion detection network in the form of a two-dimensional matrix, thereby facilitating the motion detection network to perform detection processing. The two-dimensional matrix carries spatial position information.

Optionally, the motion detection network may include: a Darknet neural network model and a YOLOv3 target detection model. According to the processing order, the two-dimensional matrix is first input into the Darknet neural network model for processing after the channel merging processing, and then the processing result is input into the YOLOv3 target detection model for processing.

Among them, the Darknet neural network is used to generate a multi-scale convolutional feature layer based on the two-dimensional matrix after channel merging; the YOLOv3 target detection model is used to generate the border positioning of the target moving object based on the multi-scale convolutional feature layer. The multi-scale convolutional feature layer is used to indicate the fusion results of different scales of the two-dimensional matrix; the border positioning includes the two-dimensional coordinates of a vertex of the border corresponding to the target moving object, as well as the width and height of the border.

It should be understood that the motion detection model including the Darknet neural network model and the YOLOv3 target detection model is a motion detection model that has been trained based on the training sample data and has excellent and stable detection capabilities. The motion detection network can also be a network model such as Faster-RCNN, RetinaNet, SSD, and of course, it can also be a combination of multiple other models, and the embodiments of the present application do not impose any restrictions on this.

It should also be understood that the determined frame location of the target moving object may be displayed on a display screen.

The moving object detection method provided in the embodiment of the present application can directly use the motion information generated in the video encoding process to perform motion analysis; combine the motion information, and interpolate and filter the motion features based on the spatiotemporal coherence of the moving object; then reconstruct them into a two-dimensional matrix form and input them into the Darknet neural network model to fuse different motion features and learn effective and comprehensive motion semantic representation; finally, based on the YOLO-v3 target detection model, motion detection is performed on the multi-scale feature layer output by the neural network model to predict the moving target in the video picture.

Compared with the existing motion detection scheme based on compressed domain, this application does not need to decode frame image data, and can directly extract reliable motion information from the compressed domain for motion analysis, so the processing speed can easily reach real-time. During the analysis, three motion features, namely motion information amount, motion vector intensity, and residual coefficient density, are designed for the three types of compressed domain motion information, namely, coding bit amount, motion vector, and residual coefficient, to characterize the motion of the video screen. In addition, the Darknet neural network model and the YOLO-v3 target detection model are combined to achieve efficient and fast video motion target detection tasks, and solve the problems of weak robustness and poor performance in complex scenarios in related methods while ensuring real-time performance.

The above text, in conjunction with Figures 1 to 8, describes a variety of scenarios and moving object detection methods applicable to embodiments of the present application. The following will describe in detail the software system, hardware system, device, and chip system of the electronic device applicable to the present application in conjunction with Figures 9 to 12. It should be understood that the software system, hardware system, device, and chip system in the embodiments of the present application can execute the various methods of the aforementioned embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the aforementioned method embodiments.

The moving object detection method provided in the embodiment of the present application can be applied to various electronic devices. In the embodiment of the present application, the electronic device can be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

Fig. 9 shows a hardware system of an electronic device applicable to the present application. The electronic device 100 can be used to implement the moving object detection method described in the above method embodiment.

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

It should be noted that the structure shown in FIG. 9 does not constitute a specific limitation on the electronic device 100 .

In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in FIG9 , or the electronic device 100 may include a combination of some of the components shown in FIG9 , or the electronic device 100 may include subcomponents of some of the components shown in FIG9 . The components shown in FIG9 may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). Different processing units may be independent devices or integrated devices. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. For example, the processor 110 may include at least one of the following interfaces: an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input/output (GPIO) interface, a SIM interface, and a USB interface.

Exemplarily, in an embodiment of the present application, the processor 110 can be used to execute the moving object detection method provided in the embodiment of the present application; for example, obtain a first image and a normal map, the normal map is used to represent the normal direction corresponding to the subject, and the first image and the normal map come from different cameras; splice the first image and the normal map to obtain a first spliced image; use the main illumination direction estimation model to process the first spliced image to determine the main illumination direction, and the main illumination direction includes the azimuth and altitude of the light source.

The connection relationship between the modules shown in Fig. 9 is only a schematic illustration and does not constitute a limitation on the connection relationship between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of multiple connection modes in the above embodiments.

The wireless communication function of the electronic device 100 can be implemented through components such as the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

The electronic device 100 can realize the display function through the GPU, the display screen 194 and the application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

Display screen 194 may be used to display images or videos.

The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can perform algorithm optimization on the noise, brightness and color of the image. The ISP can also optimize the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard red green blue (RGB), YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

The external memory interface 120 can be used to connect an external memory card, such as a secure digital (SD) card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

The internal memory 121 may be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area.

The electronic device 100 can implement audio functions, such as music playing and recording, through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor.

The audio module 170 is used to convert digital audio information into analog audio signal output, and can also be used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals.

The speaker 170A, also known as a horn, is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or make a hands-free call through the speaker 170A. The receiver 170B, also known as an earpiece, is used to convert an audio electrical signal into a sound signal.

In some embodiments, the pressure sensor 180A may be provided on the display screen 194. There are many types of pressure sensors 180A, for example, they may be resistive pressure sensors, inductive pressure sensors or capacitive pressure sensors. The capacitive pressure sensor may include at least two parallel plates having conductive materials. When force acts on the pressure sensor 180A, the capacitance between the electrodes changes, and the electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation according to the pressure sensor 180A. The electronic device 100 may also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on a short message application icon, an instruction to view a short message is executed; when a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on a short message application icon, an instruction to create a new short message is executed.

The above describes in detail the hardware system of the electronic device 100, and the following describes the software system of the electronic device 100. The software system may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the layered architecture as an example to exemplarily describe the software system of the electronic device 100.

As shown in Figure 10, a software system using a layered architecture is divided into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces.

In some embodiments, the software system can be divided into four layers, from top to bottom, namely, an application layer, an application framework layer, an Android runtime (Android Runtime) and a system library, and a kernel layer.

The application layer may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

The moving object detection method of the embodiment of the present application can be applied to camera applications, video applications, AR applications, etc.

The application framework layer provides an application programming interface (API) and a programming framework for applications in the application layer. The application framework layer may include some predefined functions.

For example, the application framework layer includes the window manager, content provider, view system, telephony manager, resource manager, and notification manager.

The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, and capture the screen.

Content providers are used to store and retrieve data and make it accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, and phone books.

The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (connected or hung up).

The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, and video files.

The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and disappear automatically after a short stay without user interaction.

Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system. The core library contains two parts: one is the function that the Java language needs to call, and the other is the Android core library.

The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

The system library can include multiple functional modules, such as: surface manager, media library, 3D graphics processing library (such as open graphics library for embedded systems (OpenGL ES) and 2D graphics engine (such as skia graphics library (SGL)).

The surface manager is used to manage the display subsystem and provide the fusion of 2D layers and 3D layers for multiple applications.

The media library supports playback and recording of multiple audio formats, playback and recording of multiple video formats, and still image files. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, moving picture experts group audio layer III (MP3), advanced audio coding (AAC), adaptive multi-rate (AMR), joint photographic experts group (JPG) and portable network graphics (PNG).

The 3D graphics processing library can be used to implement 3D graphics drawing, image rendering, compositing and layer processing.

A 2D graphics engine is a drawing engine for 2D drawings.

The kernel layer is the layer between hardware and software. The kernel layer can include driver modules such as display driver, camera driver, audio driver and sensor driver.

11 is a schematic diagram of the structure of a moving object detection device provided in an embodiment of the present application. The moving object detection device 200 includes an acquisition unit 210 and a processing unit 220 .

The acquisition unit 210 is used to acquire video code stream data and extract compression domain syntax elements, where the compression domain syntax elements are used to indicate variable information in the video code stream data.

The processing unit 220 is used to perform detection using a motion detection network according to the compressed domain syntax elements to determine the target moving object.

Optionally, as an embodiment, the processing unit 220 is further used to determine motion features according to compression domain syntax elements; generate a two-dimensional matrix according to the motion features; and input the two-dimensional matrix into a motion detection network for detection to determine the target moving object.

Optionally, as an embodiment, the processing unit 220 is further used to determine the motion features corresponding to the P frame according to the compression domain syntax elements of the P frame, and the video code stream data includes I frames, P frames and B frames; determine the motion features corresponding to the B frame according to the compression domain syntax elements of the B frame; and determine the motion features corresponding to the I frame using an interpolation method based on the motion features corresponding to the P frames before and after the I frame and/or the motion features corresponding to the B frame.

Optionally, as an embodiment, the processing unit 220 is further configured to perform smoothing processing on motion features corresponding to the I frame, the P frame and the B frame.

Optionally, as an embodiment, the compression domain syntax elements include: coding bit amount, motion vector and residual coefficient.

Optionally, as an embodiment, the motion characteristics include: motion information amount, motion vector strength, and residual coefficient density; the motion information amount corresponds to the coding bit amount, the motion vector strength corresponds to the motion vector, and the residual coefficient density corresponds to the residual coefficient.

Optionally, as an embodiment, the motion detection network includes: a Darknet neural network model and a YOLOv3 target detection model; the processing unit 220 is also used to input the two-dimensional matrix into the Darknet neural network model to obtain a multi-scale convolutional feature layer; the multi-scale convolutional feature layer is input into the YOLOv3 target detection model to determine the border positioning corresponding to the target moving object.

It should be understood that the Darknet neural network model and the YOLOv3 target detection model can be deployed in the moving object detection device 200.

It should be noted that the moving object detection device 200 is implemented in the form of a functional unit. The term "unit" here can be implemented in the form of software and/or hardware, and is not specifically limited to this.

For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combined logic circuit, and/or other suitable components that support the described functions.

Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

Fig. 12 shows a schematic diagram of the structure of an electronic device provided by the present application. The dotted line in Fig. 12 indicates that the unit or the module is optional, and the electronic device 300 can be used to implement the moving object detection method described in the above method embodiment.

The electronic device 300 includes one or more processors 301, which can support the electronic device 300 to implement the method in the method embodiment. The processor 301 can be a general-purpose processor or a special-purpose processor. For example, the processor 301 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

The processor 301 can be used to control the electronic device 300, execute software programs, and process data of the software programs.

The electronic device 300 may further include a communication unit 305 for implementing input (reception) and output (transmission) of signals.

For example, the electronic device 300 may be a chip, the communication unit 305 may be an input and/or output circuit of the chip, or the communication unit 305 may be a communication interface of the chip, and the chip may be a component of a terminal device or other electronic devices.

For another example, the electronic device 300 may be a terminal device, and the communication unit 305 may be a transceiver of the terminal device, or the communication unit 305 may be a transceiver circuit of the terminal device.

The electronic device 300 may include one or more memories 302 on which a program 304 is stored. The program 304 can be executed by the processor 301 to generate instructions 303, so that the processor 301 executes the moving object detection method described in the above method embodiment according to the instructions 303.

Optionally, data may be stored in the memory 302. Optionally, the processor 301 may read data stored in the memory 302. The data may be stored at the same storage address as the program 304, or may be stored at a different storage address than the program 304.

The processor 301 and the memory 302 may be provided separately or integrated together; for example, integrated on a system on chip (SOC) of a terminal device.

Exemplarily, the memory 302 can be used to store the related program 304 of the moving object detection method provided in the embodiment of the present application, and the processor 301 can be used to call the related program 304 of the moving object detection method stored in the memory 302 during video processing to execute the moving object detection method of the embodiment of the present application; for example, obtain video bitstream data and extract compressed domain syntax elements, the compressed domain syntax elements are used to indicate variable information in the video bitstream data; according to the compressed domain syntax elements, use the motion detection network to perform detection and determine the target moving object.

The present application also provides a computer program product, which, when executed by a processor, implements the moving object detection method described in any method embodiment of the present application.

The computer program product may be stored in the memory 302 , for example, the program is converted into an executable target file that can be executed by a processor after preprocessing, compiling, assembling and linking.

The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the moving object detection method described in any method embodiment of the present application is implemented. The computer program can be a high-level language program or an executable target program.

Optionally, the computer-readable storage medium is, for example, a memory 302. The memory 302 may be a volatile memory or a non-volatile memory, or the memory 302 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct memory bus random access memory (DR RAM).

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

In several embodiments provided in the present application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

In addition, the terms "system" and "network" are often used interchangeably in this article. The term "and/or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character "/" in this article generally indicates that the associated objects before and after are in an "or" relationship.

In short, the above is only a preferred embodiment of the technical solution of this application, and is not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application should be included in the protection scope of this application.

Claims (10)

1. A moving object detection method, characterized in that the method comprises: Acquire video code stream data, and extract compression domain syntax elements, where the compression domain syntax elements are used to indicate variable information in the video code stream data; According to the compression domain syntax elements, a motion detection network is used to perform detection to determine the target moving object. 2. The moving object detection method according to claim 1, characterized in that the step of detecting the target moving object by using a motion detection network according to the compressed domain syntax element comprises: Determining motion features according to the compressed domain syntax elements; generating a two-dimensional matrix according to the motion characteristics; The two-dimensional matrix is input into the motion detection network for detection to determine the target moving object. 3. The moving object detection method according to claim 2, characterized in that determining the motion feature according to the compressed domain syntax element comprises: Determining motion features corresponding to the P frame according to compression domain syntax elements of the P frame, wherein the video code stream data includes an I frame, a P frame, and a B frame; Determine motion features corresponding to the B frame according to the compression domain syntax elements of the B frame; The motion features corresponding to the I frame are determined by using an interpolation method according to the motion features corresponding to the P frames adjacent to and before the I frame and/or the motion features corresponding to the B frame. 4. The moving object detection method according to claim 3, characterized in that the method further comprises: The motion features corresponding to the I frame, the P frame and the B frame are smoothed. 5. The moving object detection method according to claim 4 is characterized in that the compressed domain syntax elements include: coding bit amount, motion vector and residual coefficient. 6. The moving object detection method according to claim 5, characterized in that the motion features include: motion information amount, motion vector intensity, and residual coefficient density; The amount of motion information corresponds to the amount of coded bits, the motion vector strength corresponds to the motion vector, and the residual coefficient density corresponds to the residual coefficient. 7. The moving object detection method according to claim 6, characterized in that the motion detection network comprises: a Darknet neural network model and a YOLOv3 target detection model; Inputting the two-dimensional matrix into the motion detection network for detection to determine the target moving object includes: Inputting the two-dimensional matrix into the Darknet neural network model to obtain a multi-scale convolutional feature layer; The multi-scale convolutional feature layer is input into the YOLOv3 target detection model to determine the frame location corresponding to the target moving object. 8. An electronic device, comprising a processor and a memory; The memory is used to store a computer program executable on the processor; The processor is used to execute the moving object detection method according to any one of claims 1 to 7. 9. A chip system, characterized in that it comprises: a processor, used to call and run a computer program from a memory, so that a device equipped with the chip system executes the moving object detection method as described in any one of claims 1 to 7. 10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the moving object detection method according to any one of claims 1 to 7.

Images