TWI887115B - Temporal assistant module - Google Patents

Abstract

The present invention is a temporal feature assistant module for monocular 3D object detection, wherein the temporal feature assistant module adjusts the hidden state information (H t ) of the recurrent neural network module, the long short-term memory module (LSTM module), and the gated recurrent unit module (GRU module) at the current time point, and the output state information (Y _t ) at the current time point, respectively, so as to enhance the average precision ( _AP ) of the assistant effect for detecting objects that are blocked, moved out of the detection screen, or small objects.

Description

Timing feature auxiliary module

A module for object detection, in particular a temporal feature auxiliary module for monocular 3D object detection.

Prior art, as shown in Figure 1, the operation process of the recurrent neural network is shown in Figure 1. First, the hidden state is initialized to ensure that the starting value of each set of sequence data is the same. When the sequence (Input Sequence) data begins to be input, the stored hidden state is integrated with it, and the operation is performed through the hidden layer 11 (Hidden Layer), thereby outputting a new hidden state and output (Output Sequence). This process will be repeated so that new feature information can be learned in each round of the hidden layer and the corresponding result information can be output at the same time. Output state information Y _T0 at time point T0, input state information X _T0 at time point T0, hidden state information H _T0 at time point T0, output state information Y _T1 at time point T1, input state information X _T1 at time point T1, hidden state information H _T1 at time point T1, output state information Y _T2 at time point T2, input state information X _T2 at time point T2, hidden state information H T2 at time point _T2 .

Prior art, as shown in Figure 2, the prior art recurrent neural network module 31 (Recurrent Neural Networks module) hidden layer plays an important role in the entire architecture. It enables the model to integrate the input information with the hidden state of the previous layer. Figure 2 shows the basic unit of the hidden layer, which integrates the hidden state ℎ with the input information 𝑋 at the current time point, and generates output information 𝑌 and a new hidden state ℎ through the activation function.

As shown in Figure 3, the previous technology Long Short-Term Memory (LSTM) is an improved version of the recurrent neural network. Like the traditional recurrent neural network, it also uses sequence data as input. In terms of design, in order to improve the problem of gradient explosion and disappearance when the sequence data is long, LSTM adds a cell state and three gate control units (Gates) constructed by the S-type function (Sigmoid) to the original hidden state for improvement. They are Forget Gate, Input Gate and Output Gate, respectively, so that LSTM can better learn the feature information of longer time series, as shown in Figure 3.

The previous technology Long Short-Term Memory (LSTM) will first merge the hidden state of the previous time point with the current feature information when calculating the time point t, and send it to the three control units for calculation. The forget gate part is shown in equation (2.1). It will generate a set of values 𝐹 between 0 and 1 through the results of the hidden state and the current feature. 𝐹 represents whether to forget the information in the unit state, thereby ensuring the necessity of the information used for the current state, and removing data that has been retained for too long. The input gate is shown in equations (2.2) and (2.3), which respectively represent the proportion 𝐼 of the data to update the unit state and the information 𝑆 to be updated to the unit state. The output gate is shown in equation (2.4). This gate mainly determines the data of the unit state to be output. The proportion 𝑂 to be output is calculated through the Sigmoid function. Finally, the output of each gate is calculated with the unit state and hidden state at the previous time point to obtain the information output by LSTM as shown in equations (2.5) and (2.6). 𝐹 = 𝜎0𝑊0 ∙ [ℎ , 𝑋 ] + 𝑏00 (2.1) 𝐼 = 𝜎(𝑊0 ∙ [ℎ , 𝑋 ] + 𝑏0) (2.2) 𝑆 = 𝑡𝑎𝑛ℎ(𝑊0 ∙ [ℎ , 𝑋0 ] + 𝑏0 )                                     𝑂 = 𝜎(𝑊0 ∙ [ℎ , 𝑋 ] + 𝑏0) (2.4) 𝐶                                                                                                        ℎ = 𝑂 ∗ falcon𝑎𝑛ℎ(𝐶)

Prior art, as shown in Figure 4, the prior art gated recurrent unit 51 (Gated Recurrent Unit, GRU), is a very representative improved version of LSTM. GRU adjusts the input gate and forget gate in LSTM and calls them update gate and reset gate respectively. At the same time, it merges the unit state with the hidden state and omits the output gate. Therefore, its architecture is much simpler than the traditional LSTM, as shown in Figure 4.

In the previous technology, as shown in Figure 4, in terms of data flow, the previous hidden state is first integrated with the current input data and sent to the update gate. A set of values 𝑍 between 0 and 1 is obtained through the Sigmoid function. This value determines how much of the data should be passed on, as shown in formula (2.7). In the reset gate, its main goal is to determine how much past information needs to be forgotten. Like the update gate, a set of values 𝑅 between 0 and 1 is obtained through the Sigmoid function, as shown in formula (2.8). In the part of processing the current data, as shown in formula (2.9), the information in the previous hidden state ℎ  is calculated with the 𝑅 calculated by the reset gate, and the part to be forgotten is removed to obtain the information 𝑂 to be passed on. Finally, the hidden state ℎ  and the current data 𝑂 are calculated with the update ratio 𝑍  to obtain the new hidden state ℎ , as shown in formula (2.10). 𝑍 = 𝜎(𝑊0 ∙ [ℎ , 𝑋 ]) (2.7) 𝑅 = 𝜎(𝑊0 ∙ [ℎ , 𝑋 ]) (2.8) + ℎ = (1 − 𝑍 ) ℎ + 𝑍 ∗ 𝑂 (2.10)

The present invention is a temporal feature assistant module for monocular 3D object detection, wherein the temporal feature assistant module is respectively connected to at least one of a recurrent neural network module, a long short-term memory module (LSTM module), and a gated recurrent unit module (GRU module), wherein a video frame processing a spatio-temporal feature map is processed by the temporal feature assistant module, and the temporal feature assistant module includes: a first two-dimensional convolution layer, which is composed of a hidden state information (H _t-1 ) is input to the first two-dimensional convolution layer; a second two-dimensional convolution layer is input to the second two-dimensional convolution layer by an input state information (X _t ) at a current time point; a first connection layer is output from the first two-dimensional convolution layer to the first connection layer, and from the second two-dimensional convolution layer to the first connection layer; and a third two-dimensional convolution layer is output from the first connection layer to the third two-dimensional convolution layer: wherein the recurrent neural network module (Recurrent Neural Networks module), the long short-term memory module (Long Short-Term Memory module, LSTM) are adjusted respectively through the auxiliary module of the temporal feature (temporal assistant module) module), a hidden state information (H _t ) of the gated recurrent unit module (GRU module) at the current time point, and an output state information (Y _t ) at the current time point, so as to enhance the average precision (AP) of the auxiliary effect for detecting objects that are blocked, moved out of the detection screen, or small objects.

The present invention is an auxiliary module of temporal features, which includes: a backbone network layer, the input end of the backbone network layer is connected to the input data features to extract the input data features; the input end of the temporal assistant module is connected to the output end of the backbone network layer; a neck layer, the input end of the neck layer is connected to the output end of the temporal assistant module to fuse data features; and a detection head layer, the output end of the neck layer is connected to the input end of the detection head layer.

The present invention is an auxiliary module of temporal features, which includes: a backbone layer, the input end of the backbone layer is connected to the input data features to extract the input data features; a neck layer, the input end of the neck layer is connected to the output end of the backbone layer to fuse the data features; the temporal assistant module is placed in the neck layer to integrate the data features of different scales; and a detection head layer, the output end of the neck layer is connected to the input end of the detection head layer.

The present invention is a temporal feature assistant module, which includes: a backbone layer, the input end of the backbone layer is connected to the input data feature to extract the input data feature; a neck layer, the input end of the neck layer is connected to the backbone layer to fuse the data feature; the input end of the temporal feature assistant module is connected to the output end of the backbone layer; and a detection head layer, the output end of the temporal feature assistant module is connected to the input end of the head layer.

The present invention is an auxiliary module of time series characteristics, wherein the recurrent neural network module (Recurrent Neural Networks module) is output from the third two-dimensional convolution layer to a first excitation function layer, and the first excitation function layer outputs the hidden state information (H _t ) at the current time point and the output state information (Y _t ) at the current time point respectively.

The present invention is an auxiliary module of time series features, wherein the long short-term memory module (Long Short-Term Memory module, LSTM) module, comprising: the third two-dimensional convolution layer is output and connected to a forget gate, an input gate, a second excitation function layer, and an output gate, wherein the forget gate, the input gate, and the output gate are sigmoid functions; the forget gate output terminal information is multiplied by the forget gate output terminal information to obtain a first information, the input gate output terminal information is multiplied by the second excitation function layer output terminal information to obtain a second information, and the first information and the second information are added and output to a third excitation function layer and a unit state (C _{t )} at a current time point. ); and after the second excitation function layer output terminal information is multiplied by the output gate information, the hidden state information (H _t ) at the current time point and the output state information (Y _t ) at the current time point are output respectively.

The present invention is an auxiliary module of timing characteristics, wherein the gated recurrent unit module (GRU module) comprises: the output of the third two-dimensional convolution layer is connected to a reset gate and an update gate respectively, wherein the reset gate and the update gate are sigmoid functions; the output terminal information of the reset gate is multiplied with the output terminal information of the first two-dimensional convolution layer, and then output to a second connection layer 57, and the output terminal information of the second connection layer 57 is output to a fourth two-dimensional convolution layer, The output terminal information of the fourth two-dimensional convolution layer is output to a fourth excitation function layer; and the output terminal information of the first two-dimensional convolution layer is delayed and multiplied with the update gate output terminal information to output a third information, and the update gate output terminal information is multiplied with the output terminal information of the fourth excitation function layer to output a fourth information, and after the third information and the fourth information are added, the hidden state information (H _t ) at the current time point and the output state information (Y _t ) at the current time point are output respectively.

The present invention is an auxiliary module of temporal features, which processes a video frame of a spatio-temporal feature map to detect an object, including: at least one anchor base module, which cuts the feature map into multiple grids of different proportions, and places the set at least one anchor base in each grid, captures the anchor base with the highest overlap rate, and detects an object by adjusting the offset.

The present invention is an auxiliary module of temporal features, which processes a video frame of a spatio-temporal feature map to detect objects, including: at least one anchor free module, which detects objects by finding the coordinates of the center point of the object on the feature map and predicting the distances between the center point and the upper, lower, left and right boundaries.

The present invention is an auxiliary module of time series characteristics, wherein a hidden state information (H t ) at the current time point and an output state information (Y _t ) at the current time point of the recurrent neural network module (Recurrent Neural Networks module), the long short-term memory module (Long Short-Term Memory module, LSTM module), and the gated recurrent unit module (GRU _module ) are adjusted respectively, so as to achieve the effect and purpose of enhancing the average precision (AP) of the auxiliary effect for detecting objects that are blocked, moved out of the detection screen, or small objects.

As shown in FIGS. 5 to 7 , the present invention is a temporal feature assistant module 10 for monocular 3D object detection, wherein the temporal feature assistant module 10 is respectively connected to at least one of a recurrent neural network module 501 (Recurrent Neural Networks module), a long short-term memory module 601 (Long Short-Term Memory module, LSTM module), and a gated recurrent unit module 701 (Gated Recurrent Unit module, GRU module), wherein a video frame that processes a spatio-temporal feature map is processed by the temporal feature assistant module 10 (temporal assistant module), and the temporal feature assistant module 10 (temporal assistant module) is connected to at least one of a recurrent neural network module 501 (Recurrent Neural Networks module), a long short-term memory module 601 (Long Short-Term Memory module, LSTM module), and a gated recurrent unit module 701 (GRU module). module) comprises: a first two-dimensional convolution layer, to which a hidden state information (H _t-1 ) at a previous time point is input; a second two-dimensional convolution layer, to which an input state information (X _t ) at a current time point is input; a first connection layer, to which the first two-dimensional convolution layer outputs to the first connection layer, and the second two-dimensional convolution layer outputs to the first connection layer respectively; and a third two-dimensional convolution layer 56, to which the first connection layer outputs to the third two-dimensional convolution layer 56: wherein the auxiliary module 10 (temporal assistant module) of the temporal feature is used to generate the second two-dimensional convolution layer; module), respectively adjusting a hidden state information (H t ) at a current time point of the recurrent neural network module 501 501 (Recurrent Neural Networks module, RNN module), the long short-term memory module 601 601 (Long Short-Term Memory module, LSTM module), and the gated recurrent unit module 701 701 ( _{Gated Recurrent Unit module, GRU module), and an output state information (Y t )} at a current time point, so as to enhance the average precision (AP) of the auxiliary effect for detecting objects that are covered, moved out of the detection screen, or small objects.

As shown in FIG5 , the present invention is an auxiliary module 10 of a time series feature, wherein the recurrent neural network module 501 is output by the third two-dimensional convolution layer 56 to a first excitation function layer, and the first excitation function layer outputs the hidden state information (H _t ) at the current time point and the output state information (Y _t ) at the current time point.

As shown in FIG. 6 , the invention is an auxiliary module 10 of time series features, wherein the long short-term memory module 601 (Long Short-Term Memory module, LSTM module) includes: the third two-dimensional convolution layer 56 The outputs are connected to a forget gate 61, an input gate 62, a second excitation function layer, and an output gate 63, respectively, wherein the forget gate 61, the input gate 62, and the output gate 63 are sigmoid functions; the output terminal information of the forget gate 61 is multiplied by the output terminal information of the forget gate 61 to obtain the first information, the output terminal information of the input gate 62 is multiplied by the output terminal information of the second excitation function layer to obtain the second information, and after the first information and the second information are added, they are respectively output to a third excitation function layer 56 and output to a unit state (C _{t )} at the current time point. ); and after the output terminal information of the second excitation function layer is multiplied by the output gate 63 information, the hidden state information (H _t ) at the current time point and the output state information (Y _t ) at the current time point are output respectively.

As shown in FIG. 7 , the invention is an auxiliary module 10 of timing characteristics, wherein the gated recurrent unit module 701 (GRU module) includes: the output of the third two-dimensional convolution layer 56 is connected to a reset gate 71 and an update gate 72, wherein the reset gate 71 and the update gate 72 are sigmoid functions; the output terminal information of the reset gate 71 is multiplied by the output terminal information of the first two-dimensional convolution layer, and then output to a second connection layer 57, and the output terminal information of the second connection layer 57 is output to a fourth two-dimensional convolution layer 58, and the fourth two-dimensional convolution layer 58 The output terminal information of the first two-dimensional convolution layer is output to a fourth excitation function layer 58; and the output terminal information of the first two-dimensional convolution layer is multiplied with the output terminal information of the update gate 72 after delay, and a third information is output. After the output terminal information of the update gate 72 is multiplied with the output terminal information of the fourth excitation function layer 58, a fourth information is output. After the third information and the fourth information are added, the hidden state information (H _t ) at the current time point and the output state information (Y _t ) at the current time point are output respectively.

As shown in Table 1, this invention is an auxiliary module 10 of time series features, in which the auxiliary module 10 of the improved time series features is tested. The model of VisualDet3D is used for preliminary testing. The model architecture without the time series module is defined as the baseline, and the improved RNN, LSTM, and GRU modules are added to the same position, respectively, by comparing the average precision (AP) of 2D, bird’s eye view, and 3D. That is, the values of 2D AP, BEV AP, and 3D AP are used as a preliminary comparison method for the effectiveness of the module. The preliminary data is shown in Table 1. Referring to the way KITTI classifies object occlusion rates, it is divided into three levels of difficulty: E (easy), M (moderate), and H (hard). Among them, H (hard) has the highest object occlusion rate. In terms of values, red indicates the highest accuracy in this field, and bold indicates data with higher accuracy than the baseline.

As shown in Table 1, the present invention is a time series feature auxiliary module 10, in which KITTI Car 2D AP70↑ 3D AP70↑ 3D AP 50↑ E M H E M H E M H Baseline 97.30 84.54 64.65 19.43 13.60 10.82 55.49 39.03 30.86 RNN 97.28 84.55 64.66 21.77 15.41 11.85 56.21 39.59 31.36 LSTM 97.22 84.49 67.00 21.24 15.78 12.07 59.13 41.71 32.02 GRU 97.27 86.92 67.06 20.89 14.66 11.74 57.32 41.44 31.82

According to the data in Table 1, it can be found that no matter which time series module is added, RNN, LSTM, or GRU, the accuracy of BEV and 3D is improved. Although the effect of 2D is not better than the Baseline, it is on par with LSTM and GRU. RNN lacks the design of forget gate 61 compared with LSTM, and the reference weight of time series data is the same or trade-off, which will cause the object marker box to shift, as shown in Figure 13. In this preliminary experiment, it is verified that the auxiliary module 10 with the time series feature of this creation is helpful for the effect of 3D object detection, and the average improvement of LSTM is the most obvious.

As shown in FIG8 , this invention is an auxiliary module 10 of time series features. In the architecture of FIG8 , the image is first input into the backbone network layer 81 (Backbone), and features are extracted through the backbone network layer 81 (Backbone). At this time, the feature map obtained only contains the feature information of the image. At this time, the feature information only contains the feature data in the most original image, but it also contains the most information. Therefore, the module of this invention is placed after the backbone network layer 81 (Backbone) and the time series module is added to maximize the integration of feature data at different time points, and the integrated results are continuously sent to the neck layer 82 (neck layer) for feature processing.

As shown in FIG8 , the invention is a temporal feature auxiliary module 10, which includes: a backbone network layer 81, the input end of the backbone network layer 81 is connected to the input data feature to extract the input data feature; the input end of the temporal feature auxiliary module 10 is connected to the output end of the backbone network layer 81; a neck layer 82, the input end of the neck layer 82 is connected to the output end of the temporal feature auxiliary module 10 to fuse the data feature; and a detection head layer 83, the output end of the neck layer 82 is connected to the detection head layer. 83(head layer) input port.

As shown in FIG9 , this invention is an auxiliary module 10 of time series features. In addition, the features of the backbone network layer 81 (Backbone) are continuously transmitted to the neck layer 82 (neck layer). In the neck layer 82 (neck layer), features of different scales are mainly integrated. The features of different sizes generated by the backbone network layer 81 (Backbone) are extracted and calculated, and then integrated, and a feature map with multi-scale information is output. The operation process of the neck layer 82 includes feature maps of various scales. Since the size and feature information of each scale are different, the auxiliary module 10 of the timing feature is placed in the neck layer 82, as shown in FIG. 9. By integrating the feature maps of different scales, it is hoped that the features of the original scale can be retained for timing integration. After obtaining the feature map sorted by the neck layer 82 (neck layer), it will be sent to the detection head layer 83 (head layer) for model prediction. The feature map sorted by the neck layer 82 (neck layer) has multi-scale feature information, mainly to allow the detection head layer 83 (head layer) to have better results when calculating large objects and small objects.

As shown in FIG9 , the invention is a temporal feature assistant module 10, which includes: a backbone network layer 81, the input end of which is connected to the input data feature to extract the input data feature; a neck layer 82, the input end of which is connected to the output end of the backbone network layer 81 to fuse the data feature; the temporal feature assistant module 10 (temporal assistant module) is placed in the neck layer 82 to integrate data features of different scales; and a detection head layer 83, the output end of which is connected to the detection head layer 83(head layer) input port.

As shown in FIG10 , the present invention is an auxiliary module 10 of temporal features, and the placement position is to place the auxiliary module 10 of temporal features in front of the detection head layer 83 (head layer). As shown in FIG10 , the integrated multi-scale features are integrated through the auxiliary module 10 of temporal features, so that the feature information input into the detection head layer 83 (head layer) can include not only the object features of each scale, but also the multi-scale object features of adjacent time points.

As shown in FIG10 , the invention is a temporal feature assistant module 10, which includes: a backbone network layer 81, the input end of which is connected to the input data feature to extract the input data feature; a neck layer 82, the input end of which is connected to the backbone network layer 81 to fuse the data feature; the input end of the temporal feature assistant module 10 is connected to the output end of the backbone network layer 81; and a detection head layer 83, the temporal feature assistant module 10 is connected to the output end of the backbone network layer 81; The output of the module is connected to the input of the head layer.

As shown in Table 2, the present invention is an auxiliary module 10 with timing characteristics, wherein the auxiliary module is placed in a position test: KITTI Car 2D AP70↑ 3D AP70↑ 3D AP 50↑ E M H E M H E M H Baseline 97.30 84.54 64.65 19.43 13.60 10.82 55.49 39.03 30.86 Backbone 87.39 72.21 54.78 17.09 11.25 8.58 51.78 35.05 27.01 Neck 94.50 76.99 59.58 18.58 12.56 9.81 52.75 36.42 28.53 Head before 97.33 82.19 64.70 21.24 15.78 12.07 59.13 41.71 32.02

As shown in Table 2, this work is an auxiliary module of temporal features 10. In the test of the effect of placement, this work also uses the VisualDet3D model architecture for testing. Based on the results of the feasibility experiment of the module, LSTM is selected as the module to be used. The module is placed behind the backbone network layer 81 (Backbone), in the neck layer 82 (neck layer), and before the detection head layer 83 (head layer) for experiments, and 2D AP and 3D AP are used as evaluation indicators. The test results are shown in Table 2, and the occlusion rate is also grouped with reference to KITTI. According to the above experiment, although the timing module can be added to different positions for assistance, only adding the timing module before the detection head layer 83 (head layer) can achieve the effect of assisting the output result. Adding the timing module after the backbone network layer 81 (Backbone) and the neck layer 82 (neck layer) does not achieve the effect of improvement, but will reduce the output effect. Therefore, this work proposes that adding the timing module before the detection head layer 83 (head layer) is the best position in the current test.

As shown in Figure 11, this work is an auxiliary module of time series features 10, in which Anchor Based is a method of using pre-selected boxes (Anchor) for object detection. This method cuts the feature map into multiple grids of different proportions and places the set pre-selected boxes in each grid, so as to find the pre-selected box with the highest overlap rate and detect objects by adjusting the offset. For the Anchor Based approach, the design of the pre-selected box is very important. If the size difference between the designed pre-selected box and the actual object is too large, it will increase the burden of the model in training and lead to poor convergence effect. Common pre-selected box design methods are divided into two types: empirical rules and data clustering. The empirical method mainly sets the size and parameters of the pre-selection box based on the designer's past experience, while data clustering sets the corresponding pre-selection box parameters based on the results in the statistically labeled data through clustering.

As shown in Table 3, this work is an auxiliary module of temporal features 10. In order to verify that the temporal feature auxiliary module proposed in this work can be applied to the Anchor Based model, the model architecture proposed by VisualDet3D is used for experiments. By adding the temporal auxiliary module before the model detection head, the model can integrate the image features in the observation data and send the feature map integrated by the auxiliary module to the detection head for detection tasks.

As shown in Table 3, the present invention is an auxiliary module 10 of temporal features, which processes a video frame of a spatio-temporal feature map to detect an object, including: at least one anchor base module, which cuts the feature map into a plurality of grids of different proportions, and places the at least one anchor base in each grid, extracts the anchor base with the highest overlap rate, and detects an object by adjusting the offset.

As shown in Table 3, this work is an auxiliary module of time series features 10, in which VisualDet3Det applies the experimental data of the time series auxiliary module: Car 2D AP70↑ BEV AP70↑ 3D AP70↑ BEV AP50↑ 3D P50↑ E M H E M H E M H E M H E M H Baseline 96.75 84.07 64.66 26.66 19.35 15.06 18.96 13.73 10.72 61.64 43.95 34.17 55.85 40.14 25.40 LSTM 96.75 84.07 66.06 28.48 20.55 16.12 20.90 15.27 11.77 63.87 45.44 35.13 59.12 41.86 32.06 Diff. 0.00 0.00 +1.40 +1.82 +1.21 +1.06 +1.94 +1.54 +1.05 +2.23 +1.50 +0.97 +3.27 +1.72 +6.66 Car 2D AP50↑ BEV AP50↑ 3D AP50↑ BEV AP25↑ 3D P25↑ E M H E M H E M H E M H E M H Baseline 55.98 46.22 39.29 8.39 6.71 5.09 7.44 5.83 4.64 27.13 22.07 18.48 26.34 21.35 17.56 LSTM 58.43 47.05 40.14 9.46 7.52 5.69 8.31 6.49 5.14 28.87 23.66 19.64 28.20 22.81 19.10 Diff. +2.45 +0.83 +0.84 +1.07 +0.81 +0.60 0.87 +0.66 +0.50 +1.74 +1.59 +1.16 +1.86 +1.47 +1.54 Cyclist 2D AP50↑ BEV AP50↑ 3D AP50↑ BEV AP25↑ 3D P25↑ E M H E M H E M H E M H E M H Baseline 53.09 32.25 30.43 3.59 1.98 2.00 3.04 1.72 1.65 14.54 8.22 7.75 13.47 7.50 7.47 LSTM 54.61 3.81 31.67 4.46 2.77 2.00 3.95 2.32 2.36 16.70 9.57 9.50 15.68 9.03 8.76 Diff. +1.52 +1.56 +1.24 +0.87 +0.79 +0.70 +0.91 +0.60 +0.71 +2.16 +1.35 +1.75 +2.21 +1.53 +1.29 mAP 2D↑ BEV Hard↑ 3D Hard↑ BEV Easy↑ 3D Easy↑ E M H E M H E M H E M H E M H Baseline 96.75 84.07 64.66 12.88 9.35 7.38 9.81 7.09 5.67 34.44 24.75 20.13 31.88 23.00 16.81 LSTM 69.93 54.98 45.96 14.13 10.28 8.17 11.05 8.03 6.42 36.48 26.23 21.42 34.33 24.57 19.97 Diff. +1.33 +0.80 +1.16 +1.25 +0.94 +0.79 +1.24 +0.93 +0.75 +2.04 +1.48 +1.29 +2.45 +1.57 +3.16

As shown in Table 3, this work is an auxiliary module of temporal features 10. Experimental data shows that adding a temporal auxiliary module to the Anchor Based model can improve the average prediction accuracy by about 1.4, although the auxiliary effects of individual categories vary. In addition to verifying the effect of the auxiliary module on the original model through data, the effects of this work on the object being blocked, the object partly moving out of the screen, and the detection of small objects will be verified through visualization results.

As shown in FIG14 , the present invention is an auxiliary module 10 of temporal features. In the observation data (T-1), although the vehicle in the middle of the screen is slightly obscured by the vehicle in front, it can still be roughly seen that there is a small car behind it. In the current data (T), the vehicle moves, causing the obscured area to increase, so the Baseline model cannot detect the vehicle. However, after adding the auxiliary module 10 of temporal features of the present invention for assistance, it can be found that the obscured vehicle can still be detected.

As shown in FIG15 , the present invention is an auxiliary module 10 of time series features, in which there is a car on the right side of the observed data (T-1), but when time advances to the current data (T), the car moves out of the screen due to its movement. When the Baseline model that only considers the current data is used, it is found that the car will not be detected. However, when the auxiliary module 10 of time series features of the present invention is used to refer to the observed data, the car will be detected due to the integration of the time series features.

As shown in FIG16 , the present invention is an auxiliary module 10 of time series features. In this embodiment, there is no occlusion or movement out of the screen between the observed data (T-1) and the current data (T), but there are many small objects in the screen. In the baseline model that only uses the current data, the detection effect of the small objects is poor because the features of the small objects are relatively few. In the model that adds the auxiliary time series module, the detection of small objects is improved by integrating the feature information of the observed data.

As shown in Figure 17, this creation is an auxiliary module 10 of temporal characteristics. In the last scenario where there is no occlusion, no moving out of the screen or small objects, as shown in Figure 4.8, although the above special situations do not occur in the scene, the three objects in the screen appear in the screen at the current time point or in the past time point. After adding the temporal auxiliary module, it does not affect its judgment. Therefore, the detection effect is the same as the model without the temporal auxiliary module, which proves that adding the auxiliary module 10 of the temporal characteristics of this creation does not reduce the original detection effect.

As shown in Table 3, this invention is an auxiliary module 10 of temporal features. The comparison results of the auxiliary module 10 of temporal features of this invention in the VisualDet3D model are shown in Table 4. Through experimental data, it can be verified that the auxiliary module 10 of temporal features added to the pre-selected box (Anchor Based) has different auxiliary effects in individual categories, but in terms of average prediction accuracy, the effect of the auxiliary module 10 with temporal features is improved by about 1.4. In addition to verifying the effect of the auxiliary module 10 of the timing characteristics on the original model through data, the auxiliary module 10 of the timing characteristics will be verified through visualization results to improve the effects of situations such as object shape being blocked, part of the object shape moving out of the screen, and small object detection.

As shown in Figure 12, this invention is an auxiliary module 10 of time series features, in which the Anchor Free approach is generally defined as all methods that do not use pre-selected boxes. Since the Anchor Free approach does not use pre-selected boxes, there is no need to set them in advance. By finding the coordinates of the center point of the object on the feature map and predicting the individual distances between the center point and the upper, lower, left and right boundaries, object detection is performed. For Anchor Free, there is no need to set pre-selected boxes in advance, and the computational cost will not be increased due to the need to filter a large number of pre-selected boxes. However, since there is no information about the pre-selected boxes, it is difficult for the model to converge on the regression of the distance information between the center point and the boundary.

As shown in Table 4, the present invention is an auxiliary module 10 of temporal features, which processes a video frame of a spatio-temporal feature map to detect an object, including at least one anchor free module, which detects an object by finding the coordinates of the center point of the object on the feature map and predicting the distances between the center point and the upper, lower, left and right boundaries.

As shown in Table 4, this work is an auxiliary module of timing characteristics 10, in which Monodle applies the experimental data of the timing auxiliary module: Car 2D AP70↑ BEV AP70↑ 3D AP70↑ BEV AP50↑ 3D P50↑ E M H E M H E M H E M H E M H Baseline 95.54 87.09 78.87 23.74 23.03 21.43 17.26 19.16 16.71 58.70 48.78 43.36 53.25 42.59 40.60 LSTM 95.92 87.37 79.10 28.19 23.49 21.82 21.20 19.77 16.99 60.99 49.71 43.92 56.71 43.65 41.47 Diff. +0.38 +0.28 +0.23 +4.45 +0.46 +0.39 +3.94 +0.61 +0.28 +2.29 +0.93 +0.56 +3.46 +1.06 +0.87 Car 2D AP50↑ BEV AP50↑ 3D AP50↑ BEV AP25↑ 3D P25↑ E M H E M H E M H E M H E M H Baseline 74.38 59.74 51.27 8.94 7.70 6.99 6.90 7.13 5.44 28.26 24.44 19.39 27.09 23.22 18.62 LSTM 66.21 64.13 56.02 8.32 6.52 6.34 8.31 6.49 5.62 29.17 25.19 23.53 28.84 24.76 20.55 Diff. -8.17 +4.39 +4.75 -0.62 -1.18 -0.65 -0.2 -1.1 +0.18 +0.91 +0.75 +4.14 +1.75 +1.54 +1.93 Cyclist 2D AP50↑ BEV AP50↑ 3D AP50↑ BEV AP25↑ 3D P25↑ E M H E M H E M H E M H E M H Baseline 67.55 45.55 45.09 8.79 5.48 5.49 7.20 5.40 5.40 23.67 15.25 14.01 23.43 15.05 13.80 LSTM 70.25 46.32 45.85 7.96 5.65 5.65 6.51 5.50 5.51 23.15 14.17 13.43 23.15 14.17 13.43 Diff. +2.7 +0.77 +0.76 -0.83 +0.17 +0.16 -0.69 +0.1 +0.11 -0.52 -1.08 -0.58 -0.28 -0.88 -0.37 mAP 2D↑ BEV Hard↑ 3D Hard↑ BEV Easy↑ 3D Easy↑ E M H E M H E M H E M H E M H Baseline 79.16 64.13 58.41 13.82 12.07 11.30 10.45 10.56 9.18 36.88 29.49 25.59 34.59 26.95 24.34 LSTM 77.46 65.94 60.32 14.82 11.89 11.27 11.47 10.43 9.37 37.77 29.69 26.96 36.23 27.53 25.15 Diff. -1.7 +1.81 +1.91 +1 -0.18 -0.03 +1.24 +0.93 +0.75 +0.89 +0.2 +1.37 +1.64 +0.58 +0.81

As shown in Table 4, this work is an auxiliary module of time series features 10. After experimental data analysis, adding the time series auxiliary module to the Anchor Free model framework improves the prediction accuracy by an average of 0.62. Among individual objects, the improvement of vehicles is the most stable and obvious. In addition to data comparison, visual data will also be presented based on situations such as objects being blocked, being moved out of the screen due to movement, and small object detection, to prove that adding the time series auxiliary module proposed in this work can improve the detection effect of the Anchor Free model in the above situations.

As shown in FIG18 , this invention is an auxiliary module 10 of temporal features, in which, in the case of object occlusion, as shown in FIG18 , it can be seen that the prediction of the Baseline model without the assistance of temporal features does not include the occluded vehicle, while in the model with the assistance module of temporal features, the occluded vehicle can be detected because of the additional reference to the special detection information in the observation data.

As shown in FIG19 , the present invention is a time-series feature auxiliary module 10, in which, in the example of an object moving out of the screen, as shown in FIG19 , the Baseline model without the time-series feature auxiliary module detects the object only through the image features in the current data. When the object moves out of the screen, it will not be able to accurately detect the object due to the lack of complete detection information. In the model with the time-series feature auxiliary module, the information of the object that has not moved out of the screen is referenced, so when the object moves out of the screen, it can still be detected.

As shown in FIG20 , this invention is an auxiliary module 10 of time series features. In terms of small object detection, as shown in FIG20 , in this case, there is no object moving out of the shielding, but the object is smaller in size because it is farther away from the output angle device. In the module that adds time series features for assistance, because there is observation data to reinforce the features of the current small object, the detection effect of the small object is improved.

As shown in Figure 21, this invention is an auxiliary module 10 of the timing feature. In the case where there is no occlusion, no moving out of the screen or small objects, as shown in Figure 4.13, although the above special situations do not occur in the scene, and the objects in the screen appear stably, the detection effect is not affected before and after the addition of the timing auxiliary module.

As shown in Table 5, this work is an auxiliary module of temporal features 10, in which the monocular 3D object detection model is compared: 3D AP70↑ Extra Data Car Pedestrian Cyclist Depth Temporal E M H E M H E M H CaDN V 24.87 15.63 14.47 16.51 13.37 12.21 9.68 9.09 9.09 Kinematic3D Result 13.01 9.43 7.38 1.19 0.57 0.57 0.00 0.00 0.00 VisualDet3D 19.43 13.60 10.82 6.94 5.11 4.31 2.44 1.41 1.43 Monodle 17.26 19.16 16.71 6.90 7.13 5.44 7.20 5.40 5.40 VisualDet3D LSTM 21.24 15.78 12.07 7.94 6.08 4.92 4.55 2.15 2.27 Monodle LSTM 21.20 19.77 16.99 6.70 6.03 5.62 6.51 5.50 5.51

As shown in Table 5, this work is an auxiliary module of temporal features 10. After verifying that this work can be applied to different model architectures, in this section, the comparison results of adding the auxiliary module proposed in this work and the most advanced 3D object detection model will be compared. In terms of comparison objects, the method of monocular 3D object detection is selected, and the model that only uses depth information during training or does not use depth information at all is selected as much as possible. Among the models that use depth, CaDDN is used as the comparison object, and among the models that do not use depth information, Kinematic3D, Monodle and VisualDet3D are selected as representatives, and the temporal module is added to the two models that do not use depth information for comparison. The experimental results are shown in Table 5. The table is divided into two parts. The upper part is the original model structure, and the lower part is the effect of adding the timing feature auxiliary module proposed in this work.

The above description and explanation are only the description of the preferred embodiment of the present invention. Those with ordinary knowledge of this technology can make other modifications according to the scope of the patent application defined below and the above explanation. However, these modifications should still be within the creative spirit of the present invention and within the scope of the rights of the present invention.

Y _T0 Output state information at time T0 X _T0 Input state information at time T0 H _T0 Hidden state information at time T0 Y _T1 Output state information at time T1 X _T1 Input state information at time T1 H _T1 Hidden state information at time T1 Y _T2 Output state information at time T2 X _T2 Input state information at time T2 H _T2 Hidden state information at time T2 X _t Input state information at current time Y _t Output state information at current time H _t-1 Hidden state information at previous time H _t Hidden state information at current time C _t-1 Unit state at previous time C _t Unit state at current time 21 Prior art recurrent neural network module 31 Prior art long short-term memory module 41 Prior art gated recurrent unit module 501 Recurrent neural network module (RNN module) 601 Long short-term memory module (LSTM module) 701 Gated recurrent unit module (GRU module) 11 Hidden layer 51 1st activation function layer 64 2nd activation function layer 65 3rd activation function layer 73 4th activation function layer 54 1st two-dimensional convolution layer 53 2nd two-dimensional convolution layer 56 3rd two-dimensional convolution layer 58 4th 2D convolution layer 55 1st connection layer 57 2nd connection layer 61 Forget gate 62 Input gate 63 Output gate 71 Reset gate 72 Update gate 81 Backbone network layer 82 Neck layer 10 Auxiliary module of timing characteristics 83 Detection head layer

Figure 1 is a schematic diagram of the recurrent neural network process of the prior art. Figure 2 is a schematic diagram of the recurrent neural network cell of the prior art. Figure 3 is a schematic diagram of the long short-term memory cell (LSTM cell) of the prior art. Figure 4 is a schematic diagram of the gated recurrent unit cell (GRU cell) of the prior art. Figure 5 is a schematic diagram of the recurrent neural network of the present invention. Figure 6 is a schematic diagram of the recurrent neural network cell of the present invention. Figure 7 is a schematic diagram of the long short-term memory cell (LSTM cell) of the present invention. Figure 8 is a schematic diagram of the present invention adding a timing module after the backbone network layer. Figure 9 is a schematic diagram of the present invention adding a timing module in the neck layer. Figure 10 is a schematic diagram of the present invention adding a timing module before the detection head layer. Figure 11 is a schematic diagram of the present invention's pre-selected box (Anchor Based) object detection. Figure 12 is a schematic diagram of the present invention's non-pre-selected box (Anchor Free) object detection. Figure 13 is a schematic diagram of the present invention's timing module without adding a forget gate. Figure 14 is a schematic diagram of the present invention's VisualDet3D adding a timing module to assist in the case of an object being obscured. Figure 15 is a diagram showing the auxiliary effect of adding a timing module to the VisualDet3D of this creation on the situation where the object moves out of the screen. Figure 16 is a diagram showing the auxiliary effect of adding a timing module to the VisualDet3D of this creation on the detection of small objects. Figure 17 is a diagram showing the detection effect of adding a timing module to the VisualDet3D of this creation under normal circumstances. Figure 18 is a diagram showing the auxiliary effect of adding a timing module to the Monodle of this creation on the situation where the object is blocked. Figure 19 is a diagram showing the auxiliary effect of adding a timing module to the Monodle of this creation on the situation where the object moves out of the screen. Figure 20 is a diagram showing the auxiliary effect of adding a timing module to the Monodle of this creation on the detection of small objects. Figure 21 is a diagram showing the detection effect of the Monodle in general after adding the timing module.

_Xt : Input status information at the current time point

Y _t : Output status information at the current time point

H _t-1 : Hidden state information at the previous time point

H _t : Hidden state information at the current time point

C _t-1 : The unit state at the previous time point

C _t : The unit state at the current time point

64: Second incentive function layer

65: The third incentive function layer

54: 1st 2D convolution layer

53: Second 2D convolution layer

55:1st connection layer

56: The third two-dimensional convolution layer

57: Second connection layer

61: Forgotten Gate

62: Input gate

63: Output gate

64: Second incentive function

71: Remake the gate

72: Update Gate

73: The fourth incentive function

Claims (6)

A temporal feature assistant module for monocular 3D object detection, wherein the temporal feature assistant module is connected to at least one of a recurrent neural network module (RNN module), a long short-term memory module (LSTM module), and a gated recurrent unit module (GRU module), wherein a video frame processing a spatio-temporal feature map is processed by the temporal feature assistant module, and the temporal feature assistant module comprises: a first two-dimensional convolution layer, which is composed of a hidden state information (H) at a previous time point; _t-1 ) is input to the first two-dimensional convolution layer; a second two-dimensional convolution layer is input to the second two-dimensional convolution layer by an input state information (X _t ) at the current time point; a first connection layer is output from the first two-dimensional convolution layer to the first connection layer, and from the second two-dimensional convolution layer to the first connection layer respectively; and a third two-dimensional convolution layer is output from the first connection layer to the third two-dimensional convolution layer: wherein the recurrent neural network module (Recurrent Neural Networks module), the long short-term memory module (Long Short-Term Memory module, LSTM) are adjusted respectively through the auxiliary module of the temporal feature (temporal assistant module). module), and a hidden state information (H _t ) of the gated recurrent unit module (GRU module) at the current time point, and an output state information (Y _t ) at the current time point, so as to enhance the average precision (AP) of the auxiliary effect for detecting objects that are blocked, moved out of the detection screen, or small objects; wherein the recurrent neural network module (RNN module) is output by the third two-dimensional convolution layer to a first excitation function layer, and the first excitation function layer outputs the hidden state information (H _t ) at the current time point, and outputs the output state information (Y _t ) at the current time point; The LSTM module includes: the third two-dimensional convolution layer is connected to a forget gate, an input gate, a second excitation function layer, and an output gate, wherein the forget gate, the input gate, and the output gate are sigmoid functions; the output information of the forget gate is multiplied by the output information of the forget gate to obtain the first information, the output information of the input gate is multiplied by the output information of the second excitation function layer to obtain the second information, and the first information and the second information are added and output to the third excitation function layer and a unit state (C) at the current time point. _t ); and after the output terminal information of the second excitation function layer is multiplied by the output gate information, the hidden state information (H _t ) at the current time point and the output state information (Y _t ) at the current time point are output respectively; wherein the gated recurrent unit module (GRU module) comprises: the third two-dimensional convolution layer outputs are connected to a reset gate and an update gate respectively, wherein the reset gate and the update gate are sigmoid functions; The output terminal information of the reset gate is multiplied by the output terminal information of the first two-dimensional convolution layer, and then output to a second connection layer. The output terminal information of the second connection layer is output to a fourth two-dimensional convolution layer. The output terminal information of the fourth two-dimensional convolution layer is output to a fourth excitation function layer. The output terminal information of the first two-dimensional convolution layer is multiplied by the output terminal information of the update gate after a delay, and then output a third information. The output terminal information of the update gate is multiplied by the output terminal information of the fourth excitation function layer, and then output a fourth information. After the third information and the fourth information are added, the hidden state information (H _t ) at the current time point is output respectively. , and output the output status information (Y _t ) at the current time point. The auxiliary module of the temporal feature as described in claim 1, comprising: a backbone network layer (backbone layer), the input end of the backbone network layer is connected to the input data feature to extract the input data feature; the input end of the auxiliary module of the temporal feature (temporal assistant module) is connected to the output end of the backbone network layer; a neck layer (neck layer), the input end of the neck layer is connected to the output end of the auxiliary module of the temporal feature (temporal assistant module) to fuse data features; and a detection head layer (head layer), the output end of the neck layer is connected to the input end of the detection head layer (head layer). The auxiliary module of the temporal feature as described in claim 1, comprising: a backbone layer, the input end of the backbone layer is connected to the input data feature to extract the input data feature; a neck layer, the input end of the neck layer is connected to the output end of the backbone layer to fuse the data feature; the auxiliary module of the temporal feature (temporal assistant module) is placed in the neck layer to integrate the data features of different scales; and a detection head layer (head layer), the output end of the neck layer is connected to the input end of the detection head layer (head layer). The auxiliary module of the temporal feature as described in claim 1, comprising: a backbone layer, the input end of the backbone layer is connected to the input data feature to extract the input data feature; a neck layer, the input end of the neck layer is connected to the backbone layer to fuse the data feature; the input end of the auxiliary module of the temporal feature (temporal assistant module) is connected to the output end of the backbone layer; and a detection head layer (head layer), the output end of the auxiliary module of the temporal feature (temporal assistant module) is connected to the input end of the detection head layer (head layer). The auxiliary module of temporal features as described in claim 1, wherein the video frame of the spatio-temporal feature map is processed to detect objects, including: At least one anchor base module, the at least one anchor base module cuts the feature map into multiple grids of different proportions, and places the set at least one anchor base in each grid, captures the anchor base with the highest overlap rate, and detects objects by adjusting the offset. The auxiliary module of the temporal feature as described in claim 1, wherein the video frame of the spatio-temporal feature map is processed to detect an object, including: At least one anchor free module, which detects an object by finding the coordinates of the center point of the object on the feature map and predicting the distances between the center point and the upper, lower, left and right boundaries.