CN110503073B - Dense multi-agent track prediction method for dynamic link at third view angle - Google Patents

Abstract

The invention discloses a dynamic link dense multi-agent trajectory prediction method under the third perspective, which utilizes a variational autoencoder visual component to perform data compression; the input trajectory frame X enters the dynamic cycle unit to complete the encoding network function; for the encoded data to decode. The present invention can not only simulate the space-time movement of the multi-agent fluid according to the dynamic change of the convolution kernel sampling point, but also extract the spatial features of the location of the multi-agent, and learn those pixels that are specifically sampled on the feature map according to the data , reducing spatial feature redundancy. The present invention uses a data-driven method to learn weights on the feature map according to a fixed convolution kernel, and then uses a sigmoid function to operate on the learned weight values to obtain the sampling range of spatio-temporal data, which is more in line with the objective sampling rules and improves the generalization ability of the model . The invention does not need to use intelligent body trajectory points, can realize multi-step prediction, improve model generalization ability, and reduce calculation complexity.

Description

A Dense Multi-Agent Trajectory Prediction Method Based on Dynamic Links in the Third View

technical field

The invention relates to a multi-agent trajectory prediction technology, in particular to a dynamic link dense multi-agent trajectory prediction method under a third perspective.

Background technique

In modern society, intensive multi-agent activities are becoming more and more frequent, such as large-scale concerts, sports, religious activities, and large-scale gatherings. Especially in a country with a large population like China, the prediction of dense multi-agent movement trends is one of the urgent issues in public security research. Obviously, the trajectory prediction of dense multi-agents is helpful to formulate corresponding security management strategies, design better multi-agent shunt mode, count the traffic of dense multi-agents in real time, detect the abnormal behavior of dense agents, and protect the safety of citizens. Personal safety.

At present, the trajectory prediction of dense multi-agents is still based on the data-driven fixed-connection trajectory point prediction technology. Taking the convolutional recurrent network structure as an example, since the size of the convolution kernel is fixed, it is difficult to change the position of the sampling neighbors. This technique is not only difficult to simulate the fluid spatiotemporal movement trend of multi-agents (such as multi-agent aggregation and diffusion), but also prone to sampling data redundancy. For long-term multi-step forecasting, the generalization ability of the model will be greatly reduced, not only the cost of forecasting is high, but also human resources are wasted. So far, there are still many problems to be solved urgently in the trajectory prediction of dense multi-agents.

Contents of the invention

In order to solve the above-mentioned problems in the prior art, the present invention proposes a dynamic link dense multi-agent trajectory prediction method under the third perspective that can solve the dynamic changes of spatio-temporal data and improve the model generalization ability.

In order to achieve the above object, the technical solution of the present invention is as follows: a dense multi-agent trajectory prediction method dynamically linked under the third perspective, comprising the following steps:

A. Data Compression Using Variational Autoencoder Vision Components

The variational autoencoder vision component puts the input continuous trajectory frames with temporal dependencies into the end-to-end neural network for learning, and abstracts and compresses the trajectory frame data. Specific steps are as follows:

A1. The input continuous trajectory frames X ₁ , X ₂ ,...,X _t-1 , X _t have different scales, and use the function nn.imsize(X,128,128) to adjust to the same size 128×128, where nn represents Neural network function base class name.

A2. The adjusted continuous trajectory frames are fully connected by the neural network and encoded into a vector V, so that the previous high latitude of 128×128 becomes 400 dimensions. As shown in the following formula:

V＝nn.Linear(X,400) (1)

A3. Perform a two-dimensional convolution operation on the vector V, perform down-sampling, and use the neural network to fit the vector V into a low-dimensional vector mean μ and variance δ that conform to the Gaussian distribution. The specific formula is as follows:

μ＝nn.Conv2d(V) (2)

δ＝nn.Conv2d(V) (3)

A4. Using the resampling technique, we know that sampling a trajectory frame X from N(μ,δ ² ) is equivalent to sampling an ε from the standard normal distribution N(0,1), and then let X=μ+ε×δ ; Therefore, the sampling from the original N(μ,δ ² ) is transformed into sampling from the standard Gaussian distribution N(0,1), and then the result of sampling from N(μ,δ ² ) is obtained through parameter transformation.

B. The input trajectory frame X enters the dynamic cycle unit to complete the encoding network function

After encoding the network structure, the encoded trajectory frame X enters the dynamic recurrent unit, and the vector feature extraction of the encoded trajectory frame is completed in the dynamic recurrent unit process. The specific steps are as follows formula (4):

in, Represents the Hadamard product operation, and "*" represents the convolution operation. W _xz , W _xr are respectively update gate hidden variable convolution weights, update gate input convolution weights, reset gate hidden variable convolution weights, new hidden variable convolution weights, and reset gate input convolution weights. The subscripts hz, xz, hr, hh, and xz indicate that the weight W belongs to update gate hidden variable convolution weight, update gate input convolution weight, reset gate hidden variable convolution weight, new hidden variable convolution weight, reset Gate input convolution weights. When the actual network structure is running, all values are shared in the encoded network. H _t-1 represents the hidden variable at time t-1, and k represents the neighbor spatio-temporal data point of which link is sampled.

B1. The input continuous trajectory frame processed by the visual component of the variational autoencoder first enters the update gate Z _t , and the dynamic link function is added on the basis of the update gate of the traditional gated recurrent unit, which is realized by φ(P), and P stands for convolution The kernel samples locations in the feature map. The specific implementation process of φ(P) is to use a 3×3 convolutional neural network to obtain the position offset of the spatio-temporal data on the basis of inputting the continuous trajectory frame X, and then add the position offset to the coordinates of the original input feature map. shift, and calculate the pixel value corresponding to the shifted coordinates through the bilinear difference. Finally, a 3×3 convolution kernel is used to perform a convolution operation on the value at the changed position. The update gate is used to control the degree to which state information from the previous moment is brought into the current state. The function Γ(H _t-1 ,φ(P)) is to dynamically select sampling points of spatio-temporal data.

B2. The input continuous trajectory frame enters the reset gate after the update gate operation, and the reset gate also uses φ(P) to realize the dynamic link function.

B3. After updating the gate and resetting the gate, determine the hidden variable at the current moment. The sampling range of the hidden variable at the current moment is determined by Δm _k . Δm _k is to use the convolution operation to obtain the intermediate value of the space-time data (H _t-1 , X _t ), and then use the sigmoid function to determine the probability of the sampling value, and the range is [0,1]. The specific formula is as follows:

C. Decode the encoded data

C1. Predict the most likely sequence of K time-series trajectory image frames X in the future according to the observation results of the first J input time-series trajectory image frames X;

C2. The prediction result is expressed by the following formula:

X _t+1 ,...,X _t+k ≈g _decoding (f _encoding (X _t-J+1 ,X _t-J+2 ,...,X _t )) (6)

Finish.

Compared with the prior art, the present invention has the following beneficial effects:

1. The dynamic link structure of the present invention uses a data-driven approach to first learn the coordinate position offset of the feature map, then uses bilinear interpolation to map the pixel values in the original feature map to the coordinate positions of the new feature map, and finally uses a fixed connection The convolution kernel is down-sampled to achieve dynamic link changes. Compared with the prior art, the present invention can not only simulate the space-time movement of the multi-agent fluid according to the dynamic change of the sampling point of the convolution kernel, but also extract the spatial characteristics of the location of the multi-agent. Compared with the fixed convolutional loop network structure of the previous model, the present invention not only greatly improves the structure and spatial data extraction, but also learns to sample those pixels on the feature map according to the data, reducing the redundancy of spatial features .

2. The present invention uses a data-driven approach to learn the weights on the feature map according to the fixed convolution kernel, and then uses the sigmoid function to operate the learned weight values to obtain the sampling range of the spatio-temporal data (i.e. the sampling probability of the pixel points), which is more It conforms to the objective sampling law and improves the generalization ability of the model.

3. The spatio-temporal prediction model adopted in the present invention regards the movement of dense multi-agents as a spatio-temporal data pixel prediction problem. This prediction technology does not need to use the trajectory points of the agent, can realize multi-step prediction, improve the generalization ability of the model, and reduce the computational complexity.

4. The present invention adopts the resampling technique, so that the sampling operation does not need to participate in the gradient descent operation, but instead participates in the sampling result, so that the model can reduce parameters and participate in training.

Description of drawings

The present invention has 4 accompanying drawings, wherein:

Figure 1 is a dynamic link unit structure.

Figure 2 is the codec structure.

Figure 3 is a graph of the prediction results.

Fig. 4 is a flowchart of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings. According to the process shown in Fig. 4, the multi-agent trajectory prediction method of the dynamic link (as shown in Fig. 1) under the third perspective is introduced. Firstly, the continuous temporally related trajectory frames are input to the visual component part of the variational autoencoder in the encoding network for encoding, so that the input high-dimensional trajectory frames become low-dimensional latent variables. The specific operation is to map the input time-series trajectory frame into a high-dimensional vector through a fully connected operation, and then use a convolution operation for down-sampling to obtain a low-dimensional vector representation. Low-dimensional latent variables are input to the dynamic link unit 2 in the encoding network (as shown in Figure 2) to extract the dynamic spatiotemporal data features of the trajectory. The specific operation is to first learn the position offset according to the fixed convolution kernel size, then obtain the pixel correspondence between the original feature map and the new feature map according to the bilinear difference, and finally use the same size and fixed convolution kernel to compare the new The feature map is down-sampled to complete the sampling of spatio-temporal data by the dynamic link structure. The spatio-temporal feature vectors to be extracted flow to the dynamic link unit 1 to continue spatio-temporal feature extraction. And so on, until the input of continuous trajectory frames is completed in the encoding network training. Copy the training weights of the encoding network to the dynamic link unit 1 and the dynamic link unit 2 in the decoding network structure, and then decode and output the predicted continuous trajectory frame (shown in Figure 3), the first column is the input The historical trajectory sequence, the second column is the real trajectory sequence, and the third column is the predicted trajectory sequence.

Claims (1)

1. A dense multi-agent trajectory prediction method dynamically linked under a third perspective, characterized in that: comprising the following steps: A. Data Compression Using Variational Autoencoder Vision Components The variational autoencoder vision component puts the input continuous trajectory frames with timing dependencies into the end-to-end neural network for learning, and abstracts and compresses the trajectory frame data; the specific steps are as follows: A1. The input continuous trajectory frames X ₁ , X ₂ ,...,X _t-1 ,X _t have different scales, and they are adjusted to the same size 128×128 by using the function nn.imsize(X,128,128), where nn represents the neural network function base class name; A2. The adjusted continuous trajectory frame adopts the fully connected operation of the neural network and encodes it into a vector V, so that the previous 128×128 high latitude becomes 400 dimensions; as shown in the following formula: V＝nn.Linear(X,400) (1) A3. Perform a two-dimensional convolution operation on the vector V, perform down-sampling, and use the neural network to fit the vector V into a low-dimensional vector mean μ and variance δ that conform to the Gaussian distribution. The specific formula is as follows: μ=nn.Conv2d(V) (2) δ＝nn.Conv2d(V) (3) A4. Using the resampling technique, we know that sampling a trajectory frame X from N(μ,δ ² ) is equivalent to sampling an ε from the standard normal distribution N(0,1), and then let X=μ+ε×δ ;Therefore, the sampling from the original N(μ,δ ² ) is transformed into sampling from the standard Gaussian distribution N(0,1), and then the result of sampling from N(μ,δ ² ) is obtained through parameter transformation; B. The input trajectory frame X enters the dynamic cycle unit to complete the encoding network function After encoding the network structure, the encoded trajectory frame X enters the dynamic recurrent unit, and the vector feature extraction of the encoded trajectory frame is completed in the dynamic recurrent unit process. The specific steps are as follows formula (4): in, Represents Hadamard product operation, "*" represents convolution operation; W _xz , W _xr are respectively update gate hidden variable convolution weight, update gate input convolution weight, reset gate hidden variable convolution weight, new hidden variable convolution weight, reset gate input convolution weight; subscripts hz, xz, hr , hh, xz indicate that the weight W belongs to update gate hidden variable convolution weight, update gate input convolution weight, reset gate hidden variable convolution weight, new hidden variable convolution weight, reset gate input convolution weight; When the network structure is running, ownership values are shared in the encoding network; H _t-1 represents the hidden variable at time t-1, and k represents the neighbor spatiotemporal data point of which link is sampled; B1. The input continuous trajectory frame processed by the visual component of the variational autoencoder first enters the update gate Z _t , and the dynamic link function is added on the basis of the update gate of the traditional gated recurrent unit, which is realized by φ(P), and P stands for convolution The sampling position of the kernel in the feature map; the specific implementation process of φ(P) is to use a 3×3 convolutional neural network to obtain the position offset of the spatio-temporal data on the basis of inputting the continuous trajectory frame X, and then input the feature map in the original Add the position offset on the basis of the coordinates, and calculate the pixel value corresponding to the offset coordinates through the bilinear difference; finally, use a 3×3 convolution kernel to perform a convolution operation on the value at the changed position; update The gate is used to control the degree to which the state information of the previous moment is brought into the current state; the function Γ(H _t-1 ,φ(P)) is to dynamically select the sampling points of spatio-temporal data; B2. The input continuous trajectory frame enters the reset gate after the update gate operation, and the reset gate also uses φ(P) to realize the dynamic link function; B3. After updating the gate and resetting the gate, determine the hidden variable at the current moment; the sampling range of the hidden variable at the current moment is determined by Δm _k ; Δm _k is the first convolution of the spatiotemporal data (H _t-1 ,X _t ) The operation obtains the intermediate value, and then uses the sigmoid function to determine the probability of the sampling value, the range is [0,1]; the specific formula is as follows: C. Decode the encoded data C1. Predict the most likely sequence of K time-series trajectory image frames X in the future according to the observation results of the first J input time-series trajectory image frames X; C2. The prediction result is expressed by the following formula: X _t+1 ,...,X _t+k ≈g _decoding (f _encoding (X _t-J+1 ,X _t-J+2 ,...,X _t )) (6) Finish.

Images