CN111340850A - Ground target tracking method of unmanned aerial vehicle based on twin network and central logic loss - Google Patents

Abstract

The invention discloses an unmanned aerial vehicle ground target tracking method based on a twin network and central logic loss, which aims to solve the problems of more network parameters, large calculated amount, unbalanced positive and negative training samples and the like in the existing target tracking technology and belongs to the technical field of computer image processing. The method comprises the following steps: extracting a target feature map of a first frame image at a known target position by using a first feature extraction network, and extracting a search feature map of a second frame image by using a second feature extraction network; calculating the cross correlation between the search area of the second frame image and the target area of the first frame image according to the target characteristic diagram and the search characteristic diagram to obtain a score response diagram of the second frame image, and further obtaining the target position of the second frame image according to the score response diagram of the second frame image; the first feature extraction network and the second feature extraction network are two branches of a twin convolutional network and respectively consist of a lightweight convolutional neural network.

Description

A UAV-to-ground target tracking method based on twin network and central logic loss

technical field

The invention belongs to the technical field of computer image processing, in particular to a method for tracking ground targets of unmanned aerial vehicles based on twin network and central logic loss.

Background technique

Deep learning-based visual tracking algorithms, such as the fully convolutional Siamese network-based visual tracking algorithm shown in Figure 1, have been accepted and recognized by developers and users.

In this method, the template image extracted from the first frame and the search image extracted from each frame are input into two sub-networks to extract high-level semantic features, and then the high-level semantic features are cross-correlated to obtain the similarity of each position of the template image in the search image. Spend. Usually, the parameters in the two sub-networks are shared and can be learned offline using training data. The high-level semantic features have rich semantic features related to the target category, so they have strong robustness to changes in target appearance caused by occlusion, distortion, etc., and the network does not need to be updated during the tracking process, which greatly reduces the calculation of the algorithm. It ensures the real-time performance of the algorithm. The network has two inputs: the target template image and the search area image, and the two inputs perform feature extraction through a siamese subnet of a siamese neural network that shares parameters.

However, such methods are mainly oriented to general-purpose visual tracking tasks, and cannot satisfy the UAV hardware platforms with limited computing and storage resources. First, there are a large number of weight parameters in the convolutional neural network, and saving a large number of weight parameters requires a high memory of the device. Secondly, the computing resources of embedded hardware platforms such as drones are limited, so it is difficult to achieve efficient and real-time convolution computation in convolutional neural networks.

At the same time, in the process of actual tracking, such an implementation requires intensive detection of the target at each position of the search area. For UAV aerial images, the search area generally contains many negative samples in the simple background (region 3 in Figure 2), a few difficult negative samples (region 1 in Figure 2), and positive samples containing foreground targets. samples (area 2 in Figure 2), while a large number of simple background negative samples will lead to imbalanced training samples and dominate the training of the network, resulting in model degradation.

SUMMARY OF THE INVENTION

The present invention aims to provide a UAV-to-ground target tracking method based on the twin network and central logic loss according to the characteristics of the UAV hardware platform, so as to solve the problems of many network parameters, large amount of calculation and positive problems in the existing target tracking technology. Negative training samples are not balanced.

According to the first aspect of the present invention, a method for training a visual tracking model of a UAV on a ground target, the tracking model is a twin convolution network, and the two branches of the twin convolution network are a first feature extraction network, a The second feature extraction network, the training method includes:

Obtain a video sequence dataset, which includes pairs of template images and search images;

Use the first feature extraction network to extract the target feature map of the template image, and use the second feature extraction network to extract the search feature map of the search image;

Calculate the cross-correlation between the search area of the search image and the target area of the template image according to the target feature map and the search feature map, and obtain a score response map of the search image;

Calculate the difference between the score response map and the true value according to the central logistic loss function to obtain a difference result; and

Back-propagating the difference result to adjust the weights of each layer in the Siamese neural network;

Wherein, the first feature extraction network and the second feature extraction network are respectively composed of lightweight convolutional neural networks.

Optionally, the lightweight convolutional neural network is a MobileNetV2 model.

Optionally, the central logistic loss function is:

where v∈R ^m×n is the score map output by the network, and y∈{+1,-1} is the ground-truth annotated manually. a/(1+exp(b·yv)) is a modulation factor of the logistic loss, which adaptively adjusts the contribution of each training sample to the training loss according to the input yv.

Further, when yv>0, the modulation factor assigns a first weight to the logic loss; when yv<0, the modulation factor assigns a second weight to the logic loss, and the first weight is smaller than the second weight.

According to a second aspect of the present invention, a method for visual tracking of a ground target by an unmanned aerial vehicle, comprising:

Use the first feature extraction network to extract the target feature map of the first frame image of the known target position, and use the second feature extraction network to extract the search feature map of the second frame image;

According to the target feature map and the search feature map, the cross-correlation between the search area of the second frame image and the target area of the first frame image is calculated to obtain the score response map of the second frame image, and then according to the second frame image The score response map of the image obtains the target position of the second frame image;

Among them, the first feature extraction network and the second feature extraction network are two branches of the twin convolutional network, and are respectively composed of a lightweight convolutional neural network, and the lightweight convolutional neural network is the MobileNet V2 model .

According to a third aspect of the present invention, a visual tracking device for a ground target of an unmanned aerial vehicle, comprising:

an identification unit, used for extracting the target feature map of the first frame image of the known target position using the first feature extraction network, and using the second feature extraction network to extract the search feature map of the second frame image;

a calculation unit, configured to calculate the cross-correlation between the search area of the second frame image and the target area of the first frame image according to the target feature map and the search feature map, to obtain the score response map of the second frame image;

a determining unit for obtaining the target position of the second frame image according to the score response map of the second frame image;

Wherein, the first feature extraction network and the second feature extraction network in the identification unit are two branches of the twin convolutional network, and are respectively composed of lightweight convolutional neural networks.

According to a fourth aspect of the present invention, an electronic device, comprising:

at least one processor; and

A memory communicatively connected to the processor stores instructions executable by the processor; when the instructions are executed by the processor, the processor performs visual tracking of the ground target by the UAV method, or the training method.

According to the fifth aspect of the present invention, a readable storage medium on which a computer program is stored, is characterized in that, when the computer program is executed by a processor, the described method for visual tracking of a ground target by an unmanned aerial vehicle is realized, or the training method described.

The present invention adopts the MobileNet V2 model in the lightweight network as the feature extraction sub-network of the front end of the depth frame, thereby reducing the computational complexity and the number of parameters in the convolutional neural network. At the same time, it can maintain a good balance between processing speed and accuracy, so that it can adapt to the limited storage and computing resources of the UAV hardware platform.

In addition, the present invention uses the central logic loss function to apply different weights to different training samples in the search area, solves the problem of unbalanced positive and negative training samples, avoids the problem of network degradation in offline training, and makes the learned convolution features have stronger discrimination.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that are required to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

Figure 1 is a schematic diagram of a conventional twin network structure;

Figure 2 is an actual target tracking scene, which includes simple negative samples (region 3), difficult negative samples (region 1), and positive samples containing foreground targets (region 2);

3 is a schematic structural diagram of a visual tracking network model according to an embodiment of the present invention;

Figure 4 is a schematic diagram of the MobileNet V2 network structure;

5 is a schematic flowchart of a visual tracking network training and tracking method according to an embodiment of the present invention;

6 is a schematic flow chart of a method for visual tracking of a ground target by a UAV according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a visual tracking device for a ground target of an unmanned aerial vehicle according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

In the following description, UAV (UnmannedAerial Vehicle, unmanned aerial vehicle) mainly refers to an unmanned aerial vehicle that is operated by radio remote control equipment and self-provided program control device, or is operated completely or intermittently autonomously by an onboard computer.

FIG. 3 shows a visual tracking network based on a Siamese network constructed according to an embodiment of the present invention.

As shown in Figure 3, the visual tracking network includes a first branch and a second branch with the same structure, and the distribution of the two branches includes a feature extraction network for extracting the feature map of the image. The feature extraction network of each branch in the present invention is composed of a lightweight convolutional neural network. According to an embodiment of the present invention, the lightweight convolutional neural network is a MobileNet V2 model.

Figure 4 shows the network structure of the MobileNet V2 model. MobileNet V2 uses depthwise separable convolutions as efficient building blocks. Furthermore, MobileNet V2 introduces two new architectural features: 1) linear bottleneck layers between layers; 2) connection shortcuts between bottleneck layers. This network structure enables MobileNet V2 to gain additional advantages in security, privacy and energy consumption.

In the MobileNet V2 network, the features extracted by the second convolutional layer are limited by the number of input channels. To this end, the MobileNet V2 network sets a 1×1 first convolutional layer for expansion before using the DW convolutional layer to extract features from the image, and then uses the third convolutional layer for compression after extracting features. Therefore, the MobileNet V2 network implements the expansion→convolution feature extraction→compression process, avoiding the problem of channel loss.

At the same time, ReLU functions are respectively set at the outputs of the first convolutional layer and the second convolutional layer. Due to the above-mentioned expansion→convolution feature extraction→compression process, there will be a problem after compression, that is, the ReLU function will destroy the features, and the output of the ReLU function for negative inputs is all zero. Therefore, in order to avoid further loss of features, the third The activation function of the convolutional layer adopts the Linear function.

The visual tracking network based on the twin network proposed by the present invention adopts a central logical loss function, and the central logical loss function is:

in:

v∈R ^m×n is the score map output by the network, and m×n is the size of the score map;

y∈{+1,-1} is the ground truth value annotated manually;

a/(1+exp(b·yv)) is a modulation factor of the logic loss, a and b are parameters in the modulation factor, for example, according to an embodiment of the present invention, a=2, b=1.

This modulation factor adaptively adjusts the contribution of each training sample to the training loss according to the input yv. When yv > 0, the sample is a simple sample, and the modulation factor assigns a smaller weight to the logistic loss; conversely, when yv < 0, the sample is a difficult sample, and the modulation factor assigns a larger weight to the logistic loss.

Most convolutional neural networks use logistic loss or cross entropy loss as the supervision signal in the training process of deep models. Although the models trained by this loss function have good separability, they have poor discriminative ability.

Different from closed-set problems such as object classification and recognition, target tracking is an open-set problem, which not only requires the features output by the deep model to be separable, but also must have strong discrimination. When dealing with long-tailed datasets (where most of the samples belong to very few classes, and many other classes have very few samples), how to weight the losses for different classes can be tricky. For the target tracking problem, the foreground target is easier to collect as a positive sample, while the difficult negative samples in the background that are useful for training are few.

Therefore, the present invention can adaptively adjust the ratio of positive and negative samples in an end-to-end manner by using the above-mentioned central logical loss function, so that the learned network is free from the influence of unbalanced training samples. Specifically, the present invention can deal with the problem of unbalanced foreground-background training samples by applying different weights to different samples by using the central logistic loss function.

FIG. 5 shows a schematic flow of visual tracking network training and using a trained network model to visually track ground targets of unmanned aerial vehicles according to an embodiment of the present invention.

As shown in FIG. 5 , in the present invention, the training process of the visual tracking network includes a pre-training stage and a fine-tuning stage.

In the pre-training stage, for example, the video database in the ImageNet Large-Scale Visual Recognition Challenge ILSVRC is used as the sample video sequence, and the labeled sample video sequence is used to train the visual tracking network. During training, set the maximum number of iterations and learning rate of the visual tracking network, select the network parameter initialization method and backpropagation method, and optimize the network parameters.

According to an embodiment of the present invention, the maximum number of iterations and the learning rate are set, and the back-propagation method is selected as follows:

Maximum number of iterations: 50epoch;

initial learning rate: 0.001;

Network initialization method: Xavier method

Backpropagation method: Stochastic gradient descent method.

After the pre-training stage is completed, the network parameters are further optimized through the fine-tuning stage.

In the fine-tuning stage, the present invention uses drones to collect ground targets to form a video data set, label each frame of the video data set according to the category, and divide the labeled video data set into training set, verification set, and test set, Finally, it is processed into a data type that the visual tracking network model can recognize.

The pre-trained visual tracking network is trained by using the training set and the validation set, so as to fine-tune the visual tracking network and retain the structure and parameters of the fine-tuned visual tracking network model.

Use the test set to test the fine-tuned visual tracking network to obtain the tracking accuracy. And the tracking accuracy is judged. If the tracking accuracy can meet the actual engineering needs, the visual tracking network model can be applied to the actual task of identifying the specific target on the ground by the UAV. Otherwise, it means that the training set cannot meet the actual engineering needs, and it is necessary to expand the training set and restart the pre-training and fine-tuning steps until the actual engineering needs are met.

In the training process, the present invention adopts the central logical loss function, and the central logical loss function is:

in:

v∈R ^m×n is the score map output by the network;

y∈{+1,-1} is the ground truth value annotated manually;

a/(1+exp(b·yv)) is a modulation factor of the logical loss, which adaptively adjusts the contribution of each training sample to the training loss according to the input yv.

When yv > 0, the sample is a simple sample, and the modulation factor assigns a smaller weight to the logistic loss; conversely, when yv < 0, the sample is a difficult sample, and the modulation factor assigns a larger weight to the logistic loss.

Therefore, the present invention can deal with the problem of unbalanced foreground-background training samples by applying different weights to different samples by using the central logistic loss function.

After completing the training of the network model, the visual tracking network is applied to the actual scene of the UAV tracking the ground target, and the target in the video collected by the UAV is tracked.

FIG. 5 and FIG. 6 show a schematic flow of a method for tracking a ground target of a UAV based on a twin network according to an embodiment of the present invention.

As shown in Figure 6, the method includes the following steps:

Use the first feature extraction network to extract the first frame image of the known target position, that is, the target feature map of the template image, and use the second feature extraction network to extract the second frame image, that is, the search feature map of the search image;

According to the target feature map and the search feature map, the cross-correlation between the search area of the second frame image and the target area of the first frame image is calculated, and the score response map of the second frame image is obtained. Score the response map to obtain the target position of the second frame image;

Wherein, the first feature extraction network and the second feature extraction network are two branches of the Siamese convolutional network, and are respectively composed of lightweight convolutional neural networks.

For example, by calibrating the first frame image in the video sequence, the target position of the second frame image can be obtained by using the visual tracking network; The network can get the target position of the 3rd frame image. By analogy, the target position of each frame image in the video sequence can be obtained, and the target tracking of the video sequence can be realized.

Among them, as shown in Figure 5, it also includes the score response graph obtained by the cross-correlation calculation, and the parameters of the feature extraction network are updated to further improve the accuracy and reliability of the feature extraction network.

FIG. 7 is a visual tracking device for an unmanned aerial vehicle on the ground provided by an embodiment of the present invention, including:

Recognition unit 701, for extracting the target feature map of the first frame image of the known target position by using the first feature extraction network, and using the second feature extraction network to extract the search feature map of the second frame image;

A calculation unit 702, configured to calculate the cross-correlation between the search area of the second frame image and the target area of the first frame image according to the target feature map and the search feature map, to obtain the score response map of the second frame image;

Determining unit 703, for obtaining the target position of the second frame image according to the score response map of the second frame image;

Wherein, the first feature extraction network and the second feature extraction network in the identification unit are two branches of the twin convolutional network, and are respectively composed of lightweight convolutional neural networks.

The embodiment of the present invention also provides an electronic device, including:

at least one processor; and

A memory communicatively connected to the processor stores instructions executable by the processor; when the instructions are executed by the processor, the processor performs visual tracking of the ground target by the UAV method, or the training method.

Optionally, the memory can be independent or integrated with the processor.

When the memory is provided independently, the electronic device further includes a bus for connecting the memory and the processor.

Further, an embodiment of the present invention also provides a readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the described method for visual tracking of a ground target by an unmanned aerial vehicle is implemented, or the computer program is executed. the training method described.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated. to another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or modules, and may be in electrical, mechanical or other forms.

The modules described as separate components may or may not be physically separated, and components shown as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically alone, or two or more modules may be integrated into one unit. The units formed by the above modules can be implemented in the form of hardware, or can be implemented in the form of hardware plus software functional units.

The above-mentioned integrated modules implemented in the form of software functional modules may be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium, and include several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the methods described in the various embodiments of the present application.

It should be understood that the above-mentioned processor may be a central processing unit (CPU), and may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the invention can be directly embodied as executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

The memory may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one magnetic disk memory, and may also be a U disk, a removable hard disk, a read-only memory, a magnetic disk or an optical disk, and the like.

The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into address bus, data bus, control bus and so on. For convenience of representation, the buses in the drawings of the present application are not limited to only one bus or one type of bus.

The above-mentioned storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable Except programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic or optical disks. A storage medium can be any available medium that can be accessed by a general purpose or special purpose computer.

An exemplary storage medium is coupled to the processor, such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium may reside in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium may also exist in the electronic device or the host device as discrete components.

Those skilled in the art can understand that all or part of the steps of implementing the above method embodiments may be completed by program instructions related to hardware. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps including the above method embodiments are executed; and the foregoing storage medium includes various media that can store program codes, such as ROM, RAM, magnetic disk, or optical disk. The above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that the foregoing embodiments can still be used for The technical solutions described in the examples are modified, or some or all of the technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims (9)

1. a training method of an unmanned aerial vehicle to ground target visual tracking model, the tracking model is a twin convolution network, and two branches of this twin convolution network are respectively the first feature extraction network, the second feature extraction network, It is characterized in that, the training method includes: Obtain a video sequence dataset, which includes pairs of template images and search images; Use the first feature extraction network to extract the target feature map of the template image, and use the second feature extraction network to extract the search feature map of the search image; Calculate the cross-correlation between the search area of the search image and the target area of the template image according to the target feature map and the search feature map, and obtain a score response map of the search image; Calculate the difference between the score response map and the true value according to the central logistic loss function to obtain a difference result; and Back-propagating the difference result to adjust the weights of each layer in the Siamese neural network; Wherein, the first feature extraction network and the second feature extraction network are respectively composed of lightweight convolutional neural networks. 2. The training method according to claim 1, wherein the lightweight convolutional neural network is a MobileNetV2 model. 3. The training method according to claim 1 or 2, wherein the central logistic loss function is:

where v∈R ^m×n is the score map output by the network, and y∈{+1,-1} is the ground-truth annotated manually. a/(1+exp(b·yv)) is a modulation factor of the logic loss, which adaptively adjusts the contribution of each training sample to the training loss according to the input yv, a and b are the parameters in the modulation factor. 4. The training method according to claim 3, wherein when yv>0, the modulation factor assigns a first weight to the logic loss; when yv<0, the modulation factor assigns a second weight to the logic loss, and the The first weight is smaller than the second weight. 5 . The training method according to claim 3 , wherein the parameters a=2 and b=1 in the modulation factor of the central logistic loss function. 6 . 6. A method for visual tracking of an unmanned aerial vehicle on the ground, comprising: Use the first feature extraction network to extract the target feature map of the first frame image of the known target position, and use the second feature extraction network to extract the search feature map of the second frame image; According to the target feature map and the search feature map, the cross-correlation between the search area of the second frame image and the target area of the first frame image is calculated to obtain the score response map of the second frame image, and then according to the second frame image The score response map of the image obtains the target position of the second frame image; Among them, the first feature extraction network and the second feature extraction network are two branches of the twin convolutional network, and are respectively composed of a lightweight convolutional neural network, and the lightweight convolutional neural network is the MobileNet V2 model . 7. An unmanned aerial vehicle to ground target visual tracking device, is characterized in that, comprises: an identification unit, used for extracting the target feature map of the first frame image of the known target position using the first feature extraction network, and using the second feature extraction network to extract the search feature map of the second frame image; a calculation unit, configured to calculate the cross-correlation between the search area of the second frame image and the target area of the first frame image according to the target feature map and the search feature map, to obtain the score response map of the second frame image; a determining unit for obtaining the target position of the second frame image according to the score response map of the second frame image; Wherein, the first feature extraction network and the second feature extraction network in the identification unit are two branches of the twin convolutional network, and are respectively composed of a lightweight convolutional neural network. The lightweight convolutional neural network is the MobileNet V2 model. 8. An electronic device, characterized in that, comprising: at least one processor; and a memory communicatively connected to the processor that stores instructions executable by the processor; when the instructions are executed by the processor, the processor executes the instructions of claim 6 The described UAV to ground target visual tracking method, or the training method according to any one of claims 1-5. 9. A readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by the processor, the unmanned aerial vehicle (UAV) method for visual tracking of ground targets according to claim 6 is realized, or according to the The training method described in any one of requirements 1-5.

Images