The official PyTorch implementation of our ECCV 2022 paper:

AiATrack: Attention in Attention for Transformer Visual Tracking

Shenyuan Gao, Chunluan Zhou, Chao Ma, Xinggang Wang, Junsong Yuan

[ECVA Open Access] [ArXiv Preprint] [YouTube Video] [Trained Models] [Raw Results] [SOTA Paper List]

Transformer trackers have achieved impressive advancements recently, where the attention mechanism plays an important role. However, the independent correlation computation in the attention mechanism could result in noisy and ambiguous attention weights, which inhibits further performance improvement. To address this issue, we propose an attention in attention module (named AiA), which enhances appropriate correlations and suppresses erroneous ones by seeking consensus among all correlation vectors. Our AiA module can be readily applied to both self-attention blocks and cross-attention blocks to facilitate feature aggregation and information propagation for visual tracking. Moreover, we propose a streamlined Transformer tracking framework (dubbed AiATrack), by introducing efficient feature reuse and target-background embeddings to make full use of temporal references. Experiments show that our tracker achieves state-of-the-art performance on several tracking benchmarks while running at a real-time speed.

The proposed AiATrack sets state-of-the-art results on 8 widely used benchmarks. Using ResNet-50 pre-trianed on ImageNet-1k, we can get:

Benchmark (Metrics)	AiATrack	Leaderboard
LaSOT (AUC / Norm P / P)	69.0 / 79.4 / 73.8
LaSOT Extension (AUC / Norm P / P)	47.7 / 55.6 / 55.4
TrackingNet (AUC / Norm P / P)	82.7 / 87.8 / 80.4
GOT-10k (AO / SR 0.75 / SR 0.5)	69.6 / 63.2 / 80.0
NfS30 (AUC)	67.9
OTB100 (AUC)	69.6
UAV123 (AUC)	70.6
VOT2020 (EAO / A / R)	0.530 / 0.764 / 0.827

The proposed AiATrack can run at 38 fps (frames per second) on a single NVIDIA GeForce RTX 2080 Ti.

It takes nearly two days to train our model on 8 NVIDIA GeForce RTX 2080 Ti (each of which has 11GB GPU memory).

The proposed AiATrack has 15.79M (million) model parameters.

Trained Models (containing the model we trained on four datasets and the model we trained on GOT-10k only) [download zip file]

Raw Results (containing raw tracking results on the datasets we benchmarked in the paper) [download zip file]

Download and unzip these two zip files under AiATrack project path, then both of them can be directly used by our code.

• Environment

  Our experiments are conducted with Ubuntu 18.04 and CUDA 10.1.

• Preparation

  • Clone our repository to your local project directory.

  • Download the training datasets (LaSOT, TrackingNet, GOT-10k, COCO2017) and testing datasets (NfS, OTB, UAV123) to your disk, the organized directory should look like:

    --LaSOT/
    	|--airplane
    	|...
    	|--zebra
    --TrackingNet/
    	|--TRAIN_0
    	|...
    	|--TEST
    --GOT10k/
    	|--test
    	|--train
    	|--val
    --COCO/
    	|--annotations
    	|--images
    --NFS30/
    	|--anno
    	|--sequences
    --OTB100/
    	|--Basketball
    	|...
    	|--Woman
    --UAV123/
    	|--anno
    	|--data_seq
    

  • Edit the PATH in lib/test/evaluation/local.py and lib/train/adim/local.py to the proper absolute path.

• Installation

  We use conda to manage the environment.

  conda create --name aiatrack python=3.6
  conda activate aiatrack
  sudo apt-get install ninja-build
  sudo apt-get install libturbojpeg
  bash install.sh
  

  Note that your PyTorch version must be pytorch <= 1.10.1 to successfully compile PreciseRoIPooling since <THC/THC.h> has been removed in pytorch 1.11.

• Training

  • Multiple GPU training by DDP (suppose you have 8 GPU)

    python tracking/train.py --mode multiple --nproc 8
    

  • Single GPU debugging (too slow, not recommended for training)

    python tracking/train.py
    

  • For GOT-10k evaluation, remember to set --config baseline_got.

• Evaluation

  • Make sure you have prepared the trained model.

  • On large-scale benchmarks:

    • LaSOT

      python tracking/test.py --dataset lasot
      python tracking/test.py --dataset lasot_ext
      

      Then evaluate the raw results using the official MATLAB toolkit.

    • TrackingNet

      python tracking/test.py --dataset trackingnet
      python lib/test/utils/transform_trackingnet.py --tracker_name aiatrack --cfg_name baseline
      

      Then upload test/tracking_results/aiatrack/baseline/trackingnet_submit.zip to the online evaluation server.

    • GOT-10k

      python tracking/test.py --param baseline_got --dataset got10k_test
      python lib/test/utils/transform_got10k.py --tracker_name aiatrack --cfg_name baseline_got
      

      Then upload test/tracking_results/aiatrack/baseline_got/got10k_submit.zip to the online evaluation server.

  • On small-scale benchmarks:

    • NfS30, OTB100, UAV123

      python tracking/test.py --dataset nfs
      python tracking/test.py --dataset otb
      python tracking/test.py --dataset uav
      python tracking/analysis_results.py
      

      As previous works did, the frames where the target object doesn't exist will be excluded during the analysis.

  • For multiple threads inference, just add --threads 40 after tracking/test.py (suppose you want to use 40 threads in total).

  • To show the immediate prediction results during inference, modify settings.show_result = True in lib/test/evaluation/local.py (may have bugs if you try this on a remote sever).

  • Please refer to STARK+Alpha-Refine for VOT integration and DETR Tutorial for correlation map visualization.

❤️❤️❤️Our idea is implemented base on the following projects. We really appreciate their wonderful open-source works!

• STARK [related paper]
• PyTracking [related paper]
• DETR [related paper]
• PreciseRoIPooling [related paper]

If any parts of our paper and code help your research, please consider citing us and giving a star to our repository.

@inproceedings{gao2022aiatrack,
  title={AiATrack: Attention in Attention for Transformer Visual Tracking},
  author={Gao, Shenyuan and Zhou, Chunluan and Ma, Chao and Wang, Xinggang and Yuan, Junsong},
  booktitle={European Conference on Computer Vision},
  pages={146--164},
  year={2022},
  organization={Springer}
}

If you have any questions or concerns, feel free to open issues or directly contact me through the ways on my GitHub homepage. Suggestions and collaborations are also highly welcome!