Transformer Tracking

This repository is a paper digest of Transformer-related approaches in visual tracking tasks. Currently, tasks in this repository include Unified Tracking (UT), Single Object Tracking (SOT) and 3D Single Object Tracking (3DSOT). Note that some trackers involving a Non-Local attention mechanism are also collected. Papers are listed in alphabetical order of the first character.

:link:Jump to:

• [Unified Tracking]

• [Single Object Tracking]

• [3D Single Object Tracking]

Note: I find it hard to trace all tasks that are related to tracking, including Video Object Segmentation (VOS), Multiple Object Tracking (MOT), Video Instance Segmentation (VIS), Video Object Detection (VOD) and Object Re-Identification (ReID). Hence, I discard all other tracking tasks in a previous update. If you are interested, you can find plenty of collections in this archived version. Besides, the most recent trend shows that different tracking tasks are coming to the same avenue.

:star2:Recommendation

State-of-the-Art Transformer Tracker:two_hearts::two_hearts::two_hearts:

• GRM (Generalized Relation Modeling for Transformer Tracking) [paper] [code] [video]
• AiATrack (AiATrack: Attention in Attention for Transformer Visual Tracking) [paper] [code] [video]

Up-to-Date Benchmark Results:rocket::rocket::rocket:

• Image courtesy: https://arxiv.org/abs/2302.11867

Helpful Learning Resource for Tracking:thumbsup::thumbsup::thumbsup:

• (Survey) Transformers in Single Object Tracking: An Experimental Survey [paper], Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlook [paper]
• (Talk) Discriminative Appearance-Based Tracking and Segmentation [video], Deep Visual Reasoning with Optimization-Based Network Modules [video]
• (Library) PyTracking: Visual Tracking Library Based on PyTorch [code]
• (People) Martin Danelljan@ETH [web], Bin Yan@DLUT [web]

Recent Trends:fire::fire::fire:

• Target Head: Autoregressive Temporal Modeling

  • Representative

    • Bounding box as coordinate sequence.
      • ARTrack [CVPR'23], SeqTrack [CVPR'23]

  

• Feature Backbone: Joint Feature Extraction and Interaction

  • Advantage

    • Benefit from pre-trained vision Transformer models.
    • Free from randomly initialized correlation modules.
    • More discriminative target-specific feature extraction.
    • Much faster inference and training convergence speed.
    • Simple and generic one-branch tracking framework.

  • Roadmap

    • 1st step :feet: feature interaction inside the backbone.
      • SiamAttn [CVPR'20], SBT [CVPR'22], InMo [IJCAI'22]
    • 2nd step :feet: concatenation-based feature interaction.
      • STARK [ICCV'21], SwinTrack [NeurIPS'22]
    • 3rd step :feet: joint feature extraction and interaction.
      • MixFormer [CVPR'22], OSTrack [ECCV'22], SimTrack [ECCV'22]
    • 4th step :feet: generalized and robust relation modeling.
      • GRM [CVPR'23], ROMTrack [ICCV'23].

  

:bookmark:Unified Tracking (UT)

CVPR 2024

• GLEE (General Object Foundation Model for Images and Videos at Scale) [paper] [code]
• OmniViD (OmniVid: A Generative Framework for Universal Video Understanding) [paper] [code]

CVPR 2023

• OmniTracker (OmniTracker: Unifying Object Tracking by Tracking-with-Detection) [paper] [code]
• UNINEXT (Universal Instance Perception as Object Discovery and Retrieval) [paper] [code]

ICCV 2023

• MITS (Integrating Boxes and Masks: A Multi-Object Framework for Unified Visual Tracking and Segmentation) [paper] [code]

Preprint 2023

• HQTrack (Tracking Anything in High Quality) [paper] [code]
• SAM-Track (Segment and Track Anything) [paper] [code]
• TAM (Track Anything: Segment Anything Meets Videos) [paper] [code]

CVPR 2022

• UTT (Unified Transformer Tracker for Object Tracking) [paper] [code]

ECCV 2022

• Unicorn (Towards Grand Unification of Object Tracking) [paper] [code]

:bookmark:Single Object Tracking (SOT)

CVPR 2024

• AQATrack (Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers) [paper] [code]
• ARTrackV2 (ARTrackV2: Prompting Autoregressive Tracker Where to Look and How to Describe) [paper] [code]
• DiffusionTrack (DiffusionTrack: Point Set Diffusion Model for Visual Object Tracking) [paper] [code]
• HDETrack (Event Stream-Based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline) [paper] [code]
• HIPTrack (Learning Historical Status Prompt for Accurate and Robust Visual Tracking) [paper] [code]
• OneTracker (OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning) [paper] [code]
• QueryNLT (Context-Aware Integration of Language and Visual References for Natural Language Tracking) [paper] [code]
• SDSTrack (SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking) [paper] [code]
• Un-Track (Single-Model and Any-Modality for Video Object Tracking) [paper] [code]

AAAI 2024

• BAT (Bi-Directional Adapter for Multi-Modal Tracking) [paper] [code]
• EVPTrack (Explicit Visual Prompts for Visual Object Tracking) [paper] [code]
• ODTrack (ODTrack: Online Dense Temporal Token Learning for Visual Tracking) [paper] [code]
• STCFormer (Sequential Fusion Based Multi-Granularity Consistency for Space-Time Transformer Tracking) [paper] [code]
• TATrack (Temporal Adaptive RGBT Tracking with Modality Prompt) [paper] [code]
• UVLTrack (Unifying Visual and Vision-Language Tracking via Contrastive Learning) [paper] [code]

IJCAI 2024

• USTrack (Unified Single-Stage Transformer Network for Efficient RGB-T Tracking) [paper] [code]

WACV 2024

• SMAT (Separable Self and Mixed Attention Transformers for Efficient Object Tracking) [paper] [code]
• TaMOs (Beyond SOT: It's Time to Track Multiple Generic Objects at Once) [paper] [code]

Preprint 2024

• ACTrack (ACTrack: Adding Spatio-Temporal Condition for Visual Object Tracking) [paper] [code]
• AMTTrack (Long-Term Frame-Event Visual Tracking: Benchmark Dataset and Baseline) [paper] [code]
• CRSOT (CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event Cameras) [paper] [code]
• DyTrack (Exploring Dynamic Transformer for Efficient Object Tracking) [paper] [code]
• LoRAT (Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance) [paper] [code]
• MAPNet (Multi-Attention Associate Prediction Network for Visual Tracking) [paper] [code]
• MMMP (From Two Stream to One Stream: Efficient RGB-T Tracking via Mutual Prompt Learning and Knowledge Distillation) [paper] [code]
• M3PT (Middle Fusion and Multi-Stage, Multi-Form Prompts for Robust RGB-T Tracking) [paper] [code]
• OIFTrack (Optimized Information Flow for Transformer Tracking) [paper] [code]
• STMT (Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens) [paper] [code]
• SuperSBT (Correlation-Embedded Transformer Tracking: A Single-Branch Framework) [paper] [code]

CVPR 2023

• ART (ARKitTrack: A New Diverse Dataset for Tracking Using Mobile RGB-D Data) [paper] [code]
• ARTrack (Autoregressive Visual Tracking) [paper] [code]
• DropTrack (DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking Tasks) [paper] [code]
• EMT (Resource-Efficient RGBD Aerial Tracking) [paper] [code]
• GRM (Generalized Relation Modeling for Transformer Tracking) [paper] [code]
• JointNLT (Joint Visual Grounding and Tracking with Natural Language Specification) [paper] [code]
• MAT (Representation Learning for Visual Object Tracking by Masked Appearance Transfer) [paper] [code]
• SeqTrack (SeqTrack: Sequence to Sequence Learning for Visual Object Tracking) [paper] [code]
• SwinV2 (Revealing the Dark Secrets of Masked Image Modeling) [paper] [code]
• TBSI (Bridging Search Region Interaction with Template for RGB-T Tracking) [paper] [code]
• VideoTrack (VideoTrack: Learning to Track Objects via Video Transformer) [paper] [code]
• ViPT (Visual Prompt Multi-Modal Tracking) [paper] [code]

NeurIPS 2023

• MixFormerV2 (MixFormerV2: Efficient Fully Transformer Tracking) [paper&review] [code]
• RFGM (Reading Relevant Feature from Global Representation Memory for Visual Object Tracking) [paper&review] [code]
• ZoomTrack (ZoomTrack: Target-Aware Non-Uniform Resizing for Efficient Visual Tracking) [paper&review] [code]

ICCV 2023

• Aba-ViTrack (Adaptive and Background-Aware Vision Transformer for Real-Time UAV Tracking) [paper] [code]
• AiATrack-360 (360VOT: A New Benchmark Dataset for Omnidirectional Visual Object Tracking) [paper] [code]
• CiteTracker (CiteTracker: Correlating Image and Text for Visual Tracking) [paper] [code]
• DecoupleTNL (Tracking by Natural Language Specification with Long Short-Term Context Decoupling) [paper] [code]
• F-BDMTrack (Foreground-Background Distribution Modeling Transformer for Visual Object Tracking) [paper] [code]
• HiT (Exploring Lightweight Hierarchical Vision Transformers for Efficient Visual Tracking) [paper] [code]
• HRTrack (Cross-Modal Orthogonal High-Rank Augmentation for RGB-Event Transformer-Trackers) [paper] [code]
• ROMTrack (Robust Object Modeling for Visual Tracking) [paper] [code]

AAAI 2023

• CTTrack (Compact Transformer Tracker with Correlative Masked Modeling) [paper] [code]
• GdaTFT (Global Dilated Attention and Target Focusing Network for Robust Tracking) [paper] [code]
• TATrack (Target-Aware Tracking with Long-Term Context Attention) [paper] [code]

MM 2023

• All-in-One (All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment) [paper] [code]
• UTrack (Unambiguous Object Tracking by Exploiting Target Cues) [paper] [code]

MMAsia 2023

• UPVPT (Robust Tracking via Unifying Pretrain-Finetuning and Visual Prompt Tuning) [paper] [code]

MICCAI 2023

• ConTrack (ConTrack: Contextual Transformer for Device Tracking in X-ray) [paper] [code]

WACV 2023

• E.T.Track (Efficient Visual Tracking with Exemplar Transformers) [paper] [code]

BMVC 2023

• MVT (Mobile Vision Transformer-Based Visual Object Tracking) [paper] [code]

ICRA 2023

• ClimRT (Continuity-Aware Latent Interframe Information Mining for Reliable UAV Tracking) [paper] [code]
• SGDViT (SGDViT: Saliency-Guided Dynamic vision Transformer for UAV tracking) [paper] [code]

IROS 2023

• CDT (Cascaded Denoising Transformer for UAV Nighttime Tracking) [paper] [code]
• FDNT (End-to-End Feature Decontaminated Network for UAV Tracking) [paper] [code]
• ScaleAwareDA (Scale-Aware Domain Adaptation for Robust UAV Tracking) [paper] [code]
• TOTEM (Transparent Object Tracking with Enhanced Fusion Module) [paper] [code]
• TRTrack (Boosting UAV Tracking With Voxel-Based Trajectory-Aware Pre-Training) [paper] [code]

ICASSP 2023

• MSTL (Multi-Source Templates Learning for Real-Time Aerial Tracking) [paper] [code]
• ProContEXT (ProContEXT: Exploring Progressive Context Transformer for Tracking) [paper] [code]

Preprint 2023

• AViTMP (Exploiting Image-Related Inductive Biases in Single-Branch Visual Tracking) [paper] [code]
• DATr (Leveraging the Power of Data Augmentation for Transformer-Based Tracking) [paper] [code]
• DCPT (DCPT: Darkness Clue-Prompted Tracking in Nighttime UAVs) [paper] [code]
• DETRack (Efficient Training for Visual Tracking with Deformable Transformer) [paper] [code]
• IPL (Modality-Missing RGBT Tracking via Invertible Prompt Learning and A High-Quality Data Simulation Method) [paper] [code]
• JN (Towards Efficient Training with Negative Samples in Visual Tracking) [paper] [code]
• LiteTrack (LiteTrack: Layer Pruning with Asynchronous Feature Extraction for Lightweight and Efficient Visual Tracking) [paper] [code]
• MACFT (RGB-T Tracking Based on Mixed Attention) [paper] [code]
• MixViT (MixFormer: End-to-End Tracking with Iterative Mixed Attention) [paper] [code]
• MMTrack (Towards Unified Token Learning for Vision-Language Tracking) [paper] [code]
• MPLT (RGB-T Tracking via Multi-Modal Mutual Prompt Learning) [paper] [code]
• ProFormer (RGBT Tracking via Progressive Fusion Transformer with Dynamically Guided Learning) [paper] [code]
• RTrack (RTrack: Accelerating Convergence for Visual Object Tracking via Pseudo-Boxes Exploration) [paper] [code]
• SAM-DA (SAM-DA: UAV Tracks Anything at Night with SAM-Powered Domain Adaptation) [paper] [code]
• SATracker (Beyond Visual Cues: Synchronously Exploring Target-Centric Semantics for Vision-Language Tracking) [paper] [code]
• TCTrack++ (Towards Real-World Visual Tracking with Temporal Contexts) [paper] [code]
• USTAM (USTAM: Unified Spatial-Temporal Attention MixFormer for Visual Object Tracking) [paper&review] [code]

CVPR 2022

• CMTR (Cross-Modal Target Retrieval for Tracking by Natural Language) [paper] [code]
• CSWinTT (Transformer Tracking with Cyclic Shifting Window Attention) [paper] [code]
• GTELT (Global Tracking via Ensemble of Local Trackers) [paper] [code]
• MixFormer (MixFormer: End-to-End Tracking with Iterative Mixed Attention) [paper] [code]
• RBO (Ranking-Based Siamese Visual Tracking) [paper] [code]
• SBT (Correlation-Aware Deep Tracking) [paper] [code]
• STNet (Spiking Transformers for Event-Based Single Object Tracking) [paper] [code]
• TCTrack (TCTrack: Temporal Contexts for Aerial Tracking) [paper] [code]
• ToMP (Transforming Model Prediction for Tracking) [paper] [code]
• UDAT (Unsupervised Domain Adaptation for Nighttime Aerial Tracking) [paper] [code]

NeurIPS 2022

• SwinTrack (SwinTrack: A Simple and Strong Baseline for Transformer Tracking) [paper&review] [code]

ECCV 2022

• AiATrack (AiATrack: Attention in Attention for Transformer Visual Tracking) [paper] [code]
• CIA (Hierarchical Feature Embedding for Visual Tracking) [paper] [code]
• DMTracker (Learning Dual-Fused Modality-Aware Representations for RGBD Tracking) [paper] [code]
• HCAT (Efficient Visual Tracking via Hierarchical Cross-Attention Transformer) [paper] [code]
• OSTrack (Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework) [paper] [code]
• SimTrack (Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking) [paper] [code]
• VOT2022 (The Tenth Visual Object Tracking VOT2022 Challenge Results) [paper] [code]

AAAI 2022

• APFNet (Attribute-Based Progressive Fusion Network for RGBT Tracking) [paper] [code]

IJCAI 2022

• InMo (Learning Target-Aware Representation for Visual Tracking via Informative Interactions) [paper] [code]
• SparseTT (SparseTT: Visual Tracking with Sparse Transformers) [paper] [code]

MICCAI 2022

• TLT (Transformer Lesion Tracker) [paper] [code]

WACV 2022

• SiamTPN (Siamese Transformer Pyramid Networks for Real-Time UAV Tracking) [paper] [code]

ICRA 2022

• SCT (Tracker Meets Night: A Transformer Enhancer for UAV Tracking) [paper] [code]

ACCV 2022

• TAT (Temporal-Aware Siamese Tracker: Integrate Temporal Context for 3D Object Tracking) [paper] [code]

IROS 2022

• HighlightNet (HighlightNet: Highlighting Low-Light Potential Features for Real-Time UAV Tracking) [paper] [code]
• LPAT (Local Perception-Aware Transformer for Aerial Tracking) [paper] [code]
• SiamSA (Siamese Object Tracking for Vision-Based UAM Approaching with Pairwise Scale-Channel Attention) [paper] [code]

Preprint 2022

• CEUTrack (Revisiting Color-Event Based Tracking: A Unified Network, Dataset, and Metric) [paper] [code]
• FDT (Feature-Distilled Transformer for UAV Tracking) [paper] [code]
• RAMAVT (On Deep Recurrent Reinforcement Learning for Active Visual Tracking of Space Noncooperative Objects) [paper] [code]
• SFTransT (Learning Spatial-Frequency Transformer for Visual Object Tracking) [paper] [code]
• SiamLA (Learning Localization-Aware Target Confidence for Siamese Visual Tracking) [paper] [code]
• SPT (RGBD1K: A Large-scale Dataset and Benchmark for RGB-D Object Tracking) [paper] [code]

CVPR 2021

• SiamGAT (Graph Attention Tracking) [paper] [code]
• STMTrack (STMTrack: Template-Free Visual Tracking with Space-Time Memory Networks) [paper] [code]
• TMT (Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking) [paper] [code]
• TransT (Transformer Tracking) [paper] [code]

ICCV 2021

• AutoMatch (Learn to Match: Automatic Matching Network Design for Visual Tracking) [paper] [code]
• DTT (High-Performance Discriminative Tracking with Transformers) [paper] [code]
• DualTFR (Learning Tracking Representations via Dual-Branch Fully Transformer Networks) [paper] [code]
• HiFT (HiFT: Hierarchical Feature Transformer for Aerial Tracking) [paper] [code]
• SAMN (Learning Spatio-Appearance Memory Network for High-Performance Visual Tracking) [paper] [code]
• STARK (Learning Spatio-Temporal Transformer for Visual Tracking) [paper] [code]
• TransT-M (High-Performance Transformer Tracking) [paper] [code]
• VOT2021 (The Ninth Visual Object Tracking VOT2021 Challenge Results) [paper] [code]

BMVC 2021

• TAPL (TAPL: Dynamic Part-Based Visual Tracking via Attention-Guided Part Localization) [paper] [code]

Preprint 2021

• TREG (Target Transformed Regression for Accurate Tracking) [paper] [code]
• TrTr (TrTr: Visual Tracking with Transformer) [paper] [code]

CVPR 2020

• SiamAttn (Deformable Siamese Attention Networks for Visual Object Tracking) [paper] [code]

ICPR 2020

• VTT (VTT: Long-Term Visual Tracking with Transformers) [paper] [code]

:bookmark:3D Single Object Tracking (3DSOT)

ICLR 2024

• CUTrack (Towards Category Unification of 3D Single Object Tracking on Point Clouds) [paper&review] [code]

AAAI 2024

• M3SOT (M3SOT: Multi-Frame, Multi-Field, Multi-Space 3D Single Object Tracking) [paper] [code]
• SCVTrack (Robust 3D Tracking with Quality-Aware Shape Completion) [paper] [code]
• StreamTrack (Modeling Continuous Motion for 3D Point Cloud Object Tracking) [paper] [code]

ICRA 2024

• SeqTrack3D (SeqTrack3D: Exploring Sequence Information for Robust 3D Point Cloud Tracking) [paper] [code]

Preprint 2024

• EasyTrack (EasyTrack: Efficient and Compact One-Stream 3D Point Clouds Tracker) [paper] [code]
• PillarTrack (PillarTrack: Redesigning Pillar-Based Transformer Network for Single Object Tracking on Point Clouds) [paper] [code]

CVPR 2023

• CorpNet (Correlation Pyramid Network for 3D Single Object Tracking) [paper] [code]
• CXTrack (CXTrack: Improving 3D Point Cloud Tracking with Contextual Information) [paper] [code]

ICCV 2023

• MBPTrack (MBPTrack: Improving 3D Point Cloud Tracking with Memory Networks and Box Priors) [paper] [code]
• SyncTrack (Synchronize Feature Extracting and Matching: A Single Branch Framework for 3D Object Tracking) [paper] [code]

AAAI 2023

• GLT-T (GLT-T: Global-Local Transformer Voting for 3D Single Object Tracking in Point Clouds) [paper] [code]

IJCAI 2023

• OSP2B (OSP2B: One-Stage Point-to-Box Network for 3D Siamese Tracking) [paper] [code]

IROS 2023

• PCET (Implicit and Efficient Point Cloud Completion for 3D Single Object Tracking) [paper] [code]
• STTracker (STTracker: Spatio-Temporal Tracker for 3D Single Object Tracking) [paper] [code]

Preprint 2023

• GLT-T++ (GLT-T++: Global-Local Transformer for 3D Siamese Tracking with Ranking Loss) [paper] [code]
• MCSTN (Multi-Correlation Siamese Transformer Network with Dense Connection for 3D Single Object Tracking) [paper] [code]
• MMF-Track (Multi-Modal Multi-Level Fusion for 3D Single Object Tracking) [paper] [code]
• MTM-Tracker (Motion-to-Matching: A Mixed Paradigm for 3D Single Object Tracking) [paper] [code]
• StreamTrack (Modeling Continuous Motion for 3D Point Cloud Object Tracking) [paper] [code]

CVPR 2022

• PTTR (PTTR: Relational 3D Point Cloud Object Tracking with Transformer) [paper] [code]

ECCV 2022

• CMT (CMT: Context-Matching-Guided Transformer for 3D Tracking in Point Clouds) [paper] [code]
• SpOT (SpOT: Spatiotemporal Modeling for 3D Object Tracking) [paper] [code]
• STNet (3D Siamese Transformer Network for Single Object Tracking on Point Clouds) [paper] [code]

Preprint 2022

• OST (OST: Efficient One-stream Network for 3D Single Object Tracking in Point Clouds) [paper] [code]
• PTTR++ (Exploring Point-BEV Fusion for 3D Point Cloud Object Tracking with Transformer) [paper] [code]
• RDT (Point Cloud Registration-Driven Robust Feature Matching for 3D Siamese Object Tracking) [paper] [code]

BMVC 2021

• LTTR (3D Object Tracking with Transformer) [paper] [code]

IROS 2021

• PTT (PTT: Point-Track-Transformer Module for 3D Single Object Tracking in Point Clouds) [paper] [code]