Mmaction2 Versions Save

OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark

v1.2.0

7 months ago

Highlights

  • Support the Training of ActionClip
  • Support VindLU multi-modality algorithm
  • Support MobileOne TSN/TSM

New Features

  • Support the Training of ActionClip (2620)
  • Support video retrieval dataset MSVD (2622)
  • Support VindLU multi-modality algorithm (2667)
  • Support Dense Regression Network for Video Grounding (2668)

Improvements

  • Support Video Demos (2602)
  • Support Audio Demos (2603)
  • Add README_zh-CN.md for Swin and VideoMAE (2621)
  • Support MobileOne TSN/TSM (2656)
  • Support SlowOnly K700 feature to train localization models (2673)

Bug Fixes

  • Refine ActionDataSample structure (2658)
  • Fix MPS device (2619)

v1.1.0

10 months ago

New Direction: Multi-Modal Video Understanding

We support two novel models for video recognition and retrieval based on open-domain text: ActionCLIP and CLIP4Clip. These models mark the first step of MMAction2's journey towards multi-modal video understanding. Furthermore, we also introduce a new video retrieval dataset, MSR-VTT.

img_v2_e882ffb4-84c9-4b3a-9ab6-38c251e7d95g

For more details, please refer to ActionCLIP, CLIP4Clip and MSR-VTT.

Supported by @Dai-Wenxun in #2470 and #2489.

New Config Type

MMEngine introduced the pure Python style configuration file:

  • Support navigating to base configuration file in IDE
  • Support navigating to base variable in IDE
  • Support navigating to source code of class in IDE
  • Support inheriting two configuration files containing the same field
  • Load the configuration file without other third-party requirements

Refer to the tutorial for more detailed usages.

img_v2_e882ffb4-84c9-4b3a-9ab6-38c251e7d95g

New Datasets

We are glad to support 3 new datasets:

(ICCV2019) HACS

HACS is a new large-scale dataset for recognition and temporal localization of human actions collected from Web videos.

https://github.com/open-mmlab/mmaction2/assets/58767402/7b7407e3-994a-4523-975c-5bdee3b54998

For more details, please refer to HACS.

Supported by @hukkai in #2224

(ICCV2021) MultiSports

MultiSports is a multi-person video dataset of spatio-temporally localized sports actions.

https://github.com/open-mmlab/mmaction2/assets/58767402/1f94668a-823b-46a0-9ea7-eedf0f29d1d1

For more details, please refer to MultiSports.

Supported by @cir7 in #2280

(Arxiv2022) Kinetics-710

For more details, please refer to Kinetics710.

Supported by @cir7 in #2534

Other New Features

What's Changed

New Contributors

Full Changelog: https://github.com/open-mmlab/mmaction2/compare/v1.0.0...v1.1.0

v1.0.0

1 year ago

Highlights

We are excited to announce the release of MMAction2 1.0.0 as a part of the OpenMMLab 2.0 project! MMAction2 1.0.0 introduces an updated framework structure for the core package and a new section called Projects. This section showcases various engaging and versatile applications built upon the MMAction2 foundation.

In this latest release, we have significantly refactored the core package's code to make it clearer, more comprehensible, and disentangled. This has resulted in improved performance for several existing algorithms, ensuring that they now outperform their previous versions. Additionally, we have incorporated some cutting-edge algorithms, such as VideoSwin and VideoMAE, to further enhance the capabilities of MMAction2 and provide users with a more comprehensive and powerful toolkit. The new Projects section serves as an essential addition to MMAction2, created to foster innovation and collaboration among users. This section offers the following attractive features:

  • Flexible code contribution: Unlike the core package, the Projects section allows for a more flexible environment for code contributions, enabling faster integration of state-of-the-art models and features.
  • Showcase of diverse applications: Explore various projects built upon the MMAction2 foundation, such as deployment examples and combinations of video recognition with other tasks.
  • Fostering creativity and collaboration: Encourages users to experiment, build upon the MMAction2 platform, and share their innovative applications and techniques, creating an active community of developers and researchers. Discover the possibilities within the "Projects" section and join the vibrant MMAction2 community in pushing the boundaries of video understanding applications!

Exciting Features

RGBPoseConv3D

RGBPoseConv3D is a framework that jointly uses 2D human skeletons and RGB appearance for human action recognition. It is a 3D CNN with two streams, with the architecture borrowed from SlowFast. In RGBPoseConv3D:

  • The RGB stream corresponds to the slow stream in SlowFast; The Skeleton stream corresponds to the fast stream in SlowFast.
  • The input resolution of RGB frames is 4x larger than the pseudo heatmaps.
  • Bilateral connections are used for early feature fusion between the two modalities.

Inferencer

In this release, we introduce the MMAction2Inferencer, which is a versatile API for the inference that supports multiple input types. The API enables users to easily specify and customize action recognition models, streamlining the process of performing video prediction using MMAction2.

Usage:

python demo/demo_inferencer.py ${INPUTS} [OPTIONS]
  • The INPUTS can be a video path or rawframes folder. For more detailed information on OPTIONS, please refer to Inferencer.

Example:

python demo/demo_inferencer.py zelda.mp4 --rec tsn --vid-out-dir zelda_out --label-file tools/data/kinetics/label_map_k400.txt

You can find the zelda.mp4 here. The output video is displayed below:

https://user-images.githubusercontent.com/58767402/232312742-f5eb2e8c-f015-459c-8a4d-99c331a65735.mp4

List of Novel Features

MMAction2 V1.0 introduces support for new models and datasets in the field of video understanding, including MSG3D [Project] (CVPR'2020), CTRGCN [Project] (CVPR'2021), STGCN++ (Arxiv'2022), Video Swin Transformer (CVPR'2022), VideoMAE (NeurIPS'2022), C2D (CVPR'2018), MViT V2 (CVPR'2022), UniFormer V1 (ICLR'2022), and UniFormer V2 (Arxiv'2022), as well as the spatiotemporal action detection dataset AVA-Kinetics (Arxiv'2022).

image

  • Enhanced Omni-Source: We enhanced the original omni-source technique by dynamically adjusting 3D convolutional network architecture to simultaneously utilize videos and images for training. Taking the SlowOnlyR50 8x8 as an example, the Top-1 accuracy comparison of the three training methods illustrates that our omni-source training effectively employs the additional ImageNet dataset, significantly boosting performance on Kinetics400.
  • Mulit-Stream Skeleton Pipeline: In light of MMAction2's prior support for only joint and bone modalities, we have extended support to joint motion and bone motion modalities in MMAction2 V1.0. Furthermore, we have conducted training and evaluation for these four modalities using NTU60 2D and 3D keypoint data on STGCN, 2s-AGCN, and STGCN++.
  • Repeat Augment was initially proposed as a data augmentation method for ImageNet training and has been employed in recent Video Transformer works. Whenever a video is read during training, we use multiple (typically 2-4) random samples from the video for training. This approach not only enhances the model's generalization capability but also reduces the IO pressure of video reading. We support Repeat Augment in MMAction2 V1.0 and utilize this technique in MViT V2 training. The table below compares the Top-1 accuracy on Kinetics400 before and after employing Repeat Augment:

Bug Fixes

New Contributors

Full Changelog: https://github.com/open-mmlab/mmaction2/compare/v0.24.0...v1.0.0

v1.0.0rc3

1 year ago

Highlights

  • Support Action Recognition model UniFormer V1(ICLR'2022), UniFormer V2(Arxiv'2022).
  • Support training MViT V2(CVPR'2022), and MaskFeat(CVPR'2022) fine-tuning.

New Features

  • Support UniFormer V1/V2 (#2153)
  • Support training MViT, and MaskFeat fine-tuning (#2186)
  • Support a unified inference interface: Inferencer (#2164)

Improvements

  • Support load data list from multi-backends (#2176)

Bug Fixes

  • Upgrade isort to fix CI (#2198)
  • Fix bug in skeleton demo (#2214)

Documentation

  • Add Chinese documentation for config.md (#2188)
  • Add readme for Omnisource (#2205)

v1.0.0rc2

1 year ago

Highlights

  • Support Action Recognition model VideoMAE(NeurIPS'2022), MVit V2(CVPR'2022), C2D and skeleton-based action recognition model STGCN++
  • Support Omni-Source training on ImageNet and Kinetics datasets
  • Support exporting spatial-temporal detection models to ONNX

New Features

  • Support VideoMAE (#1942)
  • Support MViT V2 (#2007)
  • Supoort C2D (#2022)
  • Support AVA-Kinetics dataset (#2080)
  • Support STGCN++ (#2156)
  • Support exporting spatial-temporal detection models to ONNX (#2148)
  • Support Omni-Source training on ImageNet and Kinetics datasets (#2143)

Improvements

  • Support repeat batch data augmentation (#2170)
  • Support calculating FLOPs tool powered by fvcore (#1997)
  • Support Spatial-temporal detection demo (#2019)
  • Add SyncBufferHook and add randomness config in train.py (#2044)
  • Refactor gradcam (#2049)
  • Support init_cfg in Swin and ViTMAE (#2055)
  • Refactor STGCN and related pipelines (#2087)
  • Refactor visualization tools (#2092)
  • Update SampleFrames transform and improve most models' performance (#1942)
  • Support real-time webcam demo (#2152)
  • Refactor and enhance 2s-AGCN (#2130)
  • Support adjusting fps in SampleFrame (#2157)

Bug Fixes

  • Fix CI upstream library dependency (#2000)
  • Fix SlowOnly readme typos and results (#2006)
  • Fix VideoSwin readme (#2010)
  • Fix tools and mim error (#2028)
  • Fix Imgaug wrapper (#2024)
  • Remove useless scripts (#2032)
  • Fix multi-view inference (#2045)
  • Update mmcv maximum version to 1.8.0 (#2047)
  • Fix torchserver dependency (#2053)
  • Fix gen_ntu_rgbd_raw script (#2076)
  • Update AVA-Kinetics experiment configs and results (#2099)
  • Add joint.pkl and bone.pkl used in multi-stream fusion tool (#2106)
  • Fix lint CI config (#2110)
  • Update testing accuracy for modified SampleFrames (#2117), (#2121), (#2122), (#2124), (#2125), (#2126), (#2129), (#2128)
  • Fix timm related bug (#1976)
  • Fix check_videos.py script (#2134)
  • Update CI maximum torch version to 1.13.0 (#2118)

Documentation

  • Add MMYOLO description in README (#2011)
  • Add v1.x introduction in README (#2023)
  • Fix link in README (#2035)
  • Refine some docs (#2038), (#2040), (#2058)
  • Update TSN/TSM Readme (#2082)
  • Add chinese document (#2083)
  • Adjust docment structure (#2088)
  • Fix Sth-Sth and Jester dataset links (#2103)
  • Fix doc link (#2131)

v1.0.0rc1

1 year ago

Highlights

  • Support Video Swin Transformer

New Features

  • Support Video Swin Transformer (#1939)

Improvements

  • Add colab tutorial for 1.x (#1956)
  • Support skeleton-based action recognition demo (#1920)

Bug Fixes

v1.0.0rc0

1 year ago

We are excited to announce the release of MMAction2 v1.0.0rc0. MMAction2 1.0.0beta is the first version of MMAction2 1.x, a part of the OpenMMLab 2.0 projects. Built upon the new training engine.

Highlights

  • New engines. MMAction2 1.x is based on MMEngine](https://github.com/open-mmlab/mmengine), which provides a general and powerful runner that allows more flexible customizations and significantly simplifies the entrypoints of high-level interfaces.

  • Unified interfaces. As a part of the OpenMMLab 2.0 projects, MMAction2 1.x unifies and refactors the interfaces and internal logics of train, testing, datasets, models, evaluation, and visualization. All the OpenMMLab 2.0 projects share the same design in those interfaces and logics to allow the emergence of multi-task/modality algorithms.

  • More documentation and tutorials. We add a bunch of documentation and tutorials to help users get started more smoothly. Read it here.

Breaking Changes

In this release, we made lots of major refactoring and modifications. Please refer to the migration guide for details and migration instructions.

v0.24.1

1 year ago

This release is meant to fix the compatibility with the latest mmcv v1.6.1

v0.24.0

2 years ago

Highlights

  • Support different seeds

New Features

  • Add lateral norm in multigrid config (#1567)
  • Add openpose 25 joints in graph config (#1578)
  • Support MLU Backend (#1608)

Bug and Typo Fixes

  • Fix local_rank (#1558)
  • Fix install typo (#1571)
  • Fix the inference API doc (#1580)
  • Fix zh-CN demo.md and getting_started.md (#1587)
  • Remove Recommonmark (#1595)
  • Fix inference with ndarray (#1603)
  • Fix the log error when IterBasedRunner is used (#1606)

v0.23.0

2 years ago

Highlights

  • Support different seeds
  • Provide multi-node training & testing script
  • Update error log

New Features

  • Support different seeds(#1502)
  • Provide multi-node training & testing script(#1521)
  • Update error log(#1546)

Documentations

  • Update gpus in Slowfast readme(#1497)
  • Fix work_dir in multigrid config(#1498)
  • Add sub bn docs(#1503)
  • Add shortcycle sampler docs(#1513)
  • Update Windows Declaration(#1520)
  • Update the link for ST-GCN(#1544)
  • Update install commands(#1549)

Bug and Typo Fixes

  • Update colab tutorial install cmds(#1522)
  • Fix num_iters_per_epoch in analyze_logs.py(#1530)
  • Fix distributed_sampler(#1532)
  • Fix cd dir error(#1545)
  • Update arg names(#1548)