Openmixup Versions Save

A collection of weights and logs for image classification experiments with modern Transformer architectures on CIFAR-100. These benchmarks are proposed for the convenience of conducting research in Mixup augmentations with Transformers since the most published benchmarks of Mixup variants with ViTs are based on ImageNet-1K. Please refer to our tech report for more details.

• Since the original resolutions of CIFAR-100 are too small for ViTs, we resize the input images to $224\times 224$ (training and testing) while not modifying the ViT architectures. This benchmark uses the DeiT setup and trains the model for 200 or 600 epochs with a batch size of 100 on CIFAR-100. The basic learning rates of DeiT and Swin are $1e-3$ and $5e-4$, which is the optimal setup in our experiments. We search and report $\alpha$ in $Beta(\alpha, \alpha)$ for all compared methods. View config files in mixups/vits.
• The best of top-1 accuracy in the last 10 training epochs is reported for ViT architectures. We released the trained models and logs in vits-mix-cifar100-weights.

ViTs' Mixup Benchmark on CIFAR-100

We provide a collection of model weights and logs for image classification networks on ImageNet-1K (download) reproduced with OpenMixup or MMLab frameworks. You can view the training setting in config files and README pages of related models. You can download all files from Baidu Cloud (cicj).

If you want us to reproduce a new model or can provide reproduced results to OpenMixup, Please contact us by GitHub issued or e-mail. This release is on updating for a long time!

ImageNet Classification with OpenMixup

A collection of weights and logs for image classification experiments with RSB A3 training setting on ImageNet-1K (download). You can view the training setting in ResNet strikes back and find the full results in MogaNet (Appendix Table A.7). You can download all files from Baidu Cloud (ss3j).

• We train all models for 100 epochs according to the RSB A3 setting on ImageNet-1K. We turn the basic learning in {8e-3, 6e-3} to get better performances.
• The best top-1 accuracy of image classification in the last 10 training epochs is reported for all experiments.

RSB A3 Image Classification on ImageNet-1K

A collection of weights and logs for image classification experiments of MogaNet on ImageNet-1K (download). You can download all files from Baidu Cloud (z8mf) at MogaNet/Classification_OpenMixup.

• We train MogaNet for 100 and 300 epochs according to the RSB A3 and DeiT settings on ImageNet-1K. Note that * denotes the refined training setting of lightweight models with 3-Augment. Refer to the Appendix of MogaNet for more training details.
• The best top-1 accuracy of image classification in the last 10 training epochs is reported for all experiments. Note that we report the classification accuracy of EMA weights for MogaNet-S, MogaNet-B, and MogaNet-L.
• As for evaluation experiments of the pre-trained weights, you can test them with tools/dist_test.sh for the classification performance or fine-tune them on downstream tasks by only loading the encoder weights, e.g., COCO detection and ADE20K segmentation.
• Warning of attn_force_fp32: During fp16 training, we force to run the gating functions with fp32 to avoid inf or nan. We found that if we use attn_force_fp32=True during training, it should also keep attn_force_fp32=True during evaluation. This might be caused by the difference between the output results of using attn_force_fp32 or not. It will not affect performances of fully fine-tuning but the results of transfer learning (e.g., COCO Mask-RCNN freezes the parameters of the first stage). We set it to true by default in OpenMixup while removing it in MogaNet implementation. For example, you can use moga_small_ema_sz224_8xb128_ep300.pth with attn_force_fp32=True while using moga_small_ema_sz224_8xb128_no_forcefp32_ep300.pth with attn_force_fp32=False.

Image Classification on ImageNet-1K

A collection of weights and logs for self-supervised learning benchmark on ImageNet-1K (download). You can find pre-training codes of compared methods in OpenMixup, VISSL, solo-learn, and the official repositories. You can download all files from Baidu Cloud: A2MIM (3q5i).

• All compared methods adopt ResNet-50 or ViT-B architectures and are pre-trained 100/300 or 800 epochs on ImageNet-1K. The pre-training and fine-tuning testing image size are $224\times 224$. The fine-tuning protocols include: RSB A3 and RSB A2 for ResNet-50, BEiT (SimMIM) for ViT-B. Refer to the paper of A2MIM for more details.
• The best top-1 accuracy of fine-tuning in the last 10 training epochs is reported for all self-supervised methods.
• Visualization of mixed samples of A2MIM are provided in zip files.
• As for pre-training and fine-tuning weights, you can evaluate them with tools/dist_test.sh or fine-tune pre-trained models tools/dist_train.sh with --load_checkpoint (loading the full checkpoints). Note that pre-trained weights stated with full_ contains the full keys of pre-trained models while backbone_ only contains the encoder weights, which can be used for downstream tasks, e.g., COCO detection and ADE20K segmentation.

Self-supervised Pre-training and Fine-tuning with ResNet-50 on ImageNet-1K

We provide the source of pre-trained weights, pre-training epochs, fine-tuning epochs and protocol, and top-1 accuracy in the following table.

Self-supervised Pre-training and Fine-tuning with ViT-B on ImageNet-1K

We provide the source of pre-trained weights, pre-training epochs, fine-tuning epochs and protocol, and top-1 accuracy in the following table.

A collection of weights and logs for mixup classification benchmark on iNaturalist-2018 (download, config). You can download all files from Baidu Cloud: iNaturalist-2018 (wy2v).

• All compared methods adopt ResNet-50 and ResNeXt-101 (32x4d) architectures and are trained 100 epochs using the PyTorch training recipe. The training and testing image size is 224 with the CenterCrop ratio of 0.85. We search $\alpha$ in $Beta(\alpha, \alpha)$ for all compared methods.
• The median of top-1 accuracy in the last 5 training epochs is reported for ResNet variants.
• Visualization of mixed samples from AutoMix and SAMix are provided in zip files. [2022-08-22] Update MixBlock keys in AutoMix and SAMix checkpoints.
• Test pre-trained weights with tools/dist_test.sh or fine-tune pre-trained models tools/dist_train.sh with --load_checkpoint.

Mixup Classification Benchmark on iNaturalist-2018

A collection of weights and logs for mixup classification benchmark on iNaturalist-2017 (download, config). You can download all files from Baidu Cloud: iNaturalist-2017 (1e7w).

• All compared methods adopt ResNet-18/50 and ResNeXt-101 (32x4d) architectures and are trained 100 epochs using the PyTorch training recipe. The training and testing image size is 224 with the CenterCrop ratio of 0.85. We search $\alpha$ in $Beta(\alpha, \alpha)$ for all compared methods.
• The median of top-1 accuracy in the last 5 training epochs is reported for ResNet variants.
• Visualization of mixed samples from AutoMix and SAMix are provided in zip files. [2022-08-22] Update MixBlock keys in AutoMix and SAMix checkpoints.
• Test pre-trained weights with tools/dist_test.sh or fine-tune pre-trained models tools/dist_train.sh with --load_checkpoint.

Mixup Classification Benchmark on iNaturalist-2017

A collection of weights and logs for mixup classification benchmark on Place205 (download, config). You can download all files from Baidu Cloud (4m94).

• All compared methods adopt ResNet-18/50 architectures and are trained 100 epochs using the PyTorch training recipe. The training and testing image size is 224 with the CenterCrop ratio of 0.85. We search $\alpha$ in $Beta(\alpha, \alpha)$ for all compared methods.
• The median of top-1 accuracy in the last 5 training epochs is reported for ResNet variants.
• Visualization of mixed samples from AutoMix and SAMix are provided in zip files. [2022-08-22] Update MixBlock keys in AutoMix and SAMix checkpoints.
• Test pre-trained weights with tools/dist_test.sh or fine-tune pre-trained models tools/dist_train.sh with --load_checkpoint.

Mixup Classification Benchmark on Place205

Highlight

• Support the online document of OpenMixup (built on Read the Docs).
• Provide README and update configs for self-supervised and supervised methods.
• Support new contrastive learning method (Barlow Twins) and Masked Image Modeling (MIM) methods (MAE, SimMIM, MaskFeat, CAE, A2MIM).
• Support new backbone networks (ConvMixer, DenseNet, MLPMixer, ResNeSt, PoolFormer, UniFormer, VAN).
• Support new Fine-tuing method (HCR).
• Support new mixup augmentation methods (SmoothMix, GridMix).
• Support more regression losses (Charbonnier loss, Focal Frequency loss, Focal L1/L2 loss, Balanced L1 loss, Balanced MSE loss).
• Support more regression metrics (regression errors and correlations) and the regression dataset.
• Support more reweight classification losses (Gradient Harmonized loss, Varifocal Focal Loss) from MMDetection.
• Model Zoos and lists of Awesome Mixups have been updated.

Bug Fixes

• Refactor code structures of openmixup.models.utils and support more network layers.
• Fix the bug of DropPath (using stochastic depth rule) in ResNet for RSB A1/A2 training settings.
• Fix bugs in self-supervised classification benchmarks (configs and implementations of VisionTransformer).
• Update INSTALL.md. We suggest you install PyTorch 1.8 or higher and mmcv-full for better usage of this repo. Since PyTorch 1.8 has bugs in AdamW optimizer, do not use PyTorch 1.8 to fine-tune ViT-based methods.
• Fix bugs in PreciseBNHook (update all BN stats) and RepeatSampler (set sync_random_seed) for RSB A1/A2.
• Fix bugs in regression metrics, MIM dataset, and benchmark configs. Notice that only l1_loss is supported by FP16 training, other regression losses (e.g., MSE and Smooth_L1 losses) will cause NAN when the target and prediction are not normalized in FP16 training.

Highlights

• Support various popular backbones (ConvNets and ViTs), various image datasets, popular mixup methods, and benchmarks for supervised learning. Config files are available (reorganized).
• Support popular self-supervised methods (e.g., BYOL, MoCo.V3, MAE, SimMIM) on both large-scale and small-scale datasets, and self-supervised benchmarks (merged from MMSelfSup). Config files are available (reorganized).
• Support analyzing tools for self-supervised learning (kNN/SVM/linear metrics and t-SNE/UMAP visualization).
• Convenient usage of configs: fast configs generation by 'auto_train.py' and configs inheriting (MMCV).
• Support mixed-precision training (NVIDIA Apex or MMCV Apex) for all methods.
• Model Zoos and lists of Awesome Mixups have been released.

Bug Fixes

• Done code refactoring follows MMSelfSup and MMClassification #3.
• Fix mixed-precision training overflow (NAN & INF in supervised mixup methods).
• Fix fine-tuning settings (ViT and Swin Transformer) as MMSelfsup.