This repository provides the official code for the paper Iterative Loop Method Combining Active and Semi-Supervised Learning for Domain Adaptive Semantic Segmentation.

Abstract Semantic segmentation is an important technique for environment perception in intelligent transportation systems. With the rapid development of convolutional neural networks (CNNs), road scene analysis can usually achieve satisfactory results in the source domain. However, guaranteeing good generalization to different target domain scenarios remains a significant challenge. Recently, semi-supervised learning and active learning have been proposed to alleviate this problem. Semi-supervised learning can improve model accuracy with massive unlabeled data, but some pseudo labels containing noise would be generated with limited or imbalanced training data. And there will be suboptimal models if human guidance is absent. Active learning can select more effective data to intervene,while the model accuracy can not be improved because the massive unlabeled data are not used. And the probability of querying sub-optimal samples will increase when the domain difference is too large, increasing annotation cost. This paper proposes an iterative loop method combining active and semi-supervised learning for domain adaptive semantic segmentation. The method first uses semi-supervised to learn massive unlabeled data to improve model accuracy and provide more accurate selection models for active learning. Secondly, combined with the predictive uncertainty sample selection strategy of active learning, manual intervention is used to correct the pseudo-labels. Finally, flexible iterative loops achieve the best performance with minimal labeling cost. Extensive experiments show that our method establishes state-of-the-art performance on tasks of GTAV→Cityscapes, SYNTHIA→Cityscapes, improving by 4.9% mIoU and 5.2% mIoU, compared to the previous best method, respectively.

For more information on ILM-ASSL, please check our [Paper].

Usage

Prerequisites

• Python 3.6.9
• Pytorch 1.8.1
• torchvision 0.9.1

Step-by-step installation

git clone https://github.com/licongguan/ILM-ASSL.git && cd ILM-ASSL
conda create -n ILM-ASSL python=3.6.9
conda activate ILM-ASSL
pip install -r requirements.txt
pip install torch==1.8.1+cu102 torchvision==0.9.1+cu102 -f https://download.pytorch.org/whl/torch_stable.html

Data Preparation

• Download The Cityscapes Dataset, The GTAV Dataset, and The SYNTHIA Dataset

First, the data folder should be structured as follows:

├── datasets/
│   ├── cityscapes/     
|   |   ├── gtFine/
|   |   ├── leftImg8bit/
│   ├── gtav/
|   |   ├── images/
|   |   ├── labels/
│   └──	synthia/
|   |   ├── RGB/
|   |   ├── LABELS/

Second, generate _labelTrainIds.png for the cityscapes dataset:

pip install cityscpaesscripts
pip install cityscpaesscripts[gui]
export CITYSCAPES_DATASET='/path_to_cityscapes'
csCreateTrainIdLabelImgs

Final, rename the gtav and synthia files for by running:

python ILM-ASSL/datasets/rename.py

Prepare Pretrained Backbone

Before training, please download ResNet101 pretrained on ImageNet-1K from one of the following:

• Google Drive
• Baidu Drive Fetch Code: 0305

After that, modify model_urls in ILM-ASSL/models/resnet.py to </path/to/resnet101.pth>

Model Zoo

GTAV to Cityscapes

We have put our model checkpoints here [Google Drive] [百度网盘] (提取码0305).

SYNTHIA to Cityscapes

ILM-ASSL Training

We provide the training scripts using Multiple GPU.

# training for GTAV to Cityscapes
# use GTAV 2000 labeled images and Cityscpaes 30(1%) labeled images
cd  experiments/gtav2cityscapes/1.0%
# use torch.distributed.launch
sh train.sh <num_gpu> <port>

ILM-ASSL Testing

sh eval.sh

Acknowledgement

This project is based on the following open-source projects: U²PL and RIPU. We thank their authors for making the source code publically available.

Citation

If you find this project useful in your research, please consider citing:

@article{guan2023iterative,
  title={Iterative Loop Method Combining Active and  Semi-Supervised Learning for Domain  Adaptive Semantic Segmentation},
  author={Guan, Licong and Yuan, Xue},
  journal={arXiv preprint arXiv:2301.13361},
  year={2023}
}

Contact

If you have any problem about our code, feel free to contact

• [email protected]

or describe your problem in Issues.