UNICORN :unicorn:

Official PyTorch implementation of Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency (ECCV 2022). Check out our webpage for video results!

This repository contains:

• scripts to download and load both datasets and pretrained models
• demo to reconstruct cars from raw images (script or interactive notebook )
• configs to train the models from scratch
• evaluation pipelines to reproduce quantitative results
• guidelines to train a model on a new dataset

@inproceedings{monnier2022unicorn,
  title={{Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency}},
  author={Monnier, Tom and Fisher, Matthew and Efros, Alexei A and Aubry, Mathieu},
  booktitle={{ECCV}},
  year={2022},
}

Installation :construction_worker:

1. Create conda environment :wrench:

conda env create -f environment.yml
conda activate unicorn

git clone https://github.com/facebookresearch/visdom
cd visdom && pip install -e .

2. Download datasets :arrow_down:

bash scripts/download_data.sh

This command will download one of the following datasets:

• ShapeNet NMR: paper / NMR paper / dataset (33Go, thanks to the DVR team for hosting the data)
• CUB-200-2011: paper / webpage / dataset (1Go)
• Pascal3D+ Cars: paper / webpage (with ftp download link, 7.5Go) / UCMR annotations (bbox + train/test split, thanks to the UCMR team for hosting them) / UNICORN annotations(3D shape ground-truth)
• CompCars: paper / webpage / dataset (12Go, thanks to the GIRAFFE team for hosting the data)
• LSUN: paper / webpage / horse dataset (69Go) / moto dataset (42Go)

3. Download pretrained models :arrow_down:

bash scripts/download_model.sh

We provide a small (200Mo) and a big (600Mo) version for each pretrained model (see training section for details). The command will download one of the following models:

• car trained on CompCars: car.pkl / car_big.pkl
• car_p3d trained on Pascal3D+: car_p3d.pkl / car_p3d_big.pkl
• bird trained on CUB: bird.pkl / bird_big.pkl
• moto trained on LSUN Motorbike: moto.pkl / moto_big.pkl
• horse trained on LSUN Horse: horse.pkl / horse_big.pkl
• sn_* trained on each ShapeNet category: airplane, bench, cabinet, car, chair, display, lamp, phone, rifle, sofa, speaker, table, vessel
• sn_big_* trained on each ShapeNet category: airplane, bench, cabinet, car, chair, display, lamp, phone, rifle, sofa, speaker, table, vessel

How to use :rocket:

1. Demo - 3D reconstruction of car images :oncoming_automobile:

You first need to download the car model (see above), then launch:

cuda=gpu_id model=car_big.pkl input=demo ./scripts/reconstruct.sh

where gpu_id is a target cuda device id, car_big.pkl corresponds to a pretrained model, demo is a folder containing the target images. Reconstruction results (.obj + gif) will be saved in a folder demo_rec.

We also provide an interactive demo to reconstruct cars from single images.

2. Train models from scratch :runner:

To launch a training from scratch, run:

cuda=gpu_id config=filename.yml tag=run_tag ./scripts/pipeline.sh

where gpu_id is a device id, filename.yml is a config in configs folder, run_tag is a tag for the experiment.

Results are saved at runs/${DATASET}/${DATE}_${run_tag} where DATASET is the dataset name specified in filename.yml and DATE is the current date in mmdd format.

3. Reproduce our quantitative results :bar_chart:

A model is evaluated at the end of training. To evaluate a pretrained model (e.g. sn_big_airplane.pkl):

1. move the model to a fake folder and rename it model.pkl (e.g. in runs/shapenet_nmr/airplane_big)
2. point to the fake tag to resume from in the config (e.g. resume: airplane_big in airplane.yml)
3. launch the training (and thus evaluation) with:

cuda=gpu_id config=sn_big/airplane.yml tag=airplane_big_eval ./scripts/pipeline.sh

For CUB, the built-in evaluation included in the training pipeline is Mask-IoU. To evaluate PCK, run:

cuda=gpu_id tag=run_tag ./scripts/kp_eval.sh

4. Train on a custom dataset :crystal_ball:

If you want to learn a model for a custom object category, here are the key things you need to do:

1. put your images in a custom_name folder inside the datasets folder
2. edit the config file custom.yml (or custom_big.yml) in the configs folder: this includes changing the dataset name to custom_name and setting all training milestones
3. launch training with:

cuda=gpu_id config=custom.yml tag=custom_run_tag ./scripts/pipeline.sh

Further information :books:

If you like this project, check out related works from our group:

• Loiseau et al. - Representing Shape Collections with Alignment-Aware Linear Models (3DV 2021)
• Monnier et al. - Unsupervised Layered Image Decomposition into Object Prototypes (ICCV 2021)
• Monnier et al. - Deep Transformation Invariant Clustering (NeurIPS 2020)
• Deprelle et al. - Learning elementary structures for 3D shape generation and matching (NeurIPS 2019)
• Groueix et al. - AtlasNet: A Papier-Mache Approach to Learning 3D Surface Generation (CVPR 2018)