An open-source impl. of Large Reconstruction Models
openlrm-mix-base-1.1
model by default. Please refer to the model card for details on the updated model architecture and training settings.git clone https://github.com/3DTopia/OpenLRM.git
cd OpenLRM
pip install -r requirements.txt
Model | Training Data | Layers | Feat. Dim | Trip. Dim. | In. Res. | Link |
---|---|---|---|---|---|---|
openlrm-obj-small-1.1 | Objaverse | 12 | 512 | 32 | 224 | HF |
openlrm-obj-base-1.1 | Objaverse | 12 | 768 | 48 | 336 | HF |
openlrm-obj-large-1.1 | Objaverse | 16 | 1024 | 80 | 448 | HF |
openlrm-mix-small-1.1 | Objaverse + MVImgNet | 12 | 512 | 32 | 224 | HF |
openlrm-mix-base-1.1 | Objaverse + MVImgNet | 12 | 768 | 48 | 336 | HF |
openlrm-mix-large-1.1 | Objaverse + MVImgNet | 16 | 1024 | 80 | 448 | HF |
Model cards with additional details can be found in model_card.md.
assets/sample_input
, and you can quickly try them.Run the inference script to get 3D assets.
You may specify which form of output to generate by setting the flags EXPORT_VIDEO=true
and EXPORT_MESH=true
.
Please set default INFER_CONFIG
according to the model you want to use. E.g., infer-b.yaml
for base models and infer-s.yaml
for small models.
An example usage is as follows:
# Example usage
EXPORT_VIDEO=true
EXPORT_MESH=true
INFER_CONFIG="./configs/infer-b.yaml"
MODEL_NAME="zxhezexin/openlrm-mix-base-1.1"
IMAGE_INPUT="./assets/sample_input/owl.png"
python -m openlrm.launch infer.lrm --infer $INFER_CONFIG model_name=$MODEL_NAME image_input=$IMAGE_INPUT export_video=$EXPORT_VIDEO export_mesh=$EXPORT_MESH
>=2.1
. Code is developed and tested under PyTorch 2.1.2
.frame_size
in the inference configs.UserWarning: xFormers is available
if xFormers
is actually working.configs/accelerate-train.yaml
, which defaults to use 8 GPUs with bf16
mixed precision.A sample training config file is provided under configs/train-sample.yaml
.
Please replace data related paths in the config file with your own paths and customize the training settings.
An example training usage is as follows:
# Example usage
ACC_CONFIG="./configs/accelerate-train.yaml"
TRAIN_CONFIG="./configs/train-sample.yaml"
accelerate launch --config_file $ACC_CONFIG -m openlrm.launch train.lrm --config $TRAIN_CONFIG
The inference pipeline is compatible with huggingface utilities for better convenience.
You need to convert the training checkpoint to inference models by running the following script.
python scripts/convert_hf.py --config <YOUR_EXACT_TRAINING_CONFIG> convert.global_step=null
The converted model will be saved under exps/releases
by default and can be used for inference following the inference guide.
If you find this work useful for your research, please consider citing:
@article{hong2023lrm,
title={Lrm: Large reconstruction model for single image to 3d},
author={Hong, Yicong and Zhang, Kai and Gu, Jiuxiang and Bi, Sai and Zhou, Yang and Liu, Difan and Liu, Feng and Sunkavalli, Kalyan and Bui, Trung and Tan, Hao},
journal={arXiv preprint arXiv:2311.04400},
year={2023}
}
@misc{openlrm,
title = {OpenLRM: Open-Source Large Reconstruction Models},
author = {Zexin He and Tengfei Wang},
year = {2023},
howpublished = {\url{https://github.com/3DTopia/OpenLRM}},
}