CVPR 2024: Language Guided Generation of 3D Embodied AI Environments.
Holodeck is based on AI2-THOR, and we currently support macOS 10.9+ or Ubuntu 14.04+.
New Feature: To add ANY new assets to AI2-THOR, please check the objathor repo!
Note: To yield better layouts, use DFS
as the solver. If you pull the repo before 12/28/2023
, you must set the argument --use_milp
to False
to use DFS
.
After cloning the repo, you can install the required dependencies using the following commands:
conda create --name holodeck python=3.9.16
conda activate holodeck
pip install -r requirements.txt
pip install --extra-index-url https://ai2thor-pypi.allenai.org ai2thor==0+6f165fdaf3cf2d03728f931f39261d14a67414d0
Download the data from google drive and extract it to the data/
folder, or use the following command to download from S3:
wget https://holodeck-ai2.s3.amazonaws.com/data.zip
unzip data.zip
You can use the following command to generate a new environment.
python main.py --query "a living room" --openai_api_key <OPENAI_API_KEY>
To be noticed, our system uses gpt-4-1106-preview
, so please ensure you have access to it.
Note: To yield better layouts, use DFS
as the solver. If you pull the repo before 12/28/2023
, you must set the argument --use_milp
to False
to use DFS
.
2020.3.25f1
.git clone https://github.com/allenai/ai2thor.git
git checkout 6f165fdaf3cf2d03728f931f39261d14a67414d0
pip uninstall Werkzeug
pip uninstall Flask
pip install Werkzeug==2.0.1
pip install Flask==2.0.1
ai2thor/unity
as project in Unity and open ai2thor/unity/Assets/Scenes/Procedural/Procedural.unity
.python connect_to_unity --scene <SCENE_JSON_FILE_PATH>
Please cite the following paper if you use this code in your work.
@article{yang2023holodeck,
title={Holodeck: Language Guided Generation of 3D Embodied AI Environments},
author={Yue Yang and Fan-Yun Sun and Luca Weihs and Eli VanderBilt and Alvaro Herrasti and Winson Han and Jiajun Wu and Nick Haber and Ranjay Krishna and Lingjie Liu and Chris Callison-Burch and Mark Yatskar and Aniruddha Kembhavi and Christopher Clark},
journal={arXiv preprint arXiv:2312.09067},
year={2023}
}