The official code for our paper "A Framework for Integrating Gesture Generation Models into Interactive Conversational Agents", published as a demonstration at AAMAS 2021.
This repo contains the official implementation for the paper A Framework for Integrating Gesture Generation Models into Interactive Conversational Agents, published as a demonstration at the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS),
by Rajmund Nagy, Taras Kucherenko, Birger Moëll, André Pereira, Hedvig Kjellström and Ulysses Bernardet.
We present a framework for integrating recent data-driven gesture generation models into interactive conversational agents in Unity. Our video demonstration is available below:
This branch contains the Blenderbot version of our implementation with a built-in chatbot and TTS. You may visit the dialogflow_demo branch for an alternative version that integrates DialogFlow to the project for speech generation.
Please follow the instructions in INSTALLATION.md to install and run the project.
Our framework is designed to be fully modular therefore it can be applied to different voices, chatbot backends, gesture generation models and 3D characters. However, using it in a new project will require some coding for which we provide guidance below.
The source code of the Unity scene with DialogFlow integration is available on this link, while the Blenderbot version is available here. The relevant C# scripts are found in the Assets/Scripts/
folder. The entry point of the python code is the main.py
file, while the bulk of the implementation is found in gesture_generator_service.py
.
ActiveMQClient.cs
and the messaging_server.py
files.MotionVisualizer.cs
file animates its model by modifying the localRotation
values of each joint.
DialogFlowCommunicator.cs
.blenderbot.py
and tts_interface.py
for details.GesturePredictor
class shows how we implemented that for Gesticulator.The authors would like to thank Lewis King for sharing the source code of his JimBot project with us.
If you use this code in your research, then please cite it:
@inproceedings{Nagy2021gesturebot,
author = {Nagy, Rajmund and Kucherenko, Taras and Moell, Birger and Pereira, Andr\'{e} and Kjellstr\"{o}m, Hedvig and Bernardet, Ulysses},
title = {A Framework for Integrating Gesture Generation Models into Interactive Conversational Agents},
year = {2021},
isbn = {9781450383073},
publisher = {International Foundation for Autonomous Agents and Multiagent Systems},
address = {Richland, SC},
booktitle = {Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems},
location = {Virtual Event, United Kingdom},
series = {AAMAS '21}
}