Python toolkit for Visual Speech Recognition
Python toolkit for Visual Speech Recognition
pyVSR is a Python toolkit aimed at running Visual Speech Recognition (VSR) experiments in a traditional framework (e.g. handcrafted visual features, Hidden Markov Models for pattern recognition).
The main goal of pyVSR is to easily reproduce VSR experiments in order to have a baseline result on most publicly available audio-visual datasets.
currently supported:
TCD-TIMIT
OuluVS2
Discrete Cosine Transform (DCT)
Active Appearance Models (AAM)
Point cloud of facial landmarks
Please refer to the attached examples.
pyVSR was re-designed to simplify its usage on multiple datasets.
Users can provide their own dictionaries of (input, output) pairs for all of pyVSR's functionalities.
The recommended way is to create an empty conda
environment and install the following dependencies:
Alternatively, you can use the environment.yml
file:
It is the user's responsibility to compile OpenFace
and HTK
.
Please refer to the documentation upstream:
OpenFace
HTK 3.5
Add the HTK binaries to the system path (e.g. /usr/local/bin/
) or to ./pyVSR/bins/htk/
Add the OpenFace binaries to ./pyVSR/bins/openface/
pyVSR was initially developed on a system running Manjaro Linux, frequently updated from the testing
repositories.
We also succesfully tested it on Windows systems.
If you are not interested in using the AAM module, you can skip installing a great amount of Python packages. We recommend running the example scripts and installing the missing dependencies (opencv, dlib, numpy).
If you use this work, please cite it as:
George Sterpu and Naomi Harte. Towards lipreading sentences using active appearance models. In AVSP, Stockholm, Sweden, August 2017.
We are always happy to hear from you:
George Sterpu sterpug [at] tcd.ie
Naomi Harte nharte [at] tcd.ie