PyTorch implementation of Emotic CNN methodology to recognize emotions in images using context information.
Humans use their facial features or expressions to convey how they feel, such as a person may smile when happy and scowl when angry. Historically, computer vision research has focussed on analyzing and learning these facial features to recognize emotions. However, these facial features are not universal and vary extensively across cultures and situations.
A scene context, as shown in the figure above, can provide additional information about the situations. This project explores the use of context in recognizing emotions in images.
The project uses the EMOTIC dataset and follows the methodology as introduced in the paper 'Context based emotion recognition using EMOTIC dataset'.
Two feature extraction modules first extract features over an image. These features are then used by a third module to predict the continuous dimensions (valence, arousal and dominance) and the discrete emotion categories.
The Emotic dataset can be used only for non-commercial research and education purposes. Please, fill out the following form to request access to the dataset and the corresponding annotations.
Download the Emotic dataset & annotations, and prepare the directory following the below structure:
├── ...
│ ├── emotic
│ | ├── ade20k
│ | ├── emodb_small
│ | ├── framesdb
│ | ├── mscoco
│ ├── Annotations
│ | ├── Annotations.mat
> python mat2py.py --data_dir proj/data/emotic19 --generate_npy
> python main.py --mode train --data_path proj/data/emotic_pre --experiment_path proj/debug_exp
> python main.py --mode test --data_path proj/data/emotic_pre --experiment_path proj/debug_exp
> python main.py --mode inference --inference_file proj/debug_exp/inference_file.txt --experiment_path proj/debug_exp
You can also train and test models on Emotic dataset by using the Colab_train_emotic notebook.
The trained models and thresholds to use for inference purposes are availble here.
Ronak Kosti, Jose Alvarez, Adria Recasens, Agata Lapedriza
[Paper] [Project Webpage] [Authors' Implementation]
@article{kosti2020context,
title={Context based emotion recognition using emotic dataset},
author={Kosti, Ronak and Alvarez, Jose M and Recasens, Adria and Lapedriza, Agata},
journal={arXiv preprint arXiv:2003.13401},
year={2020}
}