Highway Transformer: Self-Gating Enhanced Self-Attentive Networks

This repo is the demo code of Transformer-XL using Self-Dependency Unit. This work is closedly related to Gating-enhanced Transformer variants, such as Google's Switch Transformers.

Yekun Chai et. al., Highway Transformer: Self-Gating Enhanced Self-Attentive Networks (ACL 2020)

Requirements

• PyTorch >= 1.1.0
• TensorboardX >= 1.8
• Tensorboard >= 1.14
• 4 GPUs of each 8GB memory for running 12 layer Transformer-XL

Data download

bash getdata.sh

Run 6-layer Transformer-XL

cd pytorch/xl_L6_scripts && bash <script-name>.sh train --work_dir "PATH_TO_WORK_DIR"

Visualizing Your Result

cd XL-L6-results && tensorboard --logdir=.

Results

• Line plots of different model settings, where the topmost line (in red) is the baseline model (i.e., original Transformer-XL).
• After adding Self-Dependency Unit (see bottom two curves), it is clear that Highway Transformer speeds up the convergence process during training and evaluation.

training bpc	training loss

eval bpc	eval loss

Citation

For attribution in academic contexts, please cite this work as:

@inproceedings{chai-etal-2020-highway,
    title = "Highway Transformer: Self-Gating Enhanced Self-Attentive Networks",
    author = "Chai, Yekun  and
      Jin, Shuo  and
      Hou, Xinwen",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2020",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-main.616",
    pages = "6887--6900"
}