LightSeq: A High Performance Library for Sequence Processing and Generation
In the hip_dev branch, LightSeq supports CUDA backend and HIP backend（now support training only）. LightSeq transformer has a speedup about 7% comparing with FairsSeq transformer under the HIP backend. LightSeq HIP supports multiple NLP models, such as transformer, bert, gpt, etc. Users need no modification with python training. More information about the LightSeq HIP can be found here https://github.com/bytedance/lightseq/blob/hip_dev/README_HIP.md
Full Changelog: https://github.com/bytedance/lightseq/compare/v3.0.0...v3.0.1
It's been a long time since our last release (v2.2.0). For the past one year, we have focused on int8 quantization.
In this release, LightSeq supports int8 quantized training and inference. Compared with PyTorch QAT, LightSeq int8 training has a speedup of 3x without any performance loss. Compared with previous LightSeq fp16 inference, int8 engine has a speedup up to 1.7x.
LightSeq int8 engine supports multiple models, such as Transformer, BERT, GPT, etc. For int8 training, the users only need to apply quantization mode to the model using model.apply(enable_quant). For int8 inference, the users only need to use QuantTransformer instead of fp16 Transformer.
Other releases include supporting models like MoE, fix bugs, performance improvement, etc.
This version contains several features and bug fixes.
relax restriction of layer norm hidden size #137 #161 support inference during training for transformer #141 #146 #147
Add inference support and examples for BERT #145
fix save/load for training with pytorch #139 fix pos embedding index bug #144
[inference] fix warp reduce bug in inference. #74
Merge codes about training and inference. Reorganize docs and README.
It's been a long time since our last release (v1.2.0). For the past six months, we have focused on training efficiency.
In this release, LightSeq supports fast training for models in the Transformer family!
We provide highly optimized custom operators for PyTorch and TensorFlow, which cover the entire training process for Transformer-based models. Users of LightSeq can use these operators to build their own models with efficient computation.
In addition, we integrate our custom operators into popular training libraries like Fairseq, Hugging Face, NeurST, which enables a 1.5X-3X end-to-end speedup campred to the native version.
With only a small amount of code, you can enjoy the excellent performance provided by LightSeq. Try it now!
lightseq-train to accelerate fairseq training, including optimized transformer model, adam, and label smoothed losslightseq.inference
This release has API change for inference, all inference API has moved to lightseq.inference. For example, use import lightseq.inference and model = lightseq.inference.Transformer("$PB_PATH", max_batch_size)
Support Python API and multilingual nmt