Sru Versions Save

Postponed the CUDA initialization to the instantiation of SRUCells in order to ensure that the process where it takes place is where we plan to run our model.

More layer norm options; More info in _repr_()

• Support GPU/CUDA inference in Torchscript model
• Support post layer norm
• Support custom init value for weight_c
• Add unit tests for GPU inference
• Add unit tests for backward()
• Add more unit tests for Torchscript

• Support GPU/CUDA inference in torchscript model
• Support post layer norm
• Support custom init value for weight_c

Dev1:

• Support GPU/CUDA inference in torchscript model
• Support post layer norm
• Support custom init value for weight_c Dev2:
• Fix an issue

• Support GPU/CUDA inference in torchscript model
• Support post layer norm
• Support custom init value for weight_c

Fix a typo. Add an option to only use attention_last_n_layers. Replace option normalize_after with normalization_type

Changes:

• change weight_c_init from Optional[float] = None to float = 1.0

Bug fixes:

• fix a potential memory leak in custom op
• fix bug in cuda maskpad
• torchscript compatible in torch 1.5.1 now

Note that future 3.0.0 release, and future 3.0.0 dev releases might not be backwards compatible with this dev release.

Key features / changes:

• #160: SRU++ is now available. Unit tests are included for torchscript compatibility and correctness. Example language model training code is available.
• #166: fp16 training improvement. The recurrence kernel will run in float16 now when amp is enabled. This gives an additional ~10% speedup on tested language model training, ~20% reduction on GPU memory usage and no regression on final results.
• #167: Code clean-up. No autocast block needed in sru.ops.elementwise_recurrence_gpu. This would allow both Native AMP and APEX AMP to work. (Credit: @visionscaper)

Other changes:

• Fix an dtype error within adaptive embedding (#168)
• Significant speed-up on BILLONWORD training (#169)
• LICENCE update requested by IPC (#165)

Note that future release and dev releases of v3 might be backwards incompatible with this dev release.

This dev release:

• custom_m renamed to transform_module
• transform_module always used now (the weight and weight_proj parameters have been removed)
• projection_size can take in a sequence of projection_sizes, one per layer
• n_proj in SRUCell renamed to projection_size, for consistency