A GPipe implementation in PyTorch
Released on September 18, 2020.
Changed the license to BSD-3-Clause.
Released on July 29, 2020.
Released on November 29, 2019.
@skippable
for efficient skip connections. With this interface, GPipe
copies skip tensors directly to the destination device.
balance_by_size()
analyzes parameters as well.torchgpipe_balancing
module to torchgpipe.balance
.balance_by_time()
and balance_by_size()
.Released on October 8, 2019.
Released on September 30, 2019.
torchgpipe now overlaps copy and computation using the separate CUDA streams. Previously, GPU could not compute a partition while copying micro-batches across different GPUs because they all happened on the same default CUDA stream.
balance_by_size()
.GPipe.devices
from tuple
to list
.current_microbatch()
. This approach turned out to be incompatible with checkpointing.Released on June 26, 2019.
current_microbatch()
and is_recomputing()
.Released on May 14, 2019 to evaluate usability and efficiency internally.
Provided a functional GPipe implementation, including pipeline parallelism, checkpointing, and deferred batch normalization.
Supported Python 3.6+ and PyTorch 1.0.