Interactive deep learning book with multi-framework code, math, and discussions. Adopted at 500 universities from 70 countries including Stanford, MIT, Harvard, and Cambridge.
Dive into Deep Learning is now v1.0!
The Vol.1 by Cambridge University Press can be ordered on Amazon.
Thanks to all the readers and contributors in the past years!
D2L has gone 1.0.0-beta0! We thank all the 296 contributors for making this happen!
Chapter 1--11 will be forthcoming on Cambridge University Press (early 2023).
We added new JAX implementation. Get started with import jax
at https://d2l.ai/chapter_preliminaries/ndarray.html
Thank @AnirudhDagar!
With the advent of ChatGPT (sibling model to InstructGPT fine-tuned using reinforcement learning), you may get curious about how to enable ML to take decisions sequentially:
17. Reinforcement Learning 17.1. Markov Decision Process (MDP) 17.2. Value Iteration 17.3. Q-Learning
Thank Pratik Chaudhari (University of Pennsylvania and Amazon), Rasool Fakoor @rasoolfa (Amazon), and Kavosh Asadi (Amazon)!
“Everything is a special case of a Gaussian process.” Gaussian processes and deep neural networks are highly complementary and can be combined to great effect:
18. Gaussian Processes 18.1. Introduction to Gaussian Processes 18.2. Gaussian Process Priors 18.3. Gaussian Process Inference
Thank Andrew Gordon Wilson @andrewgordonwilson (New York University and Amazon)!
Tired of setting hyperparameters in a trial-and-error manner? You may wish to check out the systematic hyperparameter optimization approach:
19. Hyperparameter Optimization 19.1. What Is Hyperparameter Optimization? 19.2. Hyperparameter Optimization API 19.3. Asynchronous Random Search 19.4. Multi-Fidelity Hyperparameter Optimization 19.5. Asynchronous Successive Halving
Thank Aaron Klein @aaronkl (Amazon), Matthias Seeger @mseeger (Amazon), and Cedric Archambeau (Amazon)!
Thank @finale80 @JojiJoseph @gab-chen @Excelsior7 @shanmo @kxxt @vBarbaros @gui-miotto @bolded @atishaygarg @tuelwer @gopalakrishna-r @qingfengtommy @Mohamad-Jaallouk @biswajitsahoo1111 @315930399 for improving this book!
Add PaddlePaddle implementation for the d2l
library (compatible with classic.d2l.ai).
See the PR in d2l-zh.
We are happy to release D2L 1.0.0-alpha1.post0 We thank all the contributors who have made this open-source textbook better for everyone.
This minor release includes the following updates:
It also comes with the following bug-fixes:
We are excited to announce the release of D2L 1.0.0-alpha0! We thank all the 265 contributors who have made this open-source textbook better for everyone.
We have added the following new topics, with discussions of more recent methods such as ResNeXt, RegNet, ConvNeXt, Vision Transformer, Swin Transformer, T5, GPT-1/2/3, zero-shot, one-shot, few-shot, Gato, Imagen, Minerva, and Parti.
Besides new topics, we have significantly revised all the topics up to transformers. For example, the previous Linear Neural Networks and Multilayer Perceptrons chapters have been revamped as new chapters of Linear Neural Networks for Regression, Linear Neural Networks for Classification, and Multilayer Perceptrons.
Throughout the book we repeatedly walk through various components including the data, the model, the loss function, and the optimization algorithm. Treating components in deep learning as objects, we can define classes for these objects and their interactions. This object-oriented design for implementation will greatly streamline the presentation. Therefore, inspired by open-source libraries such as PyTorch Lightning, we have re-designed the API with three core classes:
Module
contains models, losses, and optimization methods;DataModule
provides data loaders for training and validation;Trainer
class, which allows us to train models on a variety of hardware platforms.For example, with the classic API in previous releases:
model = # Multilayer perceptron definition
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=256)
loss = nn.CrossEntropyLoss(reduction='none')
trainer = torch.optim.SGD(net.parameters(), lr=0.1)
d2l.train_ch3(model, train_iter, test_iter, loss, num_epochs=10, trainer)
With the new API:
model = # Multilayer perceptron definition
data = d2l.FashionMNIST(batch_size=256)
trainer = d2l.Trainer(max_epochs=10)
trainer.fit(model, data)
Since v1.8.0, PyTorch offers "lazy" layers where input shape specification is no longer required. For simplicity we will use "lazy" layers whenever we can, such as:
class LinearRegression(d2l.Module):
def __init__(self, lr):
super().__init__()
self.save_hyperparameters()
self.net = nn.LazyLinear(1) # Lazy layer with output dimension only
self.net.weight.data.normal_(0, 0.01)
self.net.bias.data.fill_(0)
Join us to improve ongoing translations in:
This release fixes issues when installing d2l package and running d2l notebooks on Google Colab with Python 3.7 and updates PyTorch & TensorFlow to their respective latest versions.
More concretely, this release includes the following upgrades/fixes:
reduction=none
in PyTorch loss for train_epoch_ch3
(#2007)test_feature
->test_features
of train_and_pred
in kaggle house price section (#1982)This release supports running the book with SageMaker Studio Lab for free and introduces several fixes:
Dive into Deep Learning is now available on arxiv!
We have added TensorFlow implementations up to Chapter 11 (Optimization Algorithms).
The following chapters have been significantly improved for v1.0:
Finalized chapters are being translated into Chinese (d2l-zh v2)
We have added the brand-new Chapter: Attention Mechanisms:
Attention Cues
Attention Pooling: Nadaraya-Watson Kernel Regression
Attention Scoring Functions
Bahdanau Attention
Multi-Head Attention
Self-Attention and Positional Encoding
Transformer
We have completed PyTorch implementations for Vol.1 (Chapter 1--15).
The following chapters have been significantly improved for v1.0:
The following chapters have been translated into Chinese (d2l-zh v2 Git repo, Web preview):
The community are translating the book into Turkish (d2l-tr Git repo, Web preview). The first draft of Chapter 1--7 is complete.
We have added PyTorch implementations up to Chapter 11 (Optimization Algorithms). Chapter 1--7 and Chapter 11 have also been adapted to TensorFlow.
The following chapters have been significantly improved for v1.0:
Finalized chapters are being translated into Chinese (d2l-zh v2)
d2l.Vocab
API