Dataflowr: Deep Learning DIY

Code and notebooks for the deep learning course dataflowr. Here is the schedule followed at école polytechnique in 2023:

:sunflower:Session:one: Finetuning VGG

• Module 1 - Introduction & General Overview Slides + notebook Dogs and Cats with VGG + Practicals (more dogs and cats)

:sunflower:Session:two: PyTorch tensors and Autodiff

• Module 2a - PyTorch tensors
• Module 2b - Automatic differentiation + Practicals
• MLP from scratch start of HW1
• another look at autodiff with dual numbers and Julia

:sunflower:Session:three:

• Module 3 - Loss function for classification
• Module 4 - Optimization for deep learning
• Module 5 - Stacking layers and overfitting a MLP on CIFAR10: Stacking_layers_MLP_CIFAR10.ipynb
• Module 6: Convolutional neural network
• how to regularize with dropout and uncertainty estimation with MC Dropout: Module 15 - Dropout

:sunflower:Session:four:

• Module 7 - Dataloading
• Module 8a - Embedding layers
• Module 8b - Collaborative filtering and build your own recommender system: 08_collaborative_filtering_empty.ipynb (on a larger dataset 08_collaborative_filtering_1M.ipynb)
• Module 8c - Word2vec and build your own word embedding 08_Word2vec_pytorch_empty.ipynb
• Module 16 - Batchnorm and check your understanding with 16_simple_batchnorm_eval.ipynb and more 16_batchnorm_simple.ipynb
• Module 17 - Resnets
• start of Homework 2: Class Activation Map and adversarial examples

:sunflower:Session:five:

• Module 9a: Autoencoders and code your noisy autoencoder 09_AE_NoisyAE.ipynb
• Module 10: Generative Adversarial Networks and code your GAN, Conditional GAN and InfoGAN 10_GAN_double_moon.ipynb
• Module 13: Siamese Networks and Representation Learning
• start of Homework 3: VAE for MNIST clustering and generation

:sunflower:Session:six:

• Module 11a - Recurrent Neural Networks theory
• Module 11b - Recurrent Neural Networks practice and predict engine failure with 11_predictions_RNN_empty.ipynb
• Module 11c - Batches with sequences in Pytorch

:sunflower:Session:seven:

• Module 12 - Attention and Transformers
• Correcting the PyTorch tutorial on attention in seq2seq: 12_seq2seq_attention.ipynb
• Build your own microGPT: GPT_hist.ipynb

:sunflower:Session:eight:

• Module 9b - UNets
• Module 9c - Flows
• Build your own Real NVP: Normalizing_flows_empty.ipynb

:sunflower:Session:nine:

• Module 18a - Denoising Diffusion Probabilistic Models
• Train your own DDPM on MNIST: ddpm_nano_empty.ipynb
• Finetuning on CIFAR10: ddpm_micro_sol.ipynb

For more updates:

:sunflower: All notebooks

• Module 1: Introduction & General Overview

  • Intro: finetuning VGG for dogs vs cats 01_intro.ipynb
  • Practical: Using CNN for more dogs and cats 01_practical_empty.ipynb and its solution 01_practical_sol.ipynb

• Module 2: Pytorch tensors and automatic differentiation

  • Basics on PyTorch tensors and automatic differentiation 02a_basics.ipynb
  • Linear regression from numpy to pytorch 02b_linear_reg.ipynb
  • Practical: implementing backprop from scratch 02_backprop.ipynb and its solution 02_backprop_sol.ipynb
  • Bonus: intro to JAX: autodiff the functional way autodiff_functional_empty.ipynb and its solution autodiff_functional_sol.ipynb
  • Bonus: Linear regression in JAX linear_regression_jax.ipynb
  • Bonus: automatic differentiation with dual numbers AD_with_dual_numbers_Julia.ipynb

• Homework 1: MLP from scratch

  • hw1_mlp.ipynb and its solution hw1_mlp_sol.ipynb

• Module 3: Loss functions for classification

  • An explanation of underfitting and overfitting with polynomial regression 03_polynomial_regression.ipynb

• Module 4: Optimization for deep leaning

  • Practical: code Adagrad, RMSProp, Adam, AMSGrad 04_gradient_descent_optimization_algorithms_empty.ipynb and its solution 04_gradient_descent_optimization_algorithms_sol.ipynb

• Module 5: Stacking layers

  • Practical: overfitting a MLP on CIFAR10 Stacking_layers_MLP_CIFAR10.ipynb and its solution MLP_CIFAR10.ipynb

• Module 6: Convolutional neural network

  • Practical: build a simple digit recognizer with CNN 06_convolution_digit_recognizer.ipynb

• Homework 2: Class Activation Map and adversarial examples

  • HW2_CAM_Adversarial.ipynb

• Module 8: Embedding layers, Collaborative filtering and Word2vec

  • Practical: Collaborative filtering with Movielens 100k dataset 08_collaborative_filtering_empty.ipynb
  • Practical: Refactoring code, collaborative filtering with Movielens 1M dataset 08_collaborative_filtering_1M.ipynb
  • Practical: Word Embedding (word2vec) in PyTorch 08_Word2vec_pytorch_empty.ipynb
  • Finding Synonyms and Analogies with Glove 08_Playing_with_word_embedding.ipynb

• Module 9a: Autoencoders

  • Practical: denoising autoencoder (with convolutions and transposed convolutions) 09_AE_NoisyAE.ipynb

• Module 9b - UNets

  • UNet for image segmentation UNet_image_seg.ipynb

• Module 9c - Flows

  • implementing Real NVP Normalizing_flows_empty.ipynb and its solution Normalizing_flows_sol.ipynb

• Module 10 - Generative Adversarial Networks

  • Conditional GAN and InfoGAN 10_GAN_double_moon.ipynb

• Module 11 - Recurrent Neural Networks and Batches with sequences in Pytorch

  • notebook used in the theory course: 11_RNN.ipynb
  • predicting engine failure with RNN 11_predictions_RNN_empty.ipynb

• Module 12 - Attention and Transformers

  • Correcting the PyTorch tutorial on attention in seq2seq: 12_seq2seq_attention.ipynb and its solution
  • building a simple transformer block and thinking like transformers: GPT_hist.ipynb and its solution

• Module 13 - Siamese Networks and Representation Learning

  • learning embeddings with contrastive loss: 13_siamese_triplet_mnist_empty.ipynb

• Module 15 - Dropout

  • Dropout on a toy dataset: 15a_dropout_intro.ipynb
  • playing with dropout on MNIST: 15b_dropout_mnist.ipynb

• Module 16 - Batchnorm

  • impact of batchnorm: 16_batchnorm_simple.ipynb
  • Playing with batchnorm without any training: 16_simple_batchnorm_eval.ipynb

• Module 18a - Denoising Diffusion Probabilistic Models

  • Denoising Diffusion Probabilistic Models for MNIST: ddpm_nano_empty.ipynb and its solution ddpm_nano_sol.ipynb
  • Denoising Diffusion Probabilistic Models for CIFAR10: ddpm_micro_sol.ipynb

• Module - Deep Learning on graphs

  • Inductive bias in GCN: a spectral perspective GCN_inductivebias_spectral.ipynb and for colab GCN_inductivebias_spectral-colab.ipynb
  • Graph ConvNets in PyTorch spectral_gnn.ipynb

• NERF

  • PyTorch Tiny NERF tiny_nerf_extended.ipynb

Usage

If you want to run locally, follow the instructions of Module 0 - Running the notebooks locally

2020 version of the course

Archives are available on the archive-2020 branch.