Deep Learning for Audio (DLA)

• Lecture and seminar materials for each week are in ./week* folders, see README.md for materials and instructions
• Any technical issues, ideas, bugs in course materials, contribution ideas - add an issue
• The current version of the course is conducted in autumn 2023 at the CS Faculty of HSE

Syllabus

• week01 Introduction to Course

  • Lecture: Introduction to Course
  • Seminar: Intro in pytorch

• week02 Introduction to Digital Signal Processing

  • Lecture: Signals, Fourier Transform, spectrograms, MelScale, MFCC
  • Seminar: DSP in practice, spectrogram creation, training a model for audio MNIST

• week03 Speech Recognition I

  • Lecture: Metrics, datasets, Connectionist Temporal Classification (CTC), Listen Attend and Spell (LAS), Beam Search
  • Seminar: Audio Augmentations, Beam Search, Homework discussion

• week04 Speech Recognition II

  • Lecture: RNN-T, language model fusion, Byte-Pair Encoding (BPE)
  • Seminar: --

• week05 Source Separation I

  • Lecture: A review of general Source Separation and Denoising, Encoder-Decoder-Separator architectures, Demucs family, DCCRN, FullSubNet+
  • Seminar: Metrics, Dataset of Mixtures and some tech stuff

• week06 Source Separation II

  • Lecture: Speech separation, Blind and Target Separation, Recurrent(TasNet, DPRNN, VoiceFilter) and CNN(ConvTasNet, SpEx+)
  • Seminar: WienerFilter, SincFilter and DEMUCS

• week07 Text to Speech (TTS)

  • Lecture: Tacotron, DeepVoice, GST, FastSpeech, AdaSpeech, Attention Tricks
  • Seminar: FastSpeech I

• week08 Neural Vocoders

  • Lecture: WaveNet, Parallel WaveGAN, WaveGlow, MelGAN, HiFiGAN
  • Seminar: WaveNet

• week09 Voice Conversion

  • Lecture: Disentanglement & Direct based methods
  • Seminar: TorchScript, HiFi-VC

• week10 Voice Biometry I

  • Lecture: Introduction. CMs for sythesized speech detection (LCNN, RawNet2, AASIST). GNNs
  • Seminar: ASVspoof, Sinc-layer, GNN

• week11 Voice Biometry II

  • Lecture: CMs for replay attack detection. ASV systems. SASV systems. Streaming
  • Seminar: -

• week12 Diffusion Models for Audio Generation

  • Lecture, part 1: Introduction to diffusion models from two perspectives: score matching and latent probabilistic models.
  • Lecture, part2: Diffusion models for audio synthesis and tts. WaveGrad, DiffWave, GradTTS

• bonus week Guest lecture

  • Self-Supervised models in ASR

Homeworks

• ASR Training speech recognition model
• SS Training speech separation model
• TTS Implementation of TTS model (Part 1, FastSpeech)
• NV Implementation of TTS model (Part 2, Vocoder)
• AS Implementation of Anti-spoofing Model

Resources

• Lecture recordings on YouTube (in russian)

Contributors & course staff

Course materials and teaching (in different years) were delivered by:

• Maxim Kaledin
• Alexander Markovich
• Petr Grinberg
• Aibek Alanov
• Grigory Fedorov
• Ilya Lewin
• Daniil Ivanov(previously)
• Timofey Smirnov(previously)
• Alexander Mamaev(previously)