An Updating Survey for Bayesian Deep Learning (BDL)

This is an updating survey for Bayesian Deep Learning (BDL), an constantly updated and extended version for the manuscript, 'A Survey on Bayesian Deep Learning', published in ACM Computing Surveys 2020.

Bayesian deep learning is a powerful framework for designing models across a wide range of applications. See our Nature Medicine paper for a possible application on healthcare.

Contents

• Survey
• BDL and Recommender Systems
• BDL and Domain Adaptation (and Domain Generalization, Meta Learning, etc.)
• BDL and Healthcare
• BDL and Natural Language Processing (NLP)
• BDL and Computer Vision (CV)
• BDL and Control/Planning
• BDL and Graphs (Link Prediction, Graph Neural Networks, Knowledge Graphs, etc.)
• BDL and Topic Modeling
• BDL and Speech Recognition/Synthesis
• BDL and Forecasting (Time Series Analysis)
• BDL and Distributed/Federated Learning
• BDL and Continual/Life-Long Learning
• BDL as a Framework (Miscellaneous)
• Bayesian/Probabilistic Neural Networks as Building Blocks of BDL

Survey

A Survey on Bayesian Deep Learning
by Wang et al., ACM Computing Surveys (CSUR) 2020
[PDF] [Blog] [BDL Framework in 2016]

BDL and Recommender Systems

Collaborative Deep Learning for Recommender Systems
by Wang et al., KDD 2015
[PDF] [Project Page] [2014 Arxiv Version] [Code] [MXNet Code] [TensorFlow Code] [Dataset A] [Dataset B] [Jupyter Notebook] [Slides] [Slides (Long)]

Collaborative Recurrent Autoencoder: Recommend while Learning to Fill in the Blanks
by Wang et al., NIPS 2016
[PDF]

Collaborative Knowledge Base Embedding for Recommender Systems
by Zhang et al., KDD 2016
[PDF]

Collaborative Deep Ranking: A Hybrid Pair-Wise Recommendation Algorithm with Implicit Feedback
by Ying et al., PAKDD 2016
[PDF]

Collaborative Variational Autoencoder for Recommender Systems
by Li et al., KDD 2017
[PDF]

Variational Autoencoders for Collaborative Filtering
by Liang et al., https://arxiv.org/abs/1802.05814iv.org/abs/1802.05814)

Probabilistic Metric Learning with Adaptive Margin for Top-K Recommendation
by Ma et al., KDD 2020
[PDF]

BDL and Domain Adaptation (and Domain Generalization, Meta Learning, etc.)

Probabilistic Model-Agnostic Meta-Learning
by Finn et al., NIPS 2018
[PDF]

Bayesian Model-Agnostic Meta-Learning
by Yoon et al., NIPS 2018
[PDF]

Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
by Grant et al., ICLR 2018
[PDF]

Reconciling Meta-Learning and Continual Learning with Online Mixtures of Tasks
by Jerfal et al., NIPS 2019
[PDF]

Meta-Learning Probabilistic Inference For Prediction
by Gordon et al., ICLR 2019
[PDF]

Learning to Learn with Variational Information Bottleneck for Domain Generalization
by Du et al., ECCV 2020
[PDF]

Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels
by Patacchiola et al., NIPS 2020
[PDF]

Continuously Indexed Domain Adaptation
by Wang et al., ICML 2020
[PDF]

A Bit More Bayesian: Domain-Invariant Learning with Uncertainty
by Xiao et al., ICML 2021
[PDF]

Domain-Indexing Variational Bayes: Interpretable Domain Index for Domain Adaptation
by Xu et al., ICLR 2023
[PDF]

BDL and Healthcare

Electronic Health Record Analysis via Deep Poisson Factor Models
by Henao et al., JMLR 2016
[PDF]

Structured Inference Networks for Nonlinear State Space Models
by Krishnan et al., AAAI 2017
[PDF]

Causal Effect Inference with Deep Latent-Variable Models
by Louizos et al., NIPS 2017
[PDF]

Black Box FDR
by Tansey et al., ICML 2018
[PDF]

Bidirectional Inference Networks: A Class of Deep Bayesian Networks for Health Profiling
by Wang et al., AAAI 2019
[PDF]

Sampling-free Uncertainty Estimation in Gated Recurrent Units with Applications to Normative Modeling in Neuroimaging
by Hwang et al., UAI 2019
[PDF]

Neural Jump Stochastic Differential Equations
by Jia et al., NIPS 2019
[PDF]

Towards Interpretable Clinical Diagnosis with Bayesian Network Ensembles Stacked on Entity-Aware CNNs
by Chen et al., ACL 2020
[PDF]

Continuously Indexed Domain Adaptation
by Wang et al., ICML 2020
[PDF] [Cross Referenced in BDL and Domain Adaptation]

Assessment of medication self-administration using artificial intelligence
by Zhao et al., Nature Medicine 2021
[PDF]

Neural Pharmacodynamic State Space Modeling
by Hussain et al., ICML 2021
[PDF]

Self-Interpretable Time Series Prediction with Counterfactual Explanations
by Yan et al., ICML 2023
[PDF] [Cross Referenced in BDL and Forecasting (Time Series Analysis)]

BDL and NLP

Sequence to Better Sequence: Continuous Revision of Combinatorial Structures
by Mueller et al., ICML 2017
[PDF]

QuaSE: Sequence Editing under Quantifiable Guidance
by Liao et al., EMNLP 2018
[PDF]

Dispersed Exponential Family Mixture VAEs for Interpretable Text Generation
by Shi et al., ICML 2020
[PDF]

Towards Interpretable Clinical Diagnosis with Bayesian Network Ensembles Stacked on Entity-Aware CNNs
by Chen et al., ACL 2020
[PDF] [Cross Referenced in BDL and Healthcare]

What You Say and How You Say it: Joint Modeling of Topics and Discourse in Microblog Conversations
by Zeng et al., ACL 2020
[PDF]

Latent Diffusion Energy-Based Model for Interpretable Text Modeling
by Yu et al., ICML 2022
[PDF]

Diffusion-LM Improves Controllable Text Generation
by Li et al., NeurIPS 2022
[PDF]

Tractable Control for Autoregressive Language Generation
by Zhang et al., ICML 2023
[PDF]

BDL and Computer Vision

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
by Eslami et al., NIPS 2016
[PDF]

Efficient Inference in Occlusion-aware Generative Models of Images
by Huang et al., ICLR 2016
[PDF]

Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
by Kosiorek et al., NIPS 2018
[PDF]

Gaussian Process Prior Variational Autoencoders
by Casale et al., NIPS 2018
[PDF]

Spatially Invariant Unsupervised Object Detection with Convolutional Neural Networks
by Crawford et al., AAAI 2019
[PDF]

Faster Attend-Infer-Repeat with Tractable Probabilistic Models
by Stelzner et al., ICML 2019
[PDF]

Asynchronous Temporal Fields for Action Recognition
by Sigurdsson et al., CVPR 2017
[PDF]

Generalizing Eye Tracking with Bayesian Adversarial Learning
by Wang et al., CVPR 2019
[PDF]

Sequential Neural Processes
by Singh et al., NIPS 2019
[PDF]

SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
by Lin et al., ICLR 2020
[PDF]

Being Bayesian about Categorical Probability
by Joo et al., ICML 2020
[PDF]

Learning Latent Space Energy-Based Prior Model
by Pang et al., NIPS 2020
[PDF]

Generative Neurosymbolic Machines
by Jiang et al., NIPS 2020
[PDF]

Denoising Diffusion Probabilistic Models
by Ho et al., NIPS 2020
[PDF]

A Causal View of Compositional Zero-shot Recognition
by Atzmon et al., NIPS 2020
[PDF]

Counterfactuals Uncover the Modular Structure of Deep Generative Models
by Besserve et al., ICLR 2020
[PDF]

ROOTS: Object-Centric Representation and Rendering of 3D Scenes
by Chen et al., JMLR 2021
[PDF]

Improved Denoising Diffusion Probabilistic Models
by Nichol et al., ICML 2021
[PDF]

Generative Interventions for Causal Learning.
by Mao et al., CVPR 2021
[PDF]

Adversarial Attacks are Reversible with Natural Supervision
by Mao et al., ICCV 2021
[PDF]

Counterfactual Zero-Shot and Open-Set Visual Recognition
by Yue et al., CVPR 2021
[PDF]

ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models
by Choi et al., ICCV 2021
[PDF]

Diffusion Models Beat GANs on Image Synthesis
by Dhariwal et al., NIPS 2021
[PDF]

Relational Learning with Variational Bayes
by Liu, ICLR 2022
[PDF]

High-Resolution Image Synthesis with Latent Diffusion Models
by Rombach et al., CVPR 2022
[PDF]

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
by Nichol et al., ICML 2022
[PDF]

Diffusion Models for Adversarial Purification
by Nie et al., ICML 2022
[PDF]

A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion
by Lyu et al., ICLR 2022
[PDF]

Label-Efficient Semantic Segmentation with Diffusion Models
by Baranchuk et al., ICLR 2022
[PDF]

Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality
by Watson et al., ICLR 2022
[PDF]

Flexible Diffusion Modeling of Long Videos
by Harvey et al., NIPS 2022
[PDF]

ProtoVAE: A Trustworthy Self-Explainable Prototypical Variational Model
by Gautam et al., NIPS 2022
[PDF]

Causal Transportability for Visual Recognition
by Mao et al., CVPR 2022
[PDF]

Posterior Matching for Arbitrary Conditioning
by Strauss et al., NIPS 2022
[PDF]

On the Relationship between Variational Inference and Auto-Associative Memory
by Annabi et al., NIPS 2022
[PDF]

Robust Perception through Equivariance
by Mao et al., ICML 2023
[PDF]

Object-Centric Slot Diffusion
by Jiang et al. NeurIPS 2023
[PDF]

PreDiff: Precipitation Nowcasting with Latent Diffusion Models
by Gao et al., NeurIPS 2023
[PDF]

BDL and Control/Planning

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
by Watter et al., NIPS 2015
[PDF]

Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
by Karl et al., ICLR 2017
[PDF]

Probabilistic Recurrent State-Space Models
by Doerr et al., ICML 2018
[PDF]

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
by Chua et al., NIPS 2018
[PDF]

Robust Locally-Linear Controllable Embedding
by Banijamali et al., AISTATS 2018
[PDF]

Learning Latent Dynamics for Planning from Pixels
by Hafner et al., ICML 2019
[PDF]

Planning with Diffusion for Flexible Behavior Synthesis
by Janner et al., ICML 2022
[PDF]

A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with Symbolic Reward Machines
by Zhou et al., ICML 2022
[PDF]

BDL and Graphs (Link Prediction, Graph Neural Networks, Knowledge Graphs, etc.)

Relational Deep Learning: A Deep Latent Variable Model for Link Prediction
by Wang et al., AAAI 2017
[PDF]

Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs
by Trivedi et al., ICML 2017
[PDF]

Graphite: Iterative Generative Modeling of Graphs
by Grover et al., ICML 2019
[PDF]

Relational Variational Autoencoder for Link Prediction with Multimedia Data
by Li et al., ACM MM 2017
[PDF]

Stochastic Blockmodels meet Graph Neural Networks
by Mehta et al., ICML 2019
[PDF]

Scalable Deep Generative Modeling for Sparse Graphs
by Dai et al., ICML 2020
[PDF]

PGM-Explainer: Probabilistic Graphical Model Explanations for Graph Neural Networks
by Vu et al., NIPS 2020
[PDF]

Dirichlet Graph Variational Autoencoder
by Li et al., NIPS 2020
[PDF]

Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs
by Ren et al., NIPS 2020
[PDF]

GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation
by Xu et al., ICLR 2022
[PDF]

Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations
by Jo et al., ICML 2022
[PDF]

Equivariant Diffusion for Molecule Generation in 3D
by Hoogeboom et al., ICML 2022
[PDF]

LIMO: Latent Inceptionism for Targeted Molecule Generation
by Eckmann et al., ICML 2022
[PDF]

3DLinker: An E(3) Equivariant Variational Autoencoder for Molecular Linker Design
by Huang et al., ICML 2022
[PDF]

Crystal Diffusion Variational Autoencoder for Periodic Material Generation
by Xie et al., ICLR 2022
[PDF]

OrphicX: A Causality-Inspired Latent Variable Model for Interpreting Graph Neural Networks
by Lin et al., CVPR 2022
[PDF]

BDL and Topic Modeling

Relational Stacked Denoising Autoencoder for Tag Recommendation
by Wang et al., AAAI 2015
[PDF]

Scalable Deep Poisson Factor Analysis for Topic Modeling
by Gan et al., ICML 2015
[PDF]

Deep Latent Dirichlet Allocation with Topic-layer-adaptive Stochastic Gradient Riemannian MCMC
by Cong et al., ICML 2017
[PDF]

Deep Unfolding for Topic Models
by Chien et al., TPAMI 2017
[PDF]

Neural Relational Topic Models for Scientific Article Analysis
by Bai et al., CIKM 2018
[PDF]

Dirichlet Belief Networks for Topic Structure Learning
by Zhao et al., NIPS 2018
[PDF]

Deep Relational Topic Modeling via Graph Poisson Gamma Belief Network
by Wang et al., NIPS 2020
[PDF]

Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network
by Duan et al., ICML 2021
[PDF]

Poisson-Randomised DirBN: Large Mutation is Needed in Dirichlet Belief Networks
by Fan et al., ICML 2021
[PDF]

Torsional Diffusion for Molecular Conformer Generation
by Jing et al., NIPS 2022
[PDF]

Knowledge-Aware Bayesian Deep Topic Model
by Wang et al., NIPS 2022
[PDF]

BDL and Speech Recognition/Synthesis

Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
by Hsu et al., NIPS 2017
[PDF]

Scalable Factorized Hierarchical Variational Autoencoder Training
by Hsu et al., Interspeech 2018
[PDF]

Hierarchical Generative Modeling for Controllable Speech Synthesis
by Hsu et al., ICLR 2019
[PDF]

Recurrent Poisson Process Unit for Speech Recognition
by Huang et al., AAAI 2019
[PDF]

Deep Graph Random Process for Relational-thinking-based Speech Recognition
by Huang et al., ICML 2020
[PDF]

DiffWave: A Versatile Diffusion Model for Audio Synthesis
by Kong et al., ICLR 2021
[PDF]

WaveGrad: Estimating Gradients for Waveform Generation
by Chen et al., ICLR 2021
[PDF]

Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
by Popov et al., ICML 2021
[PDF]

STRODE: Stochastic Boundary Ordinary Differential Equation
by Huang et al., ICML 2021
[PDF]

Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance
by Kim et al., ICML 2022
[PDF]

Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme
by Popov et al., ICLR 2022
[PDF]

BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis
by Lam et al., ICLR 2022
[PDF]

Unsupervised Mismatch Localization in Cross-Modal Sequential Data with Application to Mispronunciations Localization
by Wei et al., TMLR 2022
[PDF]

BDL and Forecasting (Time Series Analysis)

DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks
by Salinas et al., 2017
[PDF]

Deep State Space Models for Time Series Forecasting
by Rangapuram et al., NIPS 2018
[PDF]

Deep Factors for Forecasting
by Wang et al., ICML 2019
[PDF]

Probabilistic Forecasting with Spline Quantile Function RNNs
by Gasthaus et al., AISTATS 2019
[PDF]

Adversarial Attacks on Probabilistic Autoregressive Forecasting Models
by Dang-Nhu et al., ICML 2020
[PDF]

Neural Jump Stochastic Differential Equations
by Jia et al., NIPS 2019
[PDF]

Segmenting Hybrid Trajectories using Latent ODEs
by Shi et al., ICML 2021
[PDF]

RNN with Particle Flow for Probabilistic Spatio-temporal Forecasting
by Pal et al., ICML 2021
[PDF]

End-to-End Learning of Coherent Probabilistic Forecasts for Hierarchical Time Series
by Rangapuram et al., ICML 2021
[PDF]

Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting
by Rasul et al., ICML 2021
[PDF]

Deep Explicit Duration Switching Models for Time Series
by Ansari et al., NIPS 2021
[PDF]

Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting
by Rasul et al., ICML 2021
[PDF]

CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation
by Tashiro et al., NIPS 2021
[PDF]

TACTiS: Transformer-Attentional Copulas for Time Series
by Drouin et al., ICML 2022
[PDF]

Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Series
by Kramer et al., ICML 2022
[PDF]

Deep Variational Graph Convolutional Recurrent Network for Multivariate Time Series Anomaly Detection
by Chen et al., ICML 2022
[PDF]

Vector Quantized Time Series Generation with a Bidirectional Prior Model
by Lee at al., AISTATS 2023
[PDF]

Self-Interpretable Time Series Prediction with Counterfactual Explanations
by Yan et al., ICML 2023
[PDF] [Cross Referenced in BDL and Healthcare]

BDL and Distributed/Federated Learning

Stochastic Expectation Propagation
by Li et al., NIPS 2015
[PDF]

BDL and Continual/Life-Long Learning

Continual Learning with Deep Generative Replay
by Shin et al., NIPS 2017
[PDF]

Continual Unsupervised Representation Learning
by Rao et al., NIPS 2019
[PDF]

Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies
by Achille et al., NIPS 2018
[PDF]

Learning Latent Representations Across Multiple Data Domains Using Lifelong VAEGAN
by Ye et al., ECCV 2020
[PDF]

A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
by Lee et al., ICLR 2020
[PDF]

BDL as a Framework (Miscellaneous)

Towards Bayesian Deep Learning: A Framework and Some Existing Methods
by Wang et al., TKDE 2016
[PDF]

Composing Graphical Models with Neural Networks for Structured Representations and Fast Inference
by Johnson et al., NIPS 2016
[PDF]

Bayesian/Probabilistic Neural Networks as Building Blocks of BDL

Learning Stochastic Feedforward Networks
by Neal et al., Technical Report 1990
[PDF]

A Practical Bayesian Framework for Backprop Networks
by MacKay et al., Neural Computation 1992
[PDF]

Keeping Neural Networks Simple by Minimizing the Description Length of the Weights
by Hinton et al., COLT 1993
[PDF]

Bayesian Learning via Stochastic Gradient Langevin Dynamics
by Welling et al., ICML 2011
[PDF]

Practical Variational Inference for Neural Networks
by Alex Graves, NIPS 2011
[PDF]

Auto-Encoding Variational Bayes
by Kingma et al., ArXiv 2014
[PDF] [Code]

Deep Exponential Families
by Ranganath et al., AISTATS 2015
[PDF]

Weight Uncertainty in Neural Networks
by Blundell et al., ICML 2015
[PDF]

Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
by Hernandez-Lobato et al., ICML 2015
[PDF]

Variational Dropout and the Local Reparameterization Trick
by Kingma et al., NIPS 2015
[PDF]

The Poisson Gamma Belief Network
by Zhou et al., NIPS 2015
[PDF]

Deep Poisson Factor Modeling
by Henao et al., NIPS 2015
[PDF]

Natural-Parameter Networks: A Class of Probabilistic Neural Networks
by Wang et al., NIPS 2016
[PDF] [Project Page] [Code]

Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
by Mescheder et al., ICML 2017
[PDF]

Stick-Breaking Variational Autoencoders
by Nalisnick et al., ICLR 2017
[PDF]

Bayesian GAN
by Saatchi et al, NIPS 2017
[PDF]

Neural Expectation Maximization
by Greff et al., NIPS 2017
[PDF]

Lightweight Probabilistic Deep Networks
by Gast et al., CVPR 2018
[PDF]

Feed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers
by Shekhovtsov et al., ICLR 2018
[PDF]

Glow: Generative Flow with Invertible 1x1 Convolutions
by Kingma et al., NIPS 2018
[PDF]

Evidential Deep Learning to Quantify Classification Uncertainty
by Sensoy et al., NIPS 2018
[PDF]

ProbGAN: Towards Probabilistic GAN with Theoretical Guarantees
by He et al., ICLR 2019
[PDF] [Project Page]

Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation
by Postels et al., ICCV 2019
[PDF]

Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
by Dusenberry et al., ICML 2020
[PDF]

Neural Clustering Processes
by Pakman et al., ICML 2020
[PDF]

Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
by Kristiadi et al., ICML 2020
[PDF]

Activation-level Uncertainty in Deep Neural Networks
by Morales-Alvarez et al., ICLR 2021
[PDF]

Bayesian Deep Learning via Subnetwork Inference
by Daxberger et al., ICML 2021
[PDF]

On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks
by Seitzer et al., ICLR 2022
[PDF]

Evidential Turing Processes
by Kandemir et al., ICLR 2022
[PDF]

How Tempering Fixes Data Augmentation in Bayesian Neural Networks
by Bachmann et al., ICML 2022
[PDF]

SIMPLE: A Gradient Estimator for k-Subset Sampling
by Ahmed et al., ICLR 2023
[PDF]

Collapsed Inference for Bayesian Deep Learning
by Zeng et al., NeurIPS 2023
[PDF]

Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing
by Wang et al., NeurIPS 2023
[PDF]