Uncertainty Calibration（预估校准技术）

Calibration techniques are increasingly applied in the fields of computation advertising，medical diagnosis and meteorological forecasting etc. This is a collection of research and application papers of (uncertainty) calibration techniques. These works are summarized from the public website. If any authors do not want their paper to be listed here, please feel free to contact me（email:[email protected], Wechat:coder_hh）.

1. Papers

Post-hoc Calibrator（后处理方法）

Parametric Method（参数化方法）

• Platt scaling: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods[J]. by Platt J. Advances in large margin classifiers, 1999.
• Temperature scaling: On calibration of modern neural networks. by Chuan Guo et al. ICML, 2017.
• Beta calibration: A well-founded and easily implemented improvement on logistic calibration for binary classifiers. by Meelis Kull et al. In Artificial Intelligence and Statistics, 2017.
• Attended temperature scaling: Attended temperature scaling: a practical approach for calibrating deep neural networks. by AzadehSadatMozafari et al. arXiv , 2018.
• Dirichlet scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. by Meelis Kull et al. arXiv, 2019.

Non-parametric Method（非参数化方法）

• Hisogram Binning: Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. by Bianca Zadrozny et al. ICML, 2001.
• Isotonic Regression: Transforming classifier scores into accurate multiclass probability estimates. by Bianca Zadrozny et al. KDD, 2002.
• Bayesian Binning: Obtaining well calibrated probabilities using bayesian binning. by MahdiPakdamanNaeini et al. AAAI, 2015.
• Distribution-free calibration guarantees for histogram binning without sample splitting by Chirag Gupta et al. ICML, 2021.

Hybrid Method（参数&非参数混合方法）

• Scaling-binning:Verified Uncertainty Calibration. by Ananya Kumar et al. NeurIPS, 2020.
• Mix-n-Match:Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning. by Jize Zhang et al. ICML 2020.
• SIR:Calibrating User Response Predictions in Online Advertising. by Chao Deng et al. PKKDD, 2020. (Our work)
• MBCT:Tree-Based Feature-Aware Binning for Individual Uncertainty Calibration. by Siguang Huang et al. WWW, 2022. (Our work)

Calibrated Predictor（与模型直接结合的方法）

• Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions. by Feiyang Pan et al. WWW 2020 (Tencent).
• Posterior Probability Matters: Doubly-Adaptive Calibration for Neural Predictions in Online Advertising. by Penghui Wei et al. SIGIR 2022. (Alibaba)
  • 中文链接
• Joint Optimization of Ranking and Calibration with Contextualized Hybrid Model. by Xiang-Rong Sheng et al. KDD 2023. (Alibaba)
  • 中文链接

Calibration Theory（校准理论分析）

• Binary Classifier Calibration: Non-parametric approach. by Mahdi Pakdaman Naeini et al. arXiv 2014.
• Verified Uncertainty Calibration. by Ananya Kumar et al. NeurIPS, 2020.
• Distribution-free binary classification: prediction sets, confidence intervals and calibration. by Chirag Gupta et al. NeurIPS, 2020.
• Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning. by Jize Zhang et al. ICML 2020.
• Individual Calibration with Randomized Forecasting. by Shengjia Zhao et al. ICML, 2020.
• Post-hoc Calibration of Neural Networks. by Amir Rahimi et al. arXiv 2020.(Google)
• Non-Parametric Calibration for Classification. by Jonathan Wenger et al. AISTATS 2020.
• Don’t Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification by Yu Bai et al. ICML, 2021.
• Distribution-free calibration guarantees for histogram binning without sample splitting by Chirag Gupta et al. ICML, 2021.
• Meta-Cal: Well-controlled Post-hoc Calibration by Ranking by Xingchen Ma et al. ICML, 2021.

2. Applications（校准应用）

Advertising

• Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions. by Feiyang Pan et al. WWW 2020 (Tencent).
• Calibrating User Response Predictions in Online Advertising. by Chao Deng et al. PKKDD, 2020 (Alibaba).
• MBCT:Tree-Based Feature-Aware Binning for Individual Uncertainty Calibration. by Siguang Huang et al. WWW, 2022. (Alibaba)

3. Calibration Metrics（评价方法）

• Mitigating bias in calibration error estimation by Rebecca Roelofs et al. arXiv 2020.(Google)
• Measuring Calibration in Deep Learning by Jeremy Nixon et al. arXiv 2020.(Google)
• Calibration of Neural Networks using Splines by Kartik Gupta et al. arXiv 2020.(Google)

4. Datasets and Benchmarking（公开数据集）

• Alibaba Group's Dataset for Feature-Aware Calibration by Siguang Huang. 2022
• Alibaba Pctr Calibration Dataset by Chao Deng. 2019
• Alibaba Pcvr Calibration Dataset by Chao Deng. 2019

5. Other Links

• Research Track
  • ICML: Trustworthy Machine Learning (accountability, causality, fairness, privacy, robustness, etc.)

6. 中文材料

• 阿里巴巴展示广告校准技术演进之路
• MBCT：基于树模型的特征可感知的个性化校准方法
• AdaCalib: 后验引导的特征自适应预估校准
• 贝叶斯分层校准
• 算法实践_专栏