Yan Wang’s data science seminar talk on 10/04

Title: Prediction Done Right with Wrong Machine Learning Models

Abstract: Conformal prediction (CP) is a technique to provide theoretically justified prediction intervals under minimal assumptions. In practice, it can be applied to almost any machine learning model: linear regression, random forests, neural networks, etc. You name it. Even if you always guess 𝑌̂ 𝑛+1 = 0 irrespective of the corresponding 𝑋𝑛+1 for a future test point (𝑋𝑛+1, 𝑌𝑛+1), CP can provide a prediction interval 𝐶𝑛,𝛼(𝑋𝑛+1) such that 𝑌𝑛+1 ∈ 𝐶𝑛,𝛼(𝑋𝑛+1) with probability at least 1 − 𝛼, where 𝛼 ∈ (0,1) is a prescribed constant and 𝐶𝑛,𝛼(𝑋𝑛+1) is constructed by CP from the training set {(𝑋𝑖 , 𝑌𝑖 )}𝑖=1 𝑛 . I will introduce the basic idea of CP and some of its variants that are computationally appealing in practice. In particular, I will show that, under (really) mild conditions, by training random forests once, not only can one have a point predictor for 𝑌𝑛+1, but also a valid prediction interval that comes almost for free.