Abstract: We give sharper bounds for uniformly stable randomized algorithms in a PAC-Bayesian framework, which improve the existing results by up to a factor of square root of n (ignoring a log factor), where n is the sample size. The key idea is to bound the moment generating function of the generalization gap using concentration of weakly dependent random variables due to Bousquet et al (2020). We introduce an assumption of sub-exponential stability parameter, which allows a general treatment that we instantiate in two applications: stochastic gradient descent (SGD) and randomized coordinate descent (RCD). Our results eliminate the requirement of strong convexity from previous results, and hold for non-smooth convex problems.

 

Bio: Sijia Zhou is a PhD student in Computer Science at the University of Birmingham, under the supervision of Dr. Yunwen Lei (HKU) and Prof. Ata Kaban.  Her research focuses on machine learning and learning theory.