This Wednesday (April 17th), we are pleased to host a talk by Philippe Trunschke (PostDoc at Centrale Nantes & Nantes Université), which should be interesting to many of our members!
The seminar will take place online at the zoom link below, and will also be projected live in the room CM 1 517. https://epfl.zoom.us/j/61353461236?pwd=MnI2VkRMWlE2WUJxalRmNVJwc2JGQT09
Title: Optimal sampling for stochastic gradient descent
Abstract: Approximating high-dimensional functions often requires optimising a loss functional that can be represented as an expected value. When computing this expectation is unfeasible, a common approach is to replace the exact loss with a Monte Carlo estimate before employing a standard gradient descent scheme. This results in the well-known stochastic gradient descent method. However, using an estimated loss instead of the true loss can result in a “generalisation error”. Rigorous bounds for this error usually require strong compactness and Lipschitz continuity assumptions while providing a very slow decay with increasing sample size. This slow decay is unfavourable in settings where high accuracy is required or sample creation is costly. To address this issue, we propose a new approach that involves empirically (quasi-)projecting the gradient of the true loss onto local linearisations of the model class through an optimal weighted least squares method. The resulting optimisation scheme converges almost surely to a stationary point of the true loss, and we investigate its convergence rate.
Philipp TRUNSCHKE studied Mathematics at the Humboldt University in Berlin, specialising in statistical learning theory. He completed his doctoral studies focusing on tensor product approximation at the Technical University of Berlin 2018. Currently, he is working with Anthony NOUY in Nantes on compositional function networks and optimal sampling.
