Colloquium de Gérard Ben Arous : Dynamical spectral transition for optimization in very high dimensions

We will welcome Gérard Ben Arous (professor at the Courant Institute of Mathematical Sciences, NYU), who will give a colloquium about "Dynamical spectral transition for optimization in very high dimensions".

In recent work with Reza Gheissari (Northwestern) and Aukosh Jagannath (Waterloo), we gave a general context for the existence of projected “effective dynamics” of SGD in very high dimensions, for "summary statistics” in much smaller dimensions. These effective dynamics (and, in particular, their so-called ‘critical regime”) define a dynamical system in finite dimensions which may be quite complex, and rules the performance of the learning algorithm.

The next step is to understand how the system finds these “summary statistics”.  This is done in the last work with the same authors and with Jiaoyang Huang (Wharton, U-Penn). This is based on a dynamical spectral transition of Random Matrix Theory: along the trajectory of the optimization path, the Gram matrix or the Hessian matrix develop outliers which carry these effective dynamics. I will naturally first come back to the Random Matrix Tools needed here (the behavior of the edge of the spectrum and the BBP transition). And then illustrate the use of this point of view on a few central examples of ML:  classification for Gaussian mixtures, and the XOR task.