Statistical Seminar: Every Monday at 2:00 pm.
Time: 2:00 pm – 3:15 pm
Date: 29th March 2024
Place : 3001

Ludovic STEPHAN (EPFL) – Feature learning in two-layer neural networks with large gradient steps

Abstract:

Feature learning is an important mechanism of neural networks, and an integral part of their advantages over simpler (e.g. kernel) learning methods. In this talk, I will present how this phenomenon occurs in two-layer networks trained with large gradient steps, in which both the batch size and the learning rate grow polynomially with the dimension. In particular, we uncover an occurence of the so-called “staircase” property of learning, where important directions are learned sequentially at each new step.

Organizers:
Jaouad MOURTADA (CREST), Anna KORBA (CREST), Karim LOUNICI (CMAP)

Sponsors:
CREST-CMAP