BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//CREST - ECPv5.1.3//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:CREST
X-ORIGINAL-URL:https://crest.science
X-WR-CALDESC:Events for CREST
BEGIN:VTIMEZONE
TZID:Europe/Helsinki
BEGIN:DAYLIGHT
TZOFFSETFROM:+0200
TZOFFSETTO:+0300
TZNAME:EEST
DTSTART:20240331T010000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0300
TZOFFSETTO:+0200
TZNAME:EET
DTSTART:20241027T010000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=Europe/Helsinki:20240909T140000
DTEND;TZID=Europe/Helsinki:20240909T153000
DTSTAMP:20260710T153402
CREATED:20240903T123745Z
LAST-MODIFIED:20240903T123848Z
UID:17263-1725890400-1725895800@crest.science
SUMMARY:Etienne BOURSIER (INRIA) "Early alignment in two-layer networks training is a two-edged sword"
DESCRIPTION:Statistical Seminar: Every Monday at 2:00 pm.\nTime: 2:00 pm – 3:00 pm\nDate: 09th September 2024\nPlace: 3001 \n  \nEtienne BOURSIER (INRIA) “Early alignment in two-layer networks training is a two-edged sword” \n  \n Abstract: Training neural networks with first order optimisation methods is at the core of the empirical success of deep learning. The scale of initialisation is a crucial factor\, as small initialisations are generally associated to a feature learning regime\, for which gradient descent is implicitly biased towards simple solutions. This work provides a general and quantitative description of the early alignment phase\, originally introduced by Maennel et al. (2018) . For small initialisation and one hidden ReLU layer networks\, the early stage of the training dynamics leads to an alignment of the neurons towards key directions. This alignment induces a sparse representation of the network\, which is directly related to the implicit bias of gradient flow at convergence. This sparsity inducing alignment however comes at the expense of difficulties in minimising the training objective: we also provide a simple data example for which overparameterised networks fail to converge towards global minima and only converge to a spurious stationary point instead. \n  \nOrganizers: \nKarim LOUNICI (CMAP) \, Jaouad MOURTADA (CREST) \nSponsors:\nCREST-CMAP \n
URL:https://crest.science/event/etienne-boursier-inria-early-alignment-in-two-layer-networks-training-is-a-two-edged-sword/
CATEGORIES:Seminars,Statistics
ATTACH;FMTTYPE=:
END:VEVENT
END:VCALENDAR