Theoretical results for deep neural networks

Theoretical results for deep neural networks

Johannes Schmidt-Hieber
Leiden University


SCHEDULE Monday 14th January 2019
21st January 2019
De 14h à 16h30 Salle 2016
Thursday 17th January 2019
24th January 2019
De 14h à 16h30 Salle 2016

Theoretical results for deep neural networks


Large databases and increasing computational power have recently resulted in astonishing performances of deep neural networks for a broad range of extremely complex tasks, including image and text classification, speech recognition and game playing. These deep learning methods are build by imitating the action of the brain and there are few theoretical results as of yet. To formulate a sound mathematical framework explaining these successes is a major challenge for current research.

The course aims to give an overview about existing theory for deep neural networks with a strong emphasis on recent developments. Core topics are approximation theory and complexity bounds for the function spaces generated by deep networks. Beyond that we also discuss modelling aspects, theoretical results on the energy landscape and statistical risk bounds.


– Anthony, M., and Bartlett, P. L. Neural network learning: theoretical foundations. Cambridge University Press, 1999.
– Bach, F. Breaking the curse of dimensionality with convex neural networks. JMLR. 2017.
– Barron, A. Universal approximation bounds for superpositions of a sigmoidal function. IEEE . 1993.
– Barron, A. Approximation and estimation bounds for artificial neural networks. Machine Learning. 1994.
– Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., LeCun, Y. The loss surface of multilayer networks. Aistats. 2015.
– Goodfellow, Bengio, Courville. Deep Learning. MIT Press, 2016.
– Pinkus, A. Approximation theory of the MLP model in neural networks. Acta Numerica, 143-195, 1999.
– Schmidt-Hieber, J. Nonparametric regression using deep neural networks with ReLU activation function. ArXiv 2017.
– Telgarsky, M. Benefits of depth in neural networks. ArXiv. 2016.
– Yarotsky, D. Error bounds for approximations with deep ReLU networks. Neural Networks. 2017.