Deep Learning Theories for Problems with Low–Dimensional Structures

Abstract

Deep neural networks have demonstrated a great success on many applications, especially on problems with high-dimensional data sets. In spite of that, most existing theories are cursed by data dimension and cannot explain such a success. To bridge the gap between theories and practice, we exploit the low-dimensional  structures of data set and establish theoretical guarantees with a fast rate that is only cursed by the intrinsic dimension of the data set. This presentation addresses our recent work on function approximation and operator learning by deep neural networks. The first part function approximation on low-dimensional manifolds. For Sobolev functions defined on a low-dimensional manifold, we show that neural networks can approximate both the function value and its gradient well. The network size critically depends on the intrinsic dimension of the manifold and only weakly depends on the ambient dimension. In the second part, we consider a general encoder-decoder framework to learn Lipschitz operators between infinite dimensional spaces by feedforward neural networks. Such a framework covers most scenarios in real applications. We develop non-asymptotic upper bounds for the generalization error of the empirical risk minimizer. When the problem have low-dimensional structures, our error bounds have a fast rate depending on the intrinsic dimension. Our results show that neural networks are adaptive to the low-dimensional structures of the problem.

Reference

Deep Nonparametric Estimation of Operators between Infinite...

Learning operators between infinitely dimensional spaces is an important learning task arising in wide applications in machine learning, imaging science, mathematical modeling and simulations,...

https://arxiv.org/abs/2201.00217

Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks

Most of existing statistical theories on deep neural networks have sample complexities cursed by the data dimension and therefore cannot well explain the emp...

https://proceedings.mlr.press/v139/liu21e.html

Deep Nonparametric Estimation of Intrinsic Data Structures by...

Autoencoders have demonstrated remarkable success in learning low-dimensional latent features of high-dimensional data across various applications. Assuming that data are sampled near a...

https://arxiv.org/abs/2303.09863