Infinite width (finite depth) neural networks benefit from multi-task learning unlike shallow Gaussian Processes – an exact quantitative macroscopic characterization

12/31/2021
by   Jakob Heiss, et al.
0

We prove in this paper that optimizing wide ReLU neural networks (NNs) with at least one hidden layer using l2-regularization on the parameters enforces multi-task learning due to representation-learning - also in the limit of width to infinity. This is in contrast to multiple other results in the literature, in which idealized settings are assumed and where wide (ReLU)-NNs loose their ability to benefit from multi-task learning in the infinite width limit. We deduce the ability of multi-task learning from proving an exact quantitative macroscopic characterization of the learned NN in an appropriate function space.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset