Minimum width for universal approximation using ReLU networks on compact domain
share
The universal approximation property of width-bounded networks has been studied as a dual of the classical universal approximation theorem for depth-bounded ones. There were several attempts to characterize the minimum width w_min enabling the universal approximation property; however, only a few of them found the exact values. In this work, we show that the minimum width for the universal approximation of L^p functions from [0,1]^d_x to ℝ^d_y is exactly max{d_x,d_y,2} if an activation function is ReLU-Like (e.g., ReLU, GELU, Softplus). Compared to the known result w_min=max{d_x+1,d_y} when the domain is ℝ^d_x, our result first shows that approximation on a compact domain requires smaller width than on ℝ^d_x. We next prove a lower bound on w_min for uniform approximation using general activation functions including ReLU: w_min≥ d_y+1 if d_x<d_y≤2d_x. Together with our first result, this shows a dichotomy between L^p and uniform approximations for general activation functions and input/output dimensions.
READ FULL TEXT