"Topological properties of the set of neural networks of fixed size"Voigtlaender, FelixWe consider the set of all functions that can be implemented using neural networks of a fixed (but arbitrary) architecture. We show that this set has several pathological topological properties:1.) The set is never convex (under minimal assumptions on the activation function). More precisely, we provide an explicit bound on the number of centers of the set. 2.) The set is never closed in $L^p (p < \infty)$, under minimal assumptions on the activation function. 3.) If the activation function is $C^1$ and satisfies mild additional assumptions, then the set is not closed with respect to the topology of uniform convergence. 4.) For shallow ReLU networks, the set is closed with respect to the topology of uniform convergence. 5.) The mapping $\text{network weights} \mapsto \text{implemented network}$ is not inverse stable. That is, there are functions $f,g$ very close to each other, but such that if $f$ is implemented by a network with weights $W$, and $g$ is implemented by a network with weight $V$, then $W$ and $V$ are not close to each other. |
« back