$$ \def\E{\mathbb{E}} %expectation \def\P{\mathbb{P}} %prob \def\var{\mathbb{V}} %var \def\T{\mathrm{\scriptscriptstyle T}} %transpose \def\ind{\mathbbm{1}} %indicator \newcommand{\gw}[1]{\color{red}{(#1)}} \newcommand{\norm}[1]{\lVert#1\rVert} \newcommand{\abs}[1]{\lvert#1\rvert} $$

The expressivity or approximation theory of neural networks is a much more well-understood topic than the optimization and generalization of NNs. The survey paper Gühring et al. (2020) has a comprehensive review.