A central mystery in deep learning is why gradient-based optimizers reliably find good solutions despite training a nonconvex loss function. Most theoretical work either proves favorable properties under strong assumptions or gives worst-case bounds that are too loose to be useful in practice. In this talk, I take a different approach: rather than analyzing large networks asymptotically, I study the loss landscape of a small, concretely specified network where every critical point can be computed exactly using tools from algebraic geometry. The findings are sharp: across all data realizations and all three optimizers, the dynamically accessible critical points are in exact bijection with the local minima of the loss, as independently confirmed by Hessian eigenvalue analysis. All saddle points are completely inaccessible, with empirical basin measure zero. I also show that removing the network's scaling symmetry via an affine chart systematically degrades all three optimizers, a phenomenon explained by the fiber connectivity structure of the parameterization map. Finally, I will discuss how these findings position algebraically-certified small networks as a rigorous testbed for optimizer theory, and outline extensions to wider architectures and polynomial activation functions.