After training a neural network, what has it learned? This talk will present the analysis of two methods that address this question. First, we will discuss neural network descrambling, an explainability algorithm that was proposed by Amey et. al in 2021 for understanding the latent transformations in the weight matrices of individual neural network layers. We will show that the explanations provided by descrambling can be characterized via the singular vectors of neural network weights, and in turn these singular vectors can help explain the actions of the affine transformations within neural network layers. Second, we will discuss neural collapse--the phenomenon where a classifier's terminal features and weights converge to the vertices of a regular simplex--and study this phenomenon in the orthoplex regime where there are more classes than feature dimensions. In this case, spherical codes will play a key role in characterizing the arrangements produced under neural collapse and the emergence of a "goldilocks" region where the temperature in the cross entropy loss promotes certain spherical codes over others.