Epigenetic cell memory, the inheritance of gene expression patterns across subsequent cell divisions, is a critical property of multi-cellular organisms. It was previously found via simulations of stochastic models that the time scale separation between establishment (fast) and erasure (slow) of chromatin modifications (such as DNA methylation and histone modifications) extends the duration of cell memory, and that different asymmetries between erasure rates of chromatin modifications can lead to different gene expression patterns. We provide a mathematical framework to rigorously validate these computational findings using stochastic models of chemical reaction networks. For our study of epigenetic cell memory, these are singularly perturbed, finite state, continuous time Markov chains. We exploit special structure in our models and extend beyond existing theory to study these singularly perturbed Markov chains when the perturbation parameter is small. We also develop comparison theorems to study how different erasure rates affect the behavior of our chromatin modification circuit. The theoretical tools developed in our work not only allow us to set a rigorous mathematical basis for highlighting the effect of chromatin modification dynamics on epigenetic cell memory, but they can also be applied to other singularly perturbed Markov chains beyond the applications in this work, especially those associated with chemical reaction networks.