The Deligne–Mumford compactification of the real multiplication locus and Teichmüller curves in genus 3

In the moduli space \( \mathcal{M} \) g of genus-g Riemann surfaces, consider the locus \( \mathcal{R}{\mathcal{M}_{\mathcal{O}}} \) of Riemann surfaces whose Jacobians have real multiplication by the order \( \mathcal{O} \) in a totally real number field F of degree g. If g = 3, we compute the closure of \( \mathcal{R}{\mathcal{M}_{\mathcal{O}}} \) in the Deligne–Mumford compactification of \( \mathcal{M} \) g and the closure of the locus of eigenforms over \( \mathcal{R}{\mathcal{M}_{\mathcal{O}}} \) in the Deligne–Mumford compactification of the moduli space of holomorphic 1-forms. For higher genera, we give strong necessary conditions for a stable curve to be in the boundary of \( \mathcal{R}{\mathcal{M}_{\mathcal{O}}} \). Boundary strata of \( \mathcal{R}{\mathcal{M}_{\mathcal{O}}} \) are parameterized by configurations of elements of the field F satisfying a strong geometry of numbers type restriction.