A geometric criterion for Gelfand pairs associated with the Heisenberg group

Let K be a closed subgroup of U(n) acting on the (2n+ 1)dimensional Heisenberg group Hn by automorphisms. One calls (K,Hn) a Gelfand pair when the integrable K-invariant functions on Hn form a commutative algebra under convolution. We prove that this is the case if and only if the coadjoint orbits for G := K n Hn which meet the annihilator k⊥ of the Lie algebra k of K do so in single K-orbits. Equivalently, the representation of K on the polynomial algebra over C is multiplicity free if and only if the moment map from C to k∗ is one-to-one on K-orbits. It is also natural to conjecture that the spectrum of the quasi-regular representation of G on L(G/K) corresponds precisely to the integral coadjoint orbits that meet k⊥. We prove that the representations occurring in the quasi-regular representation are all given by integral coadjoint orbits that meet k⊥. Such orbits can, however, also give rise to representations that do not appear in L(G/K).