Nulling at short wavelengths: theoretical performance constraints and a demonstration of faint companion detection inside the diffraction limit with a rotating-baseline interferometer