LipPass: Lip Reading-based User Authentication On Smartphones Leveraging Acoustic Signals

Li Lu Shanghai Jiao Tong University, P.R. China
Jiadi Yu Shanghai Jiao Tong University, P.R. China
Yingying Chen Rutgers University, USA
Hongbo Liu Indiana University-Purdue University Indianapolis, USA
Yanmin Zhu Shanghai Jiao Tong University, P.R. China
Yunfei Liu Shanghai Jiao Tong University, P.R. China
Minglu Li Shanghai Jiao Tong University, P.R. China


To prevent users' privacy from leakage, more and more mobile devices employ biometric-based authentication approaches , such as fingerprint, face recognition, voiceprint authen-tications, etc., to enhance the privacy protection. However, these approaches are vulnerable to replay attacks. Although state-of-art solutions utilize liveness verification to combat the attacks, existing approaches are sensitive to ambient environments, such as ambient lights and surrounding audible noises. Towards this end, we explore liveness verification of user authentication leveraging users' lip movements, which are robust to noisy environments. In this paper, we propose a lip reading-based user authentication system, LipP ass, which extracts unique behavioral characteristics of users' speaking lips leveraging build-in audio devices on smartphones for user authentication. We first investigate Doppler profiles of acoustic signals caused by users' speaking lips, and find that there are unique lip movement patterns for different individuals. To characterize the lip movements, we propose a deep learning-based method to extract efficient features from Doppler profiles, and employ Support Vector Machine and Support Vector Domain Description to construct binary classifiers and spoofer detectors for user identification and spoofer detection, respectively. Afterwards, we develop a binary tree-based authentication approach to accurately identify each individual leveraging these binary classifiers and spoofer detectors with respect to registered users. Through extensive experiments involving 48 volunteers in four real environments, LipP ass can achieve 90.21% accuracy in user identification and 93.1% accuracy in spoofer detection.

You may want to know: