A Scalable Clustering Algorithm for Serendipity in Recommender Systems

High sparsity and the problem of overspecialization are challenges faced by collaborative filtering (CF) algorithms in recommender systems. In this paper, we design an approach that efficiently tackles the above problems. We address the first issue of high sparsity in CF by modifying the popular parallel seeding technique proposed by Bahmani et al. for use in a spherical setting, for the clustering of users. Experimental evaluations on highly sparse real world datasets demonstrate the better performance of our algorithm than existing Spherical K-Means algorithms. Contrary to the common belief that users are only interested in items similar to those of their previous liking, it has been well established that including serendipitous recommendations improves their satisfaction. Thus, to tackle the second problem of overspecialization, we effectuate serendipity in movie recommender systems with an end-to-end algorithm, Serendipitous Clustering for Collaborative Filtering (SC-CF) that considers diversity, unexpectedness and relevance. SC-CF takes advantage of carefully generated user profiles to then assign users to a "serendipitous cluster." The overall clustering process leverages these user profiles and improves recommendation quality through serendipity. A series of experiments on Serendipity 2018 (part of Movielens) dataset built on real user feedback has shown that SC-CF outperforms the existing popular recommendation methods like K-Means and deep learning based CF.