See and chat: automatically generating viewer-level comments on images

Image is becoming a predominant medium for social interactions. Automatically expressing opinions on an image, which we refer to as image commenting, has great potential to improve user engagement and thus becomes an emerging yet very challenging research topic. The machine-generated comments should be both relevant to image content and natural as human language. To deal with these challenges, we propose a novel two-stage approach, consisting of similar image search and comment ranking. In the first step, given an image, visually similar images are discovered by k-nearest neighbor (k-NN) search from a large image dataset. The comments associated with these images are exploited as candidates to mimic how viewers respond to this given image. In the second step, ranking canonical correlation analysis (RCCA), which is an extension of CCA by jointly learning a cross-view embedding space and a bilinear similarity function between the views of image and comment, is exploited for ranking the candidate comments. To create a benchmark for this emerging task, we collect a dataset with 426K images with 11 million associated comments. We show that our approach achieves superior performance and can suggest viewer-level comments.