Why verbalization of facial features increases false positive responses on visually-similar distractors: A computational exploration of verbal overshadowing

Why verbalization of facial features increases false positive responses on visually- similar distractors: A computational exploration of verbal overshadowing Aya Hatano (hatano.aya@c.mbox.nagoya-u.ac.jp), Taiji Ueno (taijiueno7@gmail.com), Shinji Kitagami (kitagami@cc.nagoya-u.ac.jp), and Jun Kawaguchi (kawaguchijun@nagoya-u.jp) Department of Psychology, Graduate School of Environmental Studies, Nagoya University, Furo-cho, Chikusa-ku, Nagoya City, Aichi 4648601, JAPAN Abstract Verbal overshadowing refers to a phenomenon whereby verbalization of a non-verbal stimulus (e.g., he had slant eyes) impairs subsequent non-verbal recognition accuracy. In order to understand the mechanism by which this phenomenon occurs, we constructed a computational model that was trained to generate an individual-face-specific representation upon input of a noise-filtered retinotopic face (i.e., face recognition). When the model verbalized the facial features before receiving the retinotopic input, the model incorrectly recognized a new face input as one of the different, yet visually-similar, trained items (that is, a false-alarm occurred). In contrast, this recognition error did not occur without prior verbalization. Close inspection of the model revealed that verbalization changed the internal representation such that it lacked the fine-grained information necessary to discriminate visually-similar faces. This supports the view that verbalization causes unavailability/degradation of fine- grained non-verbal representations, thus impairing recognition accuracy. Keywords: verbal overshadowing; computational modeling; verbalization face recognition; Introduction Language is the principal medium for carrying out daily communications. This is still true when communicating our non-verbal experiences, such as recounting a crime scene we have witnessed, or describing the physical appearance of a criminal. Particularly, if we do not have a record of the event such as a picture or video, then conveying an eyewitness memory relies on language. A crucial question in cognitive science, therefore, is the influence of verbalization on non-verbal memory. Many studies have revealed that language has extra-communicative functions, in that it affects such cognitive functions as perception, learning, and memory. For example, in a seminal study by Schooler and Engstler-Schooler (1990), participants watched a video of a bank robbery for 30 seconds and following which half of the participants described the appearance of the bank robber. Subsequently, all of the participants were shown a line-up that consisted of the bank robber’s photo and seven distractors. Results revealed that participants who had verbalized the bank robber’s appearance were worse at recognizing the target individual than those who had not, a phenomenon known as verbal overshadowing. The procedure of these experiments can be experienced beyond an experimental setting. For example, during criminal investigations, an eyewitness may provide a statement describing the appearance of a criminal and subsequently identify them from a line-up. In such situations, it is crucial to prevent a false accusation and to examine the credibility of the eyewitness’s testimony. Therefore, it is both theoretically and practically important to clarify the mechanism by which verbal overshadowing occurs. For this purpose, we constructed a parallel- distributed processing (PDP) model to simulate the effect of verbalization on subsequent visual recognition. A closer review of the literature allows us to gain further insight into this phenomenon and therefore to establish a more specific aim for our model. First, although not all of the past studies have split the recognition scores into positive and negative trials, false alarm is sometimes more susceptible to verbalization before recognition than hit rates; that is, participants often inaccurately identify distractors as a target rather than miss a correct target (Meissner, Brigham, & Kelley, 2001). Furthermore, recognition accuracy in this study was positively correlated with accuracy of the verbal description prior to recognition. Based on these observations, Meissner et al. proposed a recording interference account that assumed verbalization rendered the representations less accurate (compared to visual representations), thus impairing subsequent visual recognition. Second, Kitagami, Sato, & Yoshikawa (2002) revealed that verbal overshadowing is also sensitive to the degree of similarity between targets and distractors (manipulated with a morphing technique). Specifically, verbalization impaired subsequent visual recognition only when distractors were highly similar to the target (using a 9-alternative choice task with a “not present” response choice), but the impairment disappeared when similarity was low. It is also worth noting that this manipulation involved a change in the distractors, but not in the target picture itself. We revisited the original data and revealed that accuracy was impaired due to the more frequent choice of a distractor (a false alarm) rather than an incorrect choice of “not present” (a miss). Schooler (2002) explained this result with the transfer inappropriate processing shift hypothesis. This hypothesis assumes that verbalization induces a processing shift from visual to verbal, and that a shift to verbal processing makes fine- grained non-verbal information about faces unavailable. This non-verbal information is crucial for discriminating the target from others (see also, Maurer, LeGrand, & Mondroch, 2002), especially in a high-similarity condition (Kitagami et al., 2002). Although Schooler’s hypothesis does not necessarily assume a correlation between recognition accuracy and verbal description accuracy (see also Kitagami