When the Ventral Visual Stream is Not Enough: A Deep Learning Account of Medial Temporal Lobe Involvement in Perception

The medial temporal lobe (MTL) supports a constellation of memory-related behaviors. Its involvement in perceptual processing, however, has been subject to an enduring debate. This debate centers on perirhinal cortex (PRC), an MTL structure at the apex of the ventral visual stream (VVS). Here we leverage a deep learning framework that approximate visual behaviors supported by the VVS---i.e. lacking PRC. We first apply this approach retroactively, modeling 30 published concurrent visual discrimination experiments: Excluding non-diagnostic stimulus sets, there is a striking correspondence between VVS-modeled and PRC-lesioned behavior, while each are outperformed by PRC-intact participants. We corroborate these results with a novel experiment directly comparing PRC-intact performance with electrophysiological recordings from the macaque VVS. Finally, in silico experiments suggest that simple modifications to VVS-like models do not enable PRC-supported performance. By situating lesion, electrophysiological, and behavioral results within a shared computational framework, this work resolves decades of seemingly inconsistent experimental findings surrounding PRC involvement in perception.