A self-supervised deep neural network for image completion resembles early visual cortex fMRI activity patterns for occluded scenes

Svanera, M. , Morgan, A. T., Petro, L. S. and Muckli, L. (2021) A self-supervised deep neural network for image completion resembles early visual cortex fMRI activity patterns for occluded scenes. Journal of Vision, 21(7), 5. (doi: 10.1167/jov.21.7.5) (PMID:34259828)

[img] Text
243799.pdf - Published Version
Available under License Creative Commons Attribution.

9MB

Abstract

The promise of artificial intelligence in understanding biological vision relies on the comparison of computational models with brain data with the goal of capturing functional principles of visual information processing. Convolutional neural networks (CNN) have successfully matched the transformations in hierarchical processing occurring along the brain’s feedforward visual pathway extending into ventral temporal cortex. However, we are still to learn if CNNs can successfully describe feedback processes in early visual cortex. Here, we investigated similarities between human early visual cortex and a CNN with encoder/decoder architecture, trained with self-supervised learning to fill occlusions and reconstruct an unseen image. Using Representational Similarity Analysis (RSA), we compared 3T fMRI data from a non-stimulated patch of early visual cortex in human participants viewing partially occluded images, with the different CNN layer activations from the same images. Results show that our self-supervised image-completion network outperforms a classical object-recognition supervised network (VGG16) in terms of similarity to fMRI data. This provides additional evidence that optimal models of the visual system might come from less feedforward architectures trained with less supervision. We also find that CNN decoder pathway activations are more similar to brain processing compared to encoder activations, suggesting an integration of mid- and low/middle-level features in early visual cortex. Challenging an AI model and the human brain to solve the same task offers a valuable way to compare CNNs with brain data and helps to constrain our understanding of information processing such as neuronal predictive coding.

Item Type:Articles
Additional Information:Please visit the project website for more information: https://rocknroll87q.github.io/self-supervised_inpainting/
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Svanera, Dr Michele and Morgan, Mr Andrew and Petro, Dr Lucy and Muckli, Professor Lars
Authors: Svanera, M., Morgan, A. T., Petro, L. S., and Muckli, L.
Subjects:R Medicine > RC Internal medicine > RC0321 Neuroscience. Biological psychiatry. Neuropsychiatry
College/School:College of Medical Veterinary and Life Sciences > Institute of Neuroscience and Psychology
College of Science and Engineering > School of Psychology
Journal Name:Journal of Vision
Journal Abbr.:JoV
Publisher:Association for Research in Vision and Ophthalmology
ISSN:1534-7362
ISSN (Online):1534-7362
Published Online:14 July 2021
Copyright Holders:Copyright © 2021 The Authors
First Published:First published in Journal of Vision 21(7): 5
Publisher Policy:Reproduced under a Creative Commons licence
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record

Project CodeAward NoProject NamePrincipal InvestigatorFunder's NameFunder RefLead Dept
172779Human Brain ProjectLars MuckliEuropean Commission (EC)720270NP - Centre for Cognitive Neuroimaging (CCNi)
304518Human Brain Project SGA 2Lars MuckliEuropean Commission (EC)785907NP - Centre for Cognitive Neuroimaging (CCNi)