Svanera, M. , Morgan, A. T., Petro, L. S. and Muckli, L. (2021) A self-supervised deep neural network for image completion resembles early visual cortex fMRI activity patterns for occluded scenes. Journal of Vision, 21(7), 5. (doi: 10.1167/jov.21.7.5) (PMID:34259828)
![[img]](https://eprints.gla.ac.uk/style/images/fileicons/text.png) |
Text
243799.pdf - Published Version Available under License Creative Commons Attribution. 9MB |
Abstract
The promise of artificial intelligence in understanding biological vision relies on the comparison of computational models with brain data with the goal of capturing functional principles of visual information processing. Convolutional neural networks (CNN) have successfully matched the transformations in hierarchical processing occurring along the brain’s feedforward visual pathway extending into ventral temporal cortex. However, we are still to learn if CNNs can successfully describe feedback processes in early visual cortex. Here, we investigated similarities between human early visual cortex and a CNN with encoder/decoder architecture, trained with self-supervised learning to fill occlusions and reconstruct an unseen image. Using Representational Similarity Analysis (RSA), we compared 3T fMRI data from a non-stimulated patch of early visual cortex in human participants viewing partially occluded images, with the different CNN layer activations from the same images. Results show that our self-supervised image-completion network outperforms a classical object-recognition supervised network (VGG16) in terms of similarity to fMRI data. This provides additional evidence that optimal models of the visual system might come from less feedforward architectures trained with less supervision. We also find that CNN decoder pathway activations are more similar to brain processing compared to encoder activations, suggesting an integration of mid- and low/middle-level features in early visual cortex. Challenging an AI model and the human brain to solve the same task offers a valuable way to compare CNNs with brain data and helps to constrain our understanding of information processing such as neuronal predictive coding.
| Item Type: | Articles |
|---|---|
| Additional Information: | Please visit the project website for more information: https://rocknroll87q.github.io/self-supervised_inpainting/ |
| Status: | Published |
| Refereed: | Yes |
| Glasgow Author(s) Enlighten ID: | Morgan, Mr Andrew and Petro, Dr Lucy and Muckli, Professor Lars and Svanera, Dr Michele |
| Authors: | Svanera, M., Morgan, A. T., Petro, L. S., and Muckli, L. |
| Subjects: | R Medicine > RC Internal medicine > RC0321 Neuroscience. Biological psychiatry. Neuropsychiatry |
| College/School: | College of Medical Veterinary and Life Sciences > School of Psychology & Neuroscience College of Science and Engineering > School of Psychology |
| Journal Name: | Journal of Vision |
| Journal Abbr.: | JoV |
| Publisher: | Association for Research in Vision and Ophthalmology |
| ISSN: | 1534-7362 |
| ISSN (Online): | 1534-7362 |
| Published Online: | 14 July 2021 |
| Copyright Holders: | Copyright © 2021 The Authors |
| First Published: | First published in Journal of Vision 21(7): 5 |
| Publisher Policy: | Reproduced under a Creative Commons licence |
| Related URLs: |
University Staff: Request a correction | Enlighten Editors: Update this record
Altmetric
Altmetric