Understanding and Visualizing Deep Visual Saliency Models

He, S., Tavakoli, H. R., Borji, A., Mi, Y. and Pugeault, N. (2019) Understanding and Visualizing Deep Visual Saliency Models. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15-20 Jun 2019, pp. 10198-10207. ISBN 9781728132938 (doi: 10.1109/CVPR.2019.01045)

Abstract

Recently, data-driven deep saliency models have achieved high performance and have outperformed classical saliency models, as demonstrated by results on datasets such as the MIT300 and SALICON. Yet, there remains a large gap between the performance of these models and the inter-human baseline. Some outstanding questions include what have these models learned, how and where they fail, and how they can be improved. This article attempts to answer these questions by analyzing the representations learned by individual neurons located at the intermediate layers of deep saliency models. To this end, we follow the steps of existing deep saliency models, that is borrowing a pre-trained model of object recognition to encode the visual features and learning a decoder to infer the saliency. We consider two cases when the encoder is used as a fixed feature extractor and when it is fine-tuned, and compare the inner representations of the network. To study how the learned representations depend on the task, we fine-tune the same network using the same image set but for two different tasks: saliency prediction versus scene classification. Our analyses reveal that: 1) some visual regions (e.g. head, text, symbol, vehicle) are already encoded within various layers of the network pre-trained for object recognition, 2) using modern datasets, we find that fine-tuning pre-trained models for saliency prediction makes them favor some categories (e.g. head) over some others (e.g. text), 3) although deep models of saliency outperform classical models on natural images, the converse is true for synthetic stimuli (e.g. pop-out search arrays), an evidence of significant difference between human and data-driven saliency models, and 4) we confirm that, after-fine tuning, the change in inner-representations is mostly due to the task and not the domain shift in the data.

• Item Type: Conference Proceedings
• Additional Information: This research was supported by the EPSRC project DEVA EP/N035399/1. Dr Pugeault acknowledges funding from the Alan Turing Institute (EP/N510129/1). H. R. Tavakoli acknowledges NVIDIA for the donation of GPUs used in his research.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Pugeault, Dr Nicolas
• Authors: He, S., Tavakoli, H. R., Borji, A., Mi, Y., and Pugeault, N.
• College/School: College of Science and Engineering > School of Computing Science
• Publisher: IEEE
• ISSN: 2575-7075
• ISBN: 9781728132938
• Published Online: 09 January 2020
• Copyright Holders: Copyright © 2019 IEEE
• First Published: First published in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR): 10198-10207
• Publisher Policy: Reproduced under a Creative Commons License

Related URLs

• arXiv