Anciukevicius, T., Fox-Roberts, P., Rosten, E. and Henderson, P. (2023) Unsupervised Causal Generative Understanding of Images. In: 36th Conference on Neural Information Processing Systems (NeurIPS 2022), 28th November - 9th December 2022, ISBN 9781713871088
Text
282697.pdf - Published Version 4MB |
Abstract
We present a novel framework for unsupervised object-centric 3D scene understanding that generalizes robustly to out-of-distribution images. To achieve this, we design a causal generative model reflecting the physical process by which an image is produced, when a camera captures a scene containing multiple objects. This model is trained to reconstruct multi-view images via a latent representation describing the shapes, colours and positions of the 3D objects they show. It explicitly represents object instances as separate neural radiance fields, placed into a 3D scene. We then propose an inference algorithm that can infer this latent representation given a single out-of-distribution image as input -- even when it shows an unseen combination of components, unseen spatial compositions or a radically new viewpoint. We conduct extensive experiments applying our approach to test datasets that have zero probability under the training distribution. These show that it accurately reconstructs a scene's geometry, segments objects and infers their positions, despite not receiving any supervision. Our approach significantly out-performs baselines that do not capture the true causal image generation process.
Item Type: | Conference Proceedings |
---|---|
Status: | Published |
Refereed: | Yes |
Glasgow Author(s) Enlighten ID: | Henderson, Dr Paul |
Authors: | Anciukevicius, T., Fox-Roberts, P., Rosten, E., and Henderson, P. |
College/School: | College of Science and Engineering > School of Computing Science |
ISBN: | 9781713871088 |
Copyright Holders: | Copyright © The Author(s) 2022 |
First Published: | First published in Advances in Neural Information Processing Systems 35 (NeurIPS 2022) |
Publisher Policy: | Reproduced in accordance with the publisher copyright policy |
Related URLs: |
University Staff: Request a correction | Enlighten Editors: Update this record