Object-based illumination transferring and rendering for applications of mixed reality

Xu, D., Li, Z. and Cao, Q. (2022) Object-based illumination transferring and rendering for applications of mixed reality. Visual Computer, 38(12), pp. 4251-4265. (doi: 10.1007/s00371-021-02292-2)

Abstract

In applications of augmented reality or mixed reality, rendering virtual objects in real scenes with consistent illumination is crucial for realistic visualization experiences. Prior learning-based methods reported in the literature usually attempt to reconstruct complicated high dynamic range environment maps from limited input, and rely on a separate rendering pipeline to light up the virtual object. In this paper, an object-based illumination transferring and rendering algorithm is proposed to tackle this problem within a unified framework. Given a single low dynamic range image, instead of recovering lighting environment of the entire scene, the proposed algorithm directly infers the relit virtual object. It is achieved by transferring implicit illumination features which are extracted from its nearby planar surfaces. A generative adversarial network is adopted in the proposed algorithm for implicit illumination features extraction and transferring. Compared to previous works in the literature, the proposed algorithm is more robust, as it is able to efficiently recover spatially varying illumination in both indoor and outdoor scene environments. Experiments have been conducted. It is observed that notable experiment results and comparison outcomes have been obtained quantitatively and qualitatively by the proposed algorithm in different environments. It shows the effectiveness and robustness for realistic virtual object insertion and improved realism.

• Item Type: Articles
• Additional Information: This work was supported in part by the National Natural Science Foundation of China under Grant 61801391, in part by Open Project Program of the National Laboratory of Pattern Recognition under Grant 202000025, in part by China Postdoctoral Science Foundation under Grant 2018M631193.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Cao, Dr Qi
• Authors: Xu, D., Li, Z., and Cao, Q.
• College/School: College of Science and Engineering > School of Computing Science
• Journal Name: Visual Computer
• Publisher: Springer
• ISSN: 0178-2789
• ISSN (Online): 1432-2315
• Published Online: 07 October 2021
• Copyright Holders: Copyright © 2021 The Authors
• First Published: First published in Visual Computer 38(12): 4251-4265
• Publisher Policy: Reproduced under a Creative Commons License