Depth-guided deep video inpainting

Li, S., Zhu, S., Ge, Y., Zeng, B., Imran, M. A. , Abbasi, Q. H. and Cooper, J. (2024) Depth-guided deep video inpainting. IEEE Transactions on Multimedia, 26, pp. 5860-5871. (doi: 10.1109/TMM.2023.3340089)

Abstract

Video inpainting aims to fill in missing regions of a video after any undesired contents are removed from it. This technique can be applied to repair the broken video or edit the video content. In this paper, we propose a depth-guided deep video inpainting network (DGDVI) and demonstrate its effectiveness in processing challenging broken areas crossing multiple depth layers. To achieve our goal, we divide the inpainting into depth completion, content reconstruction, and content enhancement. Three corresponding modules are designed to implement a process-flow. Firstly, we develop a depth completion module based upon the spatio-temporal Transformer which is used to obtain the completed depth information for each video frame. Secondly, we design a content reconstruction module to generate initially inpainted video. With this module, the contents of the missing regions are composed via the depth-guided feature propagation. Thirdly, we construct a content enhancement module to enhance the temporal coherence and texture quality for the inpainted video. All of proposed modules are jointly optimized to guarantee the high inpainting efficiency. The experimental results demonstrate that our proposed method provides better inpainting results, both qualitatively and quantitatively, compared with the previous state-of-the-art. The code is available at https://github.com/lishibo888/DGDVI .

• Item Type: Articles
• Additional Information: This work was supported by the National Natural Science Foundation of China under Grant U20A20184, Natural Science Foundation of Sichuan Province under Grant 2023NSFSC1972, and CAAI-Huawei MindSpore Open Fund under Grant CAAIXSJLJJ-2022-060A.
• Keywords: Video inpainting, depth completion, depth-guided content reconstruction, content enhancement.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Cooper, Professor Jonathan and Imran, Professor Muhammad and Ge, Yao and Li, Shibo and Abbasi, Professor Qammer
• Authors: Li, S., Zhu, S., Ge, Y., Zeng, B., Imran, M. A., Abbasi, Q. H., and Cooper, J.
• College/School: College of Science and Engineering; College of Science and Engineering > School of Engineering > Autonomous Systems and Connectivity; College of Science and Engineering > School of Engineering > Biomedical Engineering; College of Science and Engineering > School of Engineering > Electronics and Nanoscale Engineering
• Journal Name: IEEE Transactions on Multimedia
• Publisher: IEEE
• ISSN: 1520-9210
• ISSN (Online): 1941-0077
• Published Online: 06 December 2023
• Copyright Holders: Copyright © 20223 IEEE
• First Published: First published in IEEE Transactions on Multimedia 26:5860-5871
• Publisher Policy: Reproduced in accordance with the publisher copyright policy