Immersive hybrid real-time video communication using mixed camera setups

M. Zepp; D. Chen; A. Hilsmann; P. Eisert; R. Schäfer; K. Prebeck; O. Schreer; I. Feldmann

doi:10.1117/12.3022323

20 June 2024 Immersive hybrid real-time video communication using mixed camera setups

M. Zepp, D. Chen, A. Hilsmann, P. Eisert, R. Schäfer, K. Prebeck, O. Schreer, I. Feldmann

Proceedings Volume 13000, Real-time Processing of Image, Depth, and Video Information 2024; 130000D (2024) https://doi.org/10.1117/12.3022323
Event: SPIE Photonics Europe, 2024, Strasbourg, France

Abstract

With the increase of hybrid meetings, innovative strategies are needed to enable both local and remote participants to interact and communicate in the same way as in real meetings. In the context of a standing table scenario, we present an end-to-end real-time 3D video processing workflow and demonstrator setup. This system offers on a single display two different perspectives of a remote participant for two local participants. The system consists of a 3D capture setup and processing workflow at the remote site and a novel dual-view display on the local site. The standing table scenario requires highly realistic, high resolution rendering quality on the receiving side. Hence, low-resolution depth cameras and neural network-based depth enhancement in conjunction with 8K cameras are used to achieve high quality perspectively corrected views, allowing direct eye contact for individual users. Distinct real-time view rendering is achieved without holes to provide high-quality novel views. On the local site, a novel dual-view display is used that is capable to present two different perspectives of the remote participant at two different positions in space. The display is based on a lenticular lens that is specifically designed for our use case. The prototype setup is optimized towards low latency processing and transmission in order to fulfill the constraints given from our video communication scenario.

Conference Presentation

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

M. Zepp, D. Chen, A. Hilsmann, P. Eisert, R. Schäfer, K. Prebeck, O. Schreer, and I. Feldmann "Immersive hybrid real-time video communication using mixed camera setups", Proc. SPIE 13000, Real-time Processing of Image, Depth, and Video Information 2024, 130000D (20 June 2024); https://doi.org/10.1117/12.3022323

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available