Semantic Apparatus – Towards real-time photorealistic 3D holography with deep neural networks

Cited by Lee Sonogan

A Closer Look at AI: Neural Networks and Deep Learning | Brighterion AI | A  Mastercard Company

Abstract by Liang Shi,Beichen Li,Changil Kim,Petr Kellnhofer &Wojciech Matusik 

The ability to present three-dimensional (3D) scenes with continuous depth sensation has a profound impact on virtual and augmented reality, human–computer interaction, education and training. Computer-generated holography (CGH) enables high-spatio-angular-resolution 3D projection via numerical simulation of diffraction and interference1. Yet, existing physically based methods fail to produce holograms with both per-pixel focal control and accurate occlusion2,3. The computationally taxing Fresnel diffraction simulation further places an explicit trade-off between image quality and runtime, making dynamic holography impractical4. Here we demonstrate a deep-learning-based CGH pipeline capable of synthesizing a photorealistic colour 3D hologram from a single RGB-depth image in real time. Our convolutional neural network (CNN) is extremely memory efficient (below 620 kilobytes) and runs at 60 hertz for a resolution of 1,920 × 1,080 pixels on a single consumer-grade graphics processing unit. Leveraging low-power on-device artificial intelligence acceleration chips, our CNN also runs interactively on mobile (iPhone 11 Pro at 1.1 hertz) and edge (Google Edge TPU at 2.0 hertz) devices, promising real-time performance in future-generation virtual and augmented-reality mobile headsets. We enable this pipeline by introducing a large-scale CGH dataset (MIT-CGH-4K) with 4,000 pairs of RGB-depth images and corresponding 3D holograms. Our CNN is trained with differentiable wave-based loss functions5 and physically approximates Fresnel diffraction. With an anti-aliasing phase-only encoding method, we experimentally demonstrate speckle-free, natural-looking, high-resolution 3D holograms. Our learning-based approach and the Fresnel hologram dataset will help to unlock the full potential of holography and enable applications in metasurface design6,7, optical and acoustic tweezer-based microscopic manipulation8,9,10, holographic microscopy11 and single-exposure volumetric 3D printing12,13.

Publication: Nature (Peer-Reviewed Journal and some affiliations)

Pub Date: DOI: https://doi.org/10.1038/s41586-020-03152-0

Keywords: Computational science Computational science Displays Optical manipulation and tweezers

https://www.nature.com/articles/s41586-020-03152-0 (More sections, figures and refferences)

https://entertainmentcultureonline.com/

https://ungroovygords.com/

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.