Check out NVIDIA's new neural network that estimates albedo, normals, depth, and the HDR lighting volume from a single image and allows to insert 3D objects with consistent lighting and shadows into 2D images.
Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz from NVIDIA have presented a cool neural network that can insert 3D objects into 2D images with realistic lighting. Inspired by classic volume rendering techniques, they proposed a novel Volumetric Spherical Gaussian representation for lighting, which parameterizes the exitant radiance of the 3D scene surfaces on a voxel grid. They designed a physics-based differentiable renderer that utilizes the 3D lighting representation and formulates the energy-conserving image formation process that enables joint training of all intrinsic properties with the re-rendering constraint. Their model ensures physically correct predictions and avoids the need for ground-truth HDR lighting which is not easily accessible.
"From a single image, our model jointly estimates albedo, normals, depth, and the HDR lighting volume," comments the team. "Our method predicts continuous HDR 3D spatially-varying lighting, which is critical in producing high-quality object insertion with realistic cast shadows and high-frequency details."