Nvidia Showcases Incredible Instant NeRF 2D to 3D Photo AI Processing


Nvidia researchers have developed an approach to reconstructing a 3D scene from a handful of 2D images “almost instantly.” A new blog post describes the feat, which leverages a popular new technology called neural radiance fields (NeRF), which is accelerated up to 1,000x compared to rival implementations. Nvidia’s processing speed is largely due to its AI acceleration leveraging Tensor Cores which speed up both model training and scene rendering. If you are interested but want a TLDR, take a peek at the short video embedded directly below.

Providing some context to its demo, Nvidia says that previous NeRF techniques could take hours to train for a scene and then minutes to render target scenes. Though the results of previous slower implementations were good, Nvidia researchers leveraging AI technology have put a rocket into the performance, and hence Nvidia has the confidence to describe its tech as “Instant NeRF.”

(Image credit: Nvidia)

You probably have already guessed, but this NeRF tech uses neural networks to represent and render realistic 3D scenes based on an input collection of 2D images. The above video implies that just four snaps were required to create the 3D representation we see in motion. However, the blog might be more realistic in explaining the “the neural network requires a few dozen images taken from multiple positions around the scene, as well as the camera position of each of those shots.” The neural network fills in the blanks of the full 360-degree scene and can predict the color of light radiating in any direction, from any point in 3D space for added realism. Nvidia says the technique can work around occlusions.