Tuesday, August 22nd 2023
NVIDIA Announces DLSS 3.5 Ray Reconstruction Technology, Works on GeForce 20 and Newer
At this year's Gamescom 2023, NVIDIA will be releasing several new updates for the GeForce Gaming crowd. These are led by the announcement of the new DLSS 3.5 Ray Reconstruction feature, available this Fall for all GeForce RTX GPUs (RTX 20-series and later). DLSS 3.5 introduces a new feature called "Ray Reconstruction Technology" that's specifically designed to improve the way ray traced elements look in games. While traditional rasterization calculates every single pixel, for each frame, real-time ray tracing cannot do that, for performance reasons. During rendering, only few rays are shot in a coarse grid, which leaves empty "black" gaps in-between the ray outputs. To fill those in, a denoiser is used that runs various algorithms to literally fill in the blanks.
With DLSS 3.5, NVIDIA introduces a new denoiser that's optimized to work hand-in-hand with DLSS 2 upscaling, to provide better image quality results that are more correct at the same time. This feature relies on the Tensor Cores (not the RT cores, we asked), so it is available on all GeForce RTX graphics cards (Turing and newer).The picture below shows the traditional way to do RT effects. Please note that DLSS 2 upscaling is enabled here—the image is composited at low resolution first and then scaled to native size.In a first step, the engine creates the geometry and materials, but without any shading. This information is used to create the BVH acceleration structure for ray tracing, which helps to determine where rays intersect with world geometry. Next, a number of rays is cast and their path is traced, to calculate intersections, possibly let them bounce, maybe even several times. These results are now fed to the denoiser, which turns the individual pixels into a continuous image that looks like a ray traced reflection, shadow, lighting or ambient occlusion. With upscaling enabled, the denoiser generates output at the lower render resolution, not the final native output—the denoiser isn't even aware of the final resolution. On top of that, another problem is that the upscaler doesn't know anything about rays, it just sees the pixel output from the denoiser—all the original ray tracing values are lost at that stage.The biggest problem with denoisers is that they rely on previous frames, to "collect" enough pixel data for the final image. This means that the RT output is an average of several previous frames. The slide above details such problematic cases. For example, the mirror on a moving car gets combined throughout several frames, which results in ghosting artifacts. Another problem is with subtle illumination effects and reflections that just look smeared out.NVIDIA's innovation with DLSS 3.5 is that they are combining both the denoising and the upscaling steps into a single combined step that has more information available, which promises a higher-quality output image. The low-res output is combined with the output from rasterization, the ray tracing steps and the motion vectors, and everything is painted directly into a high-res output image, 4K in this case. The DLSS 3.5 algorithm also takes into account previous frames (temporal feedback), just like DLSS 2. Once upscaling is completed, another pass is made for the DLSS 3 Frame Generation feature (when enabled).Here's some results provided by NVIDIA that show how DLSS 3.5 Ray Reconstruction promises to enhance the RT fidelity over classic denoising techniques.Ray Reconstruction has negligible performance cost of its own, on frame-rate comparisons NVIDIA showed taken on an RTX 40-series GPU, DLSS 3.5 RR offers marginally higher frame-rates than DLSS 3 FG. NVIDIA made it clear that DLSS 3.5 is not a performance enhancing feature, but the focus is on image quality. Depending on the scene, the performance will be virtually identical, slightly better or slightly worse. In theory it is possible that game developers reduce the number of rays when DLSS 3.5 is enabled, which would lower the RT performance hit, and improve framerates—still with improved image quality. There's no handholding for that though, this is purely a game dev feature and out of the scope of NVIDIA's DLSS 3.5 implementation.DLSS 3.5 will not only be available in games, but also in NVIDIA's professional D5 renderer, where it will enable real-time previews of stunning detail.When it releases this Fall, DLSS 3.5 will be enabled on all GeForce RTX GPUs through a driver update. You now have three distinct subsets of DLSS—Super Resolution (SR), or the core image upscaling tech; Frame Generation (FG) introduced with DLSS 3, which doubles frame-rates by generating alternate frames using AI; and now the new Ray Reconstruction (RR) feature. DLSS 3.5 RR will work with all RTX GPUs, as all generations include tensor cores. On older RTX 20-series "Turing" and RTX 30-series "Ampere," DLSS 3.5 will work exactly like it does on the latest RTX 40-series "Ada," but FG won't be available. Games with support for Ray Reconstruction will have an additional checkbox "enable Ray Reconstruction", just like there's a checkbox "enable Frame Generation". We confirmed with NVIDIA that running DLAA with Ray Reconstruction is supported—you don't have to use the upscaler at all times.
While the naming is a bit confusing, it's great to see that NVIDIA is constantly improving their technology. There's no news yet regarding AMD's FSR 3; perhaps an announcement might come at Gamescom. However, from a technical standpoint, we'd classify Ray Reconstruction as "DLSS 2.5", because it has absolutely nothing to do with DLSS 3 Frame Generation, and is closely interlinked with DLSS 2 upscaling. It seems NVIDIA is now releasing all similar technologies under their established "DLSS" brand name, which is further segregated by feature. For example, "DLSS 3 Frame Generation" is only supported on GeForce 40—this announcement does not change that. The new "DLSS 3 Ray Reconstruction" works on GeForce 20 and newer though, just like "DLSS 2 Upscaling" works on GeForce 20, too.
With DLSS 3.5, NVIDIA introduces a new denoiser that's optimized to work hand-in-hand with DLSS 2 upscaling, to provide better image quality results that are more correct at the same time. This feature relies on the Tensor Cores (not the RT cores, we asked), so it is available on all GeForce RTX graphics cards (Turing and newer).The picture below shows the traditional way to do RT effects. Please note that DLSS 2 upscaling is enabled here—the image is composited at low resolution first and then scaled to native size.In a first step, the engine creates the geometry and materials, but without any shading. This information is used to create the BVH acceleration structure for ray tracing, which helps to determine where rays intersect with world geometry. Next, a number of rays is cast and their path is traced, to calculate intersections, possibly let them bounce, maybe even several times. These results are now fed to the denoiser, which turns the individual pixels into a continuous image that looks like a ray traced reflection, shadow, lighting or ambient occlusion. With upscaling enabled, the denoiser generates output at the lower render resolution, not the final native output—the denoiser isn't even aware of the final resolution. On top of that, another problem is that the upscaler doesn't know anything about rays, it just sees the pixel output from the denoiser—all the original ray tracing values are lost at that stage.The biggest problem with denoisers is that they rely on previous frames, to "collect" enough pixel data for the final image. This means that the RT output is an average of several previous frames. The slide above details such problematic cases. For example, the mirror on a moving car gets combined throughout several frames, which results in ghosting artifacts. Another problem is with subtle illumination effects and reflections that just look smeared out.NVIDIA's innovation with DLSS 3.5 is that they are combining both the denoising and the upscaling steps into a single combined step that has more information available, which promises a higher-quality output image. The low-res output is combined with the output from rasterization, the ray tracing steps and the motion vectors, and everything is painted directly into a high-res output image, 4K in this case. The DLSS 3.5 algorithm also takes into account previous frames (temporal feedback), just like DLSS 2. Once upscaling is completed, another pass is made for the DLSS 3 Frame Generation feature (when enabled).Here's some results provided by NVIDIA that show how DLSS 3.5 Ray Reconstruction promises to enhance the RT fidelity over classic denoising techniques.Ray Reconstruction has negligible performance cost of its own, on frame-rate comparisons NVIDIA showed taken on an RTX 40-series GPU, DLSS 3.5 RR offers marginally higher frame-rates than DLSS 3 FG. NVIDIA made it clear that DLSS 3.5 is not a performance enhancing feature, but the focus is on image quality. Depending on the scene, the performance will be virtually identical, slightly better or slightly worse. In theory it is possible that game developers reduce the number of rays when DLSS 3.5 is enabled, which would lower the RT performance hit, and improve framerates—still with improved image quality. There's no handholding for that though, this is purely a game dev feature and out of the scope of NVIDIA's DLSS 3.5 implementation.DLSS 3.5 will not only be available in games, but also in NVIDIA's professional D5 renderer, where it will enable real-time previews of stunning detail.When it releases this Fall, DLSS 3.5 will be enabled on all GeForce RTX GPUs through a driver update. You now have three distinct subsets of DLSS—Super Resolution (SR), or the core image upscaling tech; Frame Generation (FG) introduced with DLSS 3, which doubles frame-rates by generating alternate frames using AI; and now the new Ray Reconstruction (RR) feature. DLSS 3.5 RR will work with all RTX GPUs, as all generations include tensor cores. On older RTX 20-series "Turing" and RTX 30-series "Ampere," DLSS 3.5 will work exactly like it does on the latest RTX 40-series "Ada," but FG won't be available. Games with support for Ray Reconstruction will have an additional checkbox "enable Ray Reconstruction", just like there's a checkbox "enable Frame Generation". We confirmed with NVIDIA that running DLAA with Ray Reconstruction is supported—you don't have to use the upscaler at all times.
While the naming is a bit confusing, it's great to see that NVIDIA is constantly improving their technology. There's no news yet regarding AMD's FSR 3; perhaps an announcement might come at Gamescom. However, from a technical standpoint, we'd classify Ray Reconstruction as "DLSS 2.5", because it has absolutely nothing to do with DLSS 3 Frame Generation, and is closely interlinked with DLSS 2 upscaling. It seems NVIDIA is now releasing all similar technologies under their established "DLSS" brand name, which is further segregated by feature. For example, "DLSS 3 Frame Generation" is only supported on GeForce 40—this announcement does not change that. The new "DLSS 3 Ray Reconstruction" works on GeForce 20 and newer though, just like "DLSS 2 Upscaling" works on GeForce 20, too.