What Is Neural Rendering? How It Works and Where Architects Use It

Neural rendering is the use of neural networks trained on images inside the process that turns a scene into a picture: a learned model predicts part or all of the image instead of every pixel being computed from hand-written rules. In 2026 the term covers three things above all: scenes learned from photos (NeRF and 3D Gaussian splatting), learned components in real-time engines such as NVIDIA's DLSS, and generative image models such as diffusion models.

This guide explains what the term means, how neural rendering works, how it compares with ray tracing and rasterization, the four families of methods, where architects meet them today, and what they cannot do yet.

What Is Neural Rendering?

The term comes from computer graphics research. The 2020 Eurographics survey State of the Art on Neural Rendering, by Ayush Tewari and colleagues, describes it as combining generative machine learning with physical knowledge from computer graphics, for example by building differentiable rendering into the training of a network. Its 2021 follow-up, "Advances in Neural Rendering", speaks of putting learnable components into the rendering pipeline.

There is no single settled definition. Microsoft, announcing neural rendering support in DirectX in January 2025, used the term for the whole set of techniques that use AI and machine learning to transform traditional graphics pipelines, and NVIDIA uses it for DLSS. What these uses share is that some of the picture is predicted by a model that has learned from images, rather than calculated only from the scene's geometry, materials and lights.

For architects the distinction matters in two ways. Some methods build a viewable scene from photos of a real site or building; others produce images of a design from a sketch, a model view and a sentence. Both show appearance: neither measures anything.

How Does Neural Rendering Work?

A conventional renderer computes an image from an explicit description of a scene: geometry, materials, lights and a camera, with the light worked out by equations, through rasterization and shaders or through ray and path tracing. Neural rendering replaces or supplements parts of that pipeline with neural networks whose behaviour is learned from images rather than written by hand.

The key idea is differentiability. If every step from the scene to the pixels can be differentiated, the error between a rendered image and a real photo can be passed back through the renderer, and the network's parameters, or the scene itself, are adjusted until the rendering matches the photos. This is how a NeRF learns a scene from photographs, and how inverse rendering estimates materials or lighting from pictures.

Training comes in two forms. Scene methods such as NeRF and Gaussian splatting are optimised for one scene from many photos of it taken from known camera positions. Generative models and the networks inside game and visualization engines are trained once on large image datasets and then applied to new inputs. In both cases the data decide the result: too few photos, or a dataset that lacks a kind of material or light, shows up as blur, artefacts or invented detail.

Neural Rendering vs Ray Tracing and Rasterization

Ray tracing and path tracing are older than they are often thought to be. Turner Whitted showed recursive ray tracing in 1979, and James Kajiya introduced the rendering equation and path tracing in 1986. What came later was the computing power that made them practical for everyday architectural visualization, and from 2018 hardware ray tracing on consumer graphics cards.

In practice the three are mixed rather than rivals. DLSS takes frames an engine has rasterized or ray traced and uses neural networks to upscale them, add generated frames and reconstruct ray-traced lighting; V-Ray 7 ray traces Gaussian splats next to ordinary geometry. Ray and path tracing remain the way most photorealistic renderings are made for final presentation; neural methods add learned steps where computing everything would be too slow, or where the scene exists only as photos.

Four Families of Neural Rendering

The methods called neural rendering in 2026 fall into four broad families. They solve different problems and are rarely found together in one tool.

Neural Scene Representations: NeRF and Gaussian Splatting

A neural radiance field (NeRF) stores a scene inside a neural network: given a point in space and a viewing direction, the network returns the density there and the colour seen from that direction. The original NeRF paper (Mildenhall and colleagues, ECCV 2020) optimised one scene from a set of photos with known camera positions and then rendered new views of it, a task called novel view synthesis; in that original form it does not run in real time.

3D Gaussian splatting keeps the scene as a large set of small, coloured, semi-transparent 3D Gaussians instead of a network, optimised from photos in the same differentiable way. Its 2023 paper (Kerbl and colleagues, SIGGRAPH) reported real-time novel view synthesis at 1080p, at 30 frames per second or more. How NeRF and splats compare is covered in the NeRF guide linked above.

Neural Components in Real-Time Engines: DLSS and Neural Shaders

Game and visualization engines use small networks inside an otherwise conventional pipeline. NVIDIA describes DLSS as a suite of neural rendering technologies. As of October 2026, DLSS 4.5 brings Dynamic Multi Frame Generation and a second-generation transformer model for Super Resolution and Ray Reconstruction, and DLSS 5 adds what NVIDIA calls 3D-Guided Neural Rendering on GeForce RTX 50 graphics cards, announced as available from 1 September 2026.

For developers, NVIDIA's RTX Kit groups RTX Neural Shaders, Neural Texture Compression, Neural Materials and the Neural Radiance Cache, among other tools. Microsoft announced cooperative vector support in DirectX in January 2025 to speed up the AI work of real-time rendering on graphics cards.

Differentiable and Inverse Rendering

Differentiable rendering makes the renderer part of the training: because an error in the final image can be traced back to the light direction, the reflectance of a material or the shape of an object, these values can be corrected automatically until the render matches a photo. Run the other way, the same idea becomes inverse rendering, which estimates geometry, materials and lighting from pictures.

Generative Image Models: Diffusion and GANs

The fourth family draws whole images. Diffusion models create a picture by removing noise step by step, guided by a text prompt, an input image or both; how closely the result follows the input depends on that guidance, so the geometry can change. GANs, generative adversarial networks, train a generator against a discriminator that tries to tell real images from generated ones. These models keep no 3D scene, so whether they count as neural rendering depends on the definition, but for architects they are the most visible kind.

For architects, the everyday face of image generation is the AI render tool. With AI rendering in ArchiVinci, you upload a sketch, a screenshot of a model view or a photo (JPG, PNG or WebP, up to 5 MB), describe the materials and the light in a sentence, and each run generates a new image that follows your upload, in seconds. That makes it a quick way to test light on a design: one run at a time, with a lighting button such as Golden hour or Twilight, rather than as a live preview.

Same upload and same prompt, two runs. Left: the Golden hour button. Right: the Twilight button.

The pair above was made in the Exterior module on the default engine. Twilight gives dusk with the lights on; there is no night button, so a night version is a typed request in the Edit module. The default engine, ArchiVinci 4.0, keeps the perspective and geometry of the upload; every run draws a new variation, so run it again for another option. ArchiVinci runs in your browser and needs an account; using the images for clients or marketing needs an active plan, and the plans are on the pricing page.

Where Architects Use Neural Rendering Today

In architecture, neural methods show up in four places today. None of them replaces a measured model or a render engine for the final set.

Capturing Sites and Existing Buildings

Photos or video of a site or an existing building can be turned into a NeRF or a set of Gaussian splats that you can view from new angles. Chaos writes that V-Ray 7 supports Gaussian splats, with ray tracing, parallax and view-dependent effects, so a design can be rendered in its real-world context. A capture records appearance, not dimensions: it does not replace a survey.

Real-Time Walkthroughs and Live Links

Real-time engines use neural components to keep walkthroughs smooth: AEC Magazine reported in January 2025 that D5 Render, Chaos Vantage and Unreal Engine would support DLSS 4. Live links, where a real-time renderer such as Enscape, D5 Render or Twinmotion updates the view as a Revit, Rhino or SketchUp model changes, are a different thing: the engine rasterizes or ray traces the model, and upscaling such as DLSS may speed it up, but the live update itself is not neural rendering.

Concept Images and Virtual Staging

Generative image models are where most architects meet neural methods day to day. An early sketch or a model view becomes a set of material and light options, and the options of a generative architecture and design workflow can be shown the same way. In virtual staging, a photo of an empty room is furnished for a listing; ArchiVinci's Furnish module can place appropriate furniture in such a photo, following the pieces and the style you write in its prompt, which that module requires. The default engine, ArchiVinci 4.0, keeps the perspective and geometry of your photo, so walls and windows stay where they are; say that the room is virtually staged when it goes into a listing.

Digital Twins: Capture Is Not Synchronization

NeRFs and splats can give a digital twin a realistic visual layer captured from photos or video. Keeping the twin in step with the building is the job of its data layer, with sensors, BIM and operational data, not of the rendering. NVIDIA, for example, presents Omniverse as a platform for physical AI and industrial digital twins built on OpenUSD.

Benefits and Limits of Neural Rendering

Benefits

• Views from photos: a site or a building can be seen from new angles without modelling it first.

• Smoother real-time work: upscaling and frame generation keep walkthroughs fluid on the same graphics card.

• Fast options: generative models turn a design view into material and light variations in seconds.

• Less manual tuning: learned steps take over work such as denoising, texture detail or lighting estimates that would otherwise need hand-set parameters.

Limits

• Data dependence: results are only as good as the photos or the training data. NeRFs and splats need many photos of a scene from different angles, and gaps show up as blur or floating artefacts.

• Drift from the input: a generated image can change details of the design, so check it against the drawings.

• Compute: training a scene and running neural components in real time need a capable graphics card; as of October 2026, DLSS 5 needs a GeForce RTX 50 card or NVIDIA's GeForce NOW cloud service.

• No measurement: an image, learned or ray traced, shows how a design may look. It does not measure daylight, glare or energy use; simulation tools do.

• Interpretability: a network does not explain why a pixel looks the way it does, which makes errors harder to trace.

• Authenticity: realistic synthetic images can be mistaken for photos or for a finished design, so label AI images in presentations and listings.

The Future of Neural Rendering in Architecture

The direction is hybrid. NVIDIA's RTX Kit puts learned shaders, textures and a radiance cache inside the rendering pipeline, DLSS 5 adds 3D-guided neural rendering on top of the frames an engine produces, and V-Ray 7 renders captured splats next to modelled geometry. For architects, that points to more context captured from photos, smoother real-time reviews and faster image options, rather than to one network that replaces the render engine.

What will not change soon is the check. Geometry, dimensions and performance still come from the model and from simulation; the image, however it was made, is there to show them.

Key Takeaways

• Neural rendering puts learned components into the rendering pipeline: models trained on images predict part or all of the picture.

• Its main families are neural scene representations (NeRF, 3D Gaussian splatting), neural components in real-time engines (DLSS, neural shaders), differentiable and inverse rendering, and generative image models (diffusion models, GANs).

• It complements ray tracing and rasterization rather than replacing them, and most tools mix the three.

• Only some methods run in real time: Gaussian splatting and DLSS do, the original NeRF does not.

• Architects meet it in site capture, real-time walkthroughs, concept images, virtual staging and the visual layer of digital twins.

• Generated images can drift from the input and measure nothing: check them against the drawings, use simulation for daylight and energy, and label AI images.

• AI render tools such as ArchiVinci run in the browser: upload a view, describe it, and each run generates a new image that follows the upload.

Try it in ArchiVinci

Describe a building in a sentence, or upload a sketch or a 3D model view: ArchiVinci generates concept images that follow it. A new account starts with 3 free credits.

• For early design concepts: AI concept design

Frequently Asked Questions

What does neural rendering mean?

It means making images with the help of neural networks that have learned from images, instead of computing every pixel from hand-written rules. The 2020 survey that named the field describes it as combining generative machine learning with physical knowledge from computer graphics.

Does neural rendering use machine learning?

Yes. Almost every neural rendering method contains a model trained on data, either on many photos of one scene or on a large image dataset. 3D Gaussian splatting is the usual exception: it is optimised in the same differentiable way but stores the scene as explicit splats, not inside a network.

Is DLSS neural rendering?

Yes, by NVIDIA's own description: NVIDIA calls DLSS a suite of neural rendering technologies. The engine renders frames, and neural networks upscale them, generate extra frames and reconstruct ray-traced lighting. DLSS 5, available since 1 September 2026 on GeForce RTX 50 cards, adds 3D-guided neural rendering.

What is the difference between NeRF and neural rendering?

Neural rendering is the field; NeRF is one method in it. A NeRF learns a single scene from photos and renders new views of it, while other neural rendering methods upscale engine frames, recover materials from pictures or generate whole images.

Does neural rendering replace ray tracing?

No. Ray and path tracing compute light from the scene, and they remain the standard for final architectural stills. Neural methods add learned steps, such as upscaling, denoising, new views from photos or generated images, and current tools such as DLSS and V-Ray 7 combine both.

What hardware do you need for neural rendering?

It depends on the method. Training a NeRF or Gaussian splats and running neural components in real time need a capable graphics card, and DLSS 5 needs a GeForce RTX 50 card or NVIDIA's GeForce NOW service as of October 2026. Browser AI render tools such as ArchiVinci generate the image online, so they need no local GPU. Our guide to the best GPUs for rendering compares current cards.

Can neural rendering show daylight or energy performance?

Not as an analysis. Daylight, glare and energy figures come from simulation tools that calculate them from the model. A rendered image, neural or ray traced, can illustrate a scheme or show the result of such a study, but it measures nothing itself.

What ethical challenges exist in AI-generated architectural visualization?

Ethical challenges include issues of authorship, transparency, and authenticity. Since neural renderings can produce highly realistic but synthetic images, it is crucial for professionals to clarify which visuals are AI-generated and ensure that such imagery does not misrepresent project feasibility or design intent.