Tutorials Game, Mobile

Using Variable Rate Shading to Improve Performance on Mobile Games

Søren Klit Lambæk

Developer Relations Engineer

In this article I would like to introduce a hardware optimisation technique called Variable Rate Shading (VRS) and how this technique can benefit games on mobile phones.

Introduction

Traditionally, each shaded pixel in a rendered image is being shaded individually, meaning we can shade very high details anywhere in the image which, in theory, is great. However, in practice, this can lead to wasteful GPU calculations for areas where details are less important.

In some cases, you do not need 1x1 shading of pixels to produce a high quality image. For example, for those areas that represent unlit surfaces caused by shadows naturally contain less details than brighter lit areas.

Moreover, areas which are out of focus due to camera post-effects and areas affected by motion blur naturally do not contain high details. In these cases we could benefit from letting multiple pixels be shaded by just a single calculation (like a 2x2 or 4x4 area of pixels) without losing any noticeable visual quality.

The high resolution sky texture on the left looks very much like the lower resolution sky texture on the right. This is due to the smooth colour gradients and lack of high frequency colour variation. For those reasons, there is room for a lot of optimisation.

You could argue that optimisation for handheld devices, like mobile phones, is more essential than on stationary devices, like games consoles, due to a couple of reasons.

Firstly, the hardware on handheld devices is often less powerful than conventional hardware due to smaller size and less electrical power supply. The compact size of the hardware for handheld devices are also the reason why they are more likely to suffer from temperature issues causing thermal throttling, where the performance slows down significantly.

Secondly, heavy graphics in games can quickly drain your phone's battery life. So, it is crucial to keep the GPU resources to a minimum when possible. Variable Rate Shading is a way to help doing just that.

How does Variable Rate Shading work?

In principle, Variable Rate Shading is actually a very simple method which can be implemented without having to redesign an existing rendering pipeline.

There are three ways to define areas to be optimised using Variable Rate Shading:

Let an attachment in the form of an image serve as a mask.
Execute the optimisation on a per-triangle basis.
Let the VRS optimisation be based on a per-draw call.

Use an attachment as a mask

You can provide the GPU with an image that serves as a mask. The mask contains information about what areas need to be rendered in a traditional manner by shading each pixel individually, and which areas need to be optimised by shading a group of pixels at once.

The image below is visualising such a mask by colour-coding different areas:

The blue area does not have any optimisation applied (1x1) as this area is where the player focuses on when driving.
The green area is optimised by shading four pixels (2x2) by only one shading calculation, as this area contains less details due to motion blur.
The red area can be optimised even more (4x4), as it is affected by a more aggressive motion blur.
The yellow and purple areas are also shaded with less shading calculations.

The areas defined in the image above could be static, at least while the player is driving the boat at top speed, as the boat is positioned at the centre of the image at all times. However, the level of optimisation could be reduced when another boat is passing by or when the boat slows down and therefore the motion blur is gradually reduced.

There are times where a more dynamic approach is needed, as it sometimes can be difficult to know beforehand what areas should be optimised and what areas should be shaded in a traditional manner.

In those cases, it could be beneficial to generate the mask more dynamically by rendering the geometry for the scene in an extra pass. Simply colour the geometric elements in the scene and pass it to the GPU as a mask for the Variable Rate Shading optimisation.

If the scene is rendered by using deferred lighting, an extra pass may not be needed as the mask could be based on the default geometry pass required for deferred shading.

Optimisation based on primitives

Another way of using Variable Rate Shading is taking advantage of other extensions as they allow you to define geometric elements to be optimised rather than using a mask. This can be done on a per-triangle basis or simply done by a per-draw call. Defining geometric elements could be a more efficient approach as there is no need for generating a mask as well as needing less memory bandwidth. For the per-triangle basis extension, you are able to define the optimisation level in the vertex shader. For the per-draw call method, the optimisation level can be defined before the draw call takes place. Keep in mind that the three methods can be combined if needed.

The image below is a rendering pass where all objects in a scene are shaded in different colours to define what areas should be shaded in a traditional manner (meaning no optimisation) and what areas contain less details (therefore needing less GPU calculations).

The areas defined above can be defined by all three methods. In general, by breaking a scene up in layers, where the elements nearest the camera have less optimisation and layers in the background have the most optimisation, would be an effective way to go about it.

The image below shows the same scene, but this time we see the final output where VRS is on and off.

As you may have noticed, it is very hard to tell any difference when the VRS optimisation is turned on or off.

Experiences with Variable Rate Shading so far

Some commercial games have already successfully implemented Variable Rate Shading. The image below is from Wolfenstein Young Blood.

As you may have noticed, there is barely any visual difference when VRS is on or off, but you are able to tell a difference in frame rate.

In fact, the game performs, on average, 10% or higher when VRS is turned on.

That may not sound like a lot, but considering that it is an easy optimisation to implement, there is barely any noticeable change in the visual quality. The 10% performance boost is on top of other optimisation techniques and it is actually not a bad performance boost after all.

Other games have shown an even a higher performance boost. For example, Gears Tactics has a performance boost up to 30% when using Variable Rate Shading. The image below is from that game.

Virtual Reality

Variable Rate Shading can benefit Virtual Reality as well. Not only does Virtual Reality by nature require two rendered images (one image for each eye), but the player who wears the virtual mask naturally pays most attention to the central area of the rendered image.

The areas of the rendered image that are seen from the corner of your eye naturally do not need the same amount of details as the central area of the rendered images. That means even though a static VRS mask can be used for a reasonable overall optimisation, using an eye tracker could result in an even more efficient optimisation and therefore less noticeable quality reduction.

It is crucial to have a consistent high frame rate for Virtual Reality. If the frame rate is not relatively consistent or the rendering performance is suffering for a consistent low frame rate, it quickly gets uncomfortable to wear a VR headset and the player might even get dizzy and feel physically sick.

By reducing the GPU calculations, using Variable Rate Shading not only boosts the frame rate, it also uses less battery for mobile devices. This is a huge win for systems like Samsung Gear VR where a long battery life is much appreciated as the graphics are running on a Galaxy mobile phone.

The image below shows a Variable Rate Shading mask generated by eye tracking technology for a Virtual Reality headset. The centre of the left and right images shade pixels in a traditional manner. The other colours represent different degrees of optimisation areas.

Which Samsung devices support Variable Rate Shading?

All hardware listed here supports Variable Rate Shading.

Mobile phones: Samsung Galaxy S22, S22+ and S22 Ultra
Tablets: Samsung Tab S8, S8+ and S8 Ultra

The following graphics APIs, Vulkan and OpenGL ES 2.0 (and higher), both support Variable Rate Shading.

The OpenGL extensions for the three ways of using Variable Rate Shading are the following:

GL_EXT_fragment_shading_rate_attachment for allowing to send a mask to the GPU.
GL_EXT_fragment_shading_rate_primitive for per-triangle basis, where writing a value to gl_PrimitiveShadingRateEXT in the vertex shader defines the level of optimisation.
GL_EXT_fragment_shading_rate for per-draw call, where glShadingRateEXT should be called to define the optimisation level.

The extension that enables Variable Rate Shading for Vulkan is VK_KHR_fragment_shading_rate.

Conclusion

In this article, we have established the following:

Variable Rate Shading is a hardware feature and is fairly easy to implement as it does not require any redesign of existing rendering pipelines.
Variable Rate Shading is an optimisation technique which reduces GPU calculations by allowing a group of pixels to be shaded by the same colour rather than each pixel individually.
Variable Rate Shading is particularly useful for mobile gaming as well as Samsung Gear VR, as it boosts performance and prolongs battery life.
The level of optimisation can be defined by passing a mask to the GPU that contains areas of different optimisation levels.
Some implementations have proven to boost the framerate 10% or higher, while other implementations manage to increase the frame rate up to 30%.

NOTE: Some images in this post are courtesy of UL Solutions.

Additional resources on the Samsung Developers site

The Samsung Developers site has many resources for developers looking to build for and integrate with Samsung devices and services. Stay in touch with the latest news by creating a free account and subscribing to our monthly newsletter. Visit the Marketing Resources page for information on promoting and distributing your apps. Finally, our Developer Forum is an excellent way to stay up-to-date on all things related to the Galaxy ecosystem.

Using Variable Rate Shading to Improve Performance on Mobile Games

Introduction