Problems with Transparency

Transparency rendering is no easy problem to handle. So far have all game engines I have tested have used object sorting and rendering back to front to handle the general case with transparency and then maybe used fixes for special cases when possible. Here Comes some of the problems discovered so far in stingray.  All the bottles are marked as transparent but with an opacity of 1.

Rendering Order
Object sorting errors
This images shows the rendering order problem, in the left image the bottle is drawn in front and in the right it is drawn behind. Here the objects are sorted on distance to the camera (the object’s position to the cameras position) so only by moving the camera it’s possible to change the rendering order. This problem can be even more complex when the object itself needs a internal rendering order to look correct.

Shadows for Transparent Objects
transparent_shadows
In this image the fully transparent (opacity 0) bottles  cast shadows that are too dark. The reason for this is the shadow maps only store a single depth for each pixel which means that there is only a “cast shadow” or “not cast shadow” for each object, while it here should cast a softer shadow since some of the light is passing through.

Distance Fog and Transparency
bottle_distance_fog
Here is the distance fog calculation not calculated correctly making the transparent object get more fog applied than the surroundings. This problem should be easily solvable but it’s a general problem for transparent objects in fog that the fog needs to be calculated for different depths of the same pixel.

Clouds and Transparency
bottles_topbottles_front

The left image is the bottles seen from above and in the right image the camera is placed on the left side of first image, viewing the bottles from the front.

Instancing and Transparency
bottles_instanced

To increase performance, objects of the same type are often instanced so they are all drawn in the same time. It can be seen as many objects are combined into a single object which means the objects needs to be sorted internally to be drawn in the correct order.

Depth of Field
field_of_depth
A shader technique which requires the depth of the objects to work is Depth of Field. As can be seen here the bottle, even at the same depth in the scene as the barrel, looks blurry, the reason is that the transparent objects don’t write any depth so the bottle uses the same depth as the background.

Transparency Physics

rendering_equationTo handle lighting in 3D rendering it’s needed to have a understanding of the physics behind light and a model of the reflection, refraction and absorption of light. The reflection is described by the BRDF and the refraction by the BTDF.

Physical Notation
Radiant flux is the amount of radiant power measured in watt (\text{W}) and denoted \Phi.
Irradiance is the incoming amount of radiant power over a surface and is measured in watt per square meter (\text{Wm}^{-2}).

    \[E(x)=\frac{d\Phi(x)}{d A(x)}\]

Radiance is the amount light that is received from a solid angle d\vec{\Phi}^\bot. This can be seen as the amount of light that is reflected, emitted, transmitted from a point on a surface and hits an observer, e.g. the eye. It’s measured in watt per solid angle per square meter (\text{Wsr}^{-1}\text{m}^{-2}).

    \[L(x, \vec{\omega})=\frac{d^2\Phi(x,\vec{\omega})}{d\vec{\Phi}\;dA^{\bot}(x)}\]

Bidirectional Reflectance Distribution Function (BRDF)
The BRDF is a function (f_r(x, \vec{\omega}_i \rightarrow \vec{\omega}_o)) which takes a point x at a surface, the incoming direction vector \vec{\omega}_i of light, the outgoing direction vector \vec{\omega}_o of light and returns the amount of light reflected in the outgoing direction. The incoming and outgoing vector must be part of a normal oriented hemisphere (\Omega).

Bidirectional Transmittance Distribution Function (BTDF)
The BTDF is closely related to BRDF  (f_t(x, \vec{\omega}_t \rightarrow \vec{\omega}_o)) but where the outgoing direction vector \vec{\omega}_o must be on a negative normal oriented hemisphere. The function gives the amount of light transmitted through the material.

Transparency Model
In real time rendering a simplified model is usually used. The BTDF function is usually set to only transmit light in the opposite direction of the incoming light. In reality the light also scatter in the transport medium, this effect is mostly ignored unless it has a large impact on the result, like in fog like effects.

The Rendering Equation
The rendering equation gives to total light going from a point of a surface into an observer. L(x\rightarrow \vec{\omega}): Outgoing radiance from the point x in the direction \vec{\omega}. L(x\leftarrow \vec{\omega}): Incoming radiance to the point x from the direction \vec{\omega}.

    \begin{align*} \underbrace{L(x\rightarrow \vec{\omega}_o)}_\text{outgoing} &= \underbrace{L_e(x\rightarrow \vec{\omega}_o)}_\text{emitted} + \underbrace{L_r(x\rightarrow \vec{\omega}_o)}_\text{reflected} + \underbrace{L_t(x\rightarrow \vec{\omega}_o)}_{transmitted}\\ L_r(x\rightarrow \vec{\omega}_o) &= \int_{\Omega} f_r(x, \vec{\omega}_i \rightarrow \vec{\omega}_o)\;L(x\leftarrow \vec{\omega}_i)\;(\vec{n} \bullet \vec{\omega}_i)\;d\vec{\omega}_i\\ L_t(x\rightarrow \vec{\omega}_o) &= \int_{-\Omega} f_t(x, \vec{\omega}_t \rightarrow \vec{\omega}_o)\;L(x\leftarrow \vec{\omega}_t)\;(\vec{n} \bullet \vec{\omega}_t)\;d\vec{\omega}_t \end{align*}

The reflected and transmitted components are given by the amount of incoming light from respective hemisphere, the BRDF and BTDF functions and the angle from the normal.
rendering_equation

Modern Transparency Algorithms

To handle the problems of transparency many different algorithms have been suggested, here comes a short summery of some of them:

Painters Algorithm
The idea is to work like a painter by adding the background first, and then add the next layer, maybe some distance houses and work through all the layers till you are at the front figures. In 3D rendering this would be to sort all the objects on the distance to the camera. This algorithm has many problems as can be seen in the image below, which colored rectangle is furthest away from the camera?
painters_problem

Depth Peeling
Renders the transparent objects n times, where n is the number of transparent layers in the scene. It uses the depth buffer to find the first layer, rendering it and then “peeling” it by increasing the depth, so the next rendering pass will ignore any pixels rendered in a previous pass. Depth Peeling renders the image correctly but needs to render the scene many times.

Stochastic Algorithms
To speed up the algorithm and use a fixed amount of memory, it’s possible to use sample the transparency using a random sampling pattern. The idea is to take n samples for each pixel to be rendered and try to make the number of samples for each transparent layer proportional to how much they contribute to the final color e.g. if we have 3 layers, front (a) 75% transparent, middle (b) 25% transparent and background (c) 0% transparent. We use 16 samples (4×4 pixels) and then render order is (b) -> (c) -> (a). (b) is rendered, filling 75% of the samples and also keeping track of the depth. (c) is rendered, since it’s depth is lower and covers 100%, it fills all the samples not filled by (b). Finally (a) is rendered, with the lowest depth, it writes on top of all the other samples, filling 25% of the samples.

When all the samples are gathered the average of these 16 samples is used as the color for the pixel.

Commutative Function
The rendering function for transparency is not commutative, that means that it’s not possible to switch places of the terms in the function (in the rendering this means that the layers must be rendered back to front). If it would be possible to find a function that has the same result as the transparency function but where the terms could commutative would make it possible to render the layers in any order. There are some attempts to find a function with these properties but all of them only give an approximation of the of the correct result. The equation below is the transparency equation and it’s repeated iteratively, back to front.

C_f=\alpha_1 C_1+(1-\alpha_1)C_0

Per Pixel Linked List
To get an exact result all the layers must be saved and drawn in the correct order. To accomplish this on a graphics card a method where each transparent layer’s pixels are stored in linked lists can be used and then finally sort the lists and render it. This uses both lots of memory, GPU time and require some kind of synchronization for the lists.  Approximations exist where only the n top most layers are saved to reduce the GPU usage.

Weighted Order Independent Transparency
An Commutative Function using the depth buffer to improve the closest layer. This algorithm creates one correct layer while the other are approximated.

Adaptive Transparency
Uses a combination of Per Pixel Linked List with storing the top layers and then making an approximation of the background layers using  a Commutative Function.

 

Project startup!

My first post, the blog is alive! I’m doing a master thesis about rendering transparent objects in real time applications. It’s easy to think this should be a solved problem by now since there has been transparent objects in games for many years but it’s still an open problem to find a solution that handles transparency generally.

In rasterization, opaque (none-transparent) objects are render to the screen in any order but only the objects closest to the screen are saved. With transparent object, not only can many layers contribute to the final color but also the transparent objects need to be rendered in the correct order. With an simple approach this both adds required computational power to sort the layers, and added memory usage to store the layers, both of these are not available in large amounts when doing real time graphics.

My goal is to find a solution that with the given constraints (CPU power and memory) get as good visual quality as possible.