Saturday, December 12, 2009

Screen Space Spherical Harmonic Lighting

A while ago Jon Greenberg brought up the idea of accumulating lighting in screen space using spherical harmonics, in a blog entry entitled "Has anyone tried this before?"

I've been doing deferred lighting experiments in XNA, and decided to give this technique a try. Please note I'm not doing any antialiasing and all screenshots are the definition of "programmer art" cobbled together from various freely available assets.

Screenshots show full resolution deferred lighting on top and the screen space SH technique at the bottom in the Sponza Atrium by Marko Dabrovic:

The technique was pretty simple to get up and going, and produces some interesting results. The above images are with 27 point lights, 3 spot lights, and 1 directional. The directional is the only light evaluated at full resolution per-pixel, in the apply lighting stage.

The basic idea is to use a quarter-sized lighting buffer (thus, in this case, 640x360) to accumulate 4-coefficient spherical harmonics. The nice thing is you only need the depth information to do so. I used 3 FP16 buffers to accumulate the SH constants. Points and spots are evaluated by rendering the light geometry into the scene and evaluating the SH coefficients for the light direction via cube map lookup, and then attenuating as normal. For the directional light, I evaluate that in the apply lighting shader. I'm not rendering any shadows.

For diffuse lighting, it works pretty well, although due to the low number of SH coefficients, you will get some lighting wrapping around onto backfaces, which in practice just tends to give you "softer" lighting. That may or may not be desirable.

Even though the lighting buffer is quarter-sized, you don't really lose any normal detail since SH accumulates the lighting from all directions. In my test scene, the earth models are the only ones with normal maps (deferred on the left, SH on the right)

I found that when you upsample the lighting buffer during the apply lighting stage naively, you would get halos around the edges of objects. I fixed this using a bilateral filter aware of depth discontinuities.

I was able to fake specular by extracting a dominant light direction from the SH, dotting that with the half vector, raising to the specular power, and multiplying that times the diffuse lighting result. It doesn't really give you great results, but it looks specular-ish. I tried using the lighting looked up at the reflected view vector but found that gave worse results.

Performance-wise, in my little XNA program, which I'd hardly call optimized, the SH lighting is about the same as deferred lighting when I store specular luminance instead of a specular lighting color in the lighting buffer. Here's some screen shots showing 388 lights (384 points, 3 spots, and 1 directional):

Note that there is at least one major optimization that could be performed when I'm calculating the SH coefficients for a light. Currently, my SH lookup cube map is in world space, but my light vectors are calculated in view space for points and spots. This causes me to make a matrix multiplication against the inverse view matrix in all the lighting shaders. This could probably be sped up quite a bit by calculating the SH lookup cubemap in view space each frame.

All in all, it is an interesting technique. I'm not very happy with the specular results at all, and the softness of the lighting could be a benefit or a drawback depending on the look you are going for. Jon also points out that the lighting calculations could easily be moved to the CPU on some platforms, since they only depend on depth information. I'm probably not going to explore the technique much further but thought I'd post what I'd found from the limited investigation I did.


  1. Cool of you to post this Steve. Probably worth mentioning is that the really interest part of the idea is what you can theoretically do with the screen-space SH set beyond straight-up basic lighting. The most interesting (and viable) one to me is extending SSAO into a quasi-GI solution by using the SH set directly.

  2. Wow, I was thinking about this technique almost year ago, but abandoned it mainly because of problems with specular lighting and bandwidth requirements (I was trying to implement quadratic SH with FP16 render targets - stupid me:) ). If you're interested - here is link with some thoughts on topic:

    I'm really excited that you had some success with that technique, and now I'm considering giving it a second try. However, I'm afraid that this technique could become quite slow when you have many large overlapping lights. On the other hand, cryengine3 massive lighting system looks quite similar to this approach, and it is stated that it successfully handles lights overdraw.

  3. Bandwidth is certainly a concern, although that is mitigated by rendering at 640x360 vs 1280x720. You could possibly move to 4 RGBA1010102 buffers to store the coefficients, and still get a range larger than [-1,1]. Even then though, you've only matched the bandwidth used by a full resolution deferred lighting buffer, although in my tests at that point you are almost always ALU bound anyway.

    BTW, I tried only using color information from band 0, but it doesn't look very good because of its non-directional nature. For example, a sphere with a red point light on one side and a green point light on the other would give you yellow all around.

    Part of the problem with my specular approximation is sometimes the dominant light direction is on the other side of the normal, which leads to inconsistent results which is really where it falls down right now. What I need to get is the dominant light direction in the hemisphere defined by the surface normal, but I haven't had any time to fiddle with the math to see if I can do that.

    Anyway, I'm not sure if this technique is really practical but it's fun to play around with, so I encourage you to give it a shot.

    One thing I'd be interested in seeing is how low you can make the lighting buffer resolution before quality really starts going bad -- I only went to 2x2 pixels per lighting sample because it is cheap to upsample from. But such upsampling could be done as a post process with a smaller lighting buffer into a quarter-sized one for the final application. If that could be done, this could be useful for diffuse-only fill lights, bounce lights calculated from realtime GI, or lighting LOD for lights off in the distance when used in concert with other deferred lighting techniques.

  4. Can I have a source code demo?

  5. Trying this approach for accumulating a lot of fill lights. Looks quite promissing, thank you for sharing this idea.
    Btw, how do you adress sh "ringing"?
    Does using LUT cubemap for sh basis calculation provides enought precision? I'm calculatin gthis on fly as Ati did in froblins:
    float4 SHEvalDirection (float3 vDirection)
    float4 result = 0;
    result.x = 0.282095;
    result.y = -0.488603 * vDirection.y;
    result.z = 0.488603 * vDirection.z;
    result.w = -0.488603 * vDirection.x;
    return result;

  6. I also switched to evaluating the SH basis directly in the shader, so I recommend that.

    As far as ringing, not sure if there's a good answer. You could try windowing the SH (essentially attenuating band 1), but that will lead to softer lighting. You could also try just using color information from band 0, but as I mention above, that can look odd in some cases. If you come up with a good solution, I'd sure like to know.

  7. Maybe this is not a ringing at all. This is how it looks:
    basically light sources are exactly at these "dots". I'm summing about 8-16 lights per fragment.
    In case of specular, effect is more or less logical - one light is very close to surface, it's attentuation is minimal so it gives very "strong signal". But in case of ambient it behaives as an negative light. I've just cheked attentuation function it dosn't return negative value.
    The simpliest solution would be to move light away from surface, but I just want to be sure that this is not the issue of some bigger scale.
    What do you think about this?

  8. Just going to ask the obvious question - after you've accumulated the lights and are evaluating the SH against the surface normal, you are clamping that value to be 0 or positive, right?

    intensity = max(0, dot(1, normal.yzx, shFromBuffer))

    (I've removed the transfer coefficients for clarity)

    SH can go negative. If you think about what it is doing, each band is adding and subtracting from the previous one. It's possible for band 1 to subtract enough from band 0 that it goes negative.

  9. Sigh, typo there

    intensity = max(0, dot(half4(1, normal.yzx), shFromBuffer))

  10. Yes I clamp them. Diffuse is gathered from two parts, lambert from extracted dominant direction and diffuse convolution from sh (dominant light removed). In both cases result is clamped to not go below zero.

  11. Cheked, "black spot" obtained by both dominant light and sh without dominant light. It's not negative value it's actually zero. I have suspicion that this is solely numerical precision. I've just shifted lights a little bit further from surface and bug is completetly gone.

  12. Does using the dominant directional for diffuse significantly increase the quality? It's something I wanted to try but I've had to move on to other things since playing around with this technique.

  13. It makes shading more contrast for normals that are directly facing lights, but it need to be used together with convolution from sh, other way you loose part of the shading. This is how it looks together and separated, in this particular case I'me removing dominant light from SH before convolution:

  14. Yeah i've used this evaluation before when constructing SH for an entire object. I was quadratic SH then and found you could get equivalent results extracting a directional, rendering linear SH (minus the directional), and a directional light, which was cheaper than full quadratic SH evaluation.

    I wasn't sure if it'd be worth it with the screen-space method, but those screenshots tell a different tale. Thanks for sharing.