Skin Shading in Unity3d
I've been away from AltDev for a while, a bit longer than I originally expected because after a period of crunch before IGF submission in late October, I went back to the comforts of getting a normal nights sleep, something that working a normal game development job this past summer spoiled me into enjoying. Now the Christmas holiday has given me not only the time to finally begin blogging again, but the chance to do a little side project in something I greatly enjoy: the rendering of human skin, which was also an excuse for giving the latest Unity3D 3.5 beta a whirl:
Motivation
This has entirely been a hobby project for me, because I do not foresee its application into any project I'm involved with. However, I've realized that Unity is rapidly approaching the point where someone may want to use a more complicated skin shading model than just basic wrap lighting (an example of which is provided in the documentation). With this in mind, I thought it might be of interest to do a post pulling together some of the better resources I've found on the topic as well as posting some of my more useful code so that perhaps it might help serve as a jumping off point for a more serious endeavor.
The one issue that is always at the core of skin shading is that of Subsurface Scattering, which is the effect of light bouncing around underneath the surface of the skin and re-exiting elsewhere. A simply using a Lambert model causes very harsh edges, because the scattering is what gives skin its softer appearance. Here's what the built-in "Bumped Diffuse" shader looks like, the basis that is trying to be improved:
Background: Texture Space Diffusion
No discussion of skin rendering starts with anything other than mentioning the NVIDIA Human Head demo, which has a detailed description of its implementation in GPU Gems 3. The technique relies upon Texture Space Diffusion, where the lighting for the mesh is rendered into a texture and then blurred various amounts (based off of the scattering of light inside of human skin). The results of those blurs is combined together to form the actual diffuse lighting term. I actually played around with this technique in Unity around Christmas time last year, but this proved to be difficult given the nature of the TSD and the fact that Unity did not at the time easily support some nice features like Linear Space lighting.
There have been some very useful resources on skin shading in since the publication of GPU Gems 3. There are some excellent Siggraph slides from John Hable where he details what Naughty Dog did to attempt cheaper calculations than the NVIDIA techniques in Uncharted 2. There is also a technique by Jorge Jimenez detailed in GPU Pro that does subsurface scattering calculations in screen space, which removes the cost per mesh of TSD (a serious limitation for it's use in an actual application). This seems to have garnered some adoption in game engines, but I understand that it is still a reasonably expensive technique (note: I'm less knowledgeable of Screen Space Subsurface Scattering, so take that comment with a grain of salt).
With regards to tools for doing Skin Rendering on your own time, a good quality head scan has entered into the public domain from Lee Perry-Smith and Infinite Realities. Also, Unity3d is much friendlier to doing some high quality rendering. There is now a public beta for Unity 3.5, which now features Linear Space lighting and HDR rendering conveniently built into the engine.
The Latest Hotness: Pre-Integrated Skin Shadding (PISS)
I've been impressed with the quality articles in GPU Pro 2 since I picked it up this past summer, one of which is Eric Penner's article detailing his technique "Pre-Integrated Skin Shading." He also gave a talk describing it at Siggraph, and the slides are available online. There are three main parts to it: scattering due to curvature, scattering on small details (i.e. the bump map), and scattering in shadow falloff. I did the first two, no shadows yet for me, but I'll do a follow up post if I get around to it.
The basic motivation is to pre-calculate the diffuse falloff into a look-up texture. The key to making the effect look good is that instead of having a 1-dimensional texture for NdotL, it is a 2D texture that encompasses different falloffs for different thicknesses of geometry. This will allow the falloff at the nose to differ from that in the forehead.
I've adapted the sample code to precompute the lookup texture into a little editor wizard for Unity. Simply make a folder titled "Editor" in Unity's assets and drop it in, and you'll have it extended to the editor (the function is added to the menu "GameObject/Generate Lookup Textures"). I've included the full script at the bottom of this blog post, which computes both a falloff texture that appears to be about the same as the one that Penner shows in GPU Pro 2. Here's what my lookup texture looked like when I was all said and done, this is to be sampled with 1/d in the y, and NdotL in the x:
An important note: if you activate Linear Rendering in Unity 3.5, know that the texture importer has an option to sample a texture in Linear Space. GPU Gems 3 has an entire article about the importance of being linear, I noticed that after I turned it on in Unity, the lookup texture had the effect of making the head look like it was still in Gamma Space. I then realized I needed to flip on that option in the importer. In fact you may notice that the lookup texture in GPU Pro 2 looks different than the one in Penner's Siggraph slides, this is because the one in the slides is in linear space. The one in the book is not, which is also the case with the one pictured above.
The parameter 1/d is approximated with a calculation of curvature based on similar triangles and derivatives (there's a great illustration of this in the slides), and I've included the snippet of GLSL from my shader. To see the output of curvature on my model (easily tuned with a uniform parameter in the shader to offset the calculation):
// Calculate curvature
float curvature = clamp(length(fwidth(localSurface2World[2])), 0.0, 1.0) / (length(fwidth(position)) * _TuneCurvature);
This curvature calculation is actually a serious problem with implementing the technique in Unity. This is because ddx and ddy (fwidth(x) is just: abs(ddx(x)) + abs(ddy(x))) are not supported by ARB, which means Unity can't use the shader in OpenGL. Those of you familiar with Unity, will know that CG is the shading language of choice, and it is then compiled for each platform. A thorny issue, that I have been stumped with in the past, especially because I do many of my side projects in OSX on my laptop. Tim Cooper (@stramit) came to my rescue with a solution that works decently: to write an OpenGL only shader directly in GLSL, which will support fwidth no problem. This is a *little* painful, especially because Unity seems to have dropped the glsl version of AutoLight.cginc (my 3.4 build has AutoLight.glslinc, but unfortunatey 3.5 beta 5 does not), which is going to make things like using the built-in light attenuation methods much more of a headache. In fact, the reason my sample images use a directional light is because I don't currently have point light falloffs matching *exactly* the same as Unity's. That being said, it hasn't been problematic enough to cause me to just abandon OpenGL support and move to a Windows computer. Furthermore, there's a nice wiki that served as a good jumping off point for doing GLSL in Unity.
(NOTE: Aras from Unity posted a suggestion down in the comments that might cause my choice to use GLSL directly to be un-needed. I will properly update the post after I try it out myself)
Softening Finer Details
As I mentioned, I'm sharing my experience with trying out 2 of the 3 techniques detailed by Penner. The second part is smoothing of small detailed bumps. While the pre-integrated texture will account for scattering at a broader sense, the fine details contained in the normal map are still much too harsh. Here's what my model looks like with just the pre-integrated scattering:
"Too Crispy" is my favorite way of describing the problem. Penner addresses this by proposing blending between a smoother normal map and the high detail normal map. The high detail is still used for specular calculations, but for the diffuse, you pretend that red, green, and blue all come from separate normals. This is very similar to what Hable details as the technique used in normal gameplay in Uncharted 2 in his Siggraph 2010 slides. By treating them separately the red channel can be made to be softer, Penner advises using the profile data from GPU Gems 3 to try to match real life.
In order to avoid the additional memory of having 2 normal maps, Penner uses a second sampler for the normal map that is clamped to a lower mip resolution (as opposed to using the surface normal like Uncharted 2). However, Unity does not allow easy access to samplers to my knowledge, so the only way to set up a sampler like that is to actually duplicate the texture asset, which won't save any memory. So, my solution was to just instead apply a LOD bias to the texture lookup (an optional third parameter in tex2D/texture2D in case you're not familiar). Here's what the same shot from above looks like with the blended normals applied using a mip bias of 3.0:
Specular
While many articles on skin shading focus almost entirely on the scattering of diffuse, the GPU Gems 3 article provides a good treatment of using a specular model that is better than Blinn-Phong. The authors chose the Kelemen/Szirmay-Kalos Specular BRDF, which relies on a precomputed Beckmann lookup texture and takes into account Fresnel with the Schlick approximation. Being that I had played around with that model a year ago when I was experimenting with TSD in Unity, I simply rolled that code into Penner's work. Here's the resulting specular:
A shot before applying specular:
And finally the specular and diffuse combined together:
Concluding Thoughts / Future Work
This has been a fun little side project for me, and I think it turned out decently well. I've made and fixed a few stupid mistakes along the way, and I wonder if I'll find a few more yet, don't be afraid to point out anything terrible if you spot it. My two main goals left are tighter integration with Unity : proper point/spot calculations that match CG shaders in Unity, and shadow support. When it comes to shadows I'm at a but of a crossroads between trying to hook into Unity's main shadow support vs. rolling some form of my own. I suspect rolling my own would be less useful in the effort to make the skin shader completely seamless with unity, but might be less of a headache to accomplish. Either way, if I do make make any drastic improvements, I promise I'll do a follow up post :)
Finally, if you're interested in the IGF project, Dust, that I mentioned as the root cause of my hiatus from AltDev, you can check out the project on it's website at www.adventureclubgames.com/dust/, and the trailer for it if clicking a link is too much effort:
[youtube http://www.youtube.com/watch?v=TJ5dLSh8PC0?rel=0&w=560&h=315]
Source Code
As I promised, here is the source for my Unity script that generates my lookup textures:
// GenerateLookupTexturesWizard.cs
// Place this script in Editor folder
// Generated textures are placed inside of the Editor folder as well
using UnityEditor;
using UnityEngine;using System.IO;
class GenerateLookupTexturesWizard : ScriptableWizard {
public int width = 512;
public int height = 512;
public bool generateBeckmann = true;
public bool generateDiffuseScattering = true;
[MenuItem ("GameObject/Generate Lookup Textures")]
static void CreateWizard () {
ScriptableWizard.DisplayWizard<GenerateLookupTexturesWizard>("PreIntegrate Lookup Textures", "Create");
}
float PHBeckmann(float ndoth, float m)
{
float alpha = Mathf.Acos(ndoth);
float ta = Mathf.Tan(alpha);
float val = 1f/(m*m*Mathf.Pow(ndoth,4f)) * Mathf.Exp(-(ta * ta) / (m * m));
return val;
}
Vector3 IntegrateDiffuseScatteringOnRing(float cosTheta, float skinRadius)
{
// Angle from lighting direction
float theta = Mathf.Acos(cosTheta);
Vector3 totalWeights = Vector3.zero;
Vector3 totalLight = Vector3.zero;
float a = -(Mathf.PI/2.0f);
const float inc = 0.05f;
while (a <= (Mathf.PI/2.0f))
{
float sampleAngle = theta + a;
float diffuse = Mathf.Clamp01( Mathf.Cos(sampleAngle) );
// Distance
float sampleDist = Mathf.Abs( 2.0f * skinRadius * Mathf.Sin(a * 0.5f) );
// Profile Weight
Vector3 weights = Scatter(sampleDist);
totalWeights += weights;
totalLight += diffuse * weights;
a+=inc;
}
Vector3 result = new Vector3(totalLight.x / totalWeights.x, totalLight.y / totalWeights.y, totalLight.z / totalWeights.z);
return result;
}
float Gaussian (float v, float r)
{
return 1.0f / Mathf.Sqrt(2.0f * Mathf.PI * v) * Mathf.Exp(-(r * r) / (2 * v));
}
Vector3 Scatter (float r)
{
// Values from GPU Gems 3 "Advanced Skin Rendering"
// Originally taken from real life samples
return Gaussian(0.0064f * 1.414f, r) * new Vector3(0.233f, 0.455f, 0.649f)
+ Gaussian(0.0484f * 1.414f, r) * new Vector3(0.100f, 0.336f, 0.344f)
+ Gaussian(0.1870f * 1.414f, r) * new Vector3(0.118f, 0.198f, 0.000f)
+ Gaussian(0.5670f * 1.414f, r) * new Vector3(0.113f, 0.007f, 0.007f)
+ Gaussian(1.9900f * 1.414f, r) * new Vector3(0.358f, 0.004f, 0.00001f)
+ Gaussian(7.4100f * 1.414f, r) * new Vector3(0.078f, 0.00001f, 0.00001f);
}
void OnWizardCreate () {
// Beckmann Texture for specular
if (generateBeckmann)
{
Texture2D beckmann = new Texture2D(width, height, TextureFormat.ARGB32, false);
for (int j = 0; j < height; ++j)
{
for (int i = 0; i < width; ++i)
{
float val = 0.5f * Mathf.Pow(PHBeckmann(i/(float) width, j/(float)height), 0.1f);
beckmann.SetPixel(i, j, new Color(val,val,val,val));
}
}
beckmann.Apply();
byte[] bytes = beckmann.EncodeToPNG();
DestroyImmediate(beckmann);
File.WriteAllBytes(Application.dataPath + "/Editor/BeckmannTexture.png", bytes);
}
// Diffuse Scattering
if (generateDiffuseScattering)
{
Texture2D diffuseScattering = new Texture2D(width, height, TextureFormat.ARGB32, false);
for (int j = 0; j < height; ++j)
{
for (int i = 0; i < width; ++i)
{
// Lookup by:
// x: NDotL
// y: 1 / r
float y = 2.0f * 1f / ((j + 1) / (float) height);
Vector3 val = IntegrateDiffuseScatteringOnRing(Mathf.Lerp(-1f, 1f, i/(float) width), y);
diffuseScattering.SetPixel(i, j, new Color(val.x,val.y,val.z,1f));
}
}
diffuseScattering.Apply();
byte[] bytes = diffuseScattering.EncodeToPNG();
DestroyImmediate(diffuseScattering);
File.WriteAllBytes(Application.dataPath + "/Editor/DiffuseScatteringOnRing.png", bytes);
}
}
void OnWizardUpdate () {
helpString = "Press Create to calculate texture. Saved to editor folder";
}
}
I haven't included the actual shader because it's a little messy still and currently only supports the OpenGL path. The numerous resources that I've mentioned should guide you in the right direction though.
