Dont you hate it when you spend an inordinate amount of time on a bug just to find out that something stupid was the cause? Yeah, me too. This happened to me just today. I figured I'd share what happened so that some other poor bastard may avoid this problem in the future.
A Little Background
So, the game project I'm working on right now is an "unannounced" MMO for my company Retired Astronaut Collective. We're making great progress, sure, but there's been a small visual issue that's been bugging us for a while. I took a look at it off and on over the past two weeks, but nothing really stuck out as a cause. What was this bug? Well, a picture is worth at least 100 words, so take a gander at this:
Take a close look and see if you can spot the problem. Do you see it yet? :) No, contrary to what you might be thinking, our artist didn't want to make this lady look like she was just eating out a monster's ass. That nasty ring was an unexpected visual artifact.
All of our characters in this game are created from small pieces to get maximum reuse from a small set of textures. We happily tint, rotate, scale and skew these images to composite our characters for final display. When we put this character together, we were shocked to see that she'd been digging in at the crap buffet.
Now, when I started looking at this problem, I assumed it must be an art bug. I mean, those damn artists are the source of so many bugs. Amirite? Sure I am. So, first thing I did was to look at the images that were in our sprite sheet for this character. For your viewing pleasure, here are a few of them from our morally questionable avatar:
In this simple case, we composite the gap filler (in the middle), then the head and mouth. As hard as I looked, I didn't see any cause for the unsightly makeup our lady was sporting in-game. All we knew is that it had to be the head shape that was causing this. Because, as the astute reader will notice, the crap ring is in the shape of the gap in the head shape. Great to know, but is it helpful? Not really.
My next step was to make special version of the head out of screen space sprites in the engine. To my amazement, they worked like a charm. No crap ring was present when I rendered the head that way. Awesome. Now I was convinced that my animated character render path was somehow broken. I started checking alpha modes, stepping through the render process and whatnot in an attempt to find the cause of this problem. No dice.
At this point, I got on Skype with the artist and went over my findings. We started brainstorming possible causes and stumbled on a workaround for the problem. When we disabled texture filtering then the crap ring went away -- and the character looked like shit. Hmm, curious.
"Why would filtering cause this problem?", I asked myself. Why oh why! The only thing I could think of was that the animated version of the head was being placed at subpixel locations, causing filtering to kick in and introducing the artifact. Seemed reasonable enough, but why would that cause a darkening of the pixels around the mouth edge?
The artist and I started down two separate paths to resolve this. He took the path of altering the art to remove the alpha pixels from around the rim of the mouth. I took the path of looking even closer at the source image for the head. Let's take a nice close look together, shall we?
Here's a closeup of the upper-left corner of the mouth. As expected, we see the anti-aliasing alpha blended pixels added by Photoshop. At first glance everything seems okay. But, on further investigation...
It's at this point that the forehead-slapping eureka moment came. The info that made this problem crystal clear to me was the color settings for the fully transparent pixels inside of the mouth gap:
Does anything seem off about that color to you? Look close! Right, that's it. The RGB value of the empty pixels was black! The stupid, but efficient, graphics hardware happily takes the RGB value of fully tranparent pixels into account when filtering the image for display. This caused a darkening of the alpha pixels around the ring of the mouth when rendering. The result? Shit mouth.
The solution to this problem was actually pretty straightforward. I modified our sprite sheet packer to better compute the RGB value of fully transparent pixels in the images to be packed. The algorithm is something like this:
For each tranparent pixel in the image do
Average RGB values for the surrounding non-transparent pixels.
Set the RGB component of the transparent pixel to the average.
After making that change, I re-packed the sprite sheets and took a look at the results:
I only spent about an hour on this today, with another hour spent off and on over the past two weeks. Still, it makes me sad when I spin my wheels looking for a problem that's caused by such a simple issue. Note to self:
Graphics cards are stupid (but fast) and don't think about transparent pixels like mere mortals.
Until next time, folks!
EDIT: It's been rightly pointed out that using premultiplied alpha in my engine would solve this problem. Sadly, that's not an option that I can implement at this time due to limitation in the middleware solution that I'm using (without a major rewrite of the core rendering solution). I'll stick with the current solution that works. Thanks for the suggestions, though!