Understanding the Work of 3D Graphics Gurus....

[SoulofDeity]

So I've been working on a high-performance game engine for quite some time and have spent the day trying really hard to understand how the sepia / monochrome filters and motion blurring effects work in Zelda games. Mostly, the later of them.

 

In OpenGL, there is something called an accumulation buffer which can be used to create motion trails through vector calculations. Buuutt...the N64 doesn't have one...nor do many pc ATI gpu's. The more I think about it, the more it seems like black magic...

 

 

What is known about the N64:

• It has a 16-bit color buffer
• It has a 16-bit depth buffer
• most likely has a 1-bit stencil buffer.

I've digged through the sdk a little and don't remember seeing anything about a stencil buffer, but effects like

 

 

and

 

 

suggest this is in fact true.

 

-----------------------------------------------------------------------------------------

 

For those who don't work with 3d graphics a short explanation; there are 3 main types of framebuffers in 3d graphics. The first is what you actually see like in the pictures above and is called the color buffer.

 

The second is called the depth buffer, and is used to keep track of the depth of 3d objects that have been rendered. A visual example of what this looks like is

 

 

Basically, whatever is brighter is closer and whatever is darker is further away. So, when rendering an object, if the new depth is "brighter" than the current one then the pixel gets drawn, otherwise the previous pixels are in front of the new ones.

 

 

The third is called the stencil buffer, and as the name suggests acts like a stencil to determine what should and shouldn't be rendered. A visual example of what this looks like is

 

 

When the stencil is enabled, whatever part of the color buffer that line up with the white parts of the stencil will be rendered, and whatever lines up with the black won't. Typically, stencil buffering is used to create shadows, masks (like the lens of truth eyehole), screen transition effects

 

*cough*

 

and dissolving effects like

 

------------------------------------------------------------------

 

BUT BACK ON TOPIC....

 

How does Majora's Mask achieve motion blurring? Some possibilities:

 

 

Answer: The just don't clear the last depth buffer

Problems:

• Some maps that make use of blurring like the inside of the moon have skyboxes, which would overwrite the previous frame anyway.

 

 

Answer: They don't clear the last depth buffer AND render a translucent clearcolor or skybox

Problems:

• Even if the skybox doesn't occlude the model, the map and other objects will still overwrite them.

 

 

Answer: They render to offscreen framebuffers as textures and layer them

Problems:

• Even for a poor resolution of 320x240, a single offscreen framebuffer would be 0x25800 bytes in size, or 0.14Mb, while the N64 only has 4Mb of ram (8Mb with the expansion pak). Offscreen layering requires several of these buffers and wastes a lot of what little precious ram the N64 has to offer
• This method requires altering the framebuffer manually which is cpu intensive. It may not be a big deal for your 2.3Ghz pc, but for an N64 running at approximately 6.1% of that speed struggling to maintain a mere 20fps, this would be a nightmare.
• Most N64 plugins don't support proper framebuffer emulation but still manage to somehow pull of the motion blur effect

 

 

Answer: They don't clear the color buffer and render everything at partial opacity

Problems:

• The only way to control the opacity of the rendering would be to control the environment or primitive alpha colors and mux every display list the game with them properly. Of course, anyone who screwed around with combiner modes can tell you this just doesn't happen.

 

 

 

~My First Theory:  They are using a stencil dissolve effect between frames. By alternateing checkered patterns, they could be interleaving the current frame with the last one, leaving afterimages and producing a motion blur effect.

 

 

~My Second Theory:  They are stenciling the depth buffer to prevent certain objects from being rendered over. Possibly clearing with a bright translucent color, or darkening the preserved portions of the depth buffer to fade them

 

 

 

========================================================

 

 

 

So, thoughts? Are there any problems you see with my theories, or have any of your own? I'm really interested in finding a fast solution for cross-platform motion blurring for my game engine, so chip in!

[Antidote]

If you look at some old SNES games, you'll see a pseudo motion blur, this is achieved by creating a clone of the sprite at it's previous location, they could be doing something similar for some things, but you're right it is a bit of an enigma. However I CAN confirm that the N64 has a stencil buffer, and it's used heavily in games like Goldeneye.

 

EDIT:I should mention that the ZBuffer bug could hamper speed in some instances, I'm not sure on the details though.

[SoulofDeity]

If you look at some old SNES games, you'll see a pseudo motion blur, this is achieved by creating a clone of the sprite at it's previous location, they could be doing something similar for some things, but you're right it is a bit of an enigma. However I CAN confirm that the N64 has a stencil buffer, and it's used heavily in games like Goldeneye.

 

EDIT:

I should mention that the ZBuffer bug could hamper speed in some instances, I'm not sure on the details though.

 

that could be used for some things like

 

 

but for stuff like

 

not so much...

 

at least I know for sure there is a stencil buffer though now 

[Antidote]

If you look closely you can even see the different LOD models in the first pic, I think that confirms my suggestion for one method.

 

 

EDIT:That second pic looks like they're using a Gaussian blur, that could be easily done on the N64 by using the first suggested method with a slightly transparent Gaussian blurred textured overlayed. But at the same time, it'd be a little difficult to get it to look right, they may have used some combiner trickery to get it to work as well.

 

EDIT2:

A good way to confirm one way or the other is to play the cutscene with wireframe enabled and textures disabled.

[SoulofDeity]

 

That second pic looks like they're using a Gaussian blur, that could be easily done on the N64 by using the first suggested method with a slightly transparent Gaussian blurred textured overlayed. But at the same time, it'd be a little difficult to get it to look right, they may have used some combiner trickery to get it to work as well.

 

The problem with that is that it requires at least 2 offscreen framebuffers since the game uses double-buffering, which adds up to 0.28Mb, and with TMEM limitations would require manual blending. I'd normally jump to the idea of combiner foreplay too, but each display list changes it so that can't be it either...

[Antidote]

Good point, It's probably a whole host of tricks to get it working, maybe even a combination of all your suggested methods.

[SoulofDeity]

I just had another idea; what if they rendered all the objects to the stencil buffer, but only enabled the stencil for normal rendering and not fog?

if it was done like that then the afterimages would stay onscreen and since fog is translucent, each frame would fade it more and more until it was gone.

 

then it's just a question of knowing when certain things should be removed from the stencil buffer :/

[Antidote]

That's very similar to Carmack's reverse, and definitely plausible.

 

As for the second part, they could be using the depth as a determining factor.

[SoulofDeity]

That's very similar to Carmack's reverse, and definitely plausible.

 

As for the second part, they could be using the depth as a determining factor.

 

Thats right O_O  fog affects depth! so on top of gradually fading out the after image in the color buffer, it'll fade it out in the depth buffer, and if they multiply the stencil by the depth then when the depth of the afterimage reaches 0 it'll automatically be removed from the stencil buffer!

 

[Antidote]

I can't see any point in NOT multiplying the stencil by the depth, especially in that early of a 3D console. Now it's down automagically or explicitly using pixel/fragment shaders.

[SoulofDeity]

I can't see any point in NOT multiplying the stencil by the depth, especially in that early of a 3D console. Now it's down automagically or explicitly using pixel/fragment shaders.

 

I think this may be case closed for the motion blur mystery 

 

Guess it's on to the sepia / monochrome filters now? Obviously a full framebuffer color lerp would be too cpu intensive. The color combiner supports a YUV color conversion, so one thought is that they're just cranking down the UV.

 

 

I tried looking for the instruction that does this before in OoT and only found one in the entire game, but I wasn't able to test it to see if that was it. (the sepia filter is only shown at the end cutscene, and I was lazy). Haven't looked into MM, but it might be possible. Isn't really a cross-platform thing though, any thoughts on what else they could've done?

[Antidote]

You're completely right, the N64 would completely choke if you tried to do a lerp of that size.

Monochrome is easy though, just fill the palette with black and white values.

 

EDIT:

Even JPEG supports this, so it's not too much of a stretch.

[mzxrules]

I'm not completely up to speed on 3d graphics terminology, but I do know this...

 

Occasionally when I use to play Majora's Mask on console, I'd run into a fatal error. When this occured the game would usually flip between the current frame (which usually had missing polygons) and the previous frame a few times, before finally displaying presumably the last frame drawn during the motion blur effect (which sometimes occured several scenes ago) with the bar of doom drawn on the left-hand corner.

[Mallos31]

The game also seems to save some other frames in the buffer. Some crashes cause the "song of soaring" screen to come up, then the bar of doom.

[Arcaith]

Wouldn't it just be simplest to generate a trail of alpha blended copies of the actor's model along the vector of motion and simply have a timer gradually subtract from the alpha of each copy until they all reach 0? I'd say Antidote's theory is the most likely. You'll notice that in instances where the motion blur is used, the rest of the geometry is quite simple to avoid overtaxing the n64's tiny amount of memory.

[mzxrules]

Except that the motion blur effect is used when hitting Eye of Truth Statues, which can be placed pretty much anywhere (Termina Field for instance). It's also used for the very last part of the cutscene of the Giants holding the moon above Termina.

 

I feel like the memory constraint thing is being overstated a bit considering Ocarina of Time was able to do its thing with half the memory.

[Antidote]

I just thought of something, since you're blurring everything anyway it's not necessary to use the full resolution, they could be using a compressed version of the previous frame for the blur, since you can essentially quarter the total size, you can get away with a very tiny (0.08 at minimum) buffer.

 

I'd have to look at the code for it, but that seems highly plausible considering the N64's resolution (640x480 iirc)

 

EDIT:

One other thing, does the N64 store pixels linearly or as scanlines?