Lord Ned

Also to add is that those two are the last sections before we can create our own custom collisions from scratch, theoretically with 100% support. It's composed of 5 sections: Header Vertexes Faces SpatialQueryOptimization (see post #281. Our own name for a lack of anything better. Unknown2 (see post #286. SurfaceProperty (see #286 again) Type (see #286). Both Unknown2 and Type are incomplete, and until we figure out their exact purpose we'll be unable to create custom collision files, because modifying either of these sections makes the game quite unstable.

2am in the morning. Sage of Mirrors has randomly found me on an IRC channel I was idling in. He begins to help research with me. Write tool for Sage of Mirrors to simplify discovering offsets. He eventually turns his detective skills to the "Unknown1_t" section of the format. We battle back and forth on it, trying to get ideas about what it is. Sage of Mirrors modifies the data in a way that crashes the game. Finds an exception printed out in the log left there by TWW's devs. It reads: Cryptic. But with some Romanji to English, a flicker of hope appears. "pm_bgd"... nothing translatable, even removin the pm_ (pointer member). "m_ti_num". Remove m_ and _num. ti appears to read "Ground". The format stores the ground triangles seperatly, the unknown data is simply indexes into the triangle array! We've cracked what the Unknown1 section does! I get excited and make the OBJ writer export only the ground triangles. Import it into 3ds max. Get: Sit back in shock. We were so right, we had the name matched and it made sense. But it's not right. Examin data again. Sage of Mirrors notices pattern. Indexes incriment by between 2-8 each time. Struggle. Drown in programs. Think about triangle strips. Look for pattern. Struggle some more. Attempt. Attempt again. And again. (Yes those url's are in the right order) Sage of Mirrors goes to bed. It's late. We're tired. Look at pattern some more.Show to other people. No one can find a real pattern. A friend says "There is a pattern. Draw based on the deltas". Debate what this means. Realize all the deltas added up equal the original number of faces. Change code. Get: It's the original triangle data! Except... We already had that! So what gives? The best I can tell is the "Unknown1" section is a series of triangle groups that split the map up into small chunks. Small chunks are fast to run physics tests again. Unknown1 is data used for optimization. 2 unknown lumps remain, and one partial lump. Navmeshes? Partially documented lump for player collision and hit effects? That's the last 6 hours in one post. I hope someone gets something out of it. Onwards, to another day of debugging!

I didn't modify the scale of the map when I importex/exported from max, I took the default units. I forget the exact units but the extents of the map is a pretty even number (like -1500, 1500 exactly). No significant luck on the collision format. I discovered two bits of information to add to xDan's documentation. struct Face_t { U16 verts[3]; U16 Unknown1; U16 Unknown2; }; What I figured out is that Unknown1 and Unknown2 are indexes into the "Type" structure. It is how properties (like water) are assigned to each triangle. What I don't understand is why there are two indexes. Both appear ot be indexes into the Type structure, and I would guess there are a minimum of two type structures. However, it gets more confusing: A Face_t never has the same index for Unknown1 and Unknown2, which makes me think that these Type structures need to go in pairs. However, the K_Test2 level has 5 embedded types, which means either they're not in pairs or someone reuses something. I'll give the reason for why I think both Uknown1 and Unknown2 are indexes into the Type section after I talk about my research into the Type section. struct Type_t { //Offset from start of file. This contains Romanji names for objects, things like //"floor" "wall" "water" "surfacePoly" etc. My guess is these were the original //names in Nintendo's editor and can be reused to re-create them in the OBJ. U32 nameOffset; //This value is 1,1,1 for every Type_t on some maps, or ~94% of them on others //When the map does have some non-1,1,1 the values normally range between 0.5 and 2 Vector3 Unknown1; //Unknown1 might be a scale, but doesn't make a lot of sense. U32 Unknown2; //Value such as 0 (common), 16384 55550 4282438625 2145253536 5102 3269 U32 Unknown3; //65535 (most common), 1073807359 //This value is 0,0,0 for every Type_t on some maps, or ~98% of them on others //When the map does have some non-0,0,0 the values range between -5000 and 5000 Vector3 Unknown4; U32 Unknown5; //Can be something small like 2-5 or 6551-6557, or 131082-131085 U32 Unknown6; //Either 429467295 458751 524287 589853 655359 983039 U32 Unknown7; //Values like: 65535 0 655361 1048578 46661950 etc. //Possibly flags field? Water = 256, 'Default' is 0 U32 surfaceType; }; There are a lot of unknowns in this field, but I have some reasonable defaults for some of them. Unknown1 is 1,1,1 almost all of the time, Unknown2 is 0 most often, Unknown 3 is 65535 a lot of the time and Unknown4 is 0,0,0 most of the time. However, Unknown5-7 have values that jump all over the place, there's a fairly set number of values it can be but there's no common one it's set to. (Is this indicative of another flags field? I'm not sure). They might define if it's grass or rock which the game could use to pick particle effects. What I do know is that the last U32, now named surfaceType is a flags field for the surface material. If it's set to "0", then it's solid, default. If it's set to 256 (1 << 8) then the surface is water. I imagine there's other bitshifts for Ice, lava, etc. I haven't checked yet, but only the triangles that are water point to Type_t structures that have the last value set to 256. Going back to the Face_t's Unknown1/2, a typical triangle might have the values: "{1, 2}", "{3, 4}", etc. However it can also be something like "{2, 5}" as is the case of my water triangles. What I did notice is that on triangles that are water, both Type values that the Unknown1/2 reference both have surfaceType set to 256. Other then that, I haven't made a lot of progress on the format. There' still three unknown sections, one which is 4 bytes per-entry, one that is 19 bytes per-entry, and one that is 16 bytes per entry. I have zero information, and zero idea as to the composition or reasoning behind values for these entries. When I get the time I will attempt a crude data injection, taking an exsting maps collision file, inserting my new vertexes and face geometry (pointing to the first and second Type structure for example), and then updating the remaining offsets in a hope the game can load it. Edit: The other little bits of interesting information I figured out today (Thanks to Sage_of_Mirror's prompting), is that regardless of collision model (we found the smallest one which appears as two triangles in WindViewer), it has a minimum of: 20 Vertexes 18 Faces 2 Unknown1 sections. 2 Unknown2 sections. 1 Unknown3 section. 5 Type sections. The 5 type sections might be that Nintendo's tool packs in some defaults. Inspecting the 18 faces of that model, every single triangle had Unknown1 set to 0, and Unknown2 was either 2, 3 or 4. The index "1" was not used. This just adds onto the confusing-ness of this format. :/

I began work on deciphering some of the Unknown sections from the .dzb format. I took my test file into photoshop to map out what sections the data looks like/corresponds to. I'm currently looking into Unknown2, or the two brown stripes. Each brown stripe is 19bytes long, and has an near-exact repeated pattern. At this point, I had no idea the composition of the brown stripe, other than it was mostly filled with FF, which seems pretty invalid as far as using it for numbers. I opened up a second collision format, and found the same repeating pattern in the other file, but the pattern was slightly different: test_0.dzb 01 01 FF FF 00 00 FF FF FF FF FF FF FF FF FF FF FF FF FF FF 01 01 FF FF 00 [b]01[/b] FF FF FF FF FF FF FF FF FF FF FF FF FF FF test_1.dzb (A more complicated map) 01 00 FF FF 01 01 00 02 00 03 FF FF FF FF FF FF FF FF FF FF 01 [b]01[/b] 00 00 00 00 FF FF FF FF FF FF FF FF FF FF FF FF FF FF 01 01 00 00 00 01 FF FF FF FF FF FF FF FF FF FF FF FF FF FF Of notable difference is the first map uses "xx xx FF FF xx xx" at the start, but the second map uses "xx xx FF FF xx xx xx xx", and "xx xx xx xx xx xx xx". I'm curious as to what datatype is stored here, and why it defaults to FF if it's not in use. My guess is it would be a short (U16) due to the consistent pairs of two bytes. If anyone has any thoughts, let me know. I'm just figuring this stuff out as I go What's also curious is the Unknown1 (short green strip) is only 4 bytes long which doesn't give you a lot to work with, unless it's an extra data field for another member (Eg; has the same number of entries as another unknown)

Thanks I'm using XDan's C# code (I decompiled it, sorry!) as a basis for my collision stuff, because he has it well enough to load it so I figured it was a starting place. Working on trying to figure out the Unknown's. I have some ideas as to what they'd be, my guess is per-triangle collision flags (water, ice, solid, etc.) as well as perhaps bounding boxes, etc. I'm neck deep with a notepad document and a hex editor, learning as I go and punching things into my C++ app to try and get results. Going to write an OBJ exporter now to see if we can just dump collision models for the moment. Also good work on all of the entity stuff Sage! Edit: What I've discovered so far is that the "Type" section (as documented by XDan) has a count of 5 but an offset of 0 for this collision, so I'm not sure if it's incorrect or whether it has a number sometimes but no actual value, etc. Second Edit: It finally occured to me that if my header is 48 bytes long, and the triangleOffset is 36, then the offsets are not listed from the start of the file. This then lead me to discover that none of my offsets are correct. While I fix that, I punched in the offsets manually for the moment, and got myself a collision model exported to obj: http://i.imgur.com/1XYUY.png Though this appears to not be quite correct still, it's pretty close to what it should be. Something of note that I thought about is these collision models are only written as points. I think the surface needs normals too, because looking at Sunshine glitch videos as well as a few Windwaker ones, collision testing is only one way. Normals are either determined in-game by face-order, or that's part of the unknown collision data. Final edit: Needs a y->z flip, but I finally got meshes to load right. Had a few misunderstandings with it, I think my offsets are correct, they're just additive to something other than the start of the file, maybe it's an offset past the end of the previous data*? Will try some more tomorrow. Just leaving these edits here for anyone in the future who might learn from it. *I don't think this is true. My current file has a 52 byte vertex offset, and a 36 byte triangle offset. The vertexes occupy the space between 52-292 (240 bytes long, 20 vertexes with 3 floats at 4 bytes each), adn then the face data starts IMMEDIATELY afterwards, occupying 292-452 or 160 bytes (10bytes per triangle, 16 triangles. 6 bytes goes to the index, then 4 bytes of "unknown".) This means that the 52 byte vertex offset is correct, but the 36 byte triangle offset should actually either be 0, or 292 by either theory. I'll sleep on it and see what I come up with in the morning. Use a proper Endian conversion! The offsets are now as expected when I switched endian-conversion methods. Use: _byteswap_ushort / _byteswap_ulong (<intrin.h> for MSVC instead of writing your own.

I have had some minor success with importing custom maps into Windwaker. I've only tried with two and succeeded with one. Here is the map in 3ds max, (a modified K_Test02): And now here it is in game: It unfortunately lost its ability to have lighting along the way, and my guess is it's due to texture conversions, etc. This was created by unpacking the stage, un-Yaz0' if applicable, using a 3ds max script to import the .bdm (and extract textures), and then exporting to OBJ, using obj2bdl, and then converting back to bdm via bdl2bdm. Then using the rarc repacker, NOT COMPRESSING IT, and embedding it via the ISO repacker. Unfortunately I haven't had any success with maps that start as a bdl, only ones that start as a bdm. I've been researching into the collision format some more, because I want to try and create a collision model creator to create new collision to match the geometry change.