Hardware Compatibility and DMADATA

[Airikita]

If anyone is experiencing hardware compatibility issues, is there any information regarding the dmadata table?

 

dma is used for hardware-specific access, and it might be necessary to change the table according to changes when modding, perhaps the lead cause of hardware incompatibility.

 

Has anyone tested this? I could get a test, but I don't have my own ED64.

[xdaniel]

IIRC only the Debug ROM doesn't care if the DMA table is wrong, while every other version does, although I can't remember if that's been investigated thoroughly. And IIRC it's a pain to fix the table properly; don't remember the specifics, but spinout had posted about this before - something about the file order having to be correct, which would require rebuilding the whole ROM and fixing a lot of addresses?

[Airikita]

That has been my thought... yet for reference, if it is necessary, to change it as needed.

[Airikita]

On the note of hardware compatibility... HeavyZ has notified me that mods with new textures tend to crash or do weird things on his console... he tested a mod with custom textures (maybe it had model import, but maybe not) and he mentioned there were errors...

 

My Unicat crashes the console, but apparently some other mod he tested had a similar issue.

[john_smith_account]

IIRC only the Debug ROM doesn't care if the DMA table is wrong, while every other version does, although I can't remember if that's been investigated thoroughly. And IIRC it's a pain to fix the table properly; don't remember the specifics, but spinout had posted about this before - something about the file order having to be correct, which would require rebuilding the whole ROM and fixing a lot of addresses?

The rumor that the order has to be correct is just that, a rumor.  DMA problems were blamed for early attemts to compress the debug ROM failing, when it turned out in the end to be the faultof the compression tool we were using.

 

It might be true that the table can't have gaps in the compressed entries, and it is certainly true there can be no over laps in mapping values.

 

The order does not have to be correct ...BUT... the DMA map will automaticly end when it reaches the first zeroed out entry, 00000000 00000000.

 

We didn't know that at the time.   Therefore, as a work around, the first compressed debug ROM has repeated DMA values within the table to over write DMA entries of files that were removed.  Sloppy, Sloppy Sloppy.  But it proves file order does no have to be correct.

 

The DMA issues are simple to over come, and are best fixed up as a final step of releasing a ROM mod.  To make it simple to mod the ROM, use the patch I posted long ago.  It ends the DMA table early(directly after the last address you absolutly need an entry for), and also fixes the CRC of the new table.  This removes almost all the mappings you can do without.  Without the table, the game reverts to using whatever actual physical address you give it.

 

When you mod is complete, then you can make new entires in the DMA tables for new files.  Then you fix the CRC, and *BOOM* your mod is ready for real hardware/picky emulators.

 

As a side note, the retail ROMS will accept decompressed entries as long as you fix the CRC, and as long as they fit within 32-megs.  The orginal translations of the level select menu proved that.

 

P.S. I'm still feeling like death warmed over(sadly very literal), so if you write me don't expect a fast response.

[Airikita]

The rumor that the order has to be correct is just that, a rumor.  DMA problems were blamed for early attemts to compress the debug ROM failing, when it turned out in the end to be the faultof the compression tool we were using.

 

It might be true that the table can't have gaps in the compressed entries, and it is certainly true there can be no over laps in mapping values.

 

The order does not have to be correct ...BUT... the DMA map will automaticly end when it reaches the first zeroed out entry, 00000000 00000000.

 

We didn't know that at the time.   Therefore, as a work around, the first compressed debug ROM has repeated DMA values within the table to over write DMA entries of files that were removed.  Sloppy, Sloppy Sloppy.  But it proves file order does no have to be correct.

 

The DMA issues are simple to over come, and are best fixed up as a final step of releasing a ROM mod.  To make it simple to mod the ROM, use the patch I posted long ago.  It ends the DMA table early(directly after the last address you absolutly need an entry for), and also fixes the CRC of the new table.  This removes almost all the mappings you can do without.  Without the table, the game reverts to using whatever actual physical address you give it.

 

When you mod is complete, then you can make new entires in the DMA tables for new files.  Then you fix the CRC, and *BOOM* your mod is ready for real hardware/picky emulators.

 

As a side note, the retail ROMS will accept decompressed entries as long as you fix the CRC, and as long as they fit within 32-megs.  The orginal translations of the level select menu proved that.

 

P.S. I'm still feeling like death warmed over(sadly very literal), so if you write me don't expect a fast response.

(No worries, I don't expect fast replies anyways)

 

Yeah, but do you think there would be a difference in performance compressed? Perhaps it's a silly question, since that the problem for my mod (and everyone else's who converted models) is graphical. Apparently the "fix" I did didn't work, so I'll have to run some more comparisons later and fix up Unicat.

 

I haven't been looking at my mod lately, probably won't until the holidays.

 

But... my concern was the placement of my data, which doesn't seem to be an issue with dma considering the way OoT loads the files. As long as it's a rumor, then I shouldn't have to worry about it. I anyone has more ideas for hardware compatibility, it would be good to get other insight on this topic.

 

Like you said, the compression tool mixed up things up. The emulator is being picky, and I went over the stack function a few times. I'm pretty sure I might have to do a copy/paste of the POP command likely. (Goes to read wiki)

 

--------------------------------

 

EDIT:

It's likely I'll have to compare some other objects, like snow, and the dot that faces Link when using the Hookshot. I could also just remove the Push/Pop commands, and see if that resolves anything (if that's the source of the issue, which I don't think it is).

[john_smith_account]

Yeah, but do you think there would be a difference in performance compressed? Perhaps it's a silly question, since that the problem for my mod (and everyone else's who converted models) is graphical. 

 

 

I don't think it would cause a noticeable drop in performance.  After all, the retail versions are compressed, and we can't a difference in performance between those and the Debug ROM right?

[Airikita]

I don't think it would cause a noticeable drop in performance.  After all, the retail versions are compressed, and we can't a difference in performance between those and the Debug ROM right?

Perhaps start-up time might be twice as slow, but it all gets stored in a space, so in the end, it wouldn't hinder gameplay. It's alright anyways, I was reading up on DMA for hardware, and thought the DMADATA might have something to do with it. However, from my knowledge, OoT doesn't really use the DMA table to load things. Either way, if you change the DMA, you need to recalculate the checksum... perhaps that's where it was speculated as an issue before we looked at the mechanics more.

[Jason777]

So recently there's been a huge shitstorm over things not working on hardware... Concerning custom maps (which also applies to display lists, textures, etc), how about we try to make an exact copy of a map in-game, run it through SharpOcarina or some other converter and then try to see why it doesn't work on hardware from there?

 

When I say an "exact" copy, I mean tune the model converter so much that it literally produces a 1:1 copy of a map in-game; that way we know it isn't a problem with converting.

 

Also, could I get a list of what all supposedly needs to be done in order to ensure hardware compatibility? I could try to take a stab at writing a dmatable fixer.

[Airikita]

A map would work, although we could even export a pot... do we have a model extractor? I remember a certain quitter who was working on one.

 

Also, I know it's the tool because a command was misplaced in the incorrect order in which the RDP would disagree. There's someone who can vouch for me (because I showed it to him) but he hasn't been active lately. The entire structure was improper.

[Jason777]

What exactly is the "correct" way to order display lists? Forgive me, I really don't know much about model data in the N64 other than the usual 0xE7, 0xDE, 0xFD, 0xFA, 0x01, 0xDA commands.

[Airikita]

0xE6 G_RDPLOADSYNC (usage unknown)

0xE7 G_RDPPIPESYNC (usage unknown)

 

These two were in the incorrect order (or something similar). Loading has to come before certain sets, and 0xE6 came after a set of commands. I mentioned this in xdan's topic somewhere. I have to study for now, exam is in less than an hour (which I'm ready for).

 

EDIT:

Be aware of the RDP four:

0xE6 G_RDPLOADSYNC (usage unknown)

0xE7 G_RDPPIPESYNC (usage unknown)

0xE8 G_RDPTILESYNC (usage unknown)

0xE9 G_RDPFULLSYNC (usage unknown)

 

Those determine the RDP output that feeds through the VGA input/output for video (and possibly audio).

 

I assume these are written to be in order from E6, E7, E8, and E9. I would assume that these have to follow this order.

 

 

EDIT2:

There also are these two:

0xEF G_RDPSETOTHERMODE (usage unknown)

0xF1 F3DEX2_RDPHALF_2 (usage unknown)

 

Although I would assume these are special, and I would HIGHLY assume that EF would not precede E8 due to the fact that the tile should probably be synced before setting any modes to the current tile.

[HeavyZ]

I don't think it would cause a noticeable drop in performance.  After all, the retail versions are compressed, and we can't a difference in performance between those and the Debug ROM right?

I've noticed a few graphical errors in an untouched debug rom on hardware. Very small things that I thought didn't really matter like small black lines inbetween the spaces where one large area of the ground should be connecting to another. It's very small and it's not consistant it goes away depending where you are or the distance you are from the area's... It mainly happens in places like Hyrule Field... Honestly I forgot if that happens a lot in the original OoT or not. I should load it up in my N64, examine, and edit this post later to clerify if that is indeed the case or not.. I do think that it doesn't do that on the original OoT, I may regret saying that later and making myself look very nooby/forgetful

 

Edited: Nevermind haha, no differences noticed. I only managed to spot a single thin black line if viewed at a certain spot in hyrule field between two spaces of ground, but it is the same as in the Debug rom as in is in the original OoT.

[Airikita]

http://n64.icequake.net/mirror/www.white-tower.demon.co.uk/n64/

 

 

Reality Co-Processor:

The second component of interest is the custom chip - the Reality Co-Processor (RCP). This is a 62.5Mhz chip that interfaces directly to the CPU. The RCP is designed to handle most of the audio and graphics processing. In addition to this, the chip also contains DMA logic, audio and video outputs, and a joystick input. The chip also supports timing and signals for the game cartridges.

*Groans*... this pees in my cheerios now...

 

 

Onward:

 

There are two processors inside the RCP:

• RSP: Short for Reality Signal Processor. The RSP performs all 3D manipulations and audio functions. A special feature is that this processor is configurable via microcode allowing the system to be optimised over time.

• RDP: Reality Drawing Processor. A pixel drawing processor. This unit performs all pixel-level operations including texture mapping, anti-aliasing, tri-linear interpolation, MIP mapping and z-buffering.The RDP operates on a display list to provide it's graphics output meaning the RDP is essentially an Object List Processor.

This could reference more to xdan's topic, since this relates to F3DEX2 issues. Some of the commands do DMA'ing for both processors possibly? I am seeing that there is now piping issues to address with the RDP and RSP processors, meaning the commands would have to be aligned to use the proper output.

 

What's confusing is how this also is linked to the AUDIO output... that's right, audio. Unrelated, but uses the same RCP.

[DeathBasket]

I could try to take a stab at writing a dmatable fixer.

 

This is what spinout started (whenever it was) but some stuff couldn't be found yet iirc. I also know a lot of things have hardcoded addresses which can be annoying especially in the case where it's inside a compressed file.

[Airikita]

WARNING:

-------------

I just tested running an extended cow object file on N64 hardware, if you DON'T CHANGE THE DMADATA OFFSET TO MATCH, AND RE-CALCULATE THE CRC, N64 WILL CRASH!!! So just a warning for hardware compatibility.

[spinout]

OoT uses a binary search algorithim to find files in the file table. Make it a table that is not binary search friendly and you will eventually have problems with the game not finding the file you are looking for. A binary search friendly list is one that has the values in order (smallest to largest or vice versa), so if a file is moved but it keeps the same entry the game will not be able to find the entry in the filetable.

 

 

E.G:

1, 2, 3, 4, 5, 6, 7, 8

 

If you do a binary search for 7, the algorithim will check the 4th entry (half way through), see it is too small, then check the next half way point (6), then it will see the next half way point (7), and return 7 as the result of the binary search. If you switch 2 and 7, you'll get a problem:

 

1, 7, 3, 4, 5, 6, 2, 8

Algorithim checks at 4, sees it's too small, moves up to 6, still too small, goes to read 2, and runs into undefined behavior when it reads 2.

 

This is why using the stock DMA table in OoT/MM is a pain in the ass. The Debug ROM seems to know it is decompressed and therefore does not care about file table offsets unless the table points to a compresed file - at least this is the behavior in emulators.

 

Why does the game use a binary search? It is much faster to look up files, especially when you have over 1.5k files. DMA transfers take up a lot of time so the programmers wanted to speed up the process.

 

How do we work around this in a practical manner?

 

Option 1: Make a tool which rebuilds ROMs and puts everything in order and fixes all the tables.

Option 2: Make a hack which changes how DMA transfers work.

 

I have pondered and unsuccessfully attempted the first option.

I have successfully implemented the second option, with LZO compression to boot. I would like to port this hack/tool to other ROM versions. Hardware tested with no errors.

[john_smith_account]

I've never had trouble with the DMA.  I've compressed the ROM, I've addedto MM.  I've added decompressed data to compressed retail.  I tell you I have no idea where anyone has problems with the DMA.  I'll be happy to post the work around patch here in a few minutes.

 

http://www.filedropper.com/nodma

 

Try it, i solves many woes.  Feed back needed.

[HeavyZ]

Well we went from one person on here having a flash cart to 4, at least people who frequently visit that is. Really we have 5 if we count spinout, but I don't see spinout on here a lot.

We should do hardware tests.

 

Myself, Airikita, DeathBasket, and Arcaith have N64 flash carts.

 

I don't know about the others, but I wouldn't mind testing. Anything to further our knowledge in the Zelda 64 hacking realm

 

Edit:

 

Yeah.. since I was the only one at first is the only reason I know who has them now.

[Airikita]

Someone, then, explain to me why the game crashes with my "damage on horseback" hack when Link is hurt while backing up with Epona... crashes on N64, but not Nemu?

 

Also, @spinout: Your ROM compressor is all open-source files, I don't have time to compile them to use them... why don't you release an exe we can use? Because MOST HACKERS use Windows. Thnxz.

[spinout]

What language is your hack written in? If it is written in raw MIPS you may have violated some rules that the R4300i cares about more than Nemu.

[john_smith_account]

I've never had trouble with the DMA.  I've compressed the ROM, I've addedto MM.  I've added decompressed data to compressed retail.  I tell you I have no idea where anyone has problems with the DMA.  I'll be happy to post the work around patch here in a few minutes.

 

http://www.filedropper.com/nodma

 

Try it, i solves many woes.  Feed back needed.

[Airikita]

What language is your hack written in? If it is written in raw MIPS you may have violated some rules that the R4300i cares about more than Nemu.

I hear your C-hacks crash on console, and my hacks don't. I use MIPS ASM in par with what the hex actually uses in the ROM, nothing is off from what I've coded, except a minor flaw with Epona, but I don't know what kind of restrictions registers would yield.

[spinout]

Also, @spinout: Your ROM compressor is all open-source files, I don't have time to compile them to use them... why don't you release an exe we can use? Because MOST HACKERS use Windows. Thnxz.

http://spinout182.com/?a=z&p=zfsnew ??? There has been a link to a 7z archive for ages.

 

And yeah, my C hacks all have awful hooks. I got my bombarrows working on console with some modifications but I haven't tested anything other than my ROM compressor, bomb arrows, and my octopus lol. Octopus had crazy z axis problems.

[Airikita]

Perhaps I didn't find it... anyways, thanks.