Planning for new map importer

[xdaniel]

This is something where as much input as possible is required, from beginners, advanced modders and veterans alike.

 

So here I am, contemplating where to go with this whole OoT/MM map importing business. So far, every rewrite of SO that I've started stalled for one reason or the other, and there's been two big ones as far as I can make out:

• Too little planning beforehand - Where do I start? What does the program do? What will the user need to do? What should I let the user do and what should be done automatically? And a lot more...
• Too big of a focus on sophisticated code - getting hung up on Reflection, XML, modularity of the code, etc., etc.

The second reason I have to work on alone. I have to find a balance between code that simply works and code that is clean and "professional" for the lack of a better description. The first one, the planning, is something I would very much like the input of every hacker/modder on here for.

 

Here's some questions to start:

• Question for the advanced users: How much abstraction should the program do? Which aspects of a scene or room file should be in the program in the rawest form possible, and what could be abstracted to the point that it's simple for end users to get a result, but that one would need to dive into the generated files with a hex editor to change specific, less important properties?
• Question for the beginners: What is difficult to understand or perhaps even intimidating about the current map importers, SharpOcarina in particular? What do you think is missing from SO, what could be made easier?

And that's about what comes to mind right now. I really want to create something worthwhile, a map importer that's simple to use, where you can just dive into without reading the Readme file (hi haddockd ) and still get an acceptable, playable result from, but I will need the help of everyone who might be interested in the end result.

 

Do keep in mind that I cannot promise that I'll be able to bring this to fruition. I hope this thread, with hopefully beginners and veterans alike voicing their opinions and concerns, will serve as inspiration for future programs regardless, no matter the outcome of my potential program in particular.

 

[Conker]

I don't really use things like these much, but I wish you luck.

[DeathBasket]

 

[*]Question for the advanced users: How much abstraction should the program do? Which aspects of a scene or room file should be in the program in the rawest form possible, and what could be abstracted to the point that it's simple for end users to get a result, but that one would need to dive into the generated files with a hex editor to change specific, less important properties?

 

I don't know how hard or possible it'd be but having two interfaces - a 'simple' and an 'advanced' - may go a long way in both making it easier for less experienced people while still providing options for more advanced users and as best possible eliminating the need for a hex editor entirely. I'm thinking that the program would read/write the same stuff in both cases but filter out the more obscure or advanced options in the simple interface. The simple interface could have just the basics like setting different collision types, actors/objects, etc. based on predefined lists whereas the more advanced one would allow you to go more in-depth to change each property.

 

Off the top of my head, some options that could be simplified would be:

Map time/timeflow (define a/no set time, timeflow on/off)

Indoor/outdoor/skybox

Collision types (use a list)

Actor -> object list (making sure all required objects are loaded? Would only work for ROMs that aren't hacked to change these)

Waterbox adding (room waterbox is loaded in could be calculated from its position for multiple-room maps?)

BGM (use a list, again only for unhacked ROMs)

Display List transparency/backface culling

Environment/fog colours

 

For more advanced options:

Timeflow speed?

Define own collision types

Manually edit actor number/object number

Add/edit multiple map headers for cutscene/alternate maps

Define a jpeg background

Camera settings (still not fully understood afaik)

Cutscene editor (I really don't expect this, just putting it here for completeness)

Map effects (hot/shaking screen etc.)

Flags map (to allow the user to keep track of which flags are being used by what, maybe able to attach certain actors to flags)

Set area text image

Minimap image? (not understood afaik)

Vertex colours (this would be difficult unless it is supported by the model format or maybe set vertices in defined areas to be a certain colour)

Moving/swapping textures (don't expect this to be done)

Texture combiner

Waypoint adding/editing

Some way to add header instructions and blocks of data for putting in pre-made cutscenes or whatever

 

I'm sure I could go on but just giving you an idea of what I can think of. I'm just thinking the simplest interface would have the bare minimum of options you need to make a fully functional map while switching into an advanced mode would give you almost complete control over what's going on.

[giadrosich]

SharpOcarina I could not figure out straight out of the box. This is it's major flaw. I had to get someone else to show me how to work it. I read the readme, and it was no help. Just being honest.

 

It's intimidating. There's a lot of buttons and knobs that I don't know what they do, and don't even know how to ask what they do. Like the term "waterbox". When I was first starting, my thoughts were "This makes no sense to me, what is it, how is this used, how do I put it into the map and what does it change?"

 

There's no indication you have to click two tabs to import your map, this didn't make sense until shown to me. There's also no indication on helping with going to a different area. You have to click on another tab to imput the information for that. I was going crazy trying to figure out why the whole map was taking me somewhere rather than the one area I highlighted. Once again, someone else had to tell me to click on a different tab and input the correct number at the bottom.

 

Now, obviously, this is what tutorials are for. But I think what would make ANY program awesome is a built-in tutorial mode. You download the program, open it up, and it says "hey, would you like to use the tutorial? Yes/no". If you accidentally hit no, you can then access it in a menu. Then, when you access it, for example, you choose "import map tutorial" and then it shows you what to click on outlined in little red boxes. If this doesn't make sense, I can post a screenshot of what I mean.

 

That way, the user is still having to go through the motions of importing their map, but it's being shown at the same time. You'll still have to make the map in a different program of course, then import it, but with everything they need like adding the collision to go to a different area, this could be shown. They would still have to know the numbers to input (unless you use a drop-down menu with labels of the different areas you'd like to go to), but then they wouldn't get lost.

[LingeringWraithX]

But I think what would make ANY program awesome is a built-in tutorial mode. You download the program, open it up, and it says "hey, would you like to use the tutorial? Yes/no". If you accidentally hit no, you can then access it in a menu. Then, when you access it, for example, you choose "import map tutorial" and then it shows you what to click on outlined in little red boxes. If this doesn't make sense, I can post a screenshot of what I mean. 

 This would've been great. It took a week to actually learn what some of the buttons do, It wasn't till I sign up here that I learned that an object was a "group" for actors. And one thing that bother me was actors code wasn't really explained

 

0002(Actor) 0032(Object/Group) Stalfos- 0000(Varible) = Invisible Stalfos - becomes visible with Lens of Truth, plays Mini-Boss theme

 

Having known this would have been helpful for me awhile ago 

[Strati]

Functions to add from the top of my head:

An option to chose the time flow, static day or static night.

Creating different object sets (I don't really know how the game chooses which one to load, so maybe it goes beyond info on scenes/maps)

 

The main problem is the need for external databases.

I remember constantly having to open the wiki then converting between hex/decimal to get the BGM I wanted. Or go through a battery of testing to find out the value that would make the floor ice-slippery or what each of the 12 basic environment lighting sets stand for.

Having the values to use being more accessible, in dropdown lists would save a good deal of time.

 

There is also something with alpha textures and skybox outlines. To be honest, I think I read a thread about this a while ago, but I don't know if there were any solution:

 

On the topic of intuitive interfaces, I think it's very straightforward if you know what you're going to do. The idea that someone shouldn't read the readme to use a program doesn't seem fair. For me it's very important that the person insterested in hacking a game do some research on how things work. The dungeon demo that comes together with the program serves as an example to most of the program functions, if you care to open it and study it a bit.

Edited April 10, 2013 by Strati

[giadrosich]

 

On the topic of intuitive interfaces, I think it's very straightforward if you know what you're going to do. The idea that someone shouldn't read the readme to use a program doesn't seem fair. For me it's very important that the person insterested in hacking a game do some research on how things work. The dungeon demo that comes together with the program serves as an example to most of the program functions, if you care to open it and study it a bit.

 

 

The thing is, when I first started, I couldn't figure out HOW to open up the dungeon demo, because you have to know to click on two different tabs to import it in. A readme is useful if it explains the functions and how to do something, to me that's what a readme is for. After I was told how to import a map, then I could look at it. Until you get it into the program, it's useless.

[Strati]

But it's there...The usual procedure for any program would be File->Open:

- Open Scene: Opens a previously saved scene XML and loads all associated data (model files,etc.).

There is only one XML on the Demo folder. Just open it and you have the demo in for a start.

[giadrosich]

Opening the XML didn't work for me. Yes, I did try that.

 

I have to click new scene, then import the collision model (obj file), then go to rooms and add a room. Importing the whole XML file doesn't work for some reason. It gives you an unexpected error (at least for me). *shrug*

[Secant]

While I'd love a program that's able to handle just about everything under the sun, going by your concerns from your other thread today, I'll try to keep this list brief and realistic. If possible, it may help a lot to have some features kept in, say, a separate tab/menu/whatever for more advanced, specialized maps, to keep the interface minimal enough so that it doesn't confuse new users. Thus, I'll first elaborate on what I loved about SO and what made it easier to work with than spinout's converter:

• GUI and ability to preview maps before importing
• Ability to manually choose and position actors within the GUI preview
• Greater control over parameters of map/scene header

Naturally, I'm assuming you're planning on keeping these core aspects; if so, you're already off to an excellent start. As for the basic features I think just about everyone could take advantage of:

• Skybox settings and time flow speed
• Background music
• Waterboxes
  • If at all possible (and I can't remember if this was implemented in SO or not, or if this is how waterboxes actually work), it would be really awesome to be able to modify the xy coordinates of a waterbox's corners with just the mouse, or, say, clicking on it and having a special menu for manually editing the location of the vertices
• Some form of actor database a la Utility of Time which gives descriptive names to actors as well as the variable for each actor
• Collision type editor, preferably with a possible point-and-click to select a set of polygons and then have labeled collision types
  • Bonus points for having separate labels for both walls and floors so newer users don't get confused

For most basic maps, this should suit the casual modder, I think. As for more advanced features, which I expect would come chronologically after the basic core features have been done:

• Object set modifier, and it would be really helpful if we could have a way to name these object sets to keep track of them, which might be saved in a project's XML file
• Map exit handler for transitioning to new scenes
• A special case handler for transition actors to give them the appropriate variable to work properly would help a lot as well (say, click on the room or enter the room number you're entering from, then the one you're going into)--this is something I've been hung up on for a long time and never quite got the hang of
• Vertex shading/combiner settings
  • Sort of nitpicking here, but this would help save us 3D modelers a lot of extra textures having to recreate certain effects like a shadow gradient on a map's exit and the like, and I'm basing the possibility of this feature on seeing it in Sayaka, though do correct me if I'm in error about it being doable

And that's about all I can think of. For a more complete set of features, I definitely stand behind DeathBasket's list, but these are the things I consider to be most important for what I might use.

[xdaniel]

On "simple" and "advanced" interfaces: This is a very good idea. While I think two completely different interfaces might be too much work, showing and hiding certain controls depending on the selected mode is easily done. Maybe the program could have a PropertyGrid (what ex. VC# itself uses to change the properties of controls) that allows access to all properties in a project, which is hidden in "simple" interface mode? What exactly should be part of the simple interface, tho, and what should be left to advanced users?

 

On vertex colors (and alpha): I'd be using my existing model handling library for this, used by Model2N64, which parses everything it can load into its own, internal model format anyway, and which should support vertex colors. And even if not, it should be trivial to add support for that. Of course, .obj files don't support them, so all colors would need to be set in the importer itself, so they would be lost if the original model is re-imported into the program. Because of that, they should probably be among the last things the user adds, when they don't need to change the model externally anymore.

 

On manual combiner setup editing: This is something I'd personally love to see as well. At least manual editing of this would only be available in advanced mode, while maybe a few presets could be available to the simple mode's user? Or should the program rather auto-detect the appropriate settings, ex. for translucent or untextured surfaces?

 

On implementing a tutorial: An interactive tutorial, ex. highlighting which options the user has to change or where they have to enter something, is probably beyond my capabilities. Adding some kind of "help mode", which explains options if you click on them in that mode, or adding an external tutorial in ex. HTML form with screenshots, etc. would be doable for sure.

 

On databases of actors, objects, etc.: This would also be doable. Actor or BGM selection would consist of a drop-down list, which in turn would be provided by ex. an external XML file loaded by the program - in fact, I think using and expanding upon Sorata's actor definition system idea, where the user interface is built around the actor in question via per-actor XML files, instead of the actor's properties being shoved into a single number, position, rotation and a variable, might be a good idea.

 

Finally, as for SO's demo scene not loading, that's most likely one of the stupid bugs SO has. If you open the XML file in an editor and replace all filenames inside ("collision.obj", etc.) with the full path to the file (thus "C:UsersxdanielSOcollision.obj" or whatever) it should work. Or that's what I seem to remember from the previous times this problem has come up...

 

Fake edit, replying to Naxy's post, or the parts not covered above in particular: Multiple object sets I'd certainly want to support. I'm not actively modding, but it has still kind of bugged me that this hasn't been implemented in any(?) importer yet.

 

[Secant]

Of course, .obj files don't support them, so all colors would need to be set in the importer itself, so they would be lost if the original model is re-imported into the program.

 

This actually brings up something I'd forgotten about for a while; is it worth considering a different file format for imports that may be more compatible or have more features or just be easier to work with? I seem to recall Arcaith suggesting that the .x export format was pretty solid and simple to use, and I think .3ds is so widely used and supported that it may be worth going with that as well. I do realize that because this is such a fundamental change it will probably be a pretty significant setback, hence why I feel uncertain bringing it up, but just tossing it out there for consideration.

[sairugoth]

I think DeathBasket said most of whats needed to be added, and also, I agree with Giadrosich, a tutorial is much needed. When I first started using SharpOcarina, I was so confused on how to use it, I had to ask (you)/xdan how to use it. ( but that was just me jumping the gun, I didn't even read the "ReadMe" ) But hears the thing tho. When newbies come to the GCN and want to mod, there going to look for something easy, (most likely not a hex editor) and when they see downloads to apps that are made for modding, there going to think its going to work like a charm. That is until they see a "XML" and a ".obj" file names, and then there going to say "WTF are those" The program dont really tell you that you need a whole other 3D program to actually make the rooms to actually make the ".obj" for SharpOcarina. So the newbie just downloaded a app that is useless until he/she either finds Sketch-up or some other 3D program, which I dont think is even mentioned in the "ReadMe"(correct me if I'm wrong on that one) but even if they manege to get Sketchup there's also the problem of finding a free exporting ".obj" plugin. So hears my suggestion, what if you could put Sketchup like control's in it. That why the user could just make what they want on the spot and then edit/make the climate, and then just insert into rom. It would be much simpler for new users. 

 

[xdaniel]

Naxy: At least in the case of said model handling library, adding support for a new format would "only" mean writing a "model adapter" class for it. That class is what's responsible for loading and/or saving of a specific model format, as well as creating and providing an "intermediate model" object (the previously mentioned internal model format of the library) which is what's used by the program that uses the library. Of course, the whole idea might sound quite simple, but it would still require documentation about the format and some time and effort to write a parser for it.

 

sairugoth: Yeah, SO - and most of our tools I'd guess - do require a bunch of knowledge about the games to use, from concepts (like how the collision is different from the room model) to terminology (like .obj's or waterboxes or polygon types or whatever else). That's really something that needs to change. Programs should at least be able to ease users into using them, like through a tutorial by which they learn about those words and concepts. I'm not sure about creating an actual model editor in the vein of SketchUp, tho - that is quite a task, and I personally don't think I could do that.

[sairugoth]

sairugoth: Yeah, SO - and most of our tools I'd guess - do require a bunch of knowledge about the games to use, from concepts (like how the collision is different from the room model) to terminology (like .obj's or waterboxes or polygon types or whatever else). That's really something that needs to change. Programs should at least be able to ease users into using them, like through a tutorial by which they learn about those words and concepts. I'm not sure about creating an actual model editor in the vein of SketchUp, tho - that is quite a task, and I personally don't think I could do that.

Well, it was only a suggestion

[SanguinettiMods]

I'm really agreeing with Secant and DeathBasket. I really want to push an accurate preview of how it would look in-game because it really is a hassle to open it in an emulator every single time you change a small thing. Among this, I'd also like to ask if you could look into 64 x 64, 128 x 32, etc RGBA16 Texture Support as those kinds are used in Majora's Mask, specifically in clock town. Because those add up to 4096 bytes, it should be perfectly feasible.

[xdaniel]

Are you really sure about those being 64x64 RGBA16 textures? 64*64*2 is 8192 bytes, so double the size of the texture cache. I'll have a look, but I'm pretty sure that can't be...

 

As for an accurate preview, that's another thing that's certainly doable - well, as accurate as one can make a Ucode interpreter/renderer anyway.

[DeathBasket]

Do you think it'd be doable to have accurate lighting and fog available in the preview? This is probably the one thing it took me a lot of time to get right when using SharpOcarina since the preview wasn't great as I remember.

 

 

 

On "simple" and "advanced" interfaces: This is a very good idea. While I think two completely different interfaces might be too much work, showing and hiding certain controls depending on the selected mode is easily done. Maybe the program could have a PropertyGrid (what ex. VC# itself uses to change the properties of controls) that allows access to all properties in a project, which is hidden in "simple" interface mode? What exactly should be part of the simple interface, tho, and what should be left to advanced users?

 

Showing/hiding based on whether you want advanced settings or not was probably more what I was meant to be getting at. The simple mode should be really basic and allow the user to quickly define the sorts of things needed in the map. This would be pretty similar to SharpOcarina I guess but a little more simplified - maybe stuff like double- or right- clicking a display list or collision polygon or even actor to get at a menu allowing you to edit its settings. I think things like options to automatically snap actors to the nearest polygon surface (if possible) could come in handy too. Also for actors, the user should be able to pick from a set list and have their objects automatically added with them, maybe with a section telling them how many more actors/objects are available to place in the map if needs be.

 

For the more advanced users, being able to define our own lists with actor -> object mappings, music sequences, collision types, etc. and save those files to load at any time when switching between ROMs would be great. Even further down the line, these files could then be released along with a ROM hack so that others could mess around with it without having to figure everything out again.

 

Again, not sure how much of this is feasible, just throwing out some ideas.

[xdaniel]

I think accurate fog might be more likely than accurate lighting... I've always struggled with getting lighting to appear right, while fog should be possible to get going correctly with a tiny bit of experimentation.

 

Also, regarding all the interface ideas that have been brought up, I'll have another read through them later and see what I think is feasible to implement for me. Those would be a lesser concern for the moment, tho, until the basic model conversion stuff is working. Whatever the case, I guess I'll mess around a bit in VC# later...

[Mallos31]

Accurate lighting is really really hard. I'm pretty amazed with the people at companies like Autodesk who made Maya and 3dsMax who get the lighting right so easily. It's crazy.

[haddockd]

xDan,

 

I have a more fundamental suggestion (coming from the guy who has alot going on, which is why the GUI isnt already finished): it is more important to have a solid, finished tool, than to overwhelm yourself with all these feature requests. I am not saying having a tool with all those features would not be very valuable, but perhaps get 5 or 10 solid features and make a release. Then if the mood strikes you (which you and I are notorious for...lol) you add a few more features (while you fix any major bugs). Release. Rinse and repeat until you are happy with the result. This is going to be a big undertaking so you need to ask yourself..."can I finish this with a realistic set of features?".

 

Also a note to the community, xDan is one of (if not the) most talented C#.NET programmers on the forum, so please keep it in the ball park of realistic so as to not overwhelm him. If this sounds harsh, it isnt meant to be, just wanted to share my thoughts.

 

My silly, silly, thoughts

[wareya]

Some interface rambling here.

 

Doing what it's supposed to do, well, is the most important thing; anything else should have be abstracted differently in the interface if it's present. You want to create abstraction away from the internals of the program where it's necessary (because users are using it to mod or hack the /game/, not the tool), and indirection wherever it's necessary to simplify things undestructively (like with the level editors, having input fields for actor data instead of showing a labeled hex string to edit, which is something I take for granted).

 

So, in the context of a level editor again you could include an actor list in by default, and let people see the names of entities human readably (eg "Lake Water" or "Hylian Guard"), but absolutely hiding the number of the actor itself would be bad, because with advanced use letting a user work with the classical abstraction of the actor number would be a good thing. It's a shitty example because it seems really inane and arbitrary but I feel like it's got the right notion.

 

I'll be keeping track of where this goes and give any feedback if I have any, of course.

[haddockd]

About time you showed up...heh

[giadrosich]

I agree that if you did have some sort of drop down list with actors and such, you would have to show both the name and the number.

 

Also, random side note, there are multiple entries sometimes for what seems like the same things, example:

 

000A:000E:0800:Treasure Chest (holds compass)
000A:000E:0801:Treasure Chest (holds compass)
000A:000E:080B:Treasure Chest (holds compass)
000A:000E:0820:Treasure Chest (holds dungeon map)
000A:000E:0822:Treasure Chest (holds dungeon map)
000A:000E:0823:Treasure Chest (holds dungeon map)
000A:000E:082C:Treasure Chest (holds dungeon map)

 

Why are there that many? Is this just different variables to use as examples?

[SanguinettiMods]

I agree that if you did have some sort of drop down list with actors and such, you would have to show both the name and the number.

 

Also, random side note, there are multiple entries sometimes for what seems like the same things, example:

 

000A:000E:0800:Treasure Chest (holds compass)

000A:000E:0801:Treasure Chest (holds compass)

000A:000E:080B:Treasure Chest (holds compass)

000A:000E:0820:Treasure Chest (holds dungeon map)

000A:000E:0822:Treasure Chest (holds dungeon map)

000A:000E:0823:Treasure Chest (holds dungeon map)

000A:000E:082C:Treasure Chest (holds dungeon map)

 

Why are there that many? Is this just different variables to use as examples?

That's just the chest identity. You can see that the item doesn't change.