Adding Data to NesterDC SE Library

[Silhalnor]

I am exploring the NEsterDC SE library and am looking into adding some missing images and games but to do so I must access master_nes.db (or my_nes.db that the CD Builder creates locally or ram/mynes.db on a built CD image). I have found a source of screenshots and cartridge scans and box scans but I can't open the .db files.

I am trying to use LibreOffice Base at the moment but it does not support the .db format or at least not this form of it. I can open master_nes.txt which is very informative but the software presumably reads from the .db files not the text files (although if it turns out that I am wrong on this point then that would be most convenient) so even if I edit that file I still need to save it to exactly the correct format. Any suggestions on what may be able to open and save in this format?

Oh, and if anyone is curious here are my sources of images. Note that the screenshots on the second website are not sized the same as those in Nester's library. They are a bit shorter I think. I have not yet compared the scans from the first site.
http://bootgod.dyndns.org:7777/home.php
http://www.vgmuseum.com/

[Silhalnor]

Ok, I found the source code at an archive for http://www.sch3rz0.com and finally got it to compile a database builder that came with it. It has ghttp as a dependency and I found that library but I could not find one of ghttp's own dependencies so I just cut out all the code that uses ghttp and compiled the program without it. I think it was used to access files over a network? Not really a vital feature.
http://archiv.sega-dc.de/www.sch3rz0.co ... index.html

I was actually working on my own software to convert between CSV/txt files and .db files when I got this working. The main advantage would be the ability to convert back from .db to CSV again. I'm nearly done but I'm getting some kind of undefined reference errors... possibly because I'm trying to use his C code with my own C++. I've got what I need now, a database builder, but I'm so close to done I may see about changing my portion of code to pure C and see if that fixes it. I'll happily upload what I've written if anyone wants to take a look but otherwise I'm not going to upload broken code.

I've attached a .zip with the modified dbbuilder and source. I have only compiled it for linux for now but I'll make a windows binary next time I'm using that OS or if I can figure out cross compiling. I may have to look over it's source code in a bit more detail later to figure out the commands to actually use this. I'll post them on here when I do.
And of course credit goes to the original developer. All I did to this code was remove a few lines.

[Christuserloeser]

Excellent work! This needs an update real bad.

[Silhalnor]

Oh hey, and thank you! Haven't been checking my email lately so I didn't notice you until just now. (I never get interesting mail.)

I have a deadline on something at the moment so I haven't had a chance to finish up my work on this but it is definitely on my todo list. I did successfully make a new .db file though so even if I don't get it all packaged up nicely I'll at least upload the pieces you need to do it.

If I don't post again in a few weeks PM me. I may have forgotten.

[Silhalnor]

Hey, I'm back. I haven't made an easy to use application to streamline this yet so here are some instructions.

Spoiler!

First, were are the acceptable arguments for the nesdb_builder.
I'm afraid this still only works under linux.

./nesdb_builder-linux tab INPUT_CSV OUTPUT_DB
Converts your CSV into a DB file.
./nesdb_builder-linux build INPUT_CONFIG
Seems that this option takes a config file to produce the database but I don't a guide for you here.
./nesdb_builder-linux test
No idea. Might be here for development purposes. I get a segmentation fault when I try it.
./nesdb_builder-linux dump INPUT_DB
Prints the name of every entry in a database. Works on .DB but not .CSV.

Collect and organize your files
Collect all the screenshots, scans, manuals, and everything else you may want and arrange them in your NesterDC library. They must all be named a 4 digit number, any number. I used the same number for all documents for a given game.
I have attached a tiny example library as a guide. It has documents for the Mario series and two imports. It also has has a .CSV and a .DB that you can use as templates or for reference.
FAQs can be whatever text file you want (walkthroughs are expected but summaries or histories or anything would work quite well). The library hosted here at DCEmulation doesn't include any, probably for copyright reasons.
GameGenie uses .xml files. I don't have a concise way to explain these but I have a few in the mini-library for reference.
Each game's manual must be JPGs contained in it's own subdirectory named 00.jpg, 01.jpg, 02.jpg, etc. Again, check the library for reference but note that the ones I included are tiny to keep the filesize down so you can't really reuse them.
NSF files are the only files which must be zipped. It's a special format for NES music.

Put together the CSV
Import a .CSV into a spreadsheet (I have one included in the mini-library) or create a new spreadsheet with the following fields in the top row in this order: CRC, Name, TitleShot, ScreenShot, Manual, FAQs, NSF, BoxScan, CartScan, StrategyGuide, GameGenie.
Put in the names of every game you want to add. I used the filename without the extension to ensure that it would work but you can probably use anything.
Run CRC32Reader (I have Windows and Linux versions attached) from the command line with the ROM file as the argument like so: CRC32Reader-win "Super Mario Bros. (JU) [!].nes" The ROM must not be compressed. It should give you an 8 digit hexadecimal code like "d445f698". Prefix it with "POOP" (all caps) (I no idea what it means) and paste it in the CRC field for the matching game. "POOPd445f698"
All the other fields take the 4-digit name of the corresponding file. "0845.jpg" should be put in as "0845".
When you are done export your file to CSV.

Generate the .DB
Next run nesdb_builder-linux from the command line followed by the CSV filename and then the desired output name like so: nesdb_builder-linux master_nes.csv master_nes.db
Put the master_nes.db in the root directory of the NesterDC library and now you can use the NesterDCSEBuilder.exe to compile your collection.

Got it? Yes? Great! No? What are you having trouble with?
Note that all my information here is reverse engineered from the available files. Some of my information are just guesses because I didn't want to waste a CD-R. Do you have to use 4 digit numbers? No idea. Try a 5 digit number or a name and see what happens. Maybe NSFs don't really have to be zipped. Maybe JPGs can be. Maybe it's case sensitive, maybe it's not. Maybe you don't have to prefix your CRCs with POOP.

Important but optional information
NesterDC is picky about images. First of all they must be JPGs. I haven't pinned down all the requirements and I still have pictures that don't work, but the main thing is that it requires that the pictures use an indexed color palette. At least, that's the first thing that I was able to get working. I don't know if the palette must be of a certain size but 256 colors worked for me.
One thing it isn't picky about is the image's size. I found this out when I added the cartscans for imports. You can use pictures that are tiny, large, and of any ratio and it will work just fine and display properly in the emulator. But this does mean that you can't just throw in pictures and expect them to look good. I used a high resolution scan for one game and it more than filled the whole screen. Didn't look too good.
Another thing, NesterDC sometimes crashes when I access the game's manual. I'm guessing that it's a memory issue so I recommend that you keep their JPGs small. You could even try splitting each page in half to keep them small but readable. Edit: I went back and checked my files and now I'm certain it's a memory issue. My screenshots and scans average 5-20kb but many of my manuals are high resolution and average 500kb or more. Resize yours to to 1000x400 or so and you should be fine.

I don't know what the Strategy Guide files are supposed to be. I'm going to throw out a guess and say they are text files that serve the same general purpose as the FAQs files. Please give feedback if you try this! I didn't think to experiment with it when I made mine.

I got my NSF files from http://www.zophar.net/music/nsf.html

PS. The only GameFAQs guide I included in the mini-library is one that explicitly gives permission for reproduction in the document itself as long as the source is clearly labeled.
http://www.gamefaqs.com/nes/525243-supe ... faqs/12730
Similarly, credit for the 5 NSF files go to zophar.net

PPS. Almost forgot to mention, while I did put together CRC32Reader I used code that I stripped out of db_builder so credit for those functions go to the original developer. I don't know his name offhand but I've already cited his website from before.

[Christuserloeser]

^ Excellent work, Silhalnor