help with files with (possibly) wrong checksums

This forum is for the discussion of dumping games. Get help with your dumping here.

Moderator: Moderators

kirk
Posts: 246
Joined: Wed Aug 20, 2008 9:22 pm

help with files with (possibly) wrong checksums

Post by kirk »

Hi, I've dumped a couple of games using httpd-ack-20071123, but the checksums don't match up with dumps by other people.

e.g. for F355 Challenge (previous dump info here) I have these results:
File Info wrote:disc.gdi size 87 crc d7ab4c6b md5 b3ecdf113fd42344dd84142679e4cb36 sha1 9d50b816d3f69ad192f5986ab487769965c2e082
track01.bin size 705600 crc 479e98f3 md5 a5a0fbee624c3a8b490a785f800b7d28 sha1 f2e7e8be6943cddbaa7835dc0a12dda4729aafeb
track02.raw size 1735776 crc cecd171a md5 25ae1f52389cb9de0034bcd858904362 sha1 1e45252125b0a1795373ac6c92c8839695386545
track03.bin size 1185760800 crc aa397d65 md5 41539c8d7f6f57b1920633f53528cad9 sha1 f9e581155dd2296ccb392d936b337e17c925050f
My disc has the same ring code as in that post.

For Space Channel 5 (post here):
File Info wrote:disc.gdi size 147 crc 0ef1e42a md5 e482617597842ed70c2b8204a06df57b sha1 9e4a4eb1dad09c6ae5bf4874618960b67172583d
track01.bin size 1992144 crc bc278945 md5 012ec80d1e421e98a0357b8787a25ec1 sha1 2a8b0c330229bd0b0fadcb99525725ab598561ac
track02.raw size 1735776 crc 11bbc18f md5 8bafd2d23c80e5539dbd5ed65c5b7c4c sha1 2b6fb002a412c2ac1fbe75f758e936efd59e7cb4
track03.bin size 30013872 crc e2ad3607 md5 b83b97ac92a149e707a3ad3b9d902c19 sha1 eb7a49a2bff70151693dd9452190193b7751532a
track04.raw size 1559376 crc ac45505c md5 30fb9f4d6e31683c4e59b6d0e03205e7 sha1 180d52e63e8f3f9c595153953d592b46723abe9d
track05.bin size 1153481952 crc 93765453 md5 958ddbb47733b15da371f2288f84f516 sha1 bd7b9f62c747894b63ea8c8db955e3e7d645bf05
The ring code is slightly different on mine.

Is there a problem with the hardware, or am I doing something wrong? I'm using the script from this thread.
kirk
Posts: 246
Joined: Wed Aug 20, 2008 9:22 pm

Post by kirk »

I ran CDmage on the track01.bin files for both F355 and SC5, and it found 1 corrupted sector in each. I've tried manually downloading that file from the DC a few times with the same result. Any ideas?
hotaru
Posts: 98
Joined: Thu Jun 28, 2007 8:41 pm
Location: twilight

Post by hotaru »

@kirk- welcome aboard. :D what condition are the disks in?
kirk
Posts: 246
Joined: Wed Aug 20, 2008 9:22 pm

Post by kirk »

hotaru wrote:@kirk- welcome aboard. :D
Thanks. :)
what condition are the disks in?
They are in excellent condition.

For one game (Virtual-On OT) I have two copies... the CRCs match up for both discs, but CDmage finds one error on track1 (seems to always be the last sector). Is there some CDmage setting that I'm missing? I tried downloading using other methods (IE7, Firefox) besides that script with the same problem.
hotaru
Posts: 98
Joined: Thu Jun 28, 2007 8:41 pm
Location: twilight

Post by hotaru »

@kirk- shouldn't need to config anything in CDmage 1.01.5 other than to select M1/2352 when opening TRACK01.BIN or SAMPLE_GAME_TITLE.CUE... hmmmm. walk me through your dumping & verification process including all tools used in detail from beginning to end.
kirk
Posts: 246
Joined: Wed Aug 20, 2008 9:22 pm

Post by kirk »

Hi, hotaru. I appreciate your help.

The Dreamcast is an Asian (NTSC/J) unit, model HKT-3010, made in Japan, unmodded. The physical condition is excellent; it had been in the box since I last used it years ago. It doesn't make any strange noises or have other issues.

This is a DC that I bought new years ago, mainly as a spare. I just cleaned the lens using a lens cleaner for soft lenses. I have a broadband adapter model HIT-0400. It's connected to a monitor using an official VGA box.

On the Dreamcast, I set the TCP/IP settings to have static IP using the broadband passport and it seemed to work fine (browsed to google).

I downloaded the httpd-ack 20071123 zip file, converted the CDI image to NRG fomat using cdi2nrg, and burned with Nero 6. It runs on my DC and gets an IP on my home network.

The computer and the DC are both connected to a Dell gigabit switch using straight-through CAT6 cables. I've tried different cables, and also connected them to the built-in 10/100 switch on a Cisco PIX firewall instead of the Dell.

After loading httpd-ack, it gets to the black screen with green text, so I eject the CD and insert a DC game. Since I'm testing now, I'm just downloading with a web browser. I've tried downloading track01.bin using the web browser on two computers... Firefox 2.0.0.16 on a Linux laptop, and IE7 on a Windows 2003 server.

After download the track01.bin, I create a checksum file using cfv, to check the value and to make sure there is no corruption if I move or copy the file.

Once I have the track01.bin on a Windows computer (either the Windows 2003 server, or a Windows XP virtual machine on the Linux laptop), I open CDmage (downloaded from the link in the previously mentioned thread), click File > Open, change file type to M1/2352, open the track01.bin, click Action > Scan for Corruption, click All checkable tracks and click Scan. The track01.bin from SC5 and F355 challenge both give exactly one error. For all of the troubleshooting steps I've taken, I've always got the same CRCs as posted above for these two games.

On my Linux laptop, I used tcpdump to save a package capture of the file transfer of track01.bin, and checked it in Wireshark. There aren't any packets with errors. I disabled all TCP/IP offloading with the same result.

Do you have any suggestions?
hotaru
Posts: 98
Joined: Thu Jun 28, 2007 8:41 pm
Location: twilight

Post by hotaru »

@kirk- not sure. have you tried generating file hashes for your dumps w/ WinHex 15.0 or clrmamepro 3.118? interesting that you're returning the same hash values w/ each consecutive dump...
kirk
Posts: 246
Joined: Wed Aug 20, 2008 9:22 pm

Post by kirk »

Some of the CRCs do match up, so I don't think it's a problem with the checksum program.

Anyway, I found my other Dreamcast and I'm getting the same exact results (CDmage gives an error on any track01.bin that I download). The second Dreamcast model is also HKT-3010, but it has a region mod installed. The unit is a bit older and has had the GD-ROM replaced.

I also tried burning the httpd-ack image using the actual DiscJuggler program, and the results are the same.

I'm really at a loss as to what is wrong. I only have one broadband adapter so I can't really test that. :(
User avatar
Maddog
is awesome
is awesome
Posts: 1599
Joined: Sat May 12, 2007 4:12 pm

Post by Maddog »

Hmmm, this is a really baffling problem. :?
You get correct checksums for the last track in both cases, which pretty much rules out any random hardware or software error.

How about uploading the problem tracks somewhere (rapidshare or any webspace you might have), so I can do a direct binary comparison to the verified versions I have here? This might help detecting the error, esp if it's always the same error. Checksums only can show so much (ie they show there's a difference, but can't differentiate between 1 byte diff and a wholly different archive)
Size for tracks 1-2 is small enough and you shouldn't have a problem even if your line is slow. Please upload them for both games. If you got a decent one, upload 3-4 of SC5 as well. PM the URL if you don't wish to openly post links to copyrighted material here.
Btw, CDMage shouldn't be reporting errors for Dreamcast tracks.
kirk
Posts: 246
Joined: Wed Aug 20, 2008 9:22 pm

Post by kirk »

PM sent. Regarding CDmage, I was only using it on track01.bin, which it always gives an error.
User avatar
Maddog
is awesome
is awesome
Posts: 1599
Joined: Sat May 12, 2007 4:12 pm

Post by Maddog »

I am still baffled somewhat, but at least I now got some very good clues by binary comparing the files.

Regarding data tracks (tracks 1 for both games and 3 for SC5) :

In all 3 cases, the sync field of the last sector is wrong in your dump. This results in unrecoverable read error (this is what CDMage detects, you can see the same thing by mounting the track in Alcohol or Isobuster-you can't do a sector view of last sector).
Sync field is always 00 FF FF FF FF FF FF FF FF FF FF 00 (=00 followed by 10x FF followed by 00). This is part of the CD-ROM standard.
In your dumps, last sector's sync field is being read as 00 FF FF FF FF FF FF FF FF FF 00 00, ie there are only 9 FF and last one is for some reason changed to 00. Nothing else is different in the entire data track dump.

Regarding audio tracks (tracks 2 and 4 as applicable) :

Non-zero bytes sequence is exactly the same as in the verified dumps. The difference is that the point at which the non-zero data starts (and of course ends...) is different. Difference is always the same, implying a drive that reads with a different offset!
For example, in track02 of SC5, non-zero bytes in the verified dump start at file offset AE4 (=2788), while the non-zero bytes start earlier at file offset 920h (=2336) in your dump. This 452-byte difference implies a reading offset different by 113 samples.
A similar thing applies to F355 and difference is again 452 bytes, which is consistent with previous findings.
As an experiment, your audio dumps should all have their checksums matching ours if you add the relevant amount of 452 zeroes to the start of the .raw file and remove the same amount of bytes from the end of the file.

The thought of a Dreamcast with a different drive offset is very intriguing. The only thing that weakens this theory is that we never bumped on any DC with different offset and we have several different people with all regions of DC contributing (US/PAL/JPN consoles). So far not a single console was found different, which makes it somewhat harder to believe that you happen to have two of those. Still, I cannot and will not deny the possibility.

On the other hand I have absolutely no explanation for data track sync field error for the last sector. This is such a blatant error that is simply shouldn't be there in any case and I doubt it can really be explained even by you having a DC with a different offset. It's not like it's reading the postgap or anything like that, it just changes a single byte from FF to 00.

So I guess we can safely assume there's nothing wrong with your PC, BBA, cabling, DC lens, discs or anything like that. There seems to be something weird going on with the way your DC is reading the discs. Maybe it really has a different drive model inside, but this is just a somewhat weak theory. The data sync thing is really unexplainable for me.
I still have no clue what you could do to fix the problem, apart from hacking the files to fix the sync field error and juggling audio data around as described. Wonder if anyone else has a better idea to what might be going on here... :?
kirk
Posts: 246
Joined: Wed Aug 20, 2008 9:22 pm

Post by kirk »

Thanks for the information. If you think of anything else, or need any more info, let me know.

Both of my Dreamcasts are Asian units, i.e. they came without a modem. Maybe there is some difference between the Asian and regular Japanese Dreamcasts.
kirk
Posts: 246
Joined: Wed Aug 20, 2008 9:22 pm

Post by kirk »

Maddog wrote:The thought of a Dreamcast with a different drive offset is very intriguing. The only thing that weakens this theory is that we never bumped on any DC with different offset and we have several different people with all regions of DC contributing (US/PAL/JPN consoles). So far not a single console was found different, which makes it somewhat harder to believe that you happen to have two of those. Still, I cannot and will not deny the possibility.
In Quzar's thread about the GD-ROM drive info, I posted the results for my two Dreamcasts. It would be interesting to compare the revisions to some of the Dreamcasts in use by contributors.
User avatar
Maddog
is awesome
is awesome
Posts: 1599
Joined: Sat May 12, 2007 4:12 pm

Post by Maddog »

Looking at early results from myself, Quzar and BlueCrab over DCEmulation, it looks like your Dreamcasts both have very early firmware revisions (both have pre-Japan launch dates)
This might be a plausible reason for the dumping problems you have had, especially the data track sync thing (who knows, it might have been one of the things fixed in later firmwares)

Also, it strengthens quite a lot the possibility of your Dreamcasts having a drive with different offset, explaining the audio mismatches too. IIRC, early models were built somewhat differently. Not sure of the exact differences, but I think they had heat pipe cooling and (possibly) a different drive. Wonder if anyone can confirm if it is known that the drive manufacturer had changed at some point.

I think we are getting closer to identifying the problem.
Here's my current theory:

All PAL DCs and probably all US too will have later firmwares, which have none of the dumping problems you identified. They had plenty of time to iron out early firmware problems before the Western launch dates and maybe they even changed the drive completely.

It's possible that some early Japanese and Asian models might have the earlier firmwares and/or different drive offset, resulting in problems getting tracks with the correct checksums, apart from the last data track.

This should also easily explain what happened in this thread:
http://dumpcast.thekickback.com/forum/v ... .php?t=353
Yamada also had problem getting a correct track01 with his first DC and darcagn replies there's a single byte difference. Thankfully the files he uploaded are still online and I did the comparison. Guess what? The problem is again in sync field of last sector! Also, audio offset is again 452 bytes or 113 samples! Then he gets a 2nd DC and matches our dump. I believe his first DC was old revision and the other one he got was newer and dumped it correctly.
But I think the dumping problems can be attributed to early firmwares. Looks like the problem affects only last sector of a data track followed by audio track, hence the last data track always gets a matching checksum.
Back then, we dismissed the problem as a single issue with the specific console Yamada had, but in light of the gathered information I am strongly inclined to believe that a few early JPN/Asian consoles exist that dump data tracks wrongly due to early firmware and possibly they also have a different audio offset.
Maybe this also explains why the project has a bad name in Japan, according to Yamada's other posts. :lol:
If some of the users there have older DCs, they only get matching checksums for the last track.

Of course all these theories will need investigation and I'd love to see more dumps done with early firmwares. It could prove me wrong. If Yamada or other Japanese people or owners of Japanese/Asian Dreamcasts are reading this, please supply information.

Before anyone asks:
In any case, even if my theory is proven correct, it won't affect our dumping checksums or method.
The data track error (currently attributed to early firmwares) is unquestionable (sync field is standard as I already described) so we can't accept these dumps as correct in any case. Since different audio offset seems so far to always be coming together with bad data track dumping, it's not an issue either.
The DCs with newer/fixed? firmwares seem to be dumping in a very consistent manner all the time, regardless of region.
Unfortunately, I doubt there's anything that can be done for fixing the issue with early models, apart from hacking the resulting files to comply. The good news for anyone having them is that the files will not be affected adversely for emulator use. They can't be made compliant with TOSEC checksums without hacking, but they are actually working properly. Data track error is at a place where it won't affect the emulation at all.

Awaiting more info, as and if we get it. :)
Quzar
Posts: 52
Joined: Mon Apr 23, 2007 1:22 pm
Location: Miami, FL
Contact:

Post by Quzar »

The earlier DC's did use a different drive. One drive was made by Yamaha and the other by Samsung.
I only upload the information, not images.
Post Reply