bmltool/prstool bugs

[bogglez]

Hello,

I've tried using bmltool on PSOBB files. I have issues with the file "psobb/data/bm_ene_re7_berura_a.bml" and some others.

Code: Select all

bmltool -xd bm_ene_re7_berura_a.bml 
Error decompressing file re7_b2_larm.nj.pvm: No space left on device

I have gigabytes of free disk space and use only 4% of my inodes, so I assume that a bad size is used to allocate a file?


bmltool -x works.

However, prstool -x fails for the following contained files:

Code: Select all

re7_b2_larm.nj.pvm.prs
decompress: Bad message

re7_b2_rarm.nj.pvm.prs
decompress: Bad message

re7_b_bell_body.nj.pvm.prs
decompress: Bad message

It works for all files contained which are not PVMs (NJ, NJM, etc.).


Furthermore:

Code: Select all

for i in ~/psobb/*.bml; do N=~/bml/$(basename $i .bml); mkdir -p "$N" && cd "$N"; bmltool -xd "$i"; done
Error decompressing file re7_b2_larm.nj.pvm: No space left on device
Error decompressing file re8_b_srbeast_wola_body.nj.pvm: No space left on device
Error decompressing file plAcap06.nj: Bad message
Error decompressing file plBhai00.nj: Bad message
Error decompressing file plChed00.nj: Bad message
Error decompressing file plDcap00.nj: Bad message
Error decompressing file plEhed00.nj: Bad message
Error decompressing file plFhed00.nj: Bad message
Error decompressing file plGcap00.nj: Bad message
Error decompressing file plHcap00.nj: Bad message
Error decompressing file plIcap00.nj: Bad message
Error decompressing file plJhed00.nj: Bad message
Error decompressing file plKcap00.nj: Bad message
Error decompressing file plLcap00.nj: Bad message
Error decompressing file plOhai00.nj: Bad message
Error decompressing file plPhai00.nj: Bad message
Error decompressing file plQhai00.nj: Bad message
Error decompressing file plRhai00.nj: Bad message
Error decompressing file plShai00.nj: Bad message
Error decompressing file plThai00.nj: Bad message
Error decompressing file plUhed00.nj: Bad message
Error decompressing file plVhed00.nj: Bad message
Error decompressing file plWhed00.nj: Bad message
Error decompressing file plXhed00.nj: Bad message
Error decompressing file plYhed00.nj: Bad message

So it seems to affect other files than .pvm too sometimes.

I'll continue working with what I have (worked for most files after all), but it would be nice to be able to rip all files


Another issue, some file names seem to be incorrect as they have garbage at the end.

Code: Select all

find . ! -name '*.bml' ! -name '*.njm' ! -name '*.pvm' ! -name '*.nj' ! -name '*.njs' ! -name '*.prs' ! -type d
./bm_obj_ep4_boss09_sonic/0119_bm_obj_ep4_boss09_sonic.njs??
./bm_obj_ep4_container_red/bm_obj_ep4_container_02_hakai_l.??
./bm_obj_ep4_container_red/bm_obj_ep4_container_02_hakai_l.??.pvm
./bm_obj_ep4_container_red/bm_obj_ep4_container_03_kihon.nj<?
./bm_obj_ep4_container_red/bm_obj_ep4_container_01_hakai_s.??
./bm_obj_ep4_container_red/bm_obj_ep4_container_01_hakai_s.??.pvm
./bm_obj_ep4_container/bm_obj_ep4_container_01_hakai_s.??
./bm_obj_ep4_container/bm_obj_ep4_container_03_hakai_l.??.pvm
./bm_obj_ep4_container/bm_obj_ep4_container_03_hakai_l.??
./bm_obj_ep4_container/bm_obj_ep4_container_02_kihon.nj>?
./bm_obj_ep4_container/bm_obj_ep4_container_01_hakai_s.??.pvm
./bm_ene_re8_merill_lia_low/wakeup_re8_b_beast_wola_body.njm'?
./bm_ene_re8_merill_lia_low/appear_re8_b_beast_wola_body.njm??
./bm_ene_re8_merill_lia_low/damage_re8_b_beast_wola_body.njm\?

[bogglez]

I also wrote the following script to unpack all art in PSOBB, but the tools seem to have some bugs still. I'll attach the error log

Code: Select all

#!/bin/sh

PSOBBDIR="$HOME/psobb"

rm -fr afs gsl

echo "Unpacking AFS archives."
find "$PSOBBDIR/" \( -iname '*.afs' -o -iname '*.pat' \) -exec sh -c 'OUT="afs/$(basename {})" && mkdir -p "$OUT" && cd "$OUT" && pso_artool --afs -x {}' \; | grep -v 'not an'
echo;

echo "Unpacking GSL archives."
find "$PSOBBDIR/" -iname '*.gsl' -exec sh -c 'OUT="gsl/$(basename {})" && mkdir -p "$OUT" && cd "$OUT" && pso_artool --gsl -x {} || echo "Cannot unpack {}" ' \;
echo;

echo "Unpacking BMLs that were found in AFS or GSL archives."
find "$PWD/afs" "$PWD/gsl" -iname '*.bml' -exec sh -c 'D="$(dirname {})"; F="$(basename {} .bml)"; mkdir -p "$D/$F" && cd "$D/$F" && bmltool -x {} && rm "{}" || echo "Cannot unpack {}" ' \;
echo;

echo "Decompressing PRS files that were found in AFS, GSL or BML archives."
find "$PWD/afs" "$PWD/gsl" -iname '*.prs' -exec sh -c 'prstool -x "{}" "$(dirname "{}")/$(basename "{}" .prs)" && rm "{}" || echo "Cannot decompress {}" ' \;
echo;

echo "Deleting empty directories."
find . -empty -delete

[BlueCrab]

First off, I'm assuming this is referring to the Sylverant bmltool and prstool? If so, it probably belongs in that forum not in the general programming forum (please correct me if I'm wrong and I'll move it back, but if that's the case, I'm not sure you're going to find anyone to help you with them).

Second, Sylverant bmltool was only written for the earlier versions of PSO's files (PSO for PC and PSO for Dreamcast). I never actually had any reason to mess with the Blue Burst files at all, so that's why it doesn't work there. There was a tool that was released (I think by Schthack, maybe?) for working with the Blue Burst files, but it is Windows-only. It does work under Wine though, so that's really the place to go if you want to mess with Blue Burst's files.

As for prstool, I know what is in the repository as prstool is based off of an earlier version of my PRS decompression and compression code. I don't think I changed anything major in it, but it still would probably be a good idea to grab the code in libpsoarchive and write a tool using that instead of my older PRS code. That all said, if I had to guess though, I'd assume that bmltool was spitting out garbage for the supposedly PRS-compressed files because you were working with Blue Burst stuff. Perhaps it's either not PRSed in the first place, or it's actually PR2/PRSD or something of the like. I don't have Blue Burst handy to look at right now, so I can't really suggest all that much there. As I said, you're better off trying out the old Schthack tool for Blue Burst stuff.

Speaking of which, since the schtserv site is down, I'll attach the version of the tool that I happen to have handy here to this post... To save the trouble of trying to find it. Just to disclaim anything related to it... I didn't write this tool, and I don't guarantee that it works/won't cause your computer to explode/etc.

[bogglez]

Thanks, I'll check it out

[BlueCrab]

One thing I should mention is that if you actually need to PRS-compress stuff at some point, my PRS code beats every other publicly available PRS compressor that I've seen on all inputs I personally tried. In fact, on game files directly out of PSO, I beat Sega's own compression ratio on all the files I tried.

[bogglez]

One thing I should mention is that if you actually need to PRS-compress stuff at some point, my PRS code beats every other publicly available PRS compressor that I've seen on all inputs I personally tried. In fact, on game files directly out of PSO, I beat Sega's own compression ratio on all the files I tried.

Haha, that's awesome! That said, after looking at the file formats I am quite sure there were many incompetent programmers at Sega/Sonic Team, so maybe it's not such a feat after all
It's so weird to see such poor use of compression when they read from a disc.. I expected compression everywhere.
BTW I also work with the PSOv2 for PC data. I just wanted to run my batch program on as much data as possible to find patterns since I'm trying to do this the hard way without a disassembler.

[BlueCrab]

Actually, Sega's compression on PRS is pretty good, for LZ77. None of the other publicly available PRS tools out there beat Sega's compression ratio, and I only do so because of a few little tricks I pulled from studying zlib a bit.

[Aleron Ives]

Remember that Sega chose its formats based on what the Dreamcast's SH4 could handle, too. It wouldn't have been worth it for Sega to use superior compression algorithms if doing so caused PSO to lock up while the game was compressing and decompressing data. "Downloading quest... please wait" "Decompressing quest... please wait"

[bogglez]

Remember that Sega chose its formats based on what the Dreamcast's SH4 could handle, too. It wouldn't have been worth it for Sega to use superior compression algorithms if doing so caused PSO to lock up while the game was compressing and decompressing data. "Downloading quest... please wait" "Decompressing quest... please wait"

One would use a highly asymmetric compression scheme, of course: slow but good compression, fast decompression. The developer just needs to do the compression once, it's not even performed on the DC, so irrelevant.

As soon as time(read_compressed+decompress) < time(read_uncompressed), you win.
The DC has to read data from a slow GD disc and download data using a slow modem. In other words, the reads are incredibly slow. Having to read less data helps a lot. Even the DC's CPU is fast in comparison to the disc drive/modem internet. It's the perfect scenario for compression.
In addition, you can already decompress chunks of data while you're waiting for the next chunks.

Actually, Sega's compression on PRS is pretty good, for LZ77. None of the other publicly available PRS tools out there beat Sega's compression ratio, and I only do so because of a few little tricks I pulled from studying zlib a bit.

I hadn't looked at the PRS algorithm in detail yet, but it was obviously something generic. Maybe they didn't even create it?
I'm more surprised that there's no data-tailored compression for various filetypes and how wasteful the file formats are overall both in storage and parsing efficiency. For example storing an array of int16 as int32s and other obvious problems.
In public documents they also claim they made it efficient for the SH, but their model format for example could actually consist of some DMAs/SQs/memcpys (vertices,indices,material,offset+length of parts).
Instead one needs to do recursive parsing through some pseudo-chunk-based format with optional values and oftentimes one cannot just memcpy a huge chunk of data because they don't provide a count of elements, so one needs to check the value of the chunk of data that was just read, branch, recurse, etc. etc...
I'm really wondering what's going on in the time when I see my PSO character's name/playtime and the point at which I can actually see the 3D model, feels like ages.
So don't get me wrong, I'm really talking about low-hanging fruit here

[Aleron Ives]

The developer just needs to do the compression once, it's not even performed on the DC, so irrelevant.

The DC does indeed perform PRS compression of its own. When a new player joins your team, the leader compresses the join packet before sending it to the new player. There aren't much further space gains to be had by using better compression algorithms, as PRS already gets quests to a reasonable size for transmitting over a modem. A few bytes of better compression doesn't even count at all, since PSO dices the bin and dat into 1024 byte chunks. If your improved compression doesn't cause the quest to cross over a 1024 byte multiple, then the quest still needs the same number of chunks as it did before, so your superior compression doesn't achieve anything.

As for reading from the disc, the GD-ROM drive is already equivalent to a 12X CD-ROM, so it's plenty fast enough to load the game's assets in a reasonable time; it's not like PSO has massive HD textures, after all. If anything, one has to wonder why Sega bothered to compress the ItemPMT file at all, since the space saved is insignificant compared to the capacity of a GD-ROM.

[bogglez]

The DC does indeed perform PRS compression of its own. When a new player joins your team, the leader compresses the join packet before sending it to the new player. There aren't much further space gains to be had by using better compression algorithms, as PRS already gets quests to a reasonable size for transmitting over a modem. A few bytes of better compression doesn't even count at all, since PSO dices the bin and dat into 1024 byte chunks. If your improved compression doesn't cause the quest to cross over a 1024 byte multiple, then the quest still needs the same number of chunks as it did before, so your superior compression doesn't achieve anything.

As I said, one can't slap one generic compression algorithm on everything and expect good results.
Network packet compression is obviously very different to compression of art assets. But yes, one can have huge gains here.
You're probably aware how out-of-sync PSO players actually are due to lack of game state updates. I assume the game would've benefited a lot from better packet handling (smaller, more frequent packets).
From what you're saying it seems like they compressed single packets instead of bundling them together, delta encoding, using stochastic encoding etc (all computationally cheap).

"player joined" packets are, to be frank, the worst example you could've possibly picked.. they happen very rarely and should be small to begin with.
What's much more interesting are packets like player/monster position/orientation/action which are sent multiple times per second for many entities.
The game was released for 33.6k modems (4300 bytes/s). Realistically one will never have the whole 4300 bytes due to packet loss, IP/TCP/UDP packet headers, other traffic.
I took a quick look at the Sylverant docs: a DC player position update payload (nothing else like TCP/UDP header, animation/action, orientation) seems to be 25 bytes.
If you're going to do nothing but updating player/monster positions 5 times a second (this is a low frequency for this kind of game), you'll be able to send 4300/25/5=34 such payloads (or 4 players + 30 monsters) per second before you've exhausted the modem.
This assumes the best case network scenario and ignores other data that needs to be sent (a UDP packet header for example consists of 28 bytes if I remember correctly). In other words, bandwidth is at a premium and every byte counts.
It seems like their solution was to not send the monster data at all, since PRS is not a silver bullet after all.

I also don't know what that arbitrary limit of 1024 bytes is for. The size one should get below is not 1024 anyway, but 576 (the maximum reassembly buffer size). If you send a packet of size 577 it may be split in two and one of the halves might get lost, causing round-trip times and latency.
If they pad their network packets to 1024 instead of receiving less data and padding it after it was received (decompression) that's just silly.
floor(4300/576) = 7 large updates per second. You really want to gather all your packets, compress them and fit them into those 576 bytes in order to avoid overhead from packet headers etc.

As for reading from the disc, the GD-ROM drive is already equivalent to a 12X CD-ROM, so it's plenty fast enough to load the game's assets in a reasonable time; it's not like PSO has massive HD textures, after all.

The speed of 12x = 1887438 bytes/s (looked it up) is only achieved at the outer border of the disc, since GDs use constant anglular velocity. At the center of the disc you will only have 6x = 943719 bytes/s, for example. Did anyone benchmark the DC on this btw? The actual speed is, again, probably much lower due to bad calibration of the laser, wear on the mechanical parts, dirt on the laser/disc, damage to the disc..
If you make your data smaller, more data will fit at the outer border and less time will be spent seeking, resulting in higher speed overall.
Fast read speeds are key for being able to stream data from disc (music, open worlds). Sega/Sonic Team's solution here is, again, to not do it at all. They just load everything into memory and leave the disc alone. Perhaps a valid tactic if memory is not at a premium, but on the DC it was.
You can see how slow their streaming is in practice when you look at player meshes loading in the online lobby.

If anything, one has to wonder why Sega bothered to compress the ItemPMT file at all, since the space saved is insignificant compared to the capacity of a GD-ROM.

This isn't about capacity but read speed and latency.

[Aleron Ives]

It seems like their solution was to not send the monster data at all, since PRS is not a silver bullet after all.

That's correct. The only monsters with any kind of synchronisation are bosses, since they have random attack patterns: the leader's client picks a random action for the boss to perform and then sends it to the other players. This means that whenever a player has lag, the boss freezes and warps around, which is quite annoying. It was actually a much smarter choice to not synchronise the grunt enemies, since it causes the gameplay to be smooth for each player, even if it isn't perfectly synchronised. Having Boomas freezing and warping around all the time would make playing the game a miserable experience, which is exactly what happens when people try to play Go Go Ball (aka "Lagball").

I also don't know what that arbitrary limit of 1024 bytes is for.

You're clearly not familiar with Sega, then. They make arbitrary and stupid decisions all the time.

You can see how slow their streaming is in practice when you look at player meshes loading in the online lobby.

The slow loading of player data is due to the disc's insane sort order. Player assets load much faster when the disc is sorted properly: just compare V1 to V2 to see how much of a difference even a semi-sane sort order makes (and V1 is far from an optimal sort order).

I'm not sure why you seem to be so upset about this. PSO contains lots of strange and frankly stupid design decisions. All you can do is accept them and move on, since we don't have any source code for the game.

[bogglez]

Having Boomas freezing and warping around all the time would make playing the game a miserable experience, which is exactly what happens when people try to play Go Go Ball (aka "Lagball").

Because the implementation was bad. I tried to say what could've been improved. When you replied that things could hardly be improved I tried to explain my point with some factual numbers + simple techniques (that were well known even back then), that make a huge difference. There are even more techniques which I didn't mention.. since every client runs on the same hardware, it would've been easy to use a synchronized random number generator to control the monster AI for example (and resynchronize occasionally). Only considering applying PRS to a each packet is a mistake.
Regarding lobby ball, wasn't there quite some time after a ball was kicked before it can be kicked again, so there's plenty of time for syncing? If so then lobby ball would be easier to synchronize than the actual game. They also had no dead reckoning/interpolation, I believe?

I'm not sure why you seem to be so upset about this. PSO contains lots of strange and frankly stupid design decisions. All you can do is accept them and move on, since we don't have any source code for the game.

Please don't randomly accuse me of being "upset". I'm just a programmer interested in finding out how this game was implemented. I want to learn from their mistakes, not "accept" them and "move on". I'm basically trying to find out what their source code must've looked like, based on what I find. I'm not trying to modify their code. Maybe we misunderstand each other here because it seems to me like you're trying to mod the game instead of making a new one. So for you it's not an interesting consideration, while for me it is.
I made the observation that the game/SDK code seems to be sub-par and then made it more clear to BlueCrab that I'm not talking about micro-optimizations, but basic stuff, so I listed examples. So I was merely surprised about what I learned.
You then joined the discussion saying that all these compression/efficient file format considerations are largely useless. I tried to explain to you how this is not the case at all. It seems to me like you're not interested in the technical stuff, since you say"accept" and "move on", etc., so I'll leave it at that.

[tueidj]

I would have thought you would realize most modems back then used hardware compression, so compressing small packets had insignificant benefit.

[bogglez]

I would have thought you would realize most modems back then used hardware compression, so compressing small packets had insignificant benefit.

Compressing small packets has insignificant benefit to begin with.
I'm not sure whether the DC modems supported hardware compression. It is hardly worth it anyway, read the first paragraph here https://en.wikipedia.org/wiki/Modem#Usi ... 6_kbit.2Fs

[BlueCrab]

I would have thought you would realize most modems back then used hardware compression, so compressing small packets had insignificant benefit.

Compressing small packets has insignificant benefit to begin with.
I'm not sure whether the DC modems supported hardware compression. It is hardly worth it anyway, read the first paragraph here https://en.wikipedia.org/wiki/Modem#Usi ... 6_kbit.2Fs

Compressing small packets can indeed have benefit, if those packets are filled with redundancy or constant data (i.e, all the instances of using 32-bits to store things when 16-bits would do and whatnot). Especially if you're sending lots of small packets. Plus, you would have other instances of compression probably introduced in the stack, like VJ compression of TCP/IP headers and whatnot.

Heck, even that article you linked to shows the benefit. If incompressible data is sent at 50kbit/s, and compressible data (as we've already discussed, many things in Sega's packets fit here) at closer to 160kbit/s, that's a rather huge improvement.

Also, I'm pretty sure that since the Dreamcast's modem (at least in the US) was V.90 class, it would have supported V.42/V.42bis compression. I could be wrong on that, but I think that was part of the V.90 standard that it had to support that...

Anyhow, this topic has veered wildly off the original topic. Just sayin'.