pvrtex - PVR Texture Encoder

[TapamN]

After running across a post on using FFmpeg's ELBG for encoding Cinepak (which uses vector quantization), I thought I'd try seeing how well it would work for the Dreamcast. It worked quite well, so I ended up creating a full texture convertor (including uncompressed textures). Source code is attached.

It's designed to work similarly to tvspelsfreak's texconv. I'm not going to repeat the entire contents of the README, but here's the major improvments:

* Faster texture compression and palette generation
* Can generate small codebook VQ textures
* No Qt dependency
* Support for compressed stride textures
* Better mipmap generation
* Dithering
* Support for additional output file types (adds .PVR and .DT)

For what "better mipmaps" means, here's an example. With texconv, the mipmap levels don't line up properly, so the image shifts to the side in smaller mipmap levels. pvrtex corrects for this, and uses a better filter.

Compression is generally much faster using FFmpeg's compressor (but I have found one pathological case where pvrtex is much slower: that test texture above). The results are of similar quality to texconv; you might find one spot in a texture where one compressor does slightly better than the other, but then you'll find a spot where it does slightly worse, and it works out to be about even altogether. Here are some benchmarks versus texconv:

Code: Select all

256x256 RGB565 ($ENCODER -f RGB565 -o result.tex -i tex256x256.png)
  pvrtex 0.02 seconds
  texconv 0.02 seconds

1024x1024 RGB565 Mipped ($ENCODER -f RGB565 -o result.tex -i tex1024x1024.png -m)
  pvrtex 0.80 seconds
  texconv 0.63 seconds

1024x1024 RGB565 Mipped VQ ($ENCODER -f RGB565 -o result.tex -i tex1024x1024.png -m -c)
  pvrtex 1.87 seconds
  texconv 6.75 seconds

1024x1024 ARGB4444 Mipped VQ ($ENCODER -f ARGB4444 -o result.tex -i tex1024x1024.png -m -c)
  pvrtex 1.88 seconds
  texconv 8.53 seconds

1024x1024 PAL8BPP Mipped VQ ($ENCODER -f PAL8BPP -o result.tex -i tex1024x1024.png -m -c -b)
  pvrtex 3.58 seconds
  texconv 11.5 seconds

The new texture format supports compressed texture with small codebooks, which saves RAM on small textures and improves performance a bit. It's also designed to help keep the texture data 32-byte aligned for DMA.

There's still a lot that could be done to improve it, but I think this is enough for now.

Edit: Most recent version (1.01) is in this post

[Ian Robinson]

Thank you so much, TapamN, for sharing this and doing the work. Now I have to learn how to use it and build it into my projects.

[Protofall]

Thanks again TapamN! I'll check this out when I have some time. Btw, do you have a github/gitlab/other? My game framework library auto-downloads its dependencies so it would be nice to add a download for this without the 7-zip requirement.

That said, its not urgent and I'd much prefer to get that cd driver before this small optmisation for my own code

[GyroVorbis]

Oh shit. The original Texture Converter was written by my DC dev mentor, Tvspelsfreak. It's too bad he isn't around anymore, because I think he'd love to hear about this.

Just wanted to also pay my respects and thank you kindly for sharing this with us!

How was getting rid of Qt, btw? I was the one who originally introduced him to Qt, just so he'd get so many source file formats for free.

[TapamN]

Thanks again TapamN! I'll check this out when I have some time. Btw, do you have a github/gitlab/other? My game framework library auto-downloads its dependencies so it would be nice to add a download for this without the 7-zip requirement.

I have SourceForge and GitHub accounts, that I've used for reporting bugs, but I've never uploaded anything to them yet. I'll try to remember to upload it somewhere at some point.

How was getting rid of Qt, btw? I was the one who originally introduced him to Qt, just so he'd get so many source file formats for free.

It's not really getting rid of Qt so much as not using it in the first place. It uses no code at all from texconv. I used stb_image and stb_image_write to handle reading and writing the images, stb_image_resize for mipmaps, and FFmpeg for VQ compression.

[histat]

I tried to build
avconfig.h under libavutil
seems to be missing.
It does not seem to be affected by its .

[Ian Robinson]

I tried to build
avconfig.h under libavutil
seems to be missing.
It does not seem to be affected by its .

You cant build it ?

[TapamN]

I tried to build
avconfig.h under libavutil
seems to be missing.
It does not seem to be affected by its .

Odd, you're right. I'm surprised it compiles on my system. I wonder why? Is it somehow using my system's avconfig.h?

New version with minor fixes:

• Add missing header (I don't think it's used by the compressor, but I want as few changes as possible to any of FFmpeg's files, so that if there are any updates to FFmpeg's compressor, it can be more easily added to pvrtex.)
• Displays error message when there's a problem reading a source image, instead of an assertion.
• Fix -resize down option. It was supposed to only round down non-power-of-two sizes, but would still reduce the size of a texture if it was already a power-of-two.

[histat]

it 's ok.
suggest to add install target

Code: Select all

install: $(TARGET)
	install -s $(TARGET) $(DEST)

Odd, you're right. I'm surprised it compiles on my system. I wonder why? Is it somehow using my system's avconfig.h?

I don't know your environment
but guess
try

Code: Select all

find /opt/local/include  -name avconfig.h

[Ian Robinson]

So do we have examples we can build using this with kos to benchmark the textures compared to the other one ? It would be nice to see the features in build-able kos examples showing the uses of the new features.

[TapamN]

I don't know your environment
but guess
try

Code: Select all

find /opt/local/include  -name avconfig.h

I'm using Xubuntu 20.04. All #includes using avconfig.h are local includes (#include "libavutil/avconfig.h"), so I don't think they should be using the system's headers. When I was first setting up the compressor to run out of FFmpeg, I started with just ELBG's files, and kept copying every missing header over it complained about until it would compile. Why was that single header special and not required?

So do we have examples we can build using this with kos to benchmark the textures compared to the other one ? It would be nice to see the features in build-able kos examples showing the uses of the new features.

I wrote something to view the textures over dcload, but it's very hacked together, has no real UI, and requires recompiling to change most settings, so I didn't include it.

[Ian Robinson]

I don't know your environment
but guess
try

Code: Select all

find /opt/local/include  -name avconfig.h

I'm using Xubuntu 20.04. All #includes using avconfig.h are local includes (#include "libavutil/avconfig.h"), so I don't think they should be using the system's headers. When I was first setting up the compressor to run out of FFmpeg, I started with just ELBG's files, and kept copying every missing header over it complained about until it would compile. Why was that single header special and not required?

So do we have examples we can build using this with kos to benchmark the textures compared to the other one ? It would be nice to see the features in build-able kos examples showing the uses of the new features.

I wrote something to view the textures over dcload, but it's very hacked together, has no real UI, and requires recompiling to change most settings, so I didn't include it.

Yes i have written my own to display code for normal pvr formats. I was looking at your new format and features past what we had before and example with them not the normal pvr textures.

[GyroVorbis]

Jesus, OKAY.

Since I joined the KOS crew, I've been mostly working on language, toolchain, stdlib, kernel, and low-level driver stuff... I'm a bottom-up kind of guy... anyway... I'm working my way back up to graphics and such, and am now playing with the PVR API, GLdc, rendering, textures, etc, and guess what I'm only now trying out...

The fact this even built out-of-the-box for me without a Qt dependency or download is definitely a hell of a plus, but then I see a whole myriad of things yours is doing better?

Apparently Tvspelsfreak's texconv was never even considered for being added to KOS due to the big-ass Qt dependency. I always thought it was a massive shame and that it feels wrong to have a Dreamcast SDK without such a tool built into it... is there any way you would consider letting us put this in the KOS repo? Under utils? I'm sure there will be all kinds of work to ensure the Makefile just builds for all of the platforms we target, but we can handle that... Full credit too, obviously, it's totally your tool... I just feel like the potential good this can do is so great with KOS in terms of user-experience and what people can achieve on DC, that it could really benefit from the distribution channel that is the KOS repo as opposed to just staying a post on these boards.

[GyroVorbis]

Ooops. Double-post. Meant to edit my previous one.

[TapamN]

Apparently Tvspelsfreak's texconv was never even considered for being added to KOS due to the big-ass Qt dependency. I always thought it was a massive shame and that it feels wrong to have a Dreamcast SDK without such a tool built into it... is there any way you would consider letting us put this in the KOS repo? Under utils? I'm sure there will be all kinds of work to ensure the Makefile just builds for all of the platforms we target, but we can handle that... Full credit too, obviously, it's totally your tool... I just feel like the potential good this can do is so great with KOS in terms of user-experience and what people can achieve on DC, that it could really benefit from the distribution channel that is the KOS repo as opposed to just staying a post on these boards.

Go ahead. It might nice to add KMG support to pvrtex, or add DT support to KOS the same way it supports KMG, at some point, but probably not critical.

[dRxL]

Hi TapamN,

I'm starting out with DC-dev'ing again after some odd 20 years of pause, and I was recommended to use your tool for my texture baking needs, which I've enjoyed so much thus far, that I've started working on a PR for KOS to include your tool.

My work in progress can be found here:
https://github.com/dfchil/KallistiOS/tr ... ils/pvrtex

I've only done a bit of cleanup, and removed some unused code in nvmath.h that prevented compilation on my ARM macbook, but I have two things on my todo list berfore I'll try to make a PR:

1) Add proper attribution to you
2) Add a little bit of help text for the --help option and no arguments

So regarding 1) how would you like to be attributed? Your username here, real name and contact info, something else?

For 2) I'll figure something out by copying from your README file.

[TapamN]

Wow, I'm surprised the SH4 asm compiles fine on my system, x86 GCC. I guess the real cause for it not compiling on your system would be that you're probably using Clang?

For #1, just TapamN is fine. For #2, you could probably copy and paste most of the help message for tvspelsfreak's texconv, since pvrtex is designed to be backward compatible with texconv usage.

I recently created a bash tab completion script for pvrtex (and created a "install" make target, but it looks like you already did that). I've attached it here. (I had to compress a 1.5 KB file because the forum is rejecting files with the extension ".bash".) It goes in "~/.local/share/bash-completion/completions/pvrtex.bash". For terminals that are already open, you have to make sure to run "source" on it for bash to pick up on it. I know very little bash scripting, so someone else might want to see if it needs any adjustments, but it's working for me. I did notice a minor typo in it as I was uploading it...

I haven't made any other changes otherwise, although I'm planning on adding an switch for optional mipmapping, so that instead of erroring out or forcing resizing on non-square textures, the texture will just be generated without mipmaps.

[Ian Robinson]

Still, may I request a possible conversion of the sprite sheet demo to use your texture conversion tool?

In this tutorial you will learn:
Generating a spritesheet from a directory full of single images automatically.
Converting the spritesheet into a PVR paletted texture automatically.
Loading the spritesheet into the PVR for drawing.
Setting up the palette for drawing.
Drawing a user interface (with a dynamic health bar).
Drawing animated characters.
Using the PVR's sprite drawing mode (instead of polygons)

https://dcemulation.org/index.php?title ... ritesheets

So, not using TexConv but using yours, this would provide a lot of people.
the push to try your tool, and for me, that would really help.

[TapamN]

pvrtex is designed to be backwards compatible with texconv usage, so it should just work without any changes besides the program name. To get pvrtex to generate files in the same format as texconv, the output file must have the extension .VQ or .TEX.

[dRxL]

Great news, your texture tool recently got merged into the KOS master branch.

I've made this example that uses the pvrtex tool to generate compressed and paletted textures.
https://github.com/ianmicheal/DREAMCAST ... spritecube

I also made a small single header 3 method lib for easier utilisation: https://github.com/ianmicheal/DREAMCAST ... n/pvrtex.h

Thank you for the great tool TapamN =)