Script for automatically converting textures to best format

[bogglez]

Hey everyone,

I realized I never shared the script I'm using to convert textures to the best-fitting Dreamcast format. So here it is, integrated into an example Makefile (which I suggest because texture conversion can be slow and this way it's also parallelizable).
It uses texconv by tvspelsfreak and ImageMagick for conversion/analysis.
Hope somebody will find it useful in making games.

Features:

• Converts to paletted texture if unique color count is low enough
• Converts to bumpmap format if filename is of the form *_normal*, *_bump*, *Normal* or *Bump*
• Uses A1RGB5 format if only 2 unique alpha colors are used (alpha mask), ARGB4 if more unique alpha values exist, R5G6B5 if alpha is always the same.
• Resizes textures of size >1024x1024 to 1024x1024.
• Bumpmaps are VQ compressed, but quality suffers a lot because of high count of unique colors, so resolution is reduced to 512x512.
• Generates mipmaps for square textures, unless disabled.
• Creates preview png.

Running make:

Spoiler!

Code: Select all

$ file assets/*
assets/barrelDiffuse.png:  PNG image data, 1024 x 1024, 8-bit/color RGB, non-interlaced
assets/barrelNormal.png:   PNG image data, 1024 x 1024, 8-bit/color RGB, non-interlaced
assets/barrelSpecular.png: PNG image data, 1024 x 1024, 8-bit/color RGB, non-interlaced
assets/chicken_walk.png:   PNG image data, 128 x 128, 8-bit/color RGBA, non-interlaced
assets/large.png:          PNG image data, 2048 x 2048, 8-bit/color RGB, non-interlaced

$ make
Converting input texture "assets/barrelSpecular.png" to R5G6B5 texture "baked/barrelSpecular.dtex".
Automatically generating mipmaps for square input texture "assets/barrelSpecular.png".

Converting input bumpmap "assets/barrelNormal.png" to "baked/barrelNormal.dtex".
Resizing input bumpmap to 512x512 to avoid severe quality reduction.
Automatically generating mipmaps for square input texture "assets/barrelNormal.png".

Limiting image size of "assets/large.png" to 1024x1024.
Converting input texture "assets/large.png" to R5G6B5 texture "baked/large.dtex".
Automatically generating mipmaps for square input texture "assets/large.png".

Converting input texture "assets/barrelDiffuse.png" to R5G6B5 texture "baked/barrelDiffuse.dtex".
Automatically generating mipmaps for square input texture "assets/barrelDiffuse.png".

Converting input texture "assets/chicken_walk.png" to 4-bit paletted texture "baked/chicken_walk.dtex" with separate palette file.
Not using mipmaps for input texture "assets/chicken_walk.png" although it would be possible.

$ ls baked
barrelDiffuse.dtex  barrelNormal.dtex  barrelSpecular.dtex  chicken_walk.dtex  chicken_walk.dtex.pal  large.dtex

Makefile:

Spoiler!

Code: Select all

# assets/*.png -> baked/*.dtex
all: $(patsubst assets/%.png,baked/%.dtex,$(wildcard assets/*.png))

# Special rule for files which don't need mipmaps
baked/chicken_walk.dtex: assets/chicken_walk.png
        @mkdir -p baked
        dc_convert_texture -i "$<" -o "$@" -p "baked/$@_preview.png" -d
        @echo

# Standard rule for automatically converting textures
baked/%.dtex: assets/%.png
        @mkdir -p baked
        @dc_convert_texture -i "$<" -o "$@" -p "baked/$@_preview.png"
        @echo

clean:
        $(RM) -r baked

Usage:

Code: Select all

  dc_convert_texture -i in.png -o out.dtex [-p preview.png] [-b] [-h] [-d]
  -i/--in      in.png      Input texture file to convert. Required.
  -o/--out     out.dtex    Destination path of converted texture. Required.
  -p/--preview preview.png Destination path of preview texture for checking quality reduction.
  -b/--bumpmap             Input texture will be converted to bumpmap format.
                           If the filename contains "_bump", "_normal",
                           "Bump" or "Normal" this is set automatically.
  -d/--disable-mipmaps     Disable generation of mipmaps. Use for textures which are always
                           shown on screen unscaled and without rotation.
  -h/--help                Show this help message.

dc_convert_texture:

Spoiler!

Code: Select all

#!/bin/sh
set -e

# This script converts input textures to the most fitting Dreamcast-specific texture format.
# The advantage lies in smaller file size in RAM/VRAM, lower data bus bandwidth usage and
# potentially faster texture filtering.

program_exists() { command -v "$1" >/dev/null 2>&1; }

if ! program_exists identify; then
	echo "Please install ImageMagick's \"identify\" tool."
	exit 1
fi

if ! program_exists convert; then
	echo "Please install ImageMagick's \"convert\" tool."
	exit 1
fi

if ! program_exists texconv; then
	echo "Please install texconv from https://github.com/tvspelsfreak/texconv.git"
	exit 1
fi

usage="Usage: $0 -i in.png -o out.dtex [-p preview.png] [-b] [-h] [-d]"
[ "$#" -eq 0 ] && (echo "$usage"; exit 1)

while [ $# -gt 0 ]; do
	case $1 in
	-i|--in)
		in="$2"
		shift
		;;

	-o|--out)
		out="$2"
		shift
		;;

	-b|--bumpmap)
		is_bumpmap=1
		;;

	-d|--disable-mipmap)
		disable_mipmaps=1
		;;

	-p|--preview)
		preview="$2"
		shift
		;;

	-h|--help)
		echo "$usage"
		echo "  -i/--in      in.png      Input texture file to convert. Required."
		echo "  -o/--out     out.dtex    Destination path of converted texture. Required."
		echo "  -p/--preview preview.png Destination path of preview texture for checking quality reduction."
		echo "  -b/--bumpmap             Input texture will be converted to bumpmap format."
		echo "                           If the filename contains \"_bump\", \"_normal\","
		echo "                           \"Bump\" or \"Normal\" this is set automatically."
		echo "  -d/--disable-mipmaps     Disable generation of mipmaps. Use for textures which are always"
		echo "                           shown on screen unscaled and without rotation."
		echo "  -h/--help                Show this help message."
		exit 0
		;;

	*)
		echo "Unknown option "$1"."
		exit 1
		;;
	esac
	shift
done


# Check arguments
[ -f "$in"  ] || (echo "Input file "$in" does not exist."; exit 2)
[ -z "$out" ] && (echo "Output filename is unset.";        exit 3)

original_width="$(identify  -format '%[width]'  "$in")"
original_height="$(identify -format '%[height]' "$in")"
width="$original_width"
height="$original_height"

# Determine characteristics of the input file to compress it appropriately.
case "$in" in
    *_bump*)   is_bumpmap=1;;
    *_normal*) is_bumpmap=1;;
    *Normal*)  is_bumpmap=1;;
esac

if [ "$width" -gt 1024 ] || [ "$height" -gt 1024 ]; then
	width=$(( $width  > 1024 ? 1024 : $width))
	height=$(($height > 1024 ? 1024 : $height))
	echo "Limiting image size of \"$in\" to "$width"x"$height"."
fi

if [ ! -z "$is_bumpmap" ]; then
	echo "Converting input bumpmap \"$in\" to \"$out\"."
	# Resolutions over 512x512 are prone to noise introduced by VQ, so make them smaller (> operator).
	# 1024x512 normal maps will be converted to 512x512.
	texelFormat=BUMPMAP
	if [ "$width" -gt 512 ] || [ "$height" -gt 512 ]; then
		width=$(( $width  > 512 ? 512 : $width))
		height=$(($height > 512 ? 512 : $height))
		echo "Resizing input bumpmap to "$width"x"$height" to avoid severe quality reduction."
	fi

else
	# Count the amount of unique texel values and use a palette when possible.
	uniqueColors="$(identify -format "%k" "$in")"
	if [ "$uniqueColors" -le 16 ]; then
		echo "Converting input texture \"$in\" to 4-bit paletted texture \"$out\" with separate palette file."
		texelFormat=PAL4BPP
	elif [ "$uniqueColors" -le 256 ]; then
		echo "Converting input texture \"$in\" to 8-bit paletted texture \"$out\" with separate palette file."
		texelFormat=PAL8BPP

	# Cannot use a palette, check whether the alpha channel can be omitted.
	else
		channels="$(identify -format '%[channels]' "$in")"
		# Need alpha channel.
		if [ "$channels" = "srgba" ]; then
			alphaCount="$(identify -alpha extract -unique -verbose "$i" | grep '^.*[0-9]\+: ' | wc -l)"
			# Alpha channel is used for masking only.
			if [ "$alphaCount" = 2 ]; then
				echo "Converting input texture \"$in\" to A1R5G5B5 texture \"$out\"."
				texelFormat=ARGB1555
			# Alpha channel is a gradient.
			else
				echo "Converting input texture \"$in\" to A4R4G4B4 texture \"$out\"."
				texelFormat=ARGB4444
			fi
		# Omit alpha channel.
		else
			echo "Converting input texture \"$in\" to R5G6B5 texture \"$out\"."
			texelFormat=RGB565
		fi
	fi
fi

if [ "$width" = "$height" ]; then
	is_square=1
	if [ "$disable_mipmaps" = 1 ]; then
		echo "Not using mipmaps for input texture \"$in\" although it would be possible."
	else 
		echo "Automatically generating mipmaps for square input texture \"$in\"."
	fi
elif [ -z "$disable_mipmaps" ]; then
	echo "Cannot use mipmap for non-square input texture \"$in\"."
fi


# Assemble conversion command depending on enabled options and input image properties.
cmd="texconv -o $out -c -f $texelFormat"
if [ -z "$disable_mipmaps" ] && [ "$is_square" = 1 ]; then
	cmd="$cmd -m"
fi

if [ ! -z "$preview" ]; then
	cmd="$cmd -p $preview"
fi

# Decide whether to use temporary file as input or original.
if [ "$width" -ne "$original_width" ] || [ $height -ne "$original_height" ]; then
	tmp="$out_resized.png"
	convert -resize $width"x"$height $in $tmp
	cmd="$cmd -i $tmp"
else
	cmd="$cmd -i $in"
fi

# Execute conversion
$cmd

# Remove temporary file if one was created
[ -z "$tmp" ] || rm "$tmp"

[lerabot]

Very nice!

I've been using straight PNG as texture for the PVR for a bit now and was wondering if KOS had any built-in fonction for loading those?

*I'll go googling around as well*

[bogglez]

Check the wiki in my signature

[AtariOwl]

I saw there was a way to compile texconv on debian based systems

Code: Select all

sudo apt install qt5-default qtbase5-dev
git clone https://github.com/tvspelsfreak/texconv
cd texconv
qmake
make

how would one do it on msys32?

Or can i find a compiled version?

[bogglez]

https://wiki.qt.io/MSYS2
Probably

Code: Select all

pacman -S mingw-w64-i686-qt-creator

(You should only need the 32 bit version)

Read this to find out how package management works in msys2
https://sourceforge.net/p/msys2/wiki/MS ... tallation/

[AtariOwl]

HI again

I'm so sorry, still having difficulties

I ran it through with

Code: Select all

pacman -S mingw-w64-i686-qt-creator mingw-w64-x86_64-qt-creator

That seems to have worked just fine.

However upon getting texconv and

Code: Select all

$ qmake
Could not find qmake configuration file win32-g++.
Error processing project file: C:\msys32\home\jve\texconv\texconv.pro

There is a win32-g++ directory in msys2/mingw32/share/qt5/mkspecs containing
qmake.conf
qmake.orig file
and qplatformdefs.h
So it would appear to be a missing path definition perhaps.
I wonder why its not finding it. Online questions bout it are less than helpful, suggesting one just downloads precompiled binaries.

Upon checking everything was up to date

Code: Select all

No language id given for tool chain.
No language id given for tool chain.
File C:\msys32\mingw32\share\qtcreator\QtProject\qtcreator\devices.xml not found.
File C:\msys32\mingw32\share\qtcreator\QtProject\qtcreator\devices.xml not found.
File C:\msys32\mingw32\share\qtcreator\QtProject\qtcreator\devices.xml not found.
File C:\msys32\mingw32\share\qtcreator\QtProject\qtcreator\devices.xml not found.
/

Nope apparently thats a legacy thing

[bogglez]

I've never done this on Windows, so I can only guess..

What is the output of the following command?

Code: Select all

qmake -query

You can use

Code: Select all

qmake -spec spec_here

to specify the spec manually.

You can also set the QMAKESPEC environment variable for this purpose: http://doc.qt.io/qt-5/qmake-environment ... #qmakespec

Code: Select all

export QMAKESPEC=spec_here

If none of this works, you could bruteforce the compilation..

Code: Select all

g++ -std=c++11 *.cpp

..but you would need to add the include paths with "-I path", library paths "-L path" and libraries with "-lsome_lib" yourself, so the proper way is to be preferred..

[AtariOwl]

HI

OK

first of all

Code: Select all

$ qmake -query
QT_SYSROOT:
QT_INSTALL_PREFIX:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32
QT_INSTALL_ARCHDATA:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/share/qt5
QT_INSTALL_DATA:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/share/qt5
QT_INSTALL_DOCS:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/share/qt5/doc
QT_INSTALL_HEADERS:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/include
QT_INSTALL_LIBS:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/lib
QT_INSTALL_LIBEXECS:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/share/qt5/bin
QT_INSTALL_BINS:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/bin
QT_INSTALL_TESTS:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/share/qt5/tests
QT_INSTALL_PLUGINS:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/share/qt5/plugins
QT_INSTALL_IMPORTS:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/share/qt5/imports
QT_INSTALL_QML:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/share/qt5/qml
QT_INSTALL_TRANSLATIONS:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/share/qt5/translations
QT_INSTALL_CONFIGURATION:
QT_INSTALL_EXAMPLES:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/share/qt5/examples
QT_INSTALL_DEMOS:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/share/qt5/examples
QT_HOST_PREFIX:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32
QT_HOST_DATA:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/share/qt5
QT_HOST_BINS:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/bin
QT_HOST_LIBS:C:/repo/mingw-w64-qt5/pkg/mingw-w64-i686-qt5/mingw32/lib
QMAKE_SPEC:win32-g++
QMAKE_XSPEC:win32-g++
QMAKE_VERSION:3.0
QT_VERSION:5.6.2

Does this help?

[bogglez]

Is C:/repo a valid path? Looks suspicious to me. I think manually specifying your path should fix things.

[AtariOwl]

There is no such path its very odd.

I jut installed it running from my mingw32 shell... its suspicious isn't it.
The files are actually within c:/msys32/etc...


I also tried
export QMAKESPEC=spec_here

and it ran somewhat further

Code: Select all

MINGW32 ~/texconv
$ export QMAKESPEC=/c/msys2/mingw32/share/qt5/mkspecs/qmake.conf

MINGW32 ~/texconv
$ qmake
Cannot find feature spec_pre.prf
Error processing project file: C:\msys32\home\jve\texconv\texconv.pro

That prf file is actually in
C:\msys32\mingw32\share\qt5\mkspecs\features


Indeed i truly dont want to have to set the paths manually

[AtariOwl]

Ummmmmm

Nearly there by brute force perhaps

Code: Select all

$ g++ -I/mingw32/include/QtGui -I/mingw32/include/QtCore -std=c++11 *.cpp
common.cpp: In function 'quint16 toSpherical(QRgb)':
common.cpp:61:18: error: 'M_PI' was not declared in this scope
 #define HALFPI  (M_PI / 2.0)
                  ^
common.cpp:77:10: note: in expansion of macro 'HALFPI'
  polar = HALFPI - polar;    // -HALFPI ... HALFPI
          ^~~~~~
common.cpp: In function 'QRgb toCartesian(quint16)':
common.cpp:61:18: error: 'M_PI' was not declared in this scope
 #define HALFPI  (M_PI / 2.0)
                  ^
common.cpp:91:42: note: in expansion of macro 'HALFPI'
  float S = (1.0 - ((SR >> 8) / 255.0)) * HALFPI;

M_PI included now.

Ugh
http://pastebin.com/Z9vBjW0f

[AtariOwl]

Sorry Double post somehow

[bogglez]

Seems like another messed up path
This look like a bug that you should report to the msys2 team.

The brute force method took you quite far. Only linking remains..
Try adding -lQt5Core at the very end of the g++ command. If it doesn't find the library you can s0ecify it with -L lib_dir before the -l flag. That should make most if not all Qt errors go away.

[AtariOwl]

There must be something odd about the paths, it's very strange.
I will try your sprites example, when i get it working.

Hmmm, I seem to also be having difficulty with it finding Texture Packer. That's probably unconnected?

It's windows not making it easy i suspect.

Code: Select all

$ make
rm -f spritesheet.elf
Generating spritesheet build/ui_sheet.png from assets/spritesheets/ui
TexturePacker --sheet build/ui_sheet.png \
        --format gideros --data romdisk/ui_sheet.txt \
        --size-constraints POT --max-width 1024 --max-height 1024 \
        --pack-mode Best --disable-rotation --trim-mode None \
        --trim-sprite-names \
        --algorithm Basic --png-opt-level 0 --extrude 0 --disable-auto-alias \
        assets/spritesheets/ui
make: TexturePacker: Command not found
make: *** [Makefile:49: build/ui_sheet.png] Error 127

Instead of

Code: Select all

kos-cc -std=c11 -c main.c -o main.o

Generating spritesheet build/stage1_actors_sheet.png from assets/spritesheets/stage1_actors
TexturePacker --sheet build/stage1_actors_sheet.png \
       --format gideros --data romdisk/stage1_actors_sheet.txt \
       --size-constraints POT --max-width 1024 --max-height 1024 \
       --pack-mode Best --disable-rotation --trim-mode None \
       --trim-sprite-names \
       --algorithm Basic --png-opt-level 0 --extrude 0 --disable-auto-alias \
       assets/spritesheets/stage1_actors
Resulting sprite sheet is 512x1024
Writing sprite sheet to build/stage1_actors_sheet.png
Writing romdisk/stage1_actors_sheet.txt

Converting romdisk/stage1_actors_sheet.dtex < build/stage1_actors_sheet.png

Is there an additional path i should be specifying? I allowed Texture Packer to install in its default directory.

[bogglez]

When I installed texture packer on Linux it was installed in /use/bin so it was in my PATH. I assume on Windows it's installed to c:/program files, so you will have to add that to your path or copy it to another location (I think it was just an exe without other files?). So you would have to call export PATH="/c/Program Files/TexturePacker:$PATH" or something like that, then typing TexturePacker should work.

[AtariOwl]

Aha

yes exactly that got the textpacker working... now i just need to finish texconv

[AtariOwl]

Texconv is now working and i fixed some of the demo code to load dtex files including paletted ones.

It currently loads up to 32 4BPP palettes to the first 512 and 2 8BPP palettes to last 512
I may do a version where you can specify a palette index when its called. In fact i think i will, or 16 and 3 etc.

Code: Select all

/**********************************************************
	Load in a TEXCONV dTex texture and palette 
fn is the filename
t is a dTex structure

dTex structure

typedef struct {
	uint32 w,h;
	uint32 fmt;
	pvr_ptr_t txt;
	Uint8 palette;
}dTex;


***********************************************************/

void Load_dtex(const char* fn, dTex* t){
	FILE* fp;
	header_t  hdr;
	fp = fopen(fn,"rb");
	
     pvr_set_pal_format(PVR_PAL_ARGB8888); 
	fread(&hdr,sizeof(hdr),1,fp);	// read in the header
	
	t->w = hdr.width;
	t->h = hdr.height;
	t->fmt = hdr.type;

	//Allocate texture memory
	t->txt = pvr_mem_malloc(hdr.size);

	//Temporary ram storage of texture
	void* temp = malloc(hdr.size);

	// Load texture into ram
	fread(temp,hdr.size,1,fp);

	// SQ copy into VRAM
	pvr_txr_load(temp,t->txt,hdr.size);

	//Free RAM
	free(temp);
	temp = NULL;
	fclose(fp);
	
	/* Palette loading and management */
	if( ((t->fmt >> 27) & 7) > 4 ) {
			// Append palette suffix to filepath
			char pf[64];
			strcpy(pf,fn);
			strcat(pf,".pal");
			fp = fopen(pf,"rb");
			pal_header_t phdr;
			//read in the 8-byte header
			fread(&phdr,sizeof(pal_header_t),1,fp);

		if((t->fmt >>27) & 1 ){
/* Now for 4BPP palettes */
			//setup buffer
			void *palette = malloc(phdr.numcolors*4);
			Uint32 i;
			//Make entries readable to PVR
			Uint32* colours = (Uint32*)palette;
			//Load entries in to buffer
			fread(colours,phdr.numcolors*4,1,fp);
			//Load palette entries into correct location ( first 512 bank)
			for(i = Pindex4*16; i < (Pindex4*16) + phdr.numcolors;i++) {
				pvr_set_pal_entry(i,colours[i&15]);
			}
			//Set palette Number
			t->palette = Pindex4;
			t->fmt |=  PVR_TXRFMT_4BPP_PAL(Pindex4);
		printf("\n num%d  index%d",phdr.numcolors,Pindex4);
		
		//Increase palette index to prevent overwrite
			Pindex4++;
			//32 possible palettes in 512 bank
			if(Pindex4 == 32){
				Pindex4 = 0; // overwrite
			}
			colours = NULL;
		        free(palette);

/* Now for 8BPP palettes */

		} else if((t->fmt >>27) & 2 ){
			void * palette = malloc(phdr.numcolors*4);
			Uint32 i;
			Uint32* colours = (Uint32*)palette;
			fread(colours,phdr.numcolors*4,1,fp);
						
			//Load palette entries into the second 512 bank
			for(i = (512 + Pindex8*256); i < (Pindex8*256 + 512) + phdr.numcolors;i++){
				pvr_set_pal_entry(i,colours[(i&255)]);
			}
			t->palette = Pindex8 | 0x80;
			t->fmt |=  PVR_TXRFMT_8BPP_PAL(Pindex8+2);
			Pindex8++;
			
			//Only 2 8-bit palettes can fit into second 512 bank
			if(Pindex8 == 2){
				Pindex8 = 0;
			}
			colours = NULL;
		        free(palette);
		}
		fclose(fp);
	}
	
}

[bbmario]

This is so amazing, thank you!

[tonma]

I can't make working version of texconv on my cygwin windows installation. Can compil but the executable didn't find share library.

Does someone can share a binary version for windows/cygwin ?

News : I finally understand why it's dosen't works. The default path is /lib/qt5/bin/qmake.exe and I need to use /usr/lib/qt5/bin/qmake.exe !
Same directory content, same file but not the same effect.