KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

If you have any questions on programming, this is the place to ask them, whether you're a newbie or an experienced programmer. Discussion on programming in general is also welcome. We will help you with programming homework, but we will not do your work for you! Any porting requests must be made in Developmental Ideas.
Post Reply
BigEvilCorporation
DCEmu Cool Newbie
DCEmu Cool Newbie
Posts: 15
https://www.artistsworkshop.eu/meble-kuchenne-na-wymiar-warszawa-gdzie-zamowic/
Joined: Fri Dec 16, 2016 7:18 am
Has thanked: 0
Been thanked: 0

KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by BigEvilCorporation »

Hi,

I've been banging my head against a wall for days trying to figure out why my (once working with KOS back in 2016) project is no longer booting after updating KOS to latest and greatest.

During startup, after KOS init and before main(), I get this error during static initialisation:

Code: Select all

terminate called after throwing an instance of '__gnu_cxx::__concurrence_lock_error'
  what():  __gnu_cxx::__concurrence_lock_error
arch: shutting down kernel
I can't nail down the cause - I've tried binary chopping to see if some global statics are invoking any std:: functionality, but it ended up disappearing when commenting out some seemingly unrelated code that does not run during static init. It's almost as if it's an alignment issue, a section arrangement issue, or I've hit some max executable size (image is ~680kb after stripping).

I'm using an external build system, which passes mostly the same flags to GCC as the KOS example makefiles (differences are -std=gnu++11, and rtti enabled).

I've compiled a basic Hello World sample with both the KOS makefiles and my build system, and can confirm the disassembly and section locations are identical, apart from the debug areas (because filename differences).

It would be tricky to post code snippets since the project is huge, but due to the nature of the error (and how it disappears when commenting out unrelated code) I'm convinced the problem is related to compiler/linker flags, alignment, or section locations.

I'm debugging using redream's GDB bridge, but it doesn't allow much in the way of debugging before main().

Output from gcc/g++/ld with -v option:
5>@ C.kos.C++ <dreamcast/release:dc_test>main.o
5>Using built-in specs.
5>COLLECT_GCC=C:/msys32/opt/toolchains/dc/sh-elf/bin/sh-elf-g++.exe
5>Target: sh-elf
5>Configured with: ../gcc-4.7.3/configure --target=sh-elf --prefix=/opt/toolchains/dc/sh-elf --without-headers --with-newlib --enable-languages=c --disable-libssp --disable-tls --with-multilib-list=m4-single-only,m4-nofpu,m4 --with-endian=little --with-cpu=m4-single-only CXX=g++ : (reconfigured) ../gcc-4.7.3/configure --target=sh-elf --prefix=/opt/toolchains/dc/sh-elf --with-newlib --disable-libssp --disable-tls --enable-threads=kos --enable-languages=c,c++,objc,obj-c++ --with-multilib-list=m4-single-only,m4-nofpu,m4 --with-endian=little --with-cpu=m4-single-only CXX=g++
5>Thread model: kos
5>gcc version 4.7.3 (GCC)
5>COLLECT_GCC_OPTIONS='-c' '-std=gnu++11' '-v' '-O2' '-fomit-frame-pointer' '-ml' '-m4-single-only' '-ffunction-sections' '-fdata-sections' '-Wall' '-g' '-fno-builtin' '-fno-strict-aliasing' '-fno-operator-names' '-fno-exceptions' '-I' 'C:/dev/dc_test' '-I' 'C:/dev/dc_test/ion' '-I' 'C:/dev/dc_test/dc_test' '-I' 'C:/msys32/opt/toolchains/dc/kos/include' '-I' 'C:/msys32/opt/toolchains/dc/kos/kernel/arch/dreamcast/include' '-I' 'C:/msys32/opt/toolchains/dc/kos/addons/include' '-I' 'C:/msys32/opt/toolchains/dc/kos-ports/include' '-D' '_arch_dreamcast' '-D' '_arch_sub_pristine' '-D' 'ION_BUILD_RELEASE' '-D' 'ION_PLATFORM_DREAMCAST' '-D' 'ION_PLATFORM_32BIT' '-D' 'ION_TOOLCHAIN_KOS' '-D' 'ION_ENDIAN_LITTLE' '-D' 'ION_RENDERER_OPENGL' '-D' 'ION_RENDERER_FIXED' '-D' 'ION_RENDER_SUPPORTS_GLUT' '-o' 'C:/dev/dc_test/dc_test/_Build/dreamcast/_intermediates_/dreamcast-release/bigevilcorp/dc_test/dc_test/main.o'
5> c:/msys32/opt/toolchains/dc/sh-elf/bin/../libexec/gcc/sh-elf/4.7.3/cc1plus.exe -quiet -v -I C:/dev/dc_test -I C:/dev/dc_test/ion -I C:/dev/dc_test/dc_test -I C:/msys32/opt/toolchains/dc/kos/include -I C:/msys32/opt/toolchains/dc/kos/kernel/arch/dreamcast/include -I C:/msys32/opt/toolchains/dc/kos/addons/include -I C:/msys32/opt/toolchains/dc/kos-ports/include -iprefix c:\msys32\opt\toolchains\dc\sh-elf\bin\../lib/gcc/sh-elf/4.7.3/ -D _arch_dreamcast -D _arch_sub_pristine -D ION_BUILD_RELEASE -D ION_PLATFORM_DREAMCAST -D ION_PLATFORM_32BIT -D ION_TOOLCHAIN_KOS -D ION_ENDIAN_LITTLE -D ION_RENDERER_OPENGL -D ION_RENDERER_FIXED -D ION_RENDER_SUPPORTS_GLUT C:/dev/dc_test/dc_test/main.cpp -quiet -dumpbase main.cpp -ml -m4-single-only -auxbase-strip C:/dev/dc_test/dc_test/_Build/dreamcast/_intermediates_/dreamcast-release/bigevilcorp/dc_test/dc_test/main.o -g -O2 -Wall -std=gnu++11 -version -fomit-frame-pointer -ffunction-sections -fdata-sections -fno-builtin -fno-strict-aliasing -fno-operator-names -fno-exceptions -o C:\Users\Matt\AppData\Local\Temp\cc7bEQ4v.s
5>GNU C++ (GCC) version 4.7.3 (sh-elf)
5>	compiled by GNU C version 6.2.0, GMP version 4.3.2, MPFR version 2.4.2, MPC version 0.8.1
5>GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
5>ignoring nonexistent directory "c:\msys32\opt\toolchains\dc\sh-elf\bin\../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/sys-include"
5>ignoring duplicate directory "C:/msys32/opt/toolchains/dc/sh-elf/lib/gcc/../../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/include/c++/4.7.3"
5>ignoring duplicate directory "C:/msys32/opt/toolchains/dc/sh-elf/lib/gcc/../../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/include/c++/4.7.3/sh-elf"
5>ignoring duplicate directory "C:/msys32/opt/toolchains/dc/sh-elf/lib/gcc/../../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/include/c++/4.7.3/backward"
5>ignoring duplicate directory "C:/msys32/opt/toolchains/dc/sh-elf/lib/gcc/../../lib/gcc/sh-elf/4.7.3/include"
5>ignoring duplicate directory "C:/msys32/opt/toolchains/dc/sh-elf/lib/gcc/../../lib/gcc/sh-elf/4.7.3/include-fixed"
5>ignoring nonexistent directory "C:/msys32/opt/toolchains/dc/sh-elf/lib/gcc/../../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/sys-include"
5>ignoring duplicate directory "C:/msys32/opt/toolchains/dc/sh-elf/lib/gcc/../../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/include"
5>#include "..." search starts here:
5>#include <...> search starts here:
5> C:/dev/dc_test
5> C:/dev/dc_test/ion
5> C:/dev/dc_test/dc_test
5> C:/msys32/opt/toolchains/dc/kos/include
5> C:/msys32/opt/toolchains/dc/kos/kernel/arch/dreamcast/include
5> C:/msys32/opt/toolchains/dc/kos/addons/include
5> C:/msys32/opt/toolchains/dc/kos-ports/include
5> c:\msys32\opt\toolchains\dc\sh-elf\bin\../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/include/c++/4.7.3
5> c:\msys32\opt\toolchains\dc\sh-elf\bin\../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/include/c++/4.7.3/sh-elf
5> c:\msys32\opt\toolchains\dc\sh-elf\bin\../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/include/c++/4.7.3/backward
5> c:\msys32\opt\toolchains\dc\sh-elf\bin\../lib/gcc/sh-elf/4.7.3/include
5> c:\msys32\opt\toolchains\dc\sh-elf\bin\../lib/gcc/sh-elf/4.7.3/include-fixed
5> c:\msys32\opt\toolchains\dc\sh-elf\bin\../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/include
5>End of search list.
5>GNU C++ (GCC) version 4.7.3 (sh-elf)
5>	compiled by GNU C version 6.2.0, GMP version 4.3.2, MPFR version 2.4.2, MPC version 0.8.1
5>GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
5>Compiler executable checksum: 58bc609f140e81e706072f4925b6261e
5>COLLECT_GCC_OPTIONS='-c' '-std=gnu++11' '-v' '-O2' '-fomit-frame-pointer' '-ml' '-m4-single-only' '-ffunction-sections' '-fdata-sections' '-Wall' '-g' '-fno-builtin' '-fno-strict-aliasing' '-fno-operator-names' '-fno-exceptions' '-I' 'C:/dev/dc_test' '-I' 'C:/dev/dc_test/ion' '-I' 'C:/dev/dc_test/dc_test' '-I' 'C:/msys32/opt/toolchains/dc/kos/include' '-I' 'C:/msys32/opt/toolchains/dc/kos/kernel/arch/dreamcast/include' '-I' 'C:/msys32/opt/toolchains/dc/kos/addons/include' '-I' 'C:/msys32/opt/toolchains/dc/kos-ports/include' '-D' '_arch_dreamcast' '-D' '_arch_sub_pristine' '-D' 'ION_BUILD_RELEASE' '-D' 'ION_PLATFORM_DREAMCAST' '-D' 'ION_PLATFORM_32BIT' '-D' 'ION_TOOLCHAIN_KOS' '-D' 'ION_ENDIAN_LITTLE' '-D' 'ION_RENDERER_OPENGL' '-D' 'ION_RENDERER_FIXED' '-D' 'ION_RENDER_SUPPORTS_GLUT' '-o' 'C:/dev/dc_test/dc_test/_Build/dreamcast/_intermediates_/dreamcast-release/bigevilcorp/dc_test/dc_test/main.o'
5> c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/bin/as.exe -little -o C:/dev/dc_test/dc_test/_Build/dreamcast/_intermediates_/dreamcast-release/bigevilcorp/dc_test/dc_test/main.o C:\Users\Matt\AppData\Local\Temp\cc7bEQ4v.s
5>COMPILER_PATH=c:/msys32/opt/toolchains/dc/sh-elf/bin/../libexec/gcc/sh-elf/4.7.3/;c:/msys32/opt/toolchains/dc/sh-elf/bin/../libexec/gcc/;c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/bin/
5>LIBRARY_PATH=c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/;c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/;c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/lib/
5>COLLECT_GCC_OPTIONS='-c' '-std=gnu++11' '-v' '-O2' '-fomit-frame-pointer' '-ml' '-m4-single-only' '-ffunction-sections' '-fdata-sections' '-Wall' '-g' '-fno-builtin' '-fno-strict-aliasing' '-fno-operator-names' '-fno-exceptions' '-I' 'C:/dev/dc_test' '-I' 'C:/dev/dc_test/ion' '-I' 'C:/dev/dc_test/dc_test' '-I' 'C:/msys32/opt/toolchains/dc/kos/include' '-I' 'C:/msys32/opt/toolchains/dc/kos/kernel/arch/dreamcast/include' '-I' 'C:/msys32/opt/toolchains/dc/kos/addons/include' '-I' 'C:/msys32/opt/toolchains/dc/kos-ports/include' '-D' '_arch_dreamcast' '-D' '_arch_sub_pristine' '-D' 'ION_BUILD_RELEASE' '-D' 'ION_PLATFORM_DREAMCAST' '-D' 'ION_PLATFORM_32BIT' '-D' 'ION_TOOLCHAIN_KOS' '-D' 'ION_ENDIAN_LITTLE' '-D' 'ION_RENDERER_OPENGL' '-D' 'ION_RENDERER_FIXED' '-D' 'ION_RENDER_SUPPORTS_GLUT' '-o' 'C:/dev/dc_test/dc_test/_Build/dreamcast/_intermediates_/dreamcast-release/bigevilcorp/dc_test/dc_test/main.o'
5>@ C.kos.Link <dreamcast/release:dc_test>dc_test.elf
5>Using built-in specs.
5>COLLECT_GCC=C:/msys32/opt/toolchains/dc/sh-elf/bin/sh-elf-g++.exe
5>COLLECT_LTO_WRAPPER=c:/msys32/opt/toolchains/dc/sh-elf/bin/../libexec/gcc/sh-elf/4.7.3/lto-wrapper.exe
5>Target: sh-elf
5>Configured with: ../gcc-4.7.3/configure --target=sh-elf --prefix=/opt/toolchains/dc/sh-elf --without-headers --with-newlib --enable-languages=c --disable-libssp --disable-tls --with-multilib-list=m4-single-only,m4-nofpu,m4 --with-endian=little --with-cpu=m4-single-only CXX=g++ : (reconfigured) ../gcc-4.7.3/configure --target=sh-elf --prefix=/opt/toolchains/dc/sh-elf --with-newlib --disable-libssp --disable-tls --enable-threads=kos --enable-languages=c,c++,objc,obj-c++ --with-multilib-list=m4-single-only,m4-nofpu,m4 --with-endian=little --with-cpu=m4-single-only CXX=g++
5>Thread model: kos
5>gcc version 4.7.3 (GCC)
5>COMPILER_PATH=c:/msys32/opt/toolchains/dc/sh-elf/bin/../libexec/gcc/sh-elf/4.7.3/;c:/msys32/opt/toolchains/dc/sh-elf/bin/../libexec/gcc/;c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/bin/
5>LIBRARY_PATH=c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/;c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/;c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/lib/
5>COLLECT_GCC_OPTIONS='-o' 'C:/dev/dc_test/dc_test/_Build/dreamcast/_intermediates_/dreamcast-release/bigevilcorp/dc_test/dc_test/dc_test.elf' '-LC:/msys32/opt/toolchains/dc/kos/lib/dreamcast' '-LC:/msys32/opt/toolchains/dc/kos/addons/lib/dreamcast' '-LC:/msys32/opt/toolchains/dc/kos-ports/lib' '-LC:/msys32/opt/toolchains/dc/sh-elf/sh-elf/lib' '-LC:/msys32/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/4.7.3' '-v' '-O2' '-fomit-frame-pointer' '-ml' '-m4-single-only' '-ffunction-sections' '-fdata-sections' '-Wall' '-g' '-fno-builtin' '-fno-strict-aliasing' '-fno-operator-names' '-fno-exceptions' '-v' '-T' 'C:/msys32/opt/toolchains/dc/kos/utils/ldscripts/shlelf.xc' '-nodefaultlibs'
5> c:/msys32/opt/toolchains/dc/sh-elf/bin/../libexec/gcc/sh-elf/4.7.3/collect2.exe -m shlelf -o C:/dev/dc_test/dc_test/_Build/dreamcast/_intermediates_/dreamcast-release/bigevilcorp/dc_test/dc_test/dc_test.elf c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/crt1.o c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/crti.o c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/crtbegin.o -LC:/msys32/opt/toolchains/dc/kos/lib/dreamcast -LC:/msys32/opt/toolchains/dc/kos/addons/lib/dreamcast -LC:/msys32/opt/toolchains/dc/kos-ports/lib -LC:/msys32/opt/toolchains/dc/sh-elf/sh-elf/lib -LC:/msys32/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/4.7.3 -Lc:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3 -Lc:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc -Lc:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/../../../../sh-elf/lib C:/dev/dc_test/dc_test/_Build/dreamcast/_intermediates_/dreamcast-release/bigevilcorp/dc_test/dc_test/main.o --start-group C:/msys32/opt/toolchains/dc/kos/lib/dreamcast/libkallisti.a C:/msys32/opt/toolchains/dc/sh-elf/sh-elf/lib/libc.a C:/msys32/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/4.7.3/libgcc.a C:/msys32/opt/toolchains/dc/sh-elf/sh-elf/lib/libstdc++.a --end-group --gc-sections c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/crtend.o c:/msys32/opt/toolchains/dc/sh-elf/bin/../lib/gcc/sh-elf/4.7.3/crtn.o -T C:/msys32/opt/toolchains/dc/kos/utils/ldscripts/shlelf.xc
Any clues on how to debug something like this?

Cheers
BigEvilCorporation
DCEmu Cool Newbie
DCEmu Cool Newbie
Posts: 15
Joined: Fri Dec 16, 2016 7:18 am
Has thanked: 0
Been thanked: 0

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by BigEvilCorporation »

Ok after further investigation, looks like objcopy is stripping things it shouldn't.

EDIT: the resulting ELF from this sample:

Code: Select all

#include <stdio.h>

static const char data[1024 * 1024 * 4] = { 0 };

int main()
{
	printf("H E L L O   W O R L D %c\n", data[1024]);

	return 0;
}
is only 1776kb. That data chunk is getting stripped. Why?

This is true for both the KOS makefiles and my own build setup, so I think that rules out anything I've done.

I've tried a fresh KOS toolchain build to rule out any local changes, too.
nymus
DC Developer
DC Developer
Posts: 968
Joined: Tue Feb 11, 2003 4:12 pm
Location: In a Dream
Has thanked: 5 times
Been thanked: 5 times

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by nymus »

That looks like an optimization by the compiler. The data is const and initialized so it can simply use the addressed value at compile time.

I think one would use extern or volatile to let the compiler know that the data is persistent.
behold the mind
inspired by Dreamcast
BigEvilCorporation
DCEmu Cool Newbie
DCEmu Cool Newbie
Posts: 15
Joined: Fri Dec 16, 2016 7:18 am
Has thanked: 0
Been thanked: 0

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by BigEvilCorporation »

I did try without const, and modifying the data to tell the optimiser to leave it alone, but same story.

Turns out it's a bad test - VC++ adds the whole blob to the EXE (the compiled Win32 exe grows as I increase that buffer size), but for ELF it seems to mark it as allocated in a map, and manually adds any initialised data.

Confirmed by actually putting in 4mb worth of initialisation data - my ELF is now >4mb.

So... back to the drawing board. Any other ways I can attack this?
kazade
Insane DCEmu
Insane DCEmu
Posts: 145
Joined: Tue May 02, 2017 3:11 pm
Has thanked: 3 times
Been thanked: 34 times

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by kazade »

Ok i dunno if this is related, but I discovered a while back that building KOS with a recent GCC would absolutely screw up the c++ standard library. The solution was to go back to compiling the toolchain itself with GCC 4.X.

Maybe it's that?
User avatar
lerabot
Insane DCEmu
Insane DCEmu
Posts: 134
Joined: Sun Nov 01, 2015 8:25 pm
Has thanked: 2 times
Been thanked: 19 times

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by lerabot »

I don't know if this would work on Win, but I recently compiled KOS quite easily.

I even made an updated guide on how I did it. https://github.com/dreamcastdevs/dreamc ... _toolchain
BigEvilCorporation
DCEmu Cool Newbie
DCEmu Cool Newbie
Posts: 15
Joined: Fri Dec 16, 2016 7:18 am
Has thanked: 0
Been thanked: 0

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by BigEvilCorporation »

kazade wrote: Tue Sep 11, 2018 8:48 am Ok i dunno if this is related, but I discovered a while back that building KOS with a recent GCC would absolutely screw up the c++ standard library. The solution was to go back to compiling the toolchain itself with GCC 4.X.

Maybe it's that?
That's interesting - I'm making some use of C++11 features, perhaps it's confused with something. I'll see if I can figure out how to rebuild with GCC 4, and see if that makes a difference.
BigEvilCorporation
DCEmu Cool Newbie
DCEmu Cool Newbie
Posts: 15
Joined: Fri Dec 16, 2016 7:18 am
Has thanked: 0
Been thanked: 0

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by BigEvilCorporation »

Well, this has taken a month (on and off) to track down but my game is finally running again.

TL;DR version: pay attention to your compiler warnings, kids!

Long version: The actual error in question turned out to be spurious after I moved code around and started getting different types of failures, but again all firing during global construction. Some STL errors, some KOS errors, some straight-up crashes.

Override arch_auto_init() is weakly linked so it can be overridden in user code, and I figured out how to call global constructors manually and log as they went. The table is at __ctors, and you work backwards from the end:
extern pfunc __ctors[];
extern pfunc __ctors_end[];

for (p = __ctors_end - 1; p > __ctors;)
{
    (*--P)();
}
Some of these turned out to be bad addresses, or NULL. It's the linker's responsibility to build this table, so something was upsetting it.

It turned out one of two issues (I didn't determine which) that was confusing the linker - I had a non-void function promising to return a value (a stub function for something that didn't apply to DC), which wasn't, and I had an overridden base class without a virtual destructor.

Both of these issues appeared in the compiler/linker warnings, so there's a grim lesson to be learned here.
kazade
Insane DCEmu
Insane DCEmu
Posts: 145
Joined: Tue May 02, 2017 3:11 pm
Has thanked: 3 times
Been thanked: 34 times

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by kazade »

OK, I'm hitting the same error and it's beyond weird!

I've been building my game engine using a docker container. This docker image was based on Fedora 21 (super old!) because for some reason I'd previously been unable to create a working toolchain with a GCC version that shipped in any later version.

It's important to note that I build my dc-chain in a docker container because it's built from a Dockerfile which has a fixed set of build steps.

I've just switched the Dockerfile to use Fedora 30's minimal image, and although the entire toolchain compiles fine, none of the applications will launch. All of them immediately exit with this error.

I've been through and fixed any warnings, and I'm still hitting this. Does anyone have any idea what the problem could be?

Essentially just the act of using a different GCC/OS version to compile *the toolchain* breaks this.
kazade
Insane DCEmu
Insane DCEmu
Posts: 145
Joined: Tue May 02, 2017 3:11 pm
Has thanked: 3 times
Been thanked: 34 times

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by kazade »

OK, new levels of weird...

So I compiled GCC 4.9 for Fedora 30, I then used that to compile the toolchain thinking that if there's any strange difference in output caused by compiling with GCC 8 then this should rectify it.

It didn't work, same problem.

I'm struggling to make sense of this. A toolchain compiled on Fedora 21 works, the exact same steps on Fedora 30 with GCC 4.9 or GCC 8 create a toolchain that causes this error when programs initialise.

Ultimately it might be that the toolchain is now correctly highlighting a bug, but I can't figure out why or where.
User avatar
SiZiOUS
DC Developer
DC Developer
Posts: 404
Joined: Fri Mar 05, 2004 2:22 pm
Location: France
Has thanked: 27 times
Been thanked: 19 times
Contact:

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by SiZiOUS »

In your old Fedora image, which GCC version did you used?

The GCC 4.7.3 is the latest GCC to be proven to be stable, but SWAT is using the 5.2.0 as you might see in their repository.

In the past, the KOS team tried to move to the 4.8.x branch and several issues were encountered (I'm not able to find the right thread here btw). So I guess the 4.9.x is still unstable? Maybe you should try to the 5.2.0 with the SWAT patch, but as you might see in their repository too, they tried to use newer GCC but reverted back to the 5.2.0 so it seems this version is stable for them.
mrneo240
DCEmu Freak
DCEmu Freak
Posts: 86
Joined: Wed Mar 14, 2018 12:22 am
Has thanked: 16 times
Been thanked: 19 times

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by mrneo240 »

Sizious, it's the same 4.7.3 toolchain but compiled with later versions of gcc.
User avatar
SiZiOUS
DC Developer
DC Developer
Posts: 404
Joined: Fri Mar 05, 2004 2:22 pm
Location: France
Has thanked: 27 times
Been thanked: 19 times
Contact:

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by SiZiOUS »

mrneo240 wrote: Tue Aug 20, 2019 8:08 am Sizious, it's the same 4.7.3 toolchain but compiled with later versions of gcc.
kazade wrote: Tue Sep 11, 2018 8:48 am Ok i dunno if this is related, but I discovered a while back that building KOS with a recent GCC would absolutely screw up the c++ standard library. The solution was to go back to compiling the toolchain itself with GCC 4.X.

Maybe it's that?
Oh :o
I totally misread the thread, sorry. Sorry to be off-topic. :oops:

... But this remind me something. For building DreamSDK, which provides 4.7.3 SH/ARM GCC, I used the GCC included in MinGW/MSYS, which is the 6.3.0. And I never heard about an issue like this? :|
kazade
Insane DCEmu
Insane DCEmu
Posts: 145
Joined: Tue May 02, 2017 3:11 pm
Has thanked: 3 times
Been thanked: 34 times

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by kazade »

OK, I've done some more digging, I'm making some progress.

The reason that the exception is being thrown is that locking the mutex fails with EINVAL in this function: https://github.com/KallistiOS/KallistiO ... utex.c#L85

This is likely caused by an uninitialised mutex being locked. I'm blaming some kind of static initialisation order bug here, but I'm having a hard job figuring out where it comes from. I could really do with a stack trace but I have no idea how to get gdb running and showing that kind of information.
kazade
Insane DCEmu
Insane DCEmu
Posts: 145
Joined: Tue May 02, 2017 3:11 pm
Has thanked: 3 times
Been thanked: 34 times

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by kazade »

OK, still suffering this... mrneo and I have found that it's the code in .init which is doing this, and not .ctors as BigEvilCorporation found. I have fixed all the warnings in my code, there's not thing there at -Wall that isn't just an unused variable so I think the cause is different.

My theory is that there's something slightly odd about the KOS threading stuff, and BigEvil's issue triggered it and my code is triggering it in a different way.
kazade
Insane DCEmu
Insane DCEmu
Posts: 145
Joined: Tue May 02, 2017 3:11 pm
Has thanked: 3 times
Been thanked: 34 times

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by kazade »

Right, I'm resurrecting this because there's definitely a bug in C++ support *somewhere*. Here's what I know:

1. When using C++, at some point during static initialisation, something is trying to lock a mutex before that mutex has been initialised. It's a gnu __mutex class which wraps gthread mutex which is in turn a KOS mutex.
2. It's triggered by something to do with link order. It's been seen triggered by the following:
- Having some problematic code (see BigEvils stuff above)
- Changing operating system / compiler
- Using CMake to build without using the gnu_wrappers

That final case is where I am now. I'm trying to write a toolchain file that avoids the KOS wrappers (which set their own flags like -fomit-frame-pointer etc.). CMake automatically tries to resolve circular references by linking things at least twice, so I've been unable to recreate the exact link line that the gnu_wrappers do, but although the toolchain compiles and links successfully, trying to run any code causes this exception during KOS boot up.

The code that "causes" the problem is this:

Code: Select all

if(m->type < MUTEX_TYPE_NORMAL || m->type > MUTEX_TYPE_RECURSIVE) {
        errno = EINVAL;
        rv = -1;
    }
Which is in mutex.c, so the mutex type is invalid. This is in-turn called by the locking code in concurrence.h: https://gcc.gnu.org/onlinedocs/libstdc+ ... ource.html

As you can see from that code, the KOS mutex is a member of the __mutex class, and initialised in the constructor. I can only assume then that somewhere there's a __mutex which is uninitialised at the point of locking.

If anyone has any thoughts about what could be causing this, I'm all ears!
kazade
Insane DCEmu
Insane DCEmu
Posts: 145
Joined: Tue May 02, 2017 3:11 pm
Has thanked: 3 times
Been thanked: 34 times

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by kazade »

Ok more info...

* it doesn't happen with all C++ apps, I have a simple hello world compiling and running fine with the same CMake toolchain file
* __mutex isn't used much in libstdc++, but every time it's used it's in a static local variable, returned by reference from a factory function. It's only really used in allocators and shared_ptr

static locals should be initialised on first use so it should be impossible to try to lock an unintialised mutex, especially during the Kos boot sequence.

I'm rapidly coming to the conclusion that something is corrupting the mutex on startup, whichever mutex it is.
Last edited by kazade on Tue Jan 07, 2020 3:56 am, edited 1 time in total.
kazade
Insane DCEmu
Insane DCEmu
Posts: 145
Joined: Tue May 02, 2017 3:11 pm
Has thanked: 3 times
Been thanked: 34 times

Re: KOS - static initialisation failure: __gnu_cxx::__concurrence_lock_error

Post by kazade »

If anyone fancies helping with this, here's a stripped (large) debug .elf showing the problem, and also the accompanying debuginfo and linker map: https://mega.nz/#!IV8DTK6L!QFeznqFRqleT ... vOETiTgFn0
These users thanked the author kazade for the post:
Ian Robinson
Post Reply