Crowdfunding the improvement of the gcc SH backend

[cbmuser]

Hello,

my name is Adrian and I'm both the upstream maintainer SuperH support in the Linux kernel as well as the maintainer of Debian's sh4 port. I have been made aware of this forum by the fantastic FOSDEM talk by Falco Girgis [1] who gave a unique insight into all the work people in this community have invested to support homebrew development on the Sega Dreamcast.

Naturally, one of the most important pieces of software for a target platform is the C/C++ compiler which is gcc for the SuperH architecture. Being the maintainer of Debian's sh4 port, I'm monitoring daily builds of packages for the sh4 port [2]. While the majority of packages build fine, there are a number of packages which currently fail to build on sh4 due to bugs in the SH backend of gcc.

One important issue with the SH backend is the register allocator which can fail when compiling large and complex code such as in the webkit2gtk package [3][4]. In some cases, such bugs can be worked around by reducing the optimization level from -O2 to -O1 or by omitting individual optimization flags such as -fcrossjumping [5]. It would be better though to get these bugs fixed so that workarounds are no longer necessary. Especially, since it's not always possible to resolve all compiler issues with such workarounds. Also, in the future, the register allocator is supposed to be switched to the new LRA register allocator which is still a bit wonky on SH [6].

Since the SH backend has currently a large number of unresolved bugs [7], it would be great if an experienced gcc developer could start working on these bugs and fix them. Since that requires a lot of time and effort and no developer likes to work for free, I have had the idea to start a crowdfunding campaign within this community to collect some money and create a bounty to fix these bugs which can be picked up by any developer.

I have successfully used this approach in the past in order to modernize the gcc backends for AVR [8], M68k [9] and VAX [10] and to get an experimental M68k backend added to LLVM [11], so it's definitely a very viable approach. Currently the only problem is that Bountysource which we used in the past for these crowdfunding campaigns no longer works correctly, so we would have to use something different for the SH backend.

One could use OpenCollective or Patreon for that matter which we currently use to support ongoing funding for the maintenance and development of the M68k backend in LLVM [12][13].

What do you guys think?

Thanks,
Adrian

> [1] https://fosdem.org/2024/schedule/event/ ... -with-gcc/
> [2] https://buildd.debian.org/status/archit ... &suite=sid
> [3] https://buildd.debian.org/status/fetch. ... 4115&raw=0
> [4] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81426
> [5] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93808
> [6] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55212
> [7] https://gcc.gnu.org/bugzilla/buglist.cg ... _id=415463
> [8] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92729
> [9] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91851
> [10] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95294
> [11] https://github.com/M680x0/M680x0-llvm/issues/59
> [12] https://opencollective.com/m68k-llvm-dev
> [13] https://www.patreon.com/m68k_llvm

[OniEnzeru]

I think a bounty type system is a great idea! It provides great incentive even to those who may not be super involved in our scene, but know gcc, to take a look. I've never used OpenCollective before but I'd be up for supporting whatever gets chosen. This seems vitally important to the future of KOS development, so I'm in. Thanks for all your efforts!

[Quzar]

Thank you for the well cited and thorough information. I would certainly be willing to personally contribute to such a fund, and think some of the other diehards would too here and in some of the other (less dev focused) Dreamcast communities. What may be worthwhile would also be to reach out to any of the current publishers of Dreamcast indie games who might be happy to get the word out and contribute to better backing technology being available for the developers they work with. Similarly publications like Debug Magazine that covers Dreamcast related indie dev might be interested in at least helping get the word out.

I'm not familiar with what other communities are out there using other SH chips that might be covered, but once a platform for the crowd funding is settled, could try to reach out to others as well.

From the m68k ones it looks like only about 2 dozen contributors and a few thousand dollars total. Is that the kind of range that might be expected for SH as well, or perhaps I'm misreading the scope?

[T_chan]

I want to join to say a big thank you for all the work indeed !

Concerning the funding - can't find back where I read the info/if the info is reliable, but I thought the SH was still quite a thing in Japan.
If that's the case, you might want to consider selecting a solution that targets/is well-known in Japan, since they are quite special / often don't use the same solutions as other countries.

(complete noob here concerning gcc internals, but working on my custom SH7750 disassembler for the moment as learning project... when I can find time)

[cbmuser]

I think a bounty type system is a great idea! It provides great incentive even to those who may not be super involved in our scene, but know gcc, to take a look. I've never used OpenCollective before but I'd be up for supporting whatever gets chosen.

Yes, it worked very well for the other architectures.

This seems vitally important to the future of KOS development, so I'm in. Thanks for all your efforts!

Yes, that was my guess. And thanks for the praise.

Thank you for the well cited and thorough information. I would certainly be willing to personally contribute to such a fund, and think some of the other diehards would too here and in some of the other (less dev focused) Dreamcast communities.

Great, I'm very happy to hear that!

What may be worthwhile would also be to reach out to any of the current publishers of Dreamcast indie games who might be happy to get the word out and contribute to better backing technology being available for the developers they work with. Similarly publications like Debug Magazine that covers Dreamcast related indie dev might be interested in at least helping get the word out.

I think this is something that we can do once the crowdfunding campaign is running.

For the various m68k crowdfunding campaigns, I contacted many Amiga communities, developers of software such as Amiga Forever and so on. We even had someone collecting money at AmiWest IIRC.

I'm not familiar with what other communities are out there using other SH chips that might be covered, but once a platform for the crowd funding is settled, could try to reach out to others as well.

I'm very much hoping for the multiplicator that this community will be able to provide.

From the m68k ones it looks like only about 2 dozen contributors and a few thousand dollars total. Is that the kind of range that might be expected for SH as well, or perhaps I'm misreading the scope?

As mentioned above, a lot of people actually donated in cash during AmiWest to one trusted guy from the Amiga community who then put the sum of all donations as a single donation into the bounty.

I want to join to say a big thank you for all the work indeed !

Thanks for the kind words. Much appreciated!

Concerning the funding - can't find back where I read the info/if the info is reliable, but I thought the SH was still quite a thing in Japan.

It's not as popular anymore as it used to be. In the past, you could buy so-called LANDISK devices such as the USL-5P from IO-DATA very cheap and install a custom Linux on it. I have a couple of these which I brought back from Japan which I use for development purposes. However, these devices are no longer sold as companies like Renesas have switched over to ARM.

If that's the case, you might want to consider selecting a solution that targets/is well-known in Japan, since they are quite special / often don't use the same solutions as other countries.

I have contact to several former Renesas/Hitachi engineers that worked on SuperH that I could actually ask to do the actual development work once we have the funds.

(complete noob here concerning gcc internals, but working on my custom SH7750 disassembler for the moment as learning project... when I can find time)

Sounds like a cool project!

Adrian

[Oleg Endo]

Hi,

GCC SH backend maintainer here. It's been a while (~10 years) since I've actively worked on the backend. These days my time is limited to a few critical bug fixes every now and then.

GCC backends show an interesting property, in that they continue operating and producing correct working code for quite a while, even without big maintenance efforts. However, what tends to happen every now and then are changes to the backend independent (middle-end) optimizations. Those are often based on heuristics or observations on mainstream targets like x86, ARM and so on. Changes in those parts of the compiler can degrade the generated code quality of each individual backend. So apart from active development of new target specific optimizations and improvements, there's also a need for periodic fix-up work.

The current SH related bug/improvement list that I've got in GCC's bugzilla is about 220 entries long. I'd suggest to first work together to create some work topics, e.g. "finalize transition to LRA", Maybe some other folks have some other particular requests for this or that.

I guess I could carve out some time in the mid-term for some well defined "work package". If anybody else wants to contribute I can help with reviewing patches and general discussions.

[cbmuser]

Hi Oleg,

GCC SH backend maintainer here. It's been a while (~10 years) since I've actively worked on the backend. These days my time is limited to a few critical bug fixes every now and then.

Great to see you around here. Thanks for chiming in!

GCC backends show an interesting property, in that they continue operating and producing correct working code for quite a while, even without big maintenance efforts. However, what tends to happen every now and then are changes to the backend independent (middle-end) optimizations. Those are often based on heuristics or observations on mainstream targets like x86, ARM and so on. Changes in those parts of the compiler can degrade the generated code quality of each individual backend. So apart from active development of new target specific optimizations and improvements, there's also a need for periodic fix-up work.

Thanks for explaining the reasoning behind my original request. This will certainly help more people in the community understand the need for the work of modernizing the backend.

The current SH related bug/improvement list that I've got in GCC's bugzilla is about 220 entries long. I'd suggest to first work together to create some work topics, e.g. "finalize transition to LRA", Maybe some other folks have some other particular requests for this or that.

I agree that the transition to LRA would be most pressing issue. I have already learned on the #gcc IRC channel on OFTC that moving all targets to LRA is already being prepared, so getting the LRA transition finished for the SH backend would be rather urgent to make sure we retain a working gcc compiler for SuperH.

I guess I could carve out some time in the mid-term for some well defined "work package". If anybody else wants to contribute I can help with reviewing patches and general discussions.

Sounds good. Now we just have to agree what crowd-funding platform to use.

I've made good experiences with both OpenCollective and Patreon, but it's absolutely up to you what you choose. In any case, I would recommend a platform that makes it easy for users to donate, both regularly and single-time donations.

Adrian

[cbmuser]

Hello,

I stumbled over the crowdfunding page for Stellarium on Open Collective yesterday [1] and I thought it looks pretty neat. The project accepts both one-time donations and regular donations. Plus, there is a mechanism to create an expense which is useful when working on a particular task such as fixing a bug from the gcc bugtracker.

@Oleg: What do you think? I think Open Collective fits the job neatly and is also well recognized within the open-source community as it was founded from within the community.

Adrian

> [1] https://opencollective.com/stellarium

[TapamN]

I'd be willing to help fund improvements. It would be nice if GCC would compile "char val = *char_ptr++;" as a single instruction, instead of four! [1]

Related aside: I was recently looking into -mrelax and how/why it only partially works on GCC 12.2. Until a couple of years ago, would regularly use a Jornada 690 (WinCE palmtop with SH3-DSP), running JLime Linux, and I was pretty surprised to see GCC (4.2, I think) on it correctly generating BSR instructions by default... I don't think any version of GCC I've used for DC development could reliably use BSR.

[1]

Spoiler!

Code: Select all

add     #1,r10
mov     r10,r0
add     #-16,r0
mov.b   @(15,r0),r0

An avant-garde way of doing mov.b @r10+, r0...

[cbmuser]

I'd be willing to help fund improvements. It would be nice if GCC would compile "char val = *char_ptr++;" as a single instruction, instead of four! [1]

Could you open a bug report for that in the gcc bug tracker? [1] Make sure to set sh*-*-* as the target and CC me and Oleg. Also, please prefix the subject with [SH].

Related aside: I was recently looking into -mrelax and how/why it only partially works on GCC 12.2. Until a couple of years ago, would regularly use a Jornada 690 (WinCE palmtop with SH3-DSP), running JLime Linux, and I was pretty surprised to see GCC (4.2, I think) on it correctly generating BSR instructions by default... I don't think any version of GCC I've used for DC development could reliably use BSR.

FWIW, there is a developer in #linux-sh chat on Libera IRC who works on Linux on the Journada 690. He contributed a few fixes to unbreak Linux on the Journada 690 so that the latest kernels boot again.

Feel free to join the IRC channel if you're interested in latest kernel developments on SuperH.

Adrian

> [1] https://gcc.gnu.org/bugzilla/

[Oleg Endo]

I'd be willing to help fund improvements. It would be nice if GCC would compile "char val = *char_ptr++;" as a single instruction, instead of four! [1]

This is a problem addressing mode selection. It's a well known issue. A while ago I did two rounds of google summer of code (GSoC) trying to improve the issue with a generalized addressing mode selection optimization pass. However, it turned out more difficult to realize than I had hoped and I ran out of time. Perhaps that work should be picked up.

The issue is that while older compilers (e.g. GCC 3) translated the expression *ptr++ more or less literally, it's now taken apart, pointers being analyzed and tried to optimized in various ways and so on. It's not an SH specific thing, actually all GCC backends suffer from it (e.g. M68K too .... where longer addresses result in actually slower code execution .. ouch). However, modern CPU design trends say that auto-modifying addressing modes are bad for parallelization and so on, so there are no resources flowing into fixing those issues.

Related aside: I was recently looking into -mrelax and how/why it only partially works on GCC 12.2. Until a couple of years ago, would regularly use a Jornada 690 (WinCE palmtop with SH3-DSP), running JLime Linux, and I was pretty surprised to see GCC (4.2, I think) on it correctly generating BSR instructions by default... I don't think any version of GCC I've used for DC development could reliably use BSR.

The compiler can't really generate BSR / BRA instructions itself, because there is no way for it to find out whether the instruction would be able to reach the branch target or not. The way things are done with GCC and binutils, it's only the linker who knows this. The compiler will generate annotations in the ASM code for the linker where it uses JSR / JMP instructions and their corresponding constant load instructions. The linker then will modify the code to replace the instructions. AFAIK the bug is somewhere in binutils. I have never really looked into binutils too much myself.

[Oleg Endo]

Hello,

I stumbled over the crowdfunding page for Stellarium on Open Collective yesterday [1] and I thought it looks pretty neat. The project accepts both one-time donations and regular donations. Plus, there is a mechanism to create an expense which is useful when working on a particular task such as fixing a bug from the gcc bugtracker.

@Oleg: What do you think? I think Open Collective fits the job neatly and is also well recognized within the open-source community as it was founded from within the community.

Adrian

> [1] https://opencollective.com/stellarium

I'm still checking the details how to account such type of income. From what I've seen so far, Open Collective and its implied expectations (public expense tracking??!) sounds and looks like an administrative burden for me. So I haven't decided yet.

[cbmuser]

I'm still checking the details how to account such type of income. From what I've seen so far, Open Collective and its implied expectations (public expense tracking??!) sounds and looks like an administrative burden for me. So I haven't decided yet.

I think the expense tracking is simply meant to associate funds claimed with tasks you performed. For example, if there are $5000 in the funds and you just fixed a long-standing bug, you can claim $500 for the fix and track it as an expense from the funds.

In any case: Whether you're choosing Ko-Fi, Open Collective or something else, I am definitely going to throw in some money.

PS: Could you share an URL to a Bugzilla query with the 200+ bugs you mentioned? I could start looking into some of these and check whether they have been fixed in the mean time or have become obsolete. I have a reported of these bugs after all.

Adrian

[Oleg Endo]

PS: Could you share an URL to a Bugzilla query with the 200+ bugs you mentioned? I could start looking into some of these and check whether they have been fixed in the mean time or have become obsolete. I have a reported of these bugs after all.

https://gcc.gnu.org/bugzilla/buglist.cg ... &v1=sh.%2A

[TapamN]

Could you open a bug report for that in the gcc bug tracker? [1] Make sure to set sh*-*-* as the target and CC me and Oleg. Also, please prefix the subject with [SH].

Ok, I'll try to do it sometime. It shouldn't be a big deal that I was using a patched version of 12.2? The patches are for integrating with KallistiOS (TLS, atomics); there shouldn't be anything that would affect instruction selection.

FWIW, there is a developer in #linux-sh chat on Libera IRC who works on Linux on the Journada 690. He contributed a few fixes to unbreak Linux on the Journada 690 so that the latest kernels boot again.

Feel free to join the IRC channel if you're interested in latest kernel developments on SuperH.

Maybe. I've looked into what would need to be done to get JLime to run on the Compaq Aero 8000 (SH4 WinCE laptop, part of my palmtop/SH collection), and the work needed to get it running terminal-only over serial seemed minimal. But the SH3 WinCE boot loader wouldn't run on SH4 WinCE, and wasn't able to modify it since I had trouble getting a working WinCE dev environment up on Windows 7. I added additional RAM (64MB) to my Compaq 8000, so it would have made for a fun system to run Linux on...

Actually, I remember hearing that there were SH4 based set-top boxes (maybe that was from you on the J-Core mailing list?) that would be good for SH4 Linux. Having an SH4 device that could run Linux would be convenient for me to help debug SH4 asm on. The best SH GDB experience I've had was on JLime with my Jornada (KallistiOS has no memory protection so GDB is unreliable, and emulators are painfully slow), but most of what I need to debug is 3D code that uses FPU instructions not supported by Linux's soft-float emulation (FTRV, FIPR, etc). What should I look for on eBay if I want to run SH Debian? I tried coming up with my own searches after looking at this list, but I'm not finding anything right now...

[Addressing mode and mrelax]

That's about what I would have guessed the problems would have been from.

I guess fixing mrelax would probably be fairly easy compared to some of the other issues (making sure SH works, generating working code, smarter address mode selection), since it seems like it's "just" fixing up some stuff in the linker. It would be the least important of them, though.

Would working with the M68K guys somehow be able to help with improving addressing mode usage, or would coordinating work just make things more difficult, or maybe it's too architecture specific?

[Oleg Endo]

That's about what I would have guessed the problems would have been from.

I guess fixing mrelax would probably be fairly easy compared to some of the other issues (making sure SH works, generating working code, smarter address mode selection), since it seems like it's "just" fixing up some stuff in the linker. It would be the least important of them, though.

It's only as easy as one is familiar with the code base and its inner workings.

Would working with the M68K guys somehow be able to help with improving addressing mode usage, or would coordinating work just make things more difficult, or maybe it's too architecture specific?

What I was trying to do in the GSoC (google summer of code) work was to make it a generic GCC RTL optimization pass that would work with any backend, but using SH as a starting point. The main difficulty comes from integration of this whole thing into GCC and handling of all kinds of special border cases and architecture quirks (e.g. post-increment mode available for this instruction, but not for another and so on). At the moment I don't know anybody who's actively working on anything related to M68K GCC, at least not on the official original GCC.

[cbmuser]

If anyone is interested in helping with the SuperH Linux port, the following are probably the most important issues that need to be fixed.

• glibc:
  + Inconsistent behavior when handling FPSCR on SH: https://sourceware.org/bugzilla/show_bug.cgi?id=27543
• gdb:
  + Upstream support for linux-sh: https://github.com/glaubitz/binutils-gdb/tree/linux-sh
• greenlet:
  + Add missing support for sh4: https://github.com/python-greenlet/greenlet/issues/166
• gcc:
  + Switch to LRA: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55212
• kernel:
  + Add support for device trees: https://patchwork.kernel.org/project/li ... ies=815317
• rust-libc:
  + Add target support for sh4-unknown-linux-gnu
• rustc:
  + Add target support for sh4-unknown-linux-gnu through rustc_codegen_gcc

Any input or help is much appreciated.

Thanks,
Adrian

[coltonp]

Maybe. I've looked into what would need to be done to get JLime to run on the Compaq Aero 8000 (SH4 WinCE laptop, part of my palmtop/SH collection)

Your collection inspired me to branch out to other (Non-Sega) SuperH hardware and I managed to pick up an HPW-600ET! Not sure I'd have been able to find out it was SH4 based without looking at your collection.

Edit: Forgot to share where my documenting my boards: https://github.com/cepawiel/superh-hardware/

Actually, I remember hearing that there were SH4 based set-top boxes (maybe that was from you on the J-Core mailing list?) that would be good for SH4 Linux. Having an SH4 device that could run Linux would be convenient for me to help debug SH4 asm on. The best SH GDB experience I've had was on JLime with my Jornada (KallistiOS has no memory protection so GDB is unreliable, and emulators are painfully slow), but most of what I need to debug is 3D code that uses FPU instructions not supported by Linux's soft-float emulation (FTRV, FIPR, etc). What should I look for on eBay if I want to run SH Debian? I tried coming up with my own searches after looking at this list, but I'm not finding anything right now...

I've seen a few STMicro SH4 (ST40) IP-TV boxes and managed to pick one up (Infomir MAG250 $20-$50 USD). Haven't tried that hard to use it myself beyond some poking at the JTAG TAP. It does appear someone had more success on a similar model: https://www.youtube.com/watch?v=35dPTvBf3rk so should be possible with some persistence.

While I haven't gotten Debian booting (due to minimal effort on my part), its pretty easy to get code booting on the SnapGear/SecureComputing models ($20-$50 USD) listed here: https://elinux.org/Processors#SuperH. They include a bootloader that can boot images from TFTP(tested) or RS232(untested). Alternatively the JTAG/HUDI* is easily access able and I was able to get it working just by lifting a pin on a Reset IC. I picked up one of these SG630 cards too (https://www.ebay.com/itm/134297310218) as the seller had a couple, but haven't tested it yet as I don't have any machines with PCI. I think I can just use a cheap aliexpress riser to power it, but not confirmed.

It's not as popular anymore as it used to be. In the past, you could buy so-called LANDISK devices such as the USL-5P from IO-DATA very cheap and install a custom Linux on it. I have a couple of these which I brought back from Japan which I use for development purposes. However, these devices are no longer sold as companies like Renesas have switched over to ARM.

I was keeping an eye out on and off for a few months, finally manged to pick up a few IO-Data HDL-xxxU models off Yahoo Auctions, and they just came in today! I've been poking at them, but haven't figured out how to get any signs of life yet. Interestingly they have an SH4 model I haven't seen before: "6417380" so maybe an SH7380? No mention I could find on Google

*If it would be beneficial to anyone I have 2 extra E10A-USB debuggers that are missing the official cable and I'd be happy to give them out to people who can use them. I soldered a DIY cable to one, would be easy enough to do the other. I find the IDE (HEW2) to be pretty lame, but it does work (i.e. I was able to load a KOS ELF and source level debugging worked.) They don't work with Dreamcast/NAOMI/NAOMI2/SystemSP, have already attempted that quite a few times. Alternatively I figured out how to load code into the ASERAM (again no success on Sega HW). Have been working on a RP2040 firmware (GDB<->HUDI) & accompanying ASE stub when I have time, but slow going. At the moment I have enough implemented to read/write/execute upon resetting the SH4, but haven't gotten breakpoints/stepping supported yet. Working on cleaning it up so my code is less cryptic and disorganized before I share it, hopefully within a few weeks.

[TapamN]

I'm impressed by the hardware you found. Here's another thing to add to your list: A speak-and-spell clone by Namco.

I ordered that SG630. I have two systems I can dig out with PCI slots (a P3 and P4) that I can try it in at some point. Hopefully no battery damage on those, I've been meaning to check on that...

If someone gets either the SG630 or a MAG250 running Linux, with networking and GDB, so I have a better way to debug SH4 asm, I'd be willing to Paypal them a couple hundred dollars. Getting full desktop Debian on the MAG250 (it apparently has 256MB of RAM?!) would be amazing, but being able to netboot the SG630 and SSH into it would also be acceptable. I don't have the time to do it myself right now.

What documentation on ASERAM do you have? I've tried a bit to scan for undocumented registers/HW (and found a few for the DC's GPU, although I have no clue what they do), but I didn't find anything RAM-like in the "unused" areas in the P4 area. Not really suprising, since I would guess ASERAM is only writable from the debug interface. It's probably not something that could be used on the DC without hardware modifications, but I'm still curious.

The E10A sounds interesting, but I'm not sure if any of the SH4 hardware I have would work with it, so at the moment I wouldn't get any use out of it.

[cbmuser]

Just as a heads-up: CPython 3.13 is failing with gcc-13 on sh4:

Code: Select all

sh4-linux-gnu-gcc -c -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall    -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_TIME_BITS=64 -Wdate-time -D_FORTIFY_SOURCE=2 -g -Werror=implicit-function-declaration -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector -Wformat -Werror=format-security  -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../Include/internal -I../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../Include   -fPIC -DPy_BUILD_CORE -o Python/pystate.o ../Python/pystate.c
../Python/pystate.c: In function '_PyInterpreterState_New':
../Python/pystate.c:752:1: error: unable to find a register to spill in class 'R0_REGS'
  752 | }
      | ^
../Python/pystate.c:752:1: error: this is the insn:

See: https://buildd.debian.org/status/fetch. ... 0073&raw=0

I think the SH could really need some love these days.