Restoring profiling to the KOS toolchain

If you have any questions on programming, this is the place to ask them, whether you're a newbie or an experienced programmer. Discussion on programming in general is also welcome. We will help you with programming homework, but we will not do your work for you! Any porting requests must be made in Developmental Ideas.
Post Reply
kazade
Insane DCEmu
Insane DCEmu
Posts: 139
Joined: Tue May 02, 2017 3:11 pm
Has liked: 2 times
Been liked: 16 times

Restoring profiling to the KOS toolchain

Post by kazade » Sun Dec 22, 2019 12:41 am

Hi!

I'm in desperate need of getting some real profiling data from the Dreamcast. It looks like in theory that the -pg flag should work, and we could get gmon.out generated in the /pc directory.

However, the KOS toolchain patches remove all the profiling code from crt1.S

Does anyone know if I can just copy the profiling setup asm from upstream into the custom startup file?

I don't suppose anyone has the time and know-how to have a go at this? Being able to use gprof would be a game changer!
These users liked the author kazade for the post:
ThePerfectK
User avatar
ThePerfectK
Insane DCEmu
Insane DCEmu
Posts: 121
Joined: Thu Apr 27, 2006 10:15 am
Has liked: 20 times
Been liked: 18 times

Re: Restoring profiling to the KOS toolchain

Post by ThePerfectK » Sun Dec 22, 2019 7:39 am

This is something I've been banging my head against trying to solve for a long while. My idea was to implement a poor man's profiler -- if you aren't familiar with the concept, it's a sampling profiler that works by having the user manually raise breakpoints randomly, then using those breakpoints as samples to see where the program is bottle necking. The idea is that, with random sampling, you will naturally stop execution on parts of your program where most time is being spent.

Unfortunately, I can't find any way to manually raise a breakpoint. Moop has pointed me to a method of raising a breakpoint via the SH4 with his great work, but I haven't set aside any time to look at it. What I can say, however, is that you can simulate a poor man's profiler in a rough way without raising a break point. The way I do it is to define a macro that can be placed at the top and bottom of various functions you write. The function will take the current real time and check against a time threshhold you define in memory. If enough time has passed since the last sample, have the two macros work in concert to record the time the function being run takes. This will let you make a simulated model of the execution time and path of your program. When your function ends, via the function, have it record the time, and also print the call stack so you can do a backtrace for more information. You can manually create a firegraph with something like SDL and ImGUI on a PC to format this information into something resembling a standard profiler.

Just a thought. If you come up with a better solution, please share in as much detail as possible, as I'm all ears!
Still Thinking!~~
TapamN
DCEmu Junior
DCEmu Junior
Posts: 45
Joined: Sun Oct 04, 2009 11:13 am
Has liked: 0
Been liked: 9 times

Re: Restoring profiling to the KOS toolchain

Post by TapamN » Mon Dec 23, 2019 4:04 am

I wrote sampling profiler years ago. It uses a second thread to periodically record the PC of the main thread to an array. When you exit, the samples are saved to a text file on the PC, and a script is used to pass the addresses to addr2line and determine what functions are being run, and how much time is being spent in them. I'll upload it later.
These users liked the author TapamN for the post (total 2):
ThePerfectKIan Robinson
User avatar
ThePerfectK
Insane DCEmu
Insane DCEmu
Posts: 121
Joined: Thu Apr 27, 2006 10:15 am
Has liked: 20 times
Been liked: 18 times

Re: Restoring profiling to the KOS toolchain

Post by ThePerfectK » Mon Dec 23, 2019 6:28 am

TapamN wrote:
Mon Dec 23, 2019 4:04 am
I wrote sampling profiler years ago. It uses a second thread to periodically record the PC of the main thread to an array. When you exit, the samples are saved to a text file on the PC, and a script is used to pass the addresses to addr2line and determine what functions are being run, and how much time is being spent in them. I'll upload it later.
this sounds pretty useful. Would there be any way to modify your profiler to also trace the stack so it'd be possible to know what called the current function?
Still Thinking!~~
kazade
Insane DCEmu
Insane DCEmu
Posts: 139
Joined: Tue May 02, 2017 3:11 pm
Has liked: 2 times
Been liked: 16 times

Re: Restoring profiling to the KOS toolchain

Post by kazade » Mon Dec 30, 2019 12:53 am

I've been thinking of taking exactly the same approach! Did you have any luck finding the code?
User avatar
Ian Robinson
DCEmu Cool Newbie
DCEmu Cool Newbie
Posts: 11
Joined: Mon Mar 11, 2019 7:12 am
Has liked: 11 times
Been liked: 0

Re: Restoring profiling to the KOS toolchain

Post by Ian Robinson » Wed Jan 01, 2020 2:41 pm

TapamN wrote:
Mon Dec 23, 2019 4:04 am
I wrote sampling profiler years ago. It uses a second thread to periodically record the PC of the main thread to an array. When you exit, the samples are saved to a text file on the PC, and a script is used to pass the addresses to addr2line and determine what functions are being run, and how much time is being spent in them. I'll upload it later.
Do you still have it somewhere ? it's much needed by the scene :o
TapamN
DCEmu Junior
DCEmu Junior
Posts: 45
Joined: Sun Oct 04, 2009 11:13 am
Has liked: 0
Been liked: 9 times

Re: Restoring profiling to the KOS toolchain

Post by TapamN » Thu Jan 02, 2020 5:05 pm

I still have my profiling code, but it needs a bit of cleaning up. Judging from the time stamps on the files, it looks like the oldest parts of it are at least 8 years old, and some of the more recent code is about a year or two old. My coding style has changed over time, so the source looks really inconsistent at the moment. There's also some other issues that need fixing (e.g., hard coded file names and timing values) and the only documentation for it is in my head. I'd like to include a few examples on how to use it. I think I should have it ready this weekend.
These users liked the author TapamN for the post (total 5):
ThePerfectKBB HoodIan RobinsonkazadeSiZiOUS
User avatar
Ian Robinson
DCEmu Cool Newbie
DCEmu Cool Newbie
Posts: 11
Joined: Mon Mar 11, 2019 7:12 am
Has liked: 11 times
Been liked: 0

Re: Restoring profiling to the KOS toolchain

Post by Ian Robinson » Sat Jan 04, 2020 6:31 pm

TapamN wrote:
Thu Jan 02, 2020 5:05 pm
I still have my profiling code, but it needs a bit of cleaning up. Judging from the time stamps on the files, it looks like the oldest parts of it are at least 8 years old, and some of the more recent code is about a year or two old. My coding style has changed over time, so the source looks really inconsistent at the moment. There's also some other issues that need fixing (e.g., hard coded file names and timing values) and the only documentation for it is in my head. I'd like to include a few examples on how to use it. I think I should have it ready this weekend.
Thank you so much :)
TapamN
DCEmu Junior
DCEmu Junior
Posts: 45
Joined: Sun Oct 04, 2009 11:13 am
Has liked: 0
Been liked: 9 times

Re: Restoring profiling to the KOS toolchain

Post by TapamN » Sun Jan 05, 2020 6:43 pm

Sorry, but the profiler isn't ready yet. When I was making an example of how to add the profiler to an existing program, I noticed that the profiler was generating incorrect results. After some tweaking, it seems to be working correctly now, but it took longer than I expected. I should have it ready soon.
These users liked the author TapamN for the post (total 2):
ThePerfectKIan Robinson
User avatar
ThePerfectK
Insane DCEmu
Insane DCEmu
Posts: 121
Joined: Thu Apr 27, 2006 10:15 am
Has liked: 20 times
Been liked: 18 times

Re: Restoring profiling to the KOS toolchain

Post by ThePerfectK » Sun Jan 05, 2020 8:29 pm

TapamN wrote:
Sun Jan 05, 2020 6:43 pm
Sorry, but the profiler isn't ready yet. When I was making an example of how to add the profiler to an existing program, I noticed that the profiler was generating incorrect results. After some tweaking, it seems to be working correctly now, but it took longer than I expected. I should have it ready soon.
Take your time and no rush. You're doing everyone a favor and it's not like this is paid work. A big thanks for your effort, but don't feel like you are obligated or owe anybody anything. Real life gets in the way often, and I completely understand.

Thanks again for the effort, though!
Still Thinking!~~
User avatar
Ian Robinson
DCEmu Cool Newbie
DCEmu Cool Newbie
Posts: 11
Joined: Mon Mar 11, 2019 7:12 am
Has liked: 11 times
Been liked: 0

Re: Restoring profiling to the KOS toolchain

Post by Ian Robinson » Mon Jan 06, 2020 11:51 am

ThePerfectK wrote:
Sun Jan 05, 2020 8:29 pm
TapamN wrote:
Sun Jan 05, 2020 6:43 pm
Sorry, but the profiler isn't ready yet. When I was making an example of how to add the profiler to an existing program, I noticed that the profiler was generating incorrect results. After some tweaking, it seems to be working correctly now, but it took longer than I expected. I should have it ready soon.
Take your time and no rush. You're doing everyone a favor and it's not like this is paid work. A big thanks for your effort, but don't feel like you are obligated or owe anybody anything. Real life gets in the way often, and I completely understand.

Thanks again for the effort, though!
Yes thank you for all your effort and time
User avatar
lerabot
Insane DCEmu
Insane DCEmu
Posts: 122
Joined: Sun Nov 01, 2015 8:25 pm
Has liked: 0
Been liked: 3 times

Re: Restoring profiling to the KOS toolchain

Post by lerabot » Wed Jan 08, 2020 9:49 am

Please let us know when you have anything ready :)
kazade
Insane DCEmu
Insane DCEmu
Posts: 139
Joined: Tue May 02, 2017 3:11 pm
Has liked: 2 times
Been liked: 16 times

Re: Restoring profiling to the KOS toolchain

Post by kazade » Sun Jan 19, 2020 2:41 pm

So with TapamN's confirmation that it is indeed possible to profile from a background thread, I went ahead and had a go!

The result is called dcprof, and it can be found here: https://gitlab.com/simulant/dcprof

My initial tests seem to show that it's working, but please let me know how you get on with it if you try it. When run under dcload, this code will generate a gmon.out file which is compatible with gprof.

The code is released under the MIT license, feel free to submit patches/improvements!
These users liked the author kazade for the post (total 4):
|darc|ThePerfectKBB Hoodmaslevin
User avatar
ThePerfectK
Insane DCEmu
Insane DCEmu
Posts: 121
Joined: Thu Apr 27, 2006 10:15 am
Has liked: 20 times
Been liked: 18 times

Re: Restoring profiling to the KOS toolchain

Post by ThePerfectK » Sun Jan 19, 2020 7:26 pm

kazade wrote:
Sun Jan 19, 2020 2:41 pm
So with TapamN's confirmation that it is indeed possible to profile from a background thread, I went ahead and had a go!

The result is called dcprof, and it can be found here: https://gitlab.com/simulant/dcprof

My initial tests seem to show that it's working, but please let me know how you get on with it if you try it. When run under dcload, this code will generate a gmon.out file which is compatible with gprof.

The code is released under the MIT license, feel free to submit patches/improvements!
Can't like this enough! I'd still like to see TapamN's solution if he is still working on his, but this is extremely exciting. With a proper gmon.out that means one can use all sorts of profiling visualization tools. Dude I'm so excited now!! Thanks so much!!
Still Thinking!~~
mrneo240
DCEmu Freak
DCEmu Freak
Posts: 66
Joined: Wed Mar 14, 2018 12:22 am
Has liked: 7 times
Been liked: 9 times

Re: Restoring profiling to the KOS toolchain

Post by mrneo240 » Mon Jan 20, 2020 10:16 am

No announcement but I too wrote something but requires recompile to profile, it's very different from kazade's and might give a different set of data to look at.
If interested I'll toss up a mini example.
User avatar
ThePerfectK
Insane DCEmu
Insane DCEmu
Posts: 121
Joined: Thu Apr 27, 2006 10:15 am
Has liked: 20 times
Been liked: 18 times

Re: Restoring profiling to the KOS toolchain

Post by ThePerfectK » Mon Jan 20, 2020 11:06 am

mrneo240 wrote:
Mon Jan 20, 2020 10:16 am
No announcement but I too wrote something but requires recompile to profile, it's very different from kazade's and might give a different set of data to look at.
If interested I'll toss up a mini example.
More implementations can only ever be a good thing. If nothing else, it could help verify results. Please do post your example if you don't mind!
Still Thinking!~~
User avatar
Ian Robinson
DCEmu Cool Newbie
DCEmu Cool Newbie
Posts: 11
Joined: Mon Mar 11, 2019 7:12 am
Has liked: 11 times
Been liked: 0

Re: Restoring profiling to the KOS toolchain

Post by Ian Robinson » Mon Jan 27, 2020 7:14 pm

TapamN wrote:
Sun Jan 05, 2020 6:43 pm
Sorry, but the profiler isn't ready yet. When I was making an example of how to add the profiler to an existing program, I noticed that the profiler was generating incorrect results. After some tweaking, it seems to be working correctly now, but it took longer than I expected. I should have it ready soon.
Still very much looking forward to this thanks again
Post Reply