Getting weird errors using snd_stream_poll
- showka
- DCEmu Freak
- Posts: 95
- https://www.artistsworkshop.eu/meble-kuchenne-na-wymiar-warszawa-gdzie-zamowic/
- Joined: Fri Nov 23, 2001 12:01 pm
- Location: Border Town
- Has thanked: 0
- Been thanked: 0
- Contact:
Getting weird errors using snd_stream_poll
Hi everyone.
Over the past few weeks I've been trying to get sound working on a Dreamcast game engine. I've been seeing incredibly strange errors where the system will either crash, lock up, or alter the values of variables declared const (as if the registers aren't being restored correctly). When I do get a stack pointer it always points to a malloc or thread function, but in general the main issue seems to be doing anything that takes any time at all in the call back function for a sound stream (a related issue is that using multiple threads makes this happen faster).
My question is if there are any known best practices when using these functions.
Here's a quick program I wrote demonstrating how to make the Dreamcast hang / crash / whatever if the buffer filling call back takes too long:
https://gist.github.com/TimSimpson/55569d860f34787ba65f
By the way, this was compiled against a very recent version of KallistiOS (last commit ba26256094fdee4521261c56539007198b9c11ac). I'm using C++ but have changed the code to remove exception handling and runtime type information.
This forum thread implies that if a buffer is filled in one thread, then it can be passed to the snd_stream call back function. However I have also found that if I use a thread to simply read from a file to a buffer, the program will crash if I'm also calling snd_stream_poll.
Here's a crappier example, though the fact the buffer is accessed by both threads is probably unsafe anyway:
https://gist.github.com/TimSimpson/b092aa383596b0b97579
The hard part of this is its difficult to know how long is too long with the callback. I've written some code (too much for anyone to want to grab, compile and test I assure you) that seems to work very well- and then I'll discover that after a minute or so under certain conditions (i.e. as I'm starting and stopping various sound streams randomly) it crashes.
I guess I'm just curious as to what is considered best practices when using the snd_stream stuff and how to avoid any pitfalls. Thanks.
Over the past few weeks I've been trying to get sound working on a Dreamcast game engine. I've been seeing incredibly strange errors where the system will either crash, lock up, or alter the values of variables declared const (as if the registers aren't being restored correctly). When I do get a stack pointer it always points to a malloc or thread function, but in general the main issue seems to be doing anything that takes any time at all in the call back function for a sound stream (a related issue is that using multiple threads makes this happen faster).
My question is if there are any known best practices when using these functions.
Here's a quick program I wrote demonstrating how to make the Dreamcast hang / crash / whatever if the buffer filling call back takes too long:
https://gist.github.com/TimSimpson/55569d860f34787ba65f
By the way, this was compiled against a very recent version of KallistiOS (last commit ba26256094fdee4521261c56539007198b9c11ac). I'm using C++ but have changed the code to remove exception handling and runtime type information.
This forum thread implies that if a buffer is filled in one thread, then it can be passed to the snd_stream call back function. However I have also found that if I use a thread to simply read from a file to a buffer, the program will crash if I'm also calling snd_stream_poll.
Here's a crappier example, though the fact the buffer is accessed by both threads is probably unsafe anyway:
https://gist.github.com/TimSimpson/b092aa383596b0b97579
The hard part of this is its difficult to know how long is too long with the callback. I've written some code (too much for anyone to want to grab, compile and test I assure you) that seems to work very well- and then I'll discover that after a minute or so under certain conditions (i.e. as I'm starting and stopping various sound streams randomly) it crashes.
I guess I'm just curious as to what is considered best practices when using the snd_stream stuff and how to avoid any pitfalls. Thanks.
Last edited by showka on Sun Jan 12, 2014 6:43 pm, edited 1 time in total.
- showka
- DCEmu Freak
- Posts: 95
- Joined: Fri Nov 23, 2001 12:01 pm
- Location: Border Town
- Has thanked: 0
- Been thanked: 0
- Contact:
Re: Getting werid errors using snd_stream_poll
UPDATE: So of course, after screwing around with this for several hours I finally post it and then notice an obvious bug which is that that I am not setting the samplesReturned correctly in the callback. I've fixed this and now the recreate recreates nothing, so this is a huge waste of time.
However, I *am* seeing real issues when I introduce threads and other fun things. Hopefully I can change the recreate to show these problems or just figure out whats causing them and fix it. I'd delete this post if I could, sorry for wasting anyone's time.
However, I *am* seeing real issues when I introduce threads and other fun things. Hopefully I can change the recreate to show these problems or just figure out whats causing them and fix it. I'd delete this post if I could, sorry for wasting anyone's time.
- BlueCrab
- The Crabby Overlord
- Posts: 5666
- Joined: Mon May 27, 2002 11:31 am
- Location: Sailing the Skies of Arcadia
- Has thanked: 9 times
- Been thanked: 69 times
- Contact:
Re: Getting weird errors using snd_stream_poll
I suggest taking a look at how the ogg vorbis libraries handle the sound streams (either liboggvorbisplay or libtremor). They both use threads to handle the sound stream, so they might provide potentially useful insights on what you need to do.
- showka
- DCEmu Freak
- Posts: 95
- Joined: Fri Nov 23, 2001 12:01 pm
- Location: Border Town
- Has thanked: 0
- Been thanked: 0
- Contact:
Re: Getting weird errors using snd_stream_poll
Thanks for your response BlueCrab.
I've finally created a recreate.
Edit vorbistest.c in the hello-ogg example and and add this at line 61:
With this added, the program seems to crash very quickly. My expectation before this was that the threading system would prevent this from happening. If it doesn't that's ok, as the more I think about it that would be a hell of a lot to expect from KOS, but I just would like to know what the limits are as far as how long each thread can spend working on something while sound is streaming so I don't end up dissappointed in the future after writing a bunch of code.
Since the recreate (maybe?) proves I'm not crazy, I'd like to talk about the other problems I've seen in the larger program I'm working on which is too big to post in this context. Currently, it reads a (rather large) wave file in one thread to a buffer while another thread calls snd_stream_poll and memcpy's data. The main thread is drawing some simple textures using the PVR commands (just FPS information). Usually either the thread which reads the file or the one playing the sound will eventually die.
It seems there are some things which can increase the likliness of thread death:
* Writing to the debug console (printf) can cause problems if you do it in a loop.
* If the file reading thread reads very large chunks at a time, it makes that thread more prone to dying.
* If the call back for snd_stream_poll takes too long the program is liable to crash or do weird things on returning from the callback, such as changing the values of variables.
* If there are no calls to thd_sleep in the main loop, other threads will die.
So finally, here are my questions.
1. What is the "best" way to read from a file in another thread in KOS? Is there a function where I can simply read large amounts into a buffer and retry if I get interrupted due to a context switch?
2. How long, generally, should the call back of snd_stream poll take? I've noticed if I spent too long in the call back, things break. It makes sense that the code in the call back can't be enormous, but is there a known time limit before the program crashes / hangs?
I've finally created a recreate.
Edit vorbistest.c in the hello-ogg example and and add this at line 61:
Code: Select all
for(i = 0; i < 150; i ++) {
printf("Hi! How are you doing? %d\n", i);
}
Since the recreate (maybe?) proves I'm not crazy, I'd like to talk about the other problems I've seen in the larger program I'm working on which is too big to post in this context. Currently, it reads a (rather large) wave file in one thread to a buffer while another thread calls snd_stream_poll and memcpy's data. The main thread is drawing some simple textures using the PVR commands (just FPS information). Usually either the thread which reads the file or the one playing the sound will eventually die.
It seems there are some things which can increase the likliness of thread death:
* Writing to the debug console (printf) can cause problems if you do it in a loop.
* If the file reading thread reads very large chunks at a time, it makes that thread more prone to dying.
* If the call back for snd_stream_poll takes too long the program is liable to crash or do weird things on returning from the callback, such as changing the values of variables.
* If there are no calls to thd_sleep in the main loop, other threads will die.
So finally, here are my questions.
1. What is the "best" way to read from a file in another thread in KOS? Is there a function where I can simply read large amounts into a buffer and retry if I get interrupted due to a context switch?
2. How long, generally, should the call back of snd_stream poll take? I've noticed if I spent too long in the call back, things break. It makes sense that the code in the call back can't be enormous, but is there a known time limit before the program crashes / hangs?
- showka
- DCEmu Freak
- Posts: 95
- Joined: Fri Nov 23, 2001 12:01 pm
- Location: Border Town
- Has thanked: 0
- Been thanked: 0
- Contact:
Re: Getting weird errors using snd_stream_poll
I also just realized that if a location on the pc is used, the vorbis test will fail as well. It plays the first few seconds alright, then begins repeating the same audio chunk in an endless loop.
Is the ethernet cable just too slow for this kind of thing? Despite the fact I payed a ton for it, my BBA is the crappy knock off version which apparently runs slower- could that be it? I have access to the SD card as well, though I'd imagine that wouldn't be too fast either since it goes through the serial port.
Is the ethernet cable just too slow for this kind of thing? Despite the fact I payed a ton for it, my BBA is the crappy knock off version which apparently runs slower- could that be it? I have access to the SD card as well, though I'd imagine that wouldn't be too fast either since it goes through the serial port.
- BlueCrab
- The Crabby Overlord
- Posts: 5666
- Joined: Mon May 27, 2002 11:31 am
- Location: Sailing the Skies of Arcadia
- Has thanked: 9 times
- Been thanked: 69 times
- Contact:
Re: Getting weird errors using snd_stream_poll
In theory, there shouldn't really be any issues with reading a file in one thread and doing other things in another thread (assuming there's nothing accessing the VFS in the other thread -- the VFS is explicitly NOT thread safe, which is something I should fix at some point). You shouldn't have to retry if interrupted, as the next time your thread gets swapped in the read should continue.showka wrote:1. What is the "best" way to read from a file in another thread in KOS? Is there a function where I can simply read large amounts into a buffer and retry if I get interrupted due to a context switch?
That function should finish as quickly as possible and do as little blocking as possible. This is the same as on most other low-latency sound APIs even on PCs -- taking a long time in your callback function will cause audio issues no matter what system you're doing it on. As for how long you have, that depends on a couple of factors: the size of the buffer you set up when you create the stream and how long it has been since the last time you fed data in. There's no hard limits, but as I said, you want to minimize how much time you spend in your callback. In theory, the worst that would happen if you took too long would be that the sound would start to loop, although you do seem to be experiencing other problems.2. How long, generally, should the call back of snd_stream poll take? I've noticed if I spent too long in the call back, things break. It makes sense that the code in the call back can't be enormous, but is there a known time limit before the program crashes / hangs?
If you can find a small example where something goes wrong and data that shouldn't be overwritten does get overwritten, I'd like to take a look at it. That kind of stuff should definitely not happen, regardless of how long your callback takes. In theory the worst that should happen is that your sound starts looping. If anything like what you describe is happening, then there's got to be a bug somewhere.
And you really shouldn't be reading from /pc if it is possible to avoid doing so in any time-critical code. It is slow, even with a BBA.
- showka
- DCEmu Freak
- Posts: 95
- Joined: Fri Nov 23, 2001 12:01 pm
- Location: Border Town
- Has thanked: 0
- Been thanked: 0
- Contact:
Re: Getting weird errors using snd_stream_poll
First off: THANK YOU for the info about the VFS. I assumed that was the case but its good to read it (I'm guessing it was somewhere in the docs, but still). It's also a major bummer to hear that about the BBA, as I'm not sure how I'm going to test anything now (although if it was just slow and didn't crash I'd be fine with that).In theory, there shouldn't really be any issues with reading a file in one thread and doing other things in another thread (assuming there's nothing accessing the VFS in the other thread -- the VFS is explicitly NOT thread safe, which is something I should fix at some point). You shouldn't have to retry if interrupted, as the next time your thread gets swapped in the read should continue.
Also, thanks for letting know that the amount of time taken in the callbacks is a problem. If I can isolate the issue I noticed where variables where changed after a callback I'll try to cut it down to a recreate right away.
My next question is why does adding the calls to printf in the ogg vorbis example cause it to crash? The other threads should take the same amount of time as usual, so I don't get why the main thread taking longer would cause them to die.
- showka
- DCEmu Freak
- Posts: 95
- Joined: Fri Nov 23, 2001 12:01 pm
- Location: Border Town
- Has thanked: 0
- Been thanked: 0
- Contact:
Re: Getting weird errors using snd_stream_poll
Hey - a bit of good news. Since you suggested that the size of the buffers was important, I realized I had been setting them to SND_STREAM_BUFFER_MAX, which I believe is something like 80kb. I pulled way back on that and set it down to 8kb and the results were much prettier and things generally seem to be more responsive at a steady 34fps (although now the wave file is running into some other issues).
However, after running it for awhile, the heartBeats in the loading thread still stopped incrementing. I'll try to look at it more tomorrow.
However, after running it for awhile, the heartBeats in the loading thread still stopped incrementing. I'll try to look at it more tomorrow.
- showka
- DCEmu Freak
- Posts: 95
- Joined: Fri Nov 23, 2001 12:01 pm
- Location: Border Town
- Has thanked: 0
- Been thanked: 0
- Contact:
Re: Getting weird errors using snd_stream_poll
I was able to get eliminate crashes from the program I've been describing by moving the sound poll routines in the main thread (though the wave file reader seems broken in other ways). It seems that plus the stream size of 8kb has helped immensely, though if I decrease it too much the reader thread dies right way. I also took the console writing for now.
One more question: is there a way to read from a file in KOS with a timer interrupt or call back (something like select)? It would be crazy but I'm curious if it would be possible to avoid using threads entirely.
One more question: is there a way to read from a file in KOS with a timer interrupt or call back (something like select)? It would be crazy but I'm curious if it would be possible to avoid using threads entirely.
- BlueCrab
- The Crabby Overlord
- Posts: 5666
- Joined: Mon May 27, 2002 11:31 am
- Location: Sailing the Skies of Arcadia
- Has thanked: 9 times
- Been thanked: 69 times
- Contact:
Re: Getting weird errors using snd_stream_poll
No, there is no such method of doing file I/O in KOS. The select() and poll() functions only works with sockets. In all honesty, I'm not entirely sure how they'd be applicable to what you're doing anyway. Most of the time when reading a file, your request gets handled immediately on KOS -- there generally is no blocking that happens (except with the /pc filesystem).showka wrote:One more question: is there a way to read from a file in KOS with a timer interrupt or call back (something like select)? It would be crazy but I'm curious if it would be possible to avoid using threads entirely.
Oh, and /pc is faster with a BBA than with serial, but it is still pretty slow. I didn't mean it to seem like it is completely unusable with a BBA, just much slower than burning a CD with your data and using that.
As for why adding printf calls can crash things, I believe this to be an issue with dcload losing packets (or there being a corrupted packet) or something to that effect. Every once in a while I have a similar problem with dcload-ip, and it is usually when there is a lot of interference over wifi between the router and my laptop.
- showka
- DCEmu Freak
- Posts: 95
- Joined: Fri Nov 23, 2001 12:01 pm
- Location: Border Town
- Has thanked: 0
- Been thanked: 0
- Contact:
Re: Getting weird errors using snd_stream_poll
If reading from /pc blocks, then wouldn't it follow that trying to read large chunks from it from one thread while playing sound, drawing to the screen and chewing gum in the other would lead to a crash? If the read from /pc is blocking, then when the thread goes to context switch it makes sense that it would mess up. This would also explain why the generally trusted hello-ogg example fails when reading from /pc instead of /rd.Most of the time when reading a file, your request gets handled immediately on KOS -- there generally is no blocking that happens (except with the /pc filesystem.
So, is the idea to avoid reading from /pc in this manner and stick to reading from the cd directly, maybe using romdisks or NullDC along the way to simulate the final result more accurately? If so, I need to find out if anyone still sells CDRs that aren't garbage...
- BlueCrab
- The Crabby Overlord
- Posts: 5666
- Joined: Mon May 27, 2002 11:31 am
- Location: Sailing the Skies of Arcadia
- Has thanked: 9 times
- Been thanked: 69 times
- Contact:
Re: Getting weird errors using snd_stream_poll
A context switch should not mess anything up, unless you're using dcload-serial and something else is accessing the serial port -- which brings up what might well be causing your problem... I'll have to take a look and see if that is adequately guarded against and do so if it isn't... Yay another issue to add to my pile of things to fix.showka wrote:If reading from /pc blocks, then wouldn't it follow that trying to read large chunks from it from one thread while playing sound, drawing to the screen and chewing gum in the other would lead to a crash? If the read from /pc is blocking, then when the thread goes to context switch it makes sense that it would mess up. This would also explain why the generally trusted hello-ogg example fails when reading from /pc instead of /rd.Most of the time when reading a file, your request gets handled immediately on KOS -- there generally is no blocking that happens (except with the /pc filesystem.
CD and Romdisks are your best bet. Also, with regard to CD-Rs, I've often found that the no-name ones work better on the DC than do the brand-name ones.So, is the idea to avoid reading from /pc in this manner and stick to reading from the cd directly, maybe using romdisks or NullDC along the way to simulate the final result more accurately? If so, I need to find out if anyone still sells CDRs that aren't garbage...
- showka
- DCEmu Freak
- Posts: 95
- Joined: Fri Nov 23, 2001 12:01 pm
- Location: Border Town
- Has thanked: 0
- Been thanked: 0
- Contact:
Re: Getting weird errors using snd_stream_poll
So today I started the Dreamcast with the dcload-ip disc, then swapped in a new CDR with an oggfile on it and ran a version of vorbistest.c which read from the cd drive (there were no printf statements or other alterations).
The program appeared to hang and would not end when I pressed start, constantly repeated a second or so of sound from the ogg file.
When I try the program again reading from /pc, it hangs- but it seems to get a very slight bit further into the ogg file before repeating forever.
The program appeared to hang and would not end when I pressed start, constantly repeated a second or so of sound from the ogg file.
When I try the program again reading from /pc, it hangs- but it seems to get a very slight bit further into the ogg file before repeating forever.
- BlueCrab
- The Crabby Overlord
- Posts: 5666
- Joined: Mon May 27, 2002 11:31 am
- Location: Sailing the Skies of Arcadia
- Has thanked: 9 times
- Been thanked: 69 times
- Contact:
Re: Getting weird errors using snd_stream_poll
Well, I just tested the hello-ogg example with a file on /pc, and it worked fine for me, other than a bit of looping caused by the reads from /pc being too slow to keep up with playback (which is to be expected, especially since I was using a Lan Adapter and not a BBA). Not sure what's wrong on your end...
- showka
- DCEmu Freak
- Posts: 95
- Joined: Fri Nov 23, 2001 12:01 pm
- Location: Border Town
- Has thanked: 0
- Been thanked: 0
- Contact:
Re: Getting weird errors using snd_stream_poll
Well... doo doo. At this point I'm not really sure what to do; this may be difficult to believe but I've already rebuilt KOS and the ports on both Windows and on my Ubuntu VM (which can also talk to the Dreamcast).
Should I also rebuild the toolchains (i.e. stuff like the compilers)? I last did this a year ago.
Should I also rebuild the toolchains (i.e. stuff like the compilers)? I last did this a year ago.
- BlueCrab
- The Crabby Overlord
- Posts: 5666
- Joined: Mon May 27, 2002 11:31 am
- Location: Sailing the Skies of Arcadia
- Has thanked: 9 times
- Been thanked: 69 times
- Contact:
Re: Getting weird errors using snd_stream_poll
Well, in theory it shouldn't matter. You could try that if you wanted though. I'm using the same versions that are the default in the dc-chain script that's included with KOS, by the way.
- showka
- DCEmu Freak
- Posts: 95
- Joined: Fri Nov 23, 2001 12:01 pm
- Location: Border Town
- Has thanked: 0
- Been thanked: 0
- Contact:
Re: Getting weird errors using snd_stream_poll
Just recompiled my toolchain in Linux, bringing gcc from 4.7.0 to 4.8.0, the recompiled KallistiOS and kos-ports. Same. Exact. Problem.
I then brought out my old Dreamcast, swapped out the BBA from the new one to the old one. Same thing.
If it's working for you, then I wonder if this means the speed of the connection *does* matter, and my Windows machine just isn't sending data over ethernet fast enough (though that wouldn't explain why when I tried to read from the CD the same thing happened).
I modified the Makefile to use -loggvorbisplay instead, which made the ogg example get further. But it still hung. One thing that's interesting is that before it hung and repeated ad naseum there were two times where the audio repeated a bit before it recovered. I wonder if this means a failure to receive packets is what's causing the problem.
I then brought out my old Dreamcast, swapped out the BBA from the new one to the old one. Same thing.
If it's working for you, then I wonder if this means the speed of the connection *does* matter, and my Windows machine just isn't sending data over ethernet fast enough (though that wouldn't explain why when I tried to read from the CD the same thing happened).
I modified the Makefile to use -loggvorbisplay instead, which made the ogg example get further. But it still hung. One thing that's interesting is that before it hung and repeated ad naseum there were two times where the audio repeated a bit before it recovered. I wonder if this means a failure to receive packets is what's causing the problem.
- BlueCrab
- The Crabby Overlord
- Posts: 5666
- Joined: Mon May 27, 2002 11:31 am
- Location: Sailing the Skies of Arcadia
- Has thanked: 9 times
- Been thanked: 69 times
- Contact:
Re: Getting weird errors using snd_stream_poll
I'm going to guess it is something with a packet getting lost or corrupted along the way. When using my laptop, I generally connect the DC directly to the ethernet port on the laptop, as often packets get lost if I use the laptop's wifi instead.
- showka
- DCEmu Freak
- Posts: 95
- Joined: Fri Nov 23, 2001 12:01 pm
- Location: Border Town
- Has thanked: 0
- Been thanked: 0
- Contact:
Re: Getting weird errors using snd_stream_poll
In case I'd mistated it: my ethernet cable goes from my desktop straight to the Dreamcast.
Today I refreshed and rebuilt KallistiOS and kost-ports on my netbook and ran a few demos. The behavior of the hello-ogg example was identical. It works with the romdisk, but fails reading from /pc.
Today I refreshed and rebuilt KallistiOS and kost-ports on my netbook and ran a few demos. The behavior of the hello-ogg example was identical. It works with the romdisk, but fails reading from /pc.
- BlueCrab
- The Crabby Overlord
- Posts: 5666
- Joined: Mon May 27, 2002 11:31 am
- Location: Sailing the Skies of Arcadia
- Has thanked: 9 times
- Been thanked: 69 times
- Contact:
Re: Getting weird errors using snd_stream_poll
As I mentioned in the IRC channel, I'm pretty much stumped at this point as to what's going wrong. Perhaps someone else has some ideas?