Getting weird errors using snd_stream_poll

If you have any questions on programming, this is the place to ask them, whether you're a newbie or an experienced programmer. Discussion on programming in general is also welcome. We will help you with programming homework, but we will not do your work for you! Any porting requests must be made in Developmental Ideas.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has liked: 0
Been liked: 0
Contact:

Getting weird errors using snd_stream_poll

Post by showka » Sat Jan 11, 2014 11:47 pm

Hi everyone.

Over the past few weeks I've been trying to get sound working on a Dreamcast game engine. I've been seeing incredibly strange errors where the system will either crash, lock up, or alter the values of variables declared const (as if the registers aren't being restored correctly). When I do get a stack pointer it always points to a malloc or thread function, but in general the main issue seems to be doing anything that takes any time at all in the call back function for a sound stream (a related issue is that using multiple threads makes this happen faster).

My question is if there are any known best practices when using these functions.

Here's a quick program I wrote demonstrating how to make the Dreamcast hang / crash / whatever if the buffer filling call back takes too long:
https://gist.github.com/TimSimpson/55569d860f34787ba65f

By the way, this was compiled against a very recent version of KallistiOS (last commit ba26256094fdee4521261c56539007198b9c11ac). I'm using C++ but have changed the code to remove exception handling and runtime type information.

This forum thread implies that if a buffer is filled in one thread, then it can be passed to the snd_stream call back function. However I have also found that if I use a thread to simply read from a file to a buffer, the program will crash if I'm also calling snd_stream_poll.

Here's a crappier example, though the fact the buffer is accessed by both threads is probably unsafe anyway:
https://gist.github.com/TimSimpson/b092aa383596b0b97579

The hard part of this is its difficult to know how long is too long with the callback. I've written some code (too much for anyone to want to grab, compile and test I assure you) that seems to work very well- and then I'll discover that after a minute or so under certain conditions (i.e. as I'm starting and stopping various sound streams randomly) it crashes.

I guess I'm just curious as to what is considered best practices when using the snd_stream stuff and how to avoid any pitfalls. Thanks.
Last edited by showka on Sun Jan 12, 2014 6:43 pm, edited 1 time in total.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has liked: 0
Been liked: 0
Contact:

Re: Getting werid errors using snd_stream_poll

Post by showka » Sun Jan 12, 2014 12:27 am

UPDATE: So of course, after screwing around with this for several hours I finally post it and then notice an obvious bug which is that that I am not setting the samplesReturned correctly in the callback. I've fixed this and now the recreate recreates nothing, so this is a huge waste of time.

However, I *am* seeing real issues when I introduce threads and other fun things. Hopefully I can change the recreate to show these problems or just figure out whats causing them and fix it. I'd delete this post if I could, sorry for wasting anyone's time.
User avatar
BlueCrab
The Crabby Overlord
The Crabby Overlord
Posts: 5404
Joined: Mon May 27, 2002 11:31 am
Location: Sailing the Skies of Arcadia
Has liked: 1 time
Been liked: 16 times
Contact:

Re: Getting weird errors using snd_stream_poll

Post by BlueCrab » Sun Jan 12, 2014 8:50 pm

I suggest taking a look at how the ogg vorbis libraries handle the sound streams (either liboggvorbisplay or libtremor). They both use threads to handle the sound stream, so they might provide potentially useful insights on what you need to do.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has liked: 0
Been liked: 0
Contact:

Re: Getting weird errors using snd_stream_poll

Post by showka » Sun Jan 12, 2014 11:06 pm

Thanks for your response BlueCrab.

I've finally created a recreate.

Edit vorbistest.c in the hello-ogg example and and add this at line 61:

Code: Select all

for(i = 0; i < 150; i ++) { 
    printf("Hi! How are you doing? %d\n", i);
}
With this added, the program seems to crash very quickly. My expectation before this was that the threading system would prevent this from happening. If it doesn't that's ok, as the more I think about it that would be a hell of a lot to expect from KOS, but I just would like to know what the limits are as far as how long each thread can spend working on something while sound is streaming so I don't end up dissappointed in the future after writing a bunch of code.

Since the recreate (maybe?) proves I'm not crazy, I'd like to talk about the other problems I've seen in the larger program I'm working on which is too big to post in this context. Currently, it reads a (rather large) wave file in one thread to a buffer while another thread calls snd_stream_poll and memcpy's data. The main thread is drawing some simple textures using the PVR commands (just FPS information). Usually either the thread which reads the file or the one playing the sound will eventually die.

It seems there are some things which can increase the likliness of thread death:

* Writing to the debug console (printf) can cause problems if you do it in a loop.
* If the file reading thread reads very large chunks at a time, it makes that thread more prone to dying.
* If the call back for snd_stream_poll takes too long the program is liable to crash or do weird things on returning from the callback, such as changing the values of variables.
* If there are no calls to thd_sleep in the main loop, other threads will die.

So finally, here are my questions.

1. What is the "best" way to read from a file in another thread in KOS? Is there a function where I can simply read large amounts into a buffer and retry if I get interrupted due to a context switch?

2. How long, generally, should the call back of snd_stream poll take? I've noticed if I spent too long in the call back, things break. It makes sense that the code in the call back can't be enormous, but is there a known time limit before the program crashes / hangs?
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has liked: 0
Been liked: 0
Contact:

Re: Getting weird errors using snd_stream_poll

Post by showka » Sun Jan 12, 2014 11:18 pm

I also just realized that if a location on the pc is used, the vorbis test will fail as well. It plays the first few seconds alright, then begins repeating the same audio chunk in an endless loop.

Is the ethernet cable just too slow for this kind of thing? Despite the fact I payed a ton for it, my BBA is the crappy knock off version which apparently runs slower- could that be it? I have access to the SD card as well, though I'd imagine that wouldn't be too fast either since it goes through the serial port.
User avatar
BlueCrab
The Crabby Overlord
The Crabby Overlord
Posts: 5404
Joined: Mon May 27, 2002 11:31 am
Location: Sailing the Skies of Arcadia
Has liked: 1 time
Been liked: 16 times
Contact:

Re: Getting weird errors using snd_stream_poll

Post by BlueCrab » Sun Jan 12, 2014 11:31 pm

showka wrote:1. What is the "best" way to read from a file in another thread in KOS? Is there a function where I can simply read large amounts into a buffer and retry if I get interrupted due to a context switch?
In theory, there shouldn't really be any issues with reading a file in one thread and doing other things in another thread (assuming there's nothing accessing the VFS in the other thread -- the VFS is explicitly NOT thread safe, which is something I should fix at some point). You shouldn't have to retry if interrupted, as the next time your thread gets swapped in the read should continue.
2. How long, generally, should the call back of snd_stream poll take? I've noticed if I spent too long in the call back, things break. It makes sense that the code in the call back can't be enormous, but is there a known time limit before the program crashes / hangs?
That function should finish as quickly as possible and do as little blocking as possible. This is the same as on most other low-latency sound APIs even on PCs -- taking a long time in your callback function will cause audio issues no matter what system you're doing it on. As for how long you have, that depends on a couple of factors: the size of the buffer you set up when you create the stream and how long it has been since the last time you fed data in. There's no hard limits, but as I said, you want to minimize how much time you spend in your callback. In theory, the worst that would happen if you took too long would be that the sound would start to loop, although you do seem to be experiencing other problems.

If you can find a small example where something goes wrong and data that shouldn't be overwritten does get overwritten, I'd like to take a look at it. That kind of stuff should definitely not happen, regardless of how long your callback takes. In theory the worst that should happen is that your sound starts looping. If anything like what you describe is happening, then there's got to be a bug somewhere. :?

And you really shouldn't be reading from /pc if it is possible to avoid doing so in any time-critical code. It is slow, even with a BBA. :wink:
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has liked: 0
Been liked: 0
Contact:

Re: Getting weird errors using snd_stream_poll

Post by showka » Sun Jan 12, 2014 11:48 pm

In theory, there shouldn't really be any issues with reading a file in one thread and doing other things in another thread (assuming there's nothing accessing the VFS in the other thread -- the VFS is explicitly NOT thread safe, which is something I should fix at some point). You shouldn't have to retry if interrupted, as the next time your thread gets swapped in the read should continue.
First off: THANK YOU for the info about the VFS. I assumed that was the case but its good to read it (I'm guessing it was somewhere in the docs, but still). It's also a major bummer to hear that about the BBA, as I'm not sure how I'm going to test anything now (although if it was just slow and didn't crash I'd be fine with that).

Also, thanks for letting know that the amount of time taken in the callbacks is a problem. If I can isolate the issue I noticed where variables where changed after a callback I'll try to cut it down to a recreate right away.

My next question is why does adding the calls to printf in the ogg vorbis example cause it to crash? The other threads should take the same amount of time as usual, so I don't get why the main thread taking longer would cause them to die.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has liked: 0
Been liked: 0
Contact:

Re: Getting weird errors using snd_stream_poll

Post by showka » Sun Jan 12, 2014 11:56 pm

Hey - a bit of good news. Since you suggested that the size of the buffers was important, I realized I had been setting them to SND_STREAM_BUFFER_MAX, which I believe is something like 80kb. I pulled way back on that and set it down to 8kb and the results were much prettier and things generally seem to be more responsive at a steady 34fps (although now the wave file is running into some other issues).

However, after running it for awhile, the heartBeats in the loading thread still stopped incrementing. I'll try to look at it more tomorrow.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has liked: 0
Been liked: 0
Contact:

Re: Getting weird errors using snd_stream_poll

Post by showka » Mon Jan 13, 2014 12:40 am

I was able to get eliminate crashes from the program I've been describing by moving the sound poll routines in the main thread (though the wave file reader seems broken in other ways). It seems that plus the stream size of 8kb has helped immensely, though if I decrease it too much the reader thread dies right way. I also took the console writing for now.

One more question: is there a way to read from a file in KOS with a timer interrupt or call back (something like select)? It would be crazy but I'm curious if it would be possible to avoid using threads entirely.
User avatar
BlueCrab
The Crabby Overlord
The Crabby Overlord
Posts: 5404
Joined: Mon May 27, 2002 11:31 am
Location: Sailing the Skies of Arcadia
Has liked: 1 time
Been liked: 16 times
Contact:

Re: Getting weird errors using snd_stream_poll

Post by BlueCrab » Mon Jan 13, 2014 7:52 am

showka wrote:One more question: is there a way to read from a file in KOS with a timer interrupt or call back (something like select)? It would be crazy but I'm curious if it would be possible to avoid using threads entirely.
No, there is no such method of doing file I/O in KOS. The select() and poll() functions only works with sockets. In all honesty, I'm not entirely sure how they'd be applicable to what you're doing anyway. Most of the time when reading a file, your request gets handled immediately on KOS -- there generally is no blocking that happens (except with the /pc filesystem).

Oh, and /pc is faster with a BBA than with serial, but it is still pretty slow. I didn't mean it to seem like it is completely unusable with a BBA, just much slower than burning a CD with your data and using that. :wink:

As for why adding printf calls can crash things, I believe this to be an issue with dcload losing packets (or there being a corrupted packet) or something to that effect. Every once in a while I have a similar problem with dcload-ip, and it is usually when there is a lot of interference over wifi between the router and my laptop.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has liked: 0
Been liked: 0
Contact:

Re: Getting weird errors using snd_stream_poll

Post by showka » Mon Jan 13, 2014 8:59 am

Most of the time when reading a file, your request gets handled immediately on KOS -- there generally is no blocking that happens (except with the /pc filesystem.
If reading from /pc blocks, then wouldn't it follow that trying to read large chunks from it from one thread while playing sound, drawing to the screen and chewing gum in the other would lead to a crash? If the read from /pc is blocking, then when the thread goes to context switch it makes sense that it would mess up. This would also explain why the generally trusted hello-ogg example fails when reading from /pc instead of /rd.

So, is the idea to avoid reading from /pc in this manner and stick to reading from the cd directly, maybe using romdisks or NullDC along the way to simulate the final result more accurately? If so, I need to find out if anyone still sells CDRs that aren't garbage...
User avatar
BlueCrab
The Crabby Overlord
The Crabby Overlord
Posts: 5404
Joined: Mon May 27, 2002 11:31 am
Location: Sailing the Skies of Arcadia
Has liked: 1 time
Been liked: 16 times
Contact:

Re: Getting weird errors using snd_stream_poll

Post by BlueCrab » Mon Jan 13, 2014 9:08 am

showka wrote:
Most of the time when reading a file, your request gets handled immediately on KOS -- there generally is no blocking that happens (except with the /pc filesystem.
If reading from /pc blocks, then wouldn't it follow that trying to read large chunks from it from one thread while playing sound, drawing to the screen and chewing gum in the other would lead to a crash? If the read from /pc is blocking, then when the thread goes to context switch it makes sense that it would mess up. This would also explain why the generally trusted hello-ogg example fails when reading from /pc instead of /rd.
A context switch should not mess anything up, unless you're using dcload-serial and something else is accessing the serial port -- which brings up what might well be causing your problem... I'll have to take a look and see if that is adequately guarded against and do so if it isn't... Yay another issue to add to my pile of things to fix. :P
So, is the idea to avoid reading from /pc in this manner and stick to reading from the cd directly, maybe using romdisks or NullDC along the way to simulate the final result more accurately? If so, I need to find out if anyone still sells CDRs that aren't garbage...
CD and Romdisks are your best bet. Also, with regard to CD-Rs, I've often found that the no-name ones work better on the DC than do the brand-name ones.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has liked: 0
Been liked: 0
Contact:

Re: Getting weird errors using snd_stream_poll

Post by showka » Mon Jan 13, 2014 6:00 pm

So today I started the Dreamcast with the dcload-ip disc, then swapped in a new CDR with an oggfile on it and ran a version of vorbistest.c which read from the cd drive (there were no printf statements or other alterations).

The program appeared to hang and would not end when I pressed start, constantly repeated a second or so of sound from the ogg file.

When I try the program again reading from /pc, it hangs- but it seems to get a very slight bit further into the ogg file before repeating forever.
User avatar
BlueCrab
The Crabby Overlord
The Crabby Overlord
Posts: 5404
Joined: Mon May 27, 2002 11:31 am
Location: Sailing the Skies of Arcadia
Has liked: 1 time
Been liked: 16 times
Contact:

Re: Getting weird errors using snd_stream_poll

Post by BlueCrab » Mon Jan 13, 2014 6:59 pm

Well, I just tested the hello-ogg example with a file on /pc, and it worked fine for me, other than a bit of looping caused by the reads from /pc being too slow to keep up with playback (which is to be expected, especially since I was using a Lan Adapter and not a BBA). Not sure what's wrong on your end... :?
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has liked: 0
Been liked: 0
Contact:

Re: Getting weird errors using snd_stream_poll

Post by showka » Mon Jan 13, 2014 9:08 pm

Well... doo doo. At this point I'm not really sure what to do; this may be difficult to believe but I've already rebuilt KOS and the ports on both Windows and on my Ubuntu VM (which can also talk to the Dreamcast).

Should I also rebuild the toolchains (i.e. stuff like the compilers)? I last did this a year ago.
User avatar
BlueCrab
The Crabby Overlord
The Crabby Overlord
Posts: 5404
Joined: Mon May 27, 2002 11:31 am
Location: Sailing the Skies of Arcadia
Has liked: 1 time
Been liked: 16 times
Contact:

Re: Getting weird errors using snd_stream_poll

Post by BlueCrab » Tue Jan 14, 2014 12:46 am

Well, in theory it shouldn't matter. You could try that if you wanted though. I'm using the same versions that are the default in the dc-chain script that's included with KOS, by the way.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has liked: 0
Been liked: 0
Contact:

Re: Getting weird errors using snd_stream_poll

Post by showka » Tue Jan 14, 2014 8:21 pm

Just recompiled my toolchain in Linux, bringing gcc from 4.7.0 to 4.8.0, the recompiled KallistiOS and kos-ports. Same. Exact. Problem.

I then brought out my old Dreamcast, swapped out the BBA from the new one to the old one. Same thing.

If it's working for you, then I wonder if this means the speed of the connection *does* matter, and my Windows machine just isn't sending data over ethernet fast enough (though that wouldn't explain why when I tried to read from the CD the same thing happened).

I modified the Makefile to use -loggvorbisplay instead, which made the ogg example get further. But it still hung. One thing that's interesting is that before it hung and repeated ad naseum there were two times where the audio repeated a bit before it recovered. I wonder if this means a failure to receive packets is what's causing the problem.
User avatar
BlueCrab
The Crabby Overlord
The Crabby Overlord
Posts: 5404
Joined: Mon May 27, 2002 11:31 am
Location: Sailing the Skies of Arcadia
Has liked: 1 time
Been liked: 16 times
Contact:

Re: Getting weird errors using snd_stream_poll

Post by BlueCrab » Tue Jan 14, 2014 9:14 pm

I'm going to guess it is something with a packet getting lost or corrupted along the way. When using my laptop, I generally connect the DC directly to the ethernet port on the laptop, as often packets get lost if I use the laptop's wifi instead.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has liked: 0
Been liked: 0
Contact:

Re: Getting weird errors using snd_stream_poll

Post by showka » Wed Jan 15, 2014 7:44 pm

In case I'd mistated it: my ethernet cable goes from my desktop straight to the Dreamcast.

Today I refreshed and rebuilt KallistiOS and kost-ports on my netbook and ran a few demos. The behavior of the hello-ogg example was identical. It works with the romdisk, but fails reading from /pc.
User avatar
BlueCrab
The Crabby Overlord
The Crabby Overlord
Posts: 5404
Joined: Mon May 27, 2002 11:31 am
Location: Sailing the Skies of Arcadia
Has liked: 1 time
Been liked: 16 times
Contact:

Re: Getting weird errors using snd_stream_poll

Post by BlueCrab » Wed Jan 15, 2014 10:23 pm

As I mentioned in the IRC channel, I'm pretty much stumped at this point as to what's going wrong. Perhaps someone else has some ideas?
Post Reply