Problem with Lua 5.1

If you have any questions on programming, this is the place to ask them, whether you're a newbie or an experienced programmer. Discussion on programming in general is also welcome. We will help you with programming homework, but we will not do your work for you! Any porting requests must be made in Developmental Ideas.
Post Reply
BlackAura
DC Developer
DC Developer
Posts: 9951
https://www.artistsworkshop.eu/meble-kuchenne-na-wymiar-warszawa-gdzie-zamowic/
Joined: Sun Dec 30, 2001 9:02 am
Has thanked: 0
Been thanked: 1 time

Problem with Lua 5.1

Post by BlackAura »

I've been trying to get Lua 5.1 working on the Dreamcast for a while (about an hour). As of right now, it seems to be working correctly, except for one problem - I had to bypass the KOS built-in memory allocator to get the thing to work. Otherwise, it runs perfectly with absolutely no modifications.

All of Lua's memory allocation (since 5.1) goes through a single function, the default version of which is in lauxlib.c. For some unexplained reason, it simply hard locks my Dreamcast with the default version, which looks like this:

Code: Select all

static void *l_alloc (void *ud, void *ptr, size_t osize, size_t nsize) {
	(void)ud;
	(void)osize;

	if (nsize == 0) {
		free(ptr);
		return NULL;
	}
	else
	{
		return realloc(ptr, nsize);
	}
}
Nice, simple, and should work. But it doesn't. KOS locks up and stops responding somewhere inside realloc after a few allocations, and frankly I wouldn't even know where to start trying to debug that. The test program simply called lua_open, so it's not exactly doing anything demanding or difficult - just allocating a total of less than 2KB of memory.

I stuck some instrumenting code in there, and realised it wasn't reallocating memory that was causing the problem. It was allocating the memory in the first place. Swapping out the realloc with a malloc (and a memcpy in case it actually does want a realloc) made no difference whatsoever. Still locked up.

I ran the Lua library and a simple test program (PC compiled - runs all the non-interactive test programs supplied with Lua) through Valgrind to make sure there were no memory related problems in Lua itself. There were none - no leaks, no writing to unallocated memory, no overrunning allocated memory blocks, nothing.

So I replaced the allocator with a stupid one. It allocates a 1MB block of memory and carves it up very simply (basically a stack - allocations are handled by taking the next n bytes, reallocations are handled by memcpy, and frees are totally ignored). Now it works perfectly. I can't leave it like that, because it'll run out of memory very quickly, and when that happens it'll start overwriting whatever's sitting after the memory block (no bounds checking, because I'm not fixing up a hack that I shouldn't be needing anyway).

So, does anyone have any idea what could be causing this sort of problem?
User avatar
Quzar
Dream Coder
Dream Coder
Posts: 7497
Joined: Wed Jul 31, 2002 12:14 am
Location: Miami, FL
Has thanked: 4 times
Been thanked: 9 times
Contact:

Post by Quzar »

replace it with memalign instead of malloc? (i actually have no clue, before finishing reading your post my only suggestion would have been to replace realloc with your own copy of it =P)
"When you post fewer lines of text than your signature, consider not posting at all." - A Wise Man
Phantom
DC Developer
DC Developer
Posts: 1753
Joined: Thu Jan 16, 2003 4:01 am
Location: The Netherlands
Has thanked: 0
Been thanked: 0
Contact:

Post by Phantom »

I experienced weird lockups once and tracked it down to a bug in the newlib patch (this took literally hours to track down). Ofcourse it doesn't have to be related to your problem, but just in case, make sure that you're not using the newlib patch with the bug. The version in KOS SVN has been fixed (http://svn.allusion.net/svn/kos/kos/uti ... -kos.patch).

The bug causes too little memory to be reserved for locks in newlib. In my case one lock corrupted another which was next to it in memory, leading to a deadlock later on. If you still have the newlib patch you used to build your toolchain, you can check line 122 of the patch. If it contains _LOCK_T it's the buggy version (the bug is actually very obvious).

Ofcourse it's very possible that your problem is completely unrelated... In any case, good luck!
"Nothing works" - Catweazle
BlackAura
DC Developer
DC Developer
Posts: 9951
Joined: Sun Dec 30, 2001 9:02 am
Has thanked: 0
Been thanked: 1 time

Post by BlackAura »

Hmm... The toolchain I have is pretty old, so it's probable that my copy does have that problem. Whether that is actually causing the problem or not, I have no idea.

The KOS SVN repository appears to have disappeared.
Phantom
DC Developer
DC Developer
Posts: 1753
Joined: Thu Jan 16, 2003 4:01 am
Location: The Netherlands
Has thanked: 0
Been thanked: 0
Contact:

Post by Phantom »

BlackAura wrote:The KOS SVN repository appears to have disappeared.
Has it? The link I pasted above still seems to work. The repository moved a while ago from http://www.allusion.net to svn.allusion.net.
Last edited by Phantom on Sat May 06, 2006 7:39 pm, edited 1 time in total.
"Nothing works" - Catweazle
BlackAura
DC Developer
DC Developer
Posts: 9951
Joined: Sun Dec 30, 2001 9:02 am
Has thanked: 0
Been thanked: 1 time

Post by BlackAura »

Ah, I didn't notice that link there. I had the old URL to the repository still...
Phantom
DC Developer
DC Developer
Posts: 1753
Joined: Thu Jan 16, 2003 4:01 am
Location: The Netherlands
Has thanked: 0
Been thanked: 0
Contact:

Post by Phantom »

I editted my post so it probably wasn't there yet when you read it. ;)

Anyway, the patch problem is a major bug. Even a single-threaded program could deadlock in libc because of it, So even if it turns out that it's unrelated, it's still worth fixing.
"Nothing works" - Catweazle
BlackAura
DC Developer
DC Developer
Posts: 9951
Joined: Sun Dec 30, 2001 9:02 am
Has thanked: 0
Been thanked: 1 time

Post by BlackAura »

Well, that doesn't seem to fix it. I've rebuilt both newlib by itself, and the entire toolchain. Same problem - the entire system just locks up when it's allocated some memory.

Here is what I've got thus far (basically just a makefile and a simple test program). I'd appreciate it if someone could try it out, just to see if it's something weird happening at my end.
Phantom
DC Developer
DC Developer
Posts: 1753
Joined: Thu Jan 16, 2003 4:01 am
Location: The Netherlands
Has thanked: 0
Been thanked: 0
Contact:

Post by Phantom »

I ran it, doesn't work for me either. It hangs at the 5th alloc.
"Nothing works" - Catweazle
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has thanked: 0
Been thanked: 0
Contact:

Re: Problem with Lua 5.1

Post by showka »

Any developments on this? I literally experienced the same problem today and tracked it down to the realloc inside l_alloc, tried replacing it with a malloc, and it still din't work. This happens for me in the sixth call to l_alloc. The fifth calls free, not realloc, so that's five calls to realloc.

I was searching for "malloc" and found this topic. Amazingly, searching for "Lua 5.1" has never shown this topic before.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has thanked: 0
Been thanked: 0
Contact:

Re: Problem with Lua 5.1

Post by showka »

Here is my l_alloc function:

Code: Select all

static void *l_alloc (void *ud, void *ptr, size_t osize, size_t nsize) {
  (void)ud;
  (void)osize;
  char * dummyPtr;
  /* BAD TIMMY */
  ErrOutWriteLine("from Lua_lauxlib.c");
  ErrOutWrite("l_alloc osize=");
  ErrOutWriteNumber(osize); ErrOutWriteLine("");
  ErrOutWrite("l_alloc nsize=");
  ErrOutWriteNumber(nsize); ErrOutWriteLine("");
  
  ErrOutWriteLine("Dummy alloc:");
  dummyPtr = (char *) malloc(500);
  ErrOutWriteLine("Survived dummy alloc.");
  
  if (nsize == 0) {
    ErrOutWriteLine("Uh oh, it is zero!");
    free(ptr);
    ErrOutWriteLine("Never mind, we survive.");
    return NULL;
  }
  else
  {
    if (osize != 0)
    {
        ErrOutWriteLine("Going to call realloc...");
        void * returnMe = realloc(ptr, nsize);
        ErrOutWriteLine("Call succeeded.  Returning...");
        return returnMe;
    }
    else
    {
        ErrOutWriteLine("Going to call normal malloc.");
        void * returnMe2 = malloc(nsize);
        ErrOutWriteLine("Able to malloc!");
        void * stupid = malloc(500);
        ErrOutWriteLine("Survived second dummy alloc!");
        return returnMe2;
    }
  }
}
The lines "ErrOutWriteLine" are a trace.

I just added a few lines of code that call two dummy calls to malloc and some tracing in case it locked up around those lines.

What's odd is the fail happens on the same call to l_alloc, but not on the same alloc - it actually occurs on that first dummy alloc that gets 500 bytes ("dummyPtr = (char *) malloc(500);").

I was really hoping that the issue would be the size requested (crazy as that would be) or something else, but it seems like somehow the allocator is getting into a bad state before that call is made, and has nothing to do with the size or numbers of allocations.

Has anyone else tried to use Lua 5.1? Apparently it compiles and runs fine with everything.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has thanked: 0
Been thanked: 0
Contact:

Re: Problem with Lua 5.1

Post by showka »

I figured out what it was.

The hang is occuring when lua calls l_alloc() with osize = 0 and nsize = 0.

Because nsize = 0, it seems to just be asking you to free the memory. But if you call free on it and osize = 0, the DC hangs.

After I changed the code to avoid calling free if osize = 0, the transfer window went nuts with the thousands of tracing statements I'd tainted Lua's beautiful code with. Because of all the junk I did to Lua I can't confirm if Lua 5.1 is working yet on the DC, but at least this problem is gone!

New code (sans tracing calls):

Code: Select all

static void *l_alloc (void *ud, void *ptr, size_t osize, size_t nsize) {
  (void)ud;
  (void)osize;
  
  if (nsize == 0) {
    if (osize != 0) // bug in KOS?
    {
        free(ptr);
    }
    return NULL;
  }
  else
  {
    if (osize != 0)
    {
        void * returnMe = realloc(ptr, nsize);
        return returnMe;
    }
    else
    {
        void * returnMe2 = malloc(nsize);
        return returnMe2;
    }
  }
}
EDIT - I just cleared out all those traces in the Lua code and all of the stuff I was working on in my little Lua interpretor on the PC is running beautifully. This is pretty cool.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has thanked: 0
Been thanked: 0
Contact:

Re: Problem with Lua 5.1

Post by showka »

I thought I'd write a tutorial on setting up Lua 5.1 with the DC.

Unfortunately as soon as I finished, I realized luaload() is crashing whenever the script has an error, making it almost worthless.
BlackAura
DC Developer
DC Developer
Posts: 9951
Joined: Sun Dec 30, 2001 9:02 am
Has thanked: 0
Been thanked: 1 time

Re: Problem with Lua 5.1

Post by BlackAura »

I did eventually get Lua running, but only by replacing the memory allocator with a custom one, which seemed to work OK.

Nice find though. Those changes are far simpler.

I can't recall if I had error handling working correctly or not. I don't think I ever checked any scripts with errors in them.

Unless it's crashing inside the error handler, the only other problem could be the use of setjmp, which Lua uses for error handling. I don't know if setjmp works in KOS. It's possible that a crash happens when Lua calls the longjmp function. Have a look at the LUAI_THROW macro in luaconf.h.

There's an option to use C++ exceptions instead, provided you compile Lua as C++ code. I believe that C++ exceptions do work correctly, so that might help. Or it might not.
User avatar
showka
DCEmu Freak
DCEmu Freak
Posts: 95
Joined: Fri Nov 23, 2001 12:01 pm
Location: Border Town
Has thanked: 0
Been thanked: 0
Contact:

Re: Problem with Lua 5.1

Post by showka »

You sir were completely right about LUAI_THROW.

I now have Lua handling errors which is very nice indeed.

But what I found was that "LUAI_THROW" didn't work... I think.

Here's what I found, in luaconf.h:

Code: Select all

/*
@@ LUAI_THROW/LUAI_TRY define how Lua does exception handling.
** CHANGE them if you prefer to use longjmp/setjmp even with C++
** or if want/don't to use _longjmp/_setjmp instead of regular
** longjmp/setjmp. By default, Lua handles errors with exceptions when
** compiling as C++ code, with _longjmp/_setjmp when asked to use them,
** and with longjmp/setjmp otherwise.
*/
#if defined(__cplusplus)
/* C++ exceptions */
#define LUAI_THROW(L,c)	throw(c)
#define LUAI_TRY(L,c,a)	try { a } catch(...) \
	{ if ((c)->status == 0) (c)->status = -1; }
#define luai_jmpbuf	int  /* dummy variable */

#elif defined(LUA_USE_ULONGJMP)
/* in Unix, try _longjmp/_setjmp (more efficient) */
#define LUAI_THROW(L,c)	_longjmp((c)->b, 1)
#define LUAI_TRY(L,c,a)	if (_setjmp((c)->b) == 0) { a }
#define luai_jmpbuf	jmp_buf

#else
/* default handling with long jumps */
#define LUAI_THROW(L,c)	longjmp((c)->b, 1)
#define LUAI_TRY(L,c,a)	if (setjmp((c)->b) == 0) { a }
#define luai_jmpbuf	jmp_buf

#endif
I simply changed "#if defined(__cplusplus)" to "#if defined(DONOTWANT__cplusplus)". This seems to have made it work for me.
User avatar
GyroVorbis
Elysian Shadows Developer
Elysian Shadows Developer
Posts: 1873
Joined: Mon Mar 22, 2004 4:55 pm
Location: #%^&*!!!11one Super Sonic
Has thanked: 79 times
Been thanked: 61 times
Contact:

Re: Problem with Lua 5.1

Post by GyroVorbis »

Are there any more problems with Lua 5.1 on DC? I'm looking to upgrade.
BlackAura
DC Developer
DC Developer
Posts: 9951
Joined: Sun Dec 30, 2001 9:02 am
Has thanked: 0
Been thanked: 1 time

Re: Problem with Lua 5.1

Post by BlackAura »

Now? No, works fine. You just need to make the modification to the memory allocator detailed above. If you're compiling as C++, you'll also need to disable the use of C++ exceptions, and make it use setjmp / longjmp instead (again, as above).
User avatar
emptythought
DC Developer
DC Developer
Posts: 2015
Joined: Wed Jan 30, 2002 9:14 am
Location: UNITED STATES NRN
Has thanked: 0
Been thanked: 0
Contact:

Re: Problem with Lua 5.1

Post by emptythought »

BlackAura, do you have a makefile for this version of Lua?
BlackAura
DC Developer
DC Developer
Posts: 9951
Joined: Sun Dec 30, 2001 9:02 am
Has thanked: 0
Been thanked: 1 time

Re: Problem with Lua 5.1

Post by BlackAura »

There's a Makefile in the .tar.bz2 file I posted a up there somewhere, but it's not too interesting. I think it was based on one of the example Makefiles anyway
User avatar
emptythought
DC Developer
DC Developer
Posts: 2015
Joined: Wed Jan 30, 2002 9:14 am
Location: UNITED STATES NRN
Has thanked: 0
Been thanked: 0
Contact:

Re: Problem with Lua 5.1

Post by emptythought »

Sorry, I missed that file. Thanks.
Post Reply