Problem with Lua 5.1

BlackAura · **Joined:** Sun Dec 30, 2001 8:02 am **Posts:** 9911

I've been trying to get Lua 5.1 working on the Dreamcast for a while (about an hour). As of right now, it seems to be working correctly, except for one problem - I had to bypass the KOS built-in memory allocator to get the thing to work. Otherwise, it runs perfectly with absolutely no modifications.

All of Lua's memory allocation (since 5.1) goes through a single function, the default version of which is in lauxlib.c. For some unexplained reason, it simply hard locks my Dreamcast with the default version, which looks like this:

Code:

static void *l_alloc (void *ud, void *ptr, size_t osize, size_t nsize) {
   (void)ud;
   (void)osize;

   if (nsize == 0) {
      free(ptr);
      return NULL;
   }
   else
   {
      return realloc(ptr, nsize);
   }
}

Nice, simple, and should work. But it doesn't. KOS locks up and stops responding somewhere inside realloc after a few allocations, and frankly I wouldn't even know where to start trying to debug that. The test program simply called lua_open, so it's not exactly doing anything demanding or difficult - just allocating a total of less than 2KB of memory.

I stuck some instrumenting code in there, and realised it wasn't reallocating memory that was causing the problem. It was allocating the memory in the first place. Swapping out the realloc with a malloc (and a memcpy in case it actually does want a realloc) made no difference whatsoever. Still locked up.

I ran the Lua library and a simple test program (PC compiled - runs all the non-interactive test programs supplied with Lua) through Valgrind to make sure there were no memory related problems in Lua itself. There were none - no leaks, no writing to unallocated memory, no overrunning allocated memory blocks, nothing.

So I replaced the allocator with a stupid one. It allocates a 1MB block of memory and carves it up very simply (basically a stack - allocations are handled by taking the next n bytes, reallocations are handled by memcpy, and frees are totally ignored). Now it works perfectly. I can't leave it like that, because it'll run out of memory very quickly, and when that happens it'll start overwriting whatever's sitting after the memory block (no bounds checking, because I'm not fixing up a hack that I shouldn't be needing anyway).

So, does anyone have any idea what could be causing this sort of problem?

Quzar · **Posted:** Sat May 06, 2006 2:32 pm

replace it with memalign instead of malloc? (i actually have no clue, before finishing reading your post my only suggestion would have been to replace realloc with your own copy of it =P)

Phantom · **Posted:** Sat May 06, 2006 5:04 pm

I experienced weird lockups once and tracked it down to a bug in the newlib patch (this took literally hours to track down). Ofcourse it doesn't have to be related to your problem, but just in case, make sure that you're not using the newlib patch with the bug. The version in KOS SVN has been fixed (http://svn.allusion.net/svn/kos/kos/utils/newlib/newlib-1.12.0-kos.patch).

The bug causes too little memory to be reserved for locks in newlib. In my case one lock corrupted another which was next to it in memory, leading to a deadlock later on. If you still have the newlib patch you used to build your toolchain, you can check line 122 of the patch. If it contains _LOCK_T it's the buggy version (the bug is actually very obvious).

Ofcourse it's very possible that your problem is completely unrelated... In any case, good luck!

BlackAura · **Joined:** Sun Dec 30, 2001 8:02 am **Posts:** 9911

Hmm... The toolchain I have is pretty old, so it's probable that my copy does have that problem. Whether that is actually causing the problem or not, I have no idea.

The KOS SVN repository appears to have disappeared.

Phantom · **Posted:** Sat May 06, 2006 5:35 pm

BlackAura wrote:

The KOS SVN repository appears to have disappeared.

Has it? The link I pasted above still seems to work. The repository moved a while ago from http://www.allusion.net to svn.allusion.net.

BlackAura · **Joined:** Sun Dec 30, 2001 8:02 am **Posts:** 9911

Ah, I didn't notice that link there. I had the old URL to the repository still...

Phantom · **Posted:** Sat May 06, 2006 5:50 pm

I editted my post so it probably wasn't there yet when you read it.

Anyway, the patch problem is a major bug. Even a single-threaded program could deadlock in libc because of it, So even if it turns out that it's unrelated, it's still worth fixing.

BlackAura · **Joined:** Sun Dec 30, 2001 8:02 am **Posts:** 9911

Well, that doesn't seem to fix it. I've rebuilt both newlib by itself, and the entire toolchain. Same problem - the entire system just locks up when it's allocated some memory.

Here is what I've got thus far (basically just a makefile and a simple test program). I'd appreciate it if someone could try it out, just to see if it's something weird happening at my end.

Phantom · **Posted:** Sun May 07, 2006 2:17 am

I ran it, doesn't work for me either. It hangs at the 5th alloc.

showka · **Posted:** Sat Sep 29, 2007 8:30 am

Any developments on this? I literally experienced the same problem today and tracked it down to the realloc inside l_alloc, tried replacing it with a malloc, and it still din't work. This happens for me in the sixth call to l_alloc. The fifth calls free, not realloc, so that's five calls to realloc.

I was searching for "malloc" and found this topic. Amazingly, searching for "Lua 5.1" has never shown this topic before.

showka · **Posted:** Sat Sep 29, 2007 9:06 am

Here is my l_alloc function:

Code:

static void *l_alloc (void *ud, void *ptr, size_t osize, size_t nsize) {
  (void)ud;
  (void)osize;
  char * dummyPtr;
  /* BAD TIMMY */
  ErrOutWriteLine("from Lua_lauxlib.c");
  ErrOutWrite("l_alloc osize=");
  ErrOutWriteNumber(osize); ErrOutWriteLine("");
  ErrOutWrite("l_alloc nsize=");
  ErrOutWriteNumber(nsize); ErrOutWriteLine("");
  
  ErrOutWriteLine("Dummy alloc:");
  dummyPtr = (char *) malloc(500);
  ErrOutWriteLine("Survived dummy alloc.");
  
  if (nsize == 0) {
    ErrOutWriteLine("Uh oh, it is zero!");
    free(ptr);
    ErrOutWriteLine("Never mind, we survive.");
    return NULL;
  }
  else
  {
    if (osize != 0)
    {
        ErrOutWriteLine("Going to call realloc...");
        void * returnMe = realloc(ptr, nsize);
        ErrOutWriteLine("Call succeeded.  Returning...");
        return returnMe;
    }
    else
    {
        ErrOutWriteLine("Going to call normal malloc.");
        void * returnMe2 = malloc(nsize);
        ErrOutWriteLine("Able to malloc!");
        void * stupid = malloc(500);
        ErrOutWriteLine("Survived second dummy alloc!");
        return returnMe2;
    }
  }
}

The lines "ErrOutWriteLine" are a trace.

I just added a few lines of code that call two dummy calls to malloc and some tracing in case it locked up around those lines.

What's odd is the fail happens on the same call to l_alloc, but not on the same alloc - it actually occurs on that first dummy alloc that gets 500 bytes ("dummyPtr = (char *) malloc(500);").

I was really hoping that the issue would be the size requested (crazy as that would be) or something else, but it seems like somehow the allocator is getting into a bad state before that call is made, and has nothing to do with the size or numbers of allocations.

Has anyone else tried to use Lua 5.1? Apparently it compiles and runs fine with everything.

showka · **Posted:** Sat Sep 29, 2007 11:08 am

I figured out what it was.

The hang is occuring when lua calls l_alloc() with osize = 0 and nsize = 0.

Because nsize = 0, it seems to just be asking you to free the memory. But if you call free on it and osize = 0, the DC hangs.

After I changed the code to avoid calling free if osize = 0, the transfer window went nuts with the thousands of tracing statements I'd tainted Lua's beautiful code with. Because of all the junk I did to Lua I can't confirm if Lua 5.1 is working yet on the DC, but at least this problem is gone!

New code (sans tracing calls):

Code:

static void *l_alloc (void *ud, void *ptr, size_t osize, size_t nsize) {
  (void)ud;
  (void)osize;
  
  if (nsize == 0) {
    if (osize != 0) // bug in KOS?
    {
        free(ptr);
    }
    return NULL;
  }
  else
  {
    if (osize != 0)
    {
        void * returnMe = realloc(ptr, nsize);
        return returnMe;
    }
    else
    {
        void * returnMe2 = malloc(nsize);
        return returnMe2;
    }
  }
}

EDIT - I just cleared out all those traces in the Lua code and all of the stuff I was working on in my little Lua interpretor on the PC is running beautifully. This is pretty cool.

showka · **Posted:** Sat Sep 29, 2007 4:48 pm

I thought I'd write a tutorial on setting up Lua 5.1 with the DC.

Unfortunately as soon as I finished, I realized luaload() is crashing whenever the script has an error, making it almost worthless.

BlackAura · **Joined:** Sun Dec 30, 2001 8:02 am **Posts:** 9911

I did eventually get Lua running, but only by replacing the memory allocator with a custom one, which seemed to work OK.

Nice find though. Those changes are far simpler.

I can't recall if I had error handling working correctly or not. I don't think I ever checked any scripts with errors in them.

Unless it's crashing inside the error handler, the only other problem could be the use of setjmp, which Lua uses for error handling. I don't know if setjmp works in KOS. It's possible that a crash happens when Lua calls the longjmp function. Have a look at the LUAI_THROW macro in luaconf.h.

There's an option to use C++ exceptions instead, provided you compile Lua as C++ code. I believe that C++ exceptions do work correctly, so that might help. Or it might not.

showka · **Posted:** Wed Oct 24, 2007 11:43 pm

You sir were completely right about LUAI_THROW.

I now have Lua handling errors which is very nice indeed.

But what I found was that "LUAI_THROW" didn't work... I think.

Here's what I found, in luaconf.h:

Code:

/*
@@ LUAI_THROW/LUAI_TRY define how Lua does exception handling.
** CHANGE them if you prefer to use longjmp/setjmp even with C++
** or if want/don't to use _longjmp/_setjmp instead of regular
** longjmp/setjmp. By default, Lua handles errors with exceptions when
** compiling as C++ code, with _longjmp/_setjmp when asked to use them,
** and with longjmp/setjmp otherwise.
*/
#if defined(__cplusplus)
/* C++ exceptions */
#define LUAI_THROW(L,c)   throw(c)
#define LUAI_TRY(L,c,a)   try { a } catch(...) \
   { if ((c)->status == 0) (c)->status = -1; }
#define luai_jmpbuf   int  /* dummy variable */

#elif defined(LUA_USE_ULONGJMP)
/* in Unix, try _longjmp/_setjmp (more efficient) */
#define LUAI_THROW(L,c)   _longjmp((c)->b, 1)
#define LUAI_TRY(L,c,a)   if (_setjmp((c)->b) == 0) { a }
#define luai_jmpbuf   jmp_buf

#else
/* default handling with long jumps */
#define LUAI_THROW(L,c)   longjmp((c)->b, 1)
#define LUAI_TRY(L,c,a)   if (setjmp((c)->b) == 0) { a }
#define luai_jmpbuf   jmp_buf

#endif

I simply changed "#if defined(__cplusplus)" to "#if defined(DONOTWANT__cplusplus)". This seems to have made it work for me.

GyroVorbis · **Posted:** Mon Nov 26, 2007 9:31 pm

Are there any more problems with Lua 5.1 on DC? I'm looking to upgrade.

BlackAura · **Joined:** Sun Dec 30, 2001 8:02 am **Posts:** 9911

Now? No, works fine. You just need to make the modification to the memory allocator detailed above. If you're compiling as C++, you'll also need to disable the use of C++ exceptions, and make it use setjmp / longjmp instead (again, as above).

emptythought · **Posted:** Fri Mar 07, 2008 7:56 pm

BlackAura, do you have a makefile for this version of Lua?

BlackAura · **Joined:** Sun Dec 30, 2001 8:02 am **Posts:** 9911

There's a Makefile in the .tar.bz2 file I posted a up there somewhere, but it's not too interesting. I think it was based on one of the example Makefiles anyway

emptythought · **Posted:** Sat Mar 08, 2008 3:34 am

Sorry, I missed that file. Thanks.

DCEmulation

Problem with Lua 5.1

Who is online