libfatfs - FAT16 & FAT32 for SD Cards

[BB Hood]

A library that supports reading and writing to SD that use FAT16 or FAT32 for the Sega Dreamcast. I know that due to *patent* reasons this can not be included in KallistiOS so I posted it here. If posting it is also bad feel free to delete this thread. Use at your own risk :p

https://github.com/andressbarajas/libfat2fs

[BlueCrab]

It's not copyright that is the problem, it's patents.

[SWAT]

Good job, but I have problems at mounting filesystem:

Code: Select all

FAT2FS Boot Sector: 

Boot JMP: eb:0:90
OEM Name:         
Bytes Per Sector: 512
Sectors Per Cluster: 64
Reserved Sector Count: 6288
Table Count: 2
Root Entry Count: 0
Total Sectors 16: 0
Media Type: f8
Table Size 16: 0
Sectors Per Track: 63
Head Side Count: 128
Hidden Sector Count: 8192
Total Sectors 32: 7803904
Table Size 32(Fat32 Only): 952
Starting Root Cluster(Fat32 Only): 2


Number of sectors the FAT table takes up: 952
Root directory number of sectors: 0
Root directory sector location: 8192
File allocation table sector location: 6288
File/folder data starts at sector: 8192
Total number of data sectors: 7795712
Total number of clusters: 121808

FileName: DCIM ShortName: DCIM Parent: SD  Attr: 10 Cluster: 3
...
FileName: GAMES ShortName: GAMES    Parent: SD  Attr: 10 Cluster: 25691 
Requested sbrk_base 0x8cff0000, was 0x8cfef000, diff 4096
kernel panic: out of memory; about to run over kernel stack

File list is very big, why do you parsing it all at once?
If I compile it without debug, i have only "Requested sbrk_base" error, but this not matter, DC freezes on mounting.

[BB Hood]

I parse it all at once to create a Directory tree where the nodes are either files or folders. This makes it faster to access files and folders later on when you read and write.

I did not stress test the function that parses a sd card will many sub directories. The function that parses directories uses recursion when a subdirectory is found. Sorry about that.

I did a quick fix. If the fix doesn't work then I probably need to redo that parse directory function or my whole approach to the way the library should work.

I found another bug thanks to your post, though. Fixed it too.

Git Pull the library and tell me if the fix worked for you.

Thanks for feedback on my library.

[BlueCrab]

I'd seriously suggest not trying to cache the entire directory structure of the device in RAM. In libkosext2fs, I don't actually do any cacheing at all of the directory structure, and it works well enough. The same goes for pretty much every other FS in KOS proper. It's an unnecessary overhead in RAM to do that. I'd at least make it a compile-time option whether or not to cache the directory, if nothing else.

Also, you could always make the cache work something like the data cache on a CPU, where entries that haven't been recently used get kicked out. That requires a bit more work, but would probably work a whole lot better for what MOST people would be using the FS for. In that approach, limit your cache to a sane size and go with it from there. Kick out the oldest entry when you need to put something else in. You could look at the inode cache in libkosext2fs if you'd like inspiration of how you might do that. The inode management code is a bit scary to read through (especially the part that walks directory trees), but the inode cache should be easy enough to understand.

Oh, and this is totally unrelated to the problem at hand, but why name it libfat2fs? I could understand libfatfs or some derivative, but libfat2fs kinda makes me think you're implementing some new filesystem I've never heard of called "fat2".

[BB Hood]

You make a lot of good points. I took the route that wasn't memory efficient :^/ but easy for me to work with and understand. It will take time but I will work your ideas in because they make absolute sense.

Libfat2fs because I thought your lib worked for ext filesystem. Didnt know that there is actually a ext2 (Im a windows user). Gonna change that too.

Thanks

[BlueCrab]

Libfat2fs because I thought your lib worked for ext filesystem. Didnt know that there is actually a ext2 (Im a windows user). Gonna change that too.

Nope. The "ext" filesystem has been dead for a long time and nobody has probably actively used it since like 1993. For reference the Extended File System "ext" was added to Linux in version 0.96c in April of 1992 while the Second Extended File System "ext2" was added in 0.99 in January of 1993. I had kinda guessed that the reason for your library's name might have been something like what you described.

What I actually implemented in libkosext2fs is most of ext2, including enough to get many ext3 filesystems to mount and work as well (although, I certainly don't implement the journal in ext3). It'd probably even work with some devices formatted as ext4, but I've never tried to use any of them.

You make a lot of good points. I took the route that wasn't memory efficient :^/ but easy for me to work with and understand. It will take time but I will work your ideas in because they make absolute sense.

Trial and error. You try something, see if it works, and improve from there. Rome wasn't built in a day, as they say. Heck, in libkosext2fs, I tried a few different ways of cacheing things before I decided on what I ended up implementing.

[SWAT]

I parse it all at once to create a Directory tree where the nodes are either files or folders. This makes it faster to access files and folders later on when you read and write.

I did not stress test the function that parses a sd card will many sub directories. The function that parses directories uses recursion when a subdirectory is found. Sorry about that.

I did a quick fix. If the fix doesn't work then I probably need to redo that parse directory function or my whole approach to the way the library should work.

I found another bug thanks to your post, though. Fixed it too.

Git Pull the library and tell me if the fix worked for you.

Thanks for feedback on my library.

Thanks for improvements but this not help in my case.

[BB Hood]

Okay I updated the library.

New link: https://github.com/andressbarajas/libfatfs

Now (by default) nothing is cached, so the amount of memory used by the library is small.

If you want it to be cached (like before this update), add -DFATFS_CACHEALL to KOS_CFLAGS in Makefile.

Let me know if it works now

Also, I haven't tested this new update on a FAT16 formatted SD card. My 2gb sd card cracked and broke

Use at your own risk

[SWAT]

Now it works fine! Good job.
But I would not recommend anyone to use the cache of directory tree.
In my experience, the most useful gives "cluster link map table" for an every opened file (not very small). You can see this feature (fast seek) in the library fatfs by ChaN.
It's really improving performance and memory is spent not so much.
Although perhaps you already use it.

[BB Hood]

Thanks. Yea, I use some form of a "cluster link map table". When a file/folder is opened, I create a linked list of clusters_nodes. Each node holds an int that corresponds to a cluster associated with that file/folder. No further FAT access is needed except when another cluster needs to be allocated(e.g. writting) or freed(e.g. trunc).

[SWAT]

Then everything is OK. Thanks.

[SWAT]

Why do you convert all names to upper case? It is very inconvenient
Noticed the error. When I open a folder (and possibly the file), I wrote the name in lower case, the error does not occur when opening but the result is undefined.

[BB Hood]

Fat16 and Fat32 are case insensitive filesystems so I decided to convert all file/folders to upper case.

For example,

Code: Select all

open("/sd/hello.c", O_RDWR);

will open the file hello.c in the /sd directory. hello.c could be an actual file named: HELLO.C or HeLLo.c or Hello.C. I hope this explanation is clear.

Can you post your code and the name(preserving case) of the file/folder that is not working for you?

[SWAT]

In readdir entry I would like to see the original naming rather than all in upper case. This is my personal opinion

In first case, fs_open return OK, but fs_readdir return a memory trash:

Code: Select all

Reading /sd/DS
       27749 bytes BOOT_LOADER.BIN
      450328 bytes MAIN_LOADER.BIN
         358 bytes NET.CFG
       22956 bytes SLINKIE.BIN
       17651 bytes SRLDR.BIN
      109796 bytes U-BOOT.BIN
     5123545 bytes UIMAGE
       22256 bytes IPLDR.BIN
           0 bytes UPDATE
           0 bytes APPS
           0 bytes CMDS
           0 bytes DOC
           0 bytes FIRMWARE
           0 bytes FONTS
           0 bytes GUI
           0 bytes LUA
           0 bytes MODULES
           0 bytes SHARED
     1774928 bytes DS_CORE.BIN

Reading /sd/DS/lua
  1825648368 bytes L2Ä?ís.N¡Q
   798279681 bytes âõáîkÄQ.°
   826400556 bytes ápüŸGÍR.ß
   -18659055 bytes ³.óÿ..îB
  1070800464 bytes bÐÑ2
2Ò.<ñA
   301978401 bytes Þ3¡?_üôü.÷.

It's very simple code:

Code: Select all

	dirent_t *ent;
	char dir[MAX_FN_LEN];
	getcwd(dir, MAX_FN_LEN); 
	file_t hnd = fs_open(dir, O_RDONLY | O_DIR);
	
	if (hnd < 0) {
		printf("Error opening %s\n", dir);
		return -1;
	}

	while ((ent = fs_readdir(hnd)) != NULL) {
		printf("%12d bytes %s\n", ent->size, ent->name);
	}
	fs_close(hnd);

The "lua" directory name originally written in lower case, the DS in upper case.

In second case, I try:

Code: Select all

fs_open("/sd/DS/lua/startup.lua", O_RDONLY);

.. and this return me -1

All this is accompanied by a rather long pauses in the process.

[BB Hood]

In readdir entry I would like to see the original naming rather than all in upper case. This is my personal opinion

Very good point. I updated libfatfs to now output original naming.

As for the bug, it looks like it is parsing directories incorrectly and it hard for me to duplicate unless I have your SD card
The library incorrectly parses the /sd/DS/lua directory. That's why you get a -1 for /sd/DS/lua/startup.lua because it cant find it.

Maybe this update might fix it but I doubt it. Sorry for bugs

[SWAT]

Thanks for the update, now all the names are original.
But unfortunately, still does not work. If you do not know what the reason, I have to go back to using another library

[BB Hood]

I can probably figure it out with some debug output that I added. I hate to use people to debug stuff but if you have time, update the library and run again (no need to use -DFATFS_DEBUG) and post output.

Do you know a way to some how make an image copy of the SD card to send to me?

What other library? Is it publicly available?

Again sorry for bugs and thanks for using my library.

[SWAT]

After the previous update, it all hangs when opening a file (not directory), so I can not provide the debug information.
I'm using FatFs by ChaN http://elm-chan.org/fsw/ff/00index_e.html with simple custom layer for KOS vfs. It works well and fast (faster than your for now), but I just wanted to use KOS standard for all file systems.

[BB Hood]

Alright Well I will keep trying to figure out the bugs and use that library you provided as a reference to make mines faster. Use FatFs by ChaN in the mean time.