"Why Unionfs is better than Aufs"

My most recent kernel build, with the 2.6.33.1 kernel, uses Unionfs 2.5.4. This was in response to issues discussed in this link, and Jemimah's successful move to Unionfs:

http://bkhome.org/archive/blog2/201003/26331-kernel-unionfs.html

I posted a message to the Aufs mail-list, entitled "Why Unionfs is better than Aufs", reproduced below. The comments contain the responses on the mail-list. They are very interesting responses!

Original post by me:

---

Comments

First mail-list response
Username: BarryK
This response is from J. R. Okajima, the founder and main developer of Aufs: [code]Hello Barry, Barry Kauler: > Now, it may be that these two things can be done in Aufs, but I don't know how: > > First reason ::: > If we have a "save partition" on top, we also have the SFS files in > that partition. Aufs will refuse to mount those SFS files as they are > already inside a branch. Consequently, we are forced to copy the main > SFS file into RAM and mount it from there. However, Unionfs is quite > happy with mounting SFS files that are actually present in another > branch. Because I had experienced a recursive lock (deadlock) by a loopback branch (fs image in a branch fs), aufs rejects it. But it was a long time ago. It was early stage in aufs1 era. I will check it again with recent kernel. > Second reason > Both Unionfs and Aufs have mechanisms for detecting direct writes to > branches, however they are not perfect. Unless, in the case of Aufs > you go for the mode using inotify, but that has performance penalty. udba=inotify has some overhead as you wrote. Is it so big penalty for you? CPU time or memory? If you show me some numbers about the penalty, I may re-consider the approach. > When running Puppy from Flash memory, we have the top branch as a > tmpfs in RAM, and the "save file" or "save partition" below that. This > is done to minimise writes to the Flash memory. When a package is > installed, we would like to be able to install it directly to the > "save" layer, the actual Flash memory and have all the files visible > on top. RESPONSE CONTINUED...[/code]

First response, continued
Username: BarryK
"The Comment form has a limit of 2000 characters (I must fix that!) so here is the rest of the first response: [code]To install a package, modifying a branch permission temporarly may be useful. For instance, when you have three branches, - /u = /tmpfs(rw) + /flash=ro(ro) + /squashfs(ro) - modify /u = /tmpfs(ro) + /flash=ro(rw) + /squashfs(ro) - install a package to /u _if_ the same named file doesn't exist the first /tmpfs, aufs will pass the files to the second /flash. But it may not be a generic solution. I didn't test it actually either. You may prefer bypassing aufs. > Our package installer recognises if Unionfs is being used and is able > to write direct to the "save" layer, and then is able to do a perfect > operation to make all files visible on top by doing this: > > mount -t unionfs -o remount,incgen unionfs / > > ...this flushes the caches and does a complete re-evaluation of the > layers. As far as I know, Aufs does not have the equivalent. Do you mean that the newly added files in the second layer is invisile in aufs? Usually, "mount -o remount /your/aufs" is enough for such case. Of course, if the same named file already exist in the upper layer, you cannot see the new files in the lower. Or RDU (readdir in userspace) in libau.so and rdcache=N mount option may be helpful. The 'incgen' option is easy to implement, but I don't think it meaningful. Because - simple remount is enough for most cases, but files in use will not be refreshed. - with udba=inotify, you can skip the manual remount and the files in use will be refreshed if necessary. when an inotify event is fired and aufs receives it, aufs executes something like 'decgen' for that file internally. So that the next access to the file causes the refresh. RESPONSE CONTINUES...[/code]

First response, continued
Username: BarryK
"The end of it: [code]Setting udba=inotify temporally is also recommended. # mount -o remount,udba=inotify / # install something to branch, bypassing aufs # mount -o remount,udba=reval / Reading the url you wrote, - Did you experience the corrupted filesystem after unmounting aufs? If so, let me know the detail please, particulary how you unmount it (or remount it read-only). > Posted on 18 Mar 2010, 15:33 by BarryK > Writing to branches ::: > I did actually try and explain this on the Aufs mail list once, but perhaps they never understood > what I was asking for, or didn't appreciate the value of it. I may not receive the mail. When did you post it? Did I reply it? Is it archived in the sourceforge ML? If you still have a copy, will you send it again? Thanx J. R. Okajima [/code]

Second response
Username: BarryK
"[code]> > First reason > ::: > > If we have a "save partition" on top, we also have the SFS files in > > that partition. Aufs will refuse to mount those SFS files as they are > > already inside a branch. Consequently, we are forced to copy the main > > SFS file into RAM and mount it from there. However, Unionfs is quite > > happy with mounting SFS files that are actually present in another > > branch. > > Because I had experienced a recursive lock (deadlock) by a loopback > branch (fs image in a branch fs), aufs rejects it. > But it was a long time ago. It was early stage in aufs1 era. > I will check it again with recent kernel. It may work. At least, my simple tests are passed. Here is a patch for you. Test it well, if you want. J. R. Okajima diff --git a/fs/aufs/loop.c b/fs/aufs/loop.c index 5fceec7..2a78a8d 100644 --- a/fs/aufs/loop.c +++ b/fs/aufs/loop.c @@ -32,6 +32,7 @@ int au_test_loopback_overlap(struct super_block *sb, struct dentry *h_d1, struct inode *h_inode; struct loop_device *l; + return 0; h_inode = h_d1->d_inode; if (MAJOR(h_inode->i_sb->s_dev) != LOOP_MAJOR) return 0;[/code]

Third response
Username: BarryK
"This is a response from me: [code]Thank you very much for your very detailed response. This is the major difference from the Unionfs mail-list, where there is often no response. The idea of temporarily remounting with 'udba=inotify' looks like a great idea! I'll try it. I will also try the patch to mount a Squashfs file that is present in another branch. This is great, thanks again for your responsiveness. Note, regarding my previous attempt to explain one or more of these issues, that was a long time ago, and I don't recall the details -- probably my explanation back then was too vague. Regarding f.s. corruption, it is another Puppy Linux developer (Jemimah) who was having trouble. I did have a one-off problem but it didn't repeat so I don't really know what that problem was. I'll do another kernel compile (with the patch applied to Aufs) and will see if I can get any f.s. errors. One interesting point about the 2.6.33 kernel is that they have dropped the ext2 and ext3 drivers as the new ext4 driver handles those also. -- probably has nothing to do with this topic, just an interesting item of news.[/code]

scrolling box
Username: ICPUG
"Is it just me but these scrolling boxes do not display properly in my browser (IE6 enforced by employer). Tried copying contents to a Word processor, which helped but still left bits missing.

copydown errors
Username: Jemimah
"I'm having some trouble with I/O errors during shutdown when it's copying files from RAM to the save file. I haven't investigated thoroughly yet, but I thought I'd ask if you had any ideas. Are there any changes you made to the snapmerge script to accommodate unionfs? Also I've had a couple problems with whiteout files in unionfs, so I'm really sort of hoping to go back to AUFS if possible.

snapmergepuppy
Username: BarryK
"snapmergepuppy has been designed to work with both Aufs and Unionfs. In fact it was initially used with Unionfs, and it was example copydown code provided by the Unionfs project that was utilised in creating the first snapmergepuppy. I haven't worked on that script for quite some time. Yes, problems with whiteout files is the main reason why people moved to Aufs, en masse. Very few people are still using Unionfs. Unionfs also had intervals of instability, in which the developers were unresponsive to bug reports -- my personal experience and one reason for one of my earlier moves to Aufs. I am seriously thinking of going back to Aufs, as J. R. Okajima has answered my questions and shown me that Aufs can do the special things I want. I am also thinking of using the 2.6.32.x kernel, as I'm very suspicious of that ext4 driver taking over responsibility for ext2 and ext3.

2.6.32.x
Username: kirk
"Barry, I think I may be having problems with ext3 as well. When you compiled 2.6.33.1, in your .config did you have ext2=n ext3=n ext4=y and CONFIG_EXT4_USE_FOR_EXT23=y? I had ext2 and ext3 set to yes, so I think that's still using the old driver. I was thinking of reconfiguring with ext4 driver for ext2/3. But If you have tried it I won't. See this for more info: http://cateee.net/lkddb/web-lkddb/EXT4_USE_FOR_EXT23.html Phoronix has a article that seems to indicate problems with ext3, but they don't say how they configured their kernel. http://www.phoronix.com/scan.php?page=article&item=linux_2624_2633&num=2

2.6.32.10
Username: kirk
"I switched my save file to ext3 in my 64bit build when I changed from 2.6.33 to 2.6.33.1. I also enabled NFS exports with aufs. Then I started having intermittent boot problems. Every once in a while it would hang near the end of rc.sysint. So I thought maybe it was the NFS aufs option. I ended up recompiling 2.6.33.1 three times, all with the same result. So I just compiled 2.6.32.10. So far so good. No nouveau, but Nvidia supports their stuff pretty well.

2.6.33.1 ext2/3/4
Username: BarryK
"Kirk, I configured like this: [code]CONFIG_EXT2_FS=y CONFIG_EXT2_FS_XATTR=y # CONFIG_EXT2_FS_POSIX_ACL is not set # CONFIG_EXT2_FS_SECURITY is not set CONFIG_EXT2_FS_XIP=y CONFIG_EXT3_FS=y # CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set CONFIG_EXT3_FS_XATTR=y # CONFIG_EXT3_FS_POSIX_ACL is not set # CONFIG_EXT3_FS_SECURITY is not set CONFIG_EXT4_FS=y CONFIG_EXT4_FS_XATTR=y # CONFIG_EXT4_FS_POSIX_ACL is not set # CONFIG_EXT4_FS_SECURITY is not set # CONFIG_EXT4_DEBUG is not set[/code] So it looks like I'm still using the old ext2 and ext3 drivers. Problem for me, I want to investigate btrfs, new f.s. that does compression plus lots more, and I see lots of bugfixes going into 2.6.34-rc2. I think that 2.6.32 is the first release of the kernel that has a usable/stable btrfs.

2.6.32.11?
Username: BarryK
"Well, if they backport the btrfs fixes into the next 2.6.32.x then I'm very interested. Yeah, Nouveau has been a disaster for me, so not interested in that until much later when it matures.

EXT3 default journal option?
Username: jamesbond
"Barry just a little note. I noticed in your kernel config that CONFIG_EXT3_DEFAULTS_TO_ORDERED is commented out. I remember reading in your blog somewhere that recently Linus et al changes the default journal option to "data=writeback" which can cause issues during crash etc. I suppose that little option is used to force EXT3 driver to use "data=journal" which was the default for many years. http://cateee.net/lkddb/web-lkddb/EXT3_DEFAULTS_TO_ORDERED.html Not sure whether it's something you've forgotten to turn on, or whether you have finally decided to go with the new default of "data=writeback". cheers!

Ext3
Username: kirk
"Thanks jamesbond, I missed that too. I just skimmed over that same link you posted and went with no. That was a mistake. It looks like problem I thought I was having with ext3 was do to a tv tuner card. I think the card's going bad or the module/firmware has a problem. So Back to 2.6.33.1. Barry, A warning about 2.6.34, there may be a problem with squashfs and the new lzma patches. I did quite a bit of testing with that. Occasionally I'd get squashfs read errors, with lzma compression or zlib. The squashfs developer is looking into it. That was with 2.6.33 and .32, maybe it's fixed now.

ext4
Username: BarryK
"Ah, maybe that's a clue. I think Jemimah configured 2.6.33.1 with the ext4 driver to handle ext2/3, and she has experience a lot of trouble. I had some inode trouble but it was a one-off and I don't know the cause. The 2.6.33.1 kernel has been running fine for me, except for the known crash using OpenOffice with Unionfs 2.5.4.

unionfs
Username: Jemimah
"I've been playing with unionfs some more and I thought I'd post what I've learned so far in case anyone else wants to mess with it. The current version of unionfs doesn't seem to be able to deal with whiteout files in the readonly layers. This only creates an issue if you've got a RAM layer and a save file. For example, if you create a new directory, then do a snapmerge, you won't be able to delete that directory, because it will tell you the directory is not empty. It looks empty to you because unionfs hides the .wh_dir_opaque file, but it's there in the save file and can't be removed except by deleting it directly from the save branch. Even worse, if a file and its whiteout file both exist in the same branch, unionfs gives an IO error, and then you have a file you can neither modify or delete. The snapmergepuppy script doesn't check for this condition, and creates it in the save file whenever you create and delete files. It's pretty easy to fix the snapmerge script to deal with the whiteout file IO errors, but the dir_opaque file problem is more complicated. I think the only correct solution is to copy the files up in the init script, and wait to copy them down until shutdown. Other than that, I think unionfs is working fine with the unlink patch and it's been stable for me in pupmode 12. Is there a reason you don't do a 'mount -t unionfs -o remount,incgen unionfs /' in the snapmerge script?

unionfs
Username: BarryK
""Is there a reason you don't do a 'mount -t unionfs -o remount,incgen unionfs /' in the snapmerge script?" I can't remember enough details of my thinking at the time I was working on it. Quite awhile ago. Maybe because it was supposed to work for both Aufs and Unionfs. Or, maybe I didn't think it was necessary.

Tags: puppy