"Why Unionfs is better than Aufs"

My most recent kernel build, with the 2.6.33.1 kernel, uses Unionfs 2.5.4. This was in response to issues discussed in this link, and Jemimah's successful move to Unionfs:

http://bkhome.org/blog/?viewDetailed=01440

I posted a message to the Aufs mail-list, entitled "Why Unionfs is better than Aufs", reproduced below. The comments contain the responses on the mail-list. They are very interesting responses!

Original post by me:
Over the years with Puppy Linux I have gone backwards and forwards

between using Aufs and Unionfs.

They each have advantages and disadvantages, but Unionfs has two major
advantages that make me prefer it. The problem is that we have had
ongoing stability issues with Unionfs and the developers are extremely
slow to fix them.

That was the reason for my last move from Unionfs to Aufs. However,
for the latest experiments with the 2.6.33.x kernel we are once again
playing with Unionfs.

I need to explain why I want to use Unionfs, if it is stable, rather than Aufs.

We have been discussing these reasons on my blog:

http://bkhome.org/blog/?viewDetailed=01440

Now, it may be that these two things can be done in Aufs, but I don't know how:

First reason
Puppy Linux has the layers (branches) with basically the r/w "save
file" or "save partition" on top, the main Puppy f.s. below as a r/o
Squashfs file (what we call a SFS file), and optionally more SFS files
as layers below that -- for example a complete c/c++ development
environment, or OpenOffice.

If we have a "save partition" on top, we also have the SFS files in
that partition. Aufs will refuse to mount those SFS files as they are
already inside a branch. Consequently, we are forced to copy the main
SFS file into RAM and mount it from there. However, Unionfs is quite
happy with mounting SFS files that are actually present in another
branch.

Second reason
Both Unionfs and Aufs have mechanisms for detecting direct writes to
branches, however they are not perfect. Unless, in the case of Aufs
you go for the mode using inotify, but that has performance penalty.

When running Puppy from Flash memory, we have the top branch as a
tmpfs in RAM, and the "save file" or "save partition" below that. This
is done to minimise writes to the Flash memory. When a package is
installed, we would like to be able to install it directly to the
"save" layer, the actual Flash memory and have all the files visible
on top.

Our package installer recognises if Unionfs is being used and is able
to write direct to the "save" layer, and then is able to do a perfect
operation to make all files visible on top by doing this:

mount -t unionfs -o remount,incgen unionfs /

...this flushes the caches and does a complete re-evaluation of the
layers. As far as I know, Aufs does not have the equivalent.

Regards,
Barry Kauler


Read Comments for mail-list responses:

Posted on 24 Mar 2010, 8:13


Comments:

Posted on 24 Mar 2010, 8:19 by BarryK
First mail-list response
This response is from J. R. Okajima, the founder and main developer of Aufs:

Hello Barry,


Barry Kauler:
> Now, it may be that these two things can be done in Aufs, but I don't know how:
>
> First reason
:::
> If we have a "save partition" on top, we also have the SFS files in
> that partition. Aufs will refuse to mount those SFS files as they are
> already inside a branch. Consequently, we are forced to copy the main
> SFS file into RAM and mount it from there. However, Unionfs is quite
> happy with mounting SFS files that are actually present in another
> branch.

Because I had experienced a recursive lock (deadlock) by a loopback
branch (fs image in a branch fs), aufs rejects it.
But it was a long time ago. It was early stage in aufs1 era.
I will check it again with recent kernel.


> Second reason
> Both Unionfs and Aufs have mechanisms for detecting direct writes to
> branches, however they are not perfect. Unless, in the case of Aufs
> you go for the mode using inotify, but that has performance penalty.

udba=inotify has some overhead as you wrote. Is it so big penalty for
you? CPU time or memory? If you show me some numbers about the penalty,
I may re-consider the approach.


> When running Puppy from Flash memory, we have the top branch as a
> tmpfs in RAM, and the "save file" or "save partition" below that. This
> is done to minimise writes to the Flash memory. When a package is
> installed, we would like to be able to install it directly to the
> "save" layer, the actual Flash memory and have all the files visible
> on top.

RESPONSE CONTINUED...




Posted on 24 Mar 2010, 8:23 by BarryK
First response, continued
The Comment form has a limit of 2000 characters (I must fix that!) so here is the rest of the first response:

To install a package, modifying a branch permission temporarly may be

useful. For instance, when you have three branches,
- /u = /tmpfs(rw) + /flash=ro(ro) + /squashfs(ro)
- modify
/u = /tmpfs(ro) + /flash=ro(rw) + /squashfs(ro)
- install a package to /u
_if_ the same named file doesn't exist the first /tmpfs, aufs will
pass the files to the second /flash.

But it may not be a generic solution. I didn't test it actually either.
You may prefer bypassing aufs.

> Our package installer recognises if Unionfs is being used and is able
> to write direct to the "save" layer, and then is able to do a perfect
> operation to make all files visible on top by doing this:
>
> mount -t unionfs -o remount,incgen unionfs /
>
> ...this flushes the caches and does a complete re-evaluation of the
> layers. As far as I know, Aufs does not have the equivalent.

Do you mean that the newly added files in the second layer is invisile
in aufs?
Usually, "mount -o remount /your/aufs" is enough for such case.
Of course, if the same named file already exist in the upper layer, you
cannot see the new files in the lower.
Or RDU (readdir in userspace) in libau.so and rdcache=N mount option may
be helpful.

The 'incgen' option is easy to implement, but I don't think it
meaningful. Because
- simple remount is enough for most cases, but files in use will not be
refreshed.
- with udba=inotify, you can skip the manual remount and the files in
use will be refreshed if necessary. when an inotify event is fired and
aufs receives it, aufs executes something like 'decgen' for that file
internally. So that the next access to the file causes the refresh.

RESPONSE CONTINUES...



Posted on 24 Mar 2010, 8:24 by BarryK
First response, continued
The end of it:

Setting udba=inotify temporally is also recommended.

# mount -o remount,udba=inotify /
# install something to branch, bypassing aufs
# mount -o remount,udba=reval /

Reading the url you wrote,
- Did you experience the corrupted filesystem after unmounting aufs?
If so, let me know the detail please, particulary how you unmount it
(or remount it read-only).

> Posted on 18 Mar 2010, 15:33 by BarryK
> Writing to branches
:::
> I did actually try and explain this on the Aufs mail list once, but perhaps they never understood
> what I was asking for, or didn't appreciate the value of it.

I may not receive the mail.
When did you post it? Did I reply it? Is it archived in the
sourceforge ML? If you still have a copy, will you send it again?


Thanx
J. R. Okajima




Posted on 24 Mar 2010, 8:26 by BarryK
Second response
> > First reason

> :::
> > If we have a "save partition" on top, we also have the SFS files in
> > that partition. Aufs will refuse to mount those SFS files as they are
> > already inside a branch. Consequently, we are forced to copy the main
> > SFS file into RAM and mount it from there. However, Unionfs is quite
> > happy with mounting SFS files that are actually present in another
> > branch.
>
> Because I had experienced a recursive lock (deadlock) by a loopback
> branch (fs image in a branch fs), aufs rejects it.
> But it was a long time ago. It was early stage in aufs1 era.
> I will check it again with recent kernel.

It may work.
At least, my simple tests are passed.
Here is a patch for you. Test it well, if you want.


J. R. Okajima


diff --git a/fs/aufs/loop.c b/fs/aufs/loop.c
index 5fceec7..2a78a8d 100644
--- a/fs/aufs/loop.c
+++ b/fs/aufs/loop.c
@@ -32,6 +32,7 @@ int au_test_loopback_overlap(struct super_block *sb, struct dentry *h_d1,
struct inode *h_inode;
struct loop_device *l;

+ return 0;
h_inode = h_d1->d_inode;
if (MAJOR(h_inode->i_sb->s_dev) != LOOP_MAJOR)
return 0;



Posted on 24 Mar 2010, 8:41 by BarryK
Third response
This is a response from me:

Thank you very much for your very detailed response. This is the major

difference from the Unionfs mail-list, where there is often no
response.

The idea of temporarily remounting with 'udba=inotify' looks like a
great idea! I'll try it.

I will also try the patch to mount a Squashfs file that is present in
another branch.

This is great, thanks again for your responsiveness. Note, regarding
my previous attempt to explain one or more of these issues, that was a
long time ago, and I don't recall the details -- probably my
explanation back then was too vague.

Regarding f.s. corruption, it is another Puppy Linux developer
(Jemimah) who was having trouble. I did have a one-off problem but it
didn't repeat so I don't really know what that problem was. I'll do
another kernel compile (with the patch applied to Aufs) and will see
if I can get any f.s. errors.

One interesting point about the 2.6.33 kernel is that they have
dropped the ext2 and ext3 drivers as the new ext4 driver handles those
also. -- probably has nothing to do with this topic, just an
interesting item of news.




Posted on 24 Mar 2010, 20:16 by ICPUG
scrolling box
Is it just me but these scrolling boxes do not display properly in my browser (IE6 enforced by employer).

Tried copying contents to a Word processor, which helped but still left bits missing.


Posted on 24 Mar 2010, 23:44 by Jemimah
copydown errors
I'm having some trouble with I/O errors during shutdown when it's copying files from RAM to the save file. I haven't investigated thoroughly yet, but I thought I'd ask if you had any ideas. Are there any changes you made to the snapmerge script to accommodate unionfs?

Also I've had a couple problems with whiteout files in unionfs, so I'm really sort of hoping to go back to AUFS if possible.


Posted on 25 Mar 2010, 7:59 by BarryK
snapmergepuppy
snapmergepuppy has been designed to work with both Aufs and Unionfs. In fact it was initially used with Unionfs, and it was example copydown code provided by the Unionfs project that was utilised in creating the first snapmergepuppy. I haven't worked on that script for quite some time.

Yes, problems with whiteout files is the main reason why people moved to Aufs, en masse. Very few people are still using Unionfs. Unionfs also had intervals of instability, in which the developers were unresponsive to bug reports -- my personal experience and one reason for one of my earlier moves to Aufs.

I am seriously thinking of going back to Aufs, as J. R. Okajima has answered my questions and shown me that Aufs can do the special things I want. I am also thinking of using the 2.6.32.x kernel, as I'm very suspicious of that ext4 driver taking over responsibility for ext2 and ext3.



Posted on 25 Mar 2010, 24:30 by kirk
2.6.32.x
Barry,

I think I may be having problems with ext3 as well. When you compiled 2.6.33.1, in your .config did you have ext2=n ext3=n ext4=y and CONFIG_EXT4_USE_FOR_EXT23=y? I had ext2 and ext3 set to yes, so I think that's still using the old driver. I was thinking of reconfiguring with ext4 driver for ext2/3. But If you have tried it I won't. See this for more info:

http://cateee.net/lkddb/web-lkddb/EXT4_USE_FOR_EXT23.html

Phoronix has a article that seems to indicate problems with ext3, but they don't say how they configured their kernel.

http://www.phoronix.com/scan.php?page=article&item=linux_2624_2633&num=2


Posted on 26 Mar 2010, 8:35 by kirk
2.6.32.10
I switched my save file to ext3 in my 64bit build when I changed from 2.6.33 to 2.6.33.1. I also enabled NFS exports with aufs. Then I started having intermittent boot problems. Every once in a while it would hang near the end of rc.sysint. So I thought maybe it was the NFS aufs option. I ended up recompiling 2.6.33.1 three times, all with the same result. So I just compiled 2.6.32.10. So far so good. No nouveau, but Nvidia supports their stuff pretty well.



Posted on 26 Mar 2010, 9:13 by BarryK
2.6.33.1 ext2/3/4
Kirk,
I configured like this:

CONFIG_EXT2_FS=y

CONFIG_EXT2_FS_XATTR=y
# CONFIG_EXT2_FS_POSIX_ACL is not set
# CONFIG_EXT2_FS_SECURITY is not set
CONFIG_EXT2_FS_XIP=y
CONFIG_EXT3_FS=y
# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
CONFIG_EXT3_FS_XATTR=y
# CONFIG_EXT3_FS_POSIX_ACL is not set
# CONFIG_EXT3_FS_SECURITY is not set
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_XATTR=y
# CONFIG_EXT4_FS_POSIX_ACL is not set
# CONFIG_EXT4_FS_SECURITY is not set
# CONFIG_EXT4_DEBUG is not set


So it looks like I'm still using the old ext2 and ext3 drivers.

Problem for me, I want to investigate btrfs, new f.s. that does compression plus lots more, and I see lots of bugfixes going into 2.6.34-rc2.
I think that 2.6.32 is the first release of the kernel that has a usable/stable btrfs.



Posted on 26 Mar 2010, 9:22 by BarryK
2.6.32.11?
Well, if they backport the btrfs fixes into the next 2.6.32.x then I'm very interested.

Yeah, Nouveau has been a disaster for me, so not interested in that until much later when it matures.



Posted on 26 Mar 2010, 16:40 by jamesbond
EXT3 default journal option?
Barry just a little note. I noticed in your kernel config that CONFIG_EXT3_DEFAULTS_TO_ORDERED is commented out. I remember reading in your blog somewhere that recently Linus et al changes the default journal option to "data=writeback" which can cause issues during crash etc.

I suppose that little option is used to force EXT3 driver to use "data=journal" which was the default for many years. http://cateee.net/lkddb/web-lkddb/EXT3_DEFAULTS_TO_ORDERED.html

Not sure whether it's something you've forgotten to turn on, or whether you have finally decided to go with the new default of "data=writeback".

cheers!


Posted on 27 Mar 2010, 9:17 by kirk
Ext3
Thanks jamesbond, I missed that too. I just skimmed over that same link you posted and went with no. That was a mistake. It looks like problem I thought I was having with ext3 was do to a tv tuner card. I think the card's going bad or the module/firmware has a problem. So Back to 2.6.33.1.

Barry,

A warning about 2.6.34, there may be a problem with squashfs and the new lzma patches. I did quite a bit of testing with that. Occasionally I'd get squashfs read errors, with lzma compression or zlib. The squashfs developer is looking into it. That was with 2.6.33 and .32, maybe it's fixed now.


Posted on 27 Mar 2010, 16:12 by BarryK
ext4
Ah, maybe that's a clue. I think Jemimah configured 2.6.33.1 with the ext4 driver to handle ext2/3, and she has experience a lot of trouble. I had some inode trouble but it was a one-off and I don't know the cause. The 2.6.33.1 kernel has been running fine for me, except for the known crash using OpenOffice with Unionfs 2.5.4.



Posted on 30 Mar 2010, 22:41 by Jemimah
unionfs
I've been playing with unionfs some more and I thought I'd post what I've learned so far in case anyone else wants to mess with it.

The current version of unionfs doesn't seem to be able to deal with whiteout files in the readonly layers. This only creates an issue if you've got a RAM layer and a save file.

For example, if you create a new directory, then do a snapmerge, you won't be able to delete that directory, because it will tell you the directory is not empty. It looks empty to you because unionfs hides the .wh_dir_opaque file, but it's there in the save file and can't be removed except by deleting it directly from the save branch.

Even worse, if a file and its whiteout file both exist in the same branch, unionfs gives an IO error, and then you have a file you can neither modify or delete. The snapmergepuppy script doesn't check for this condition, and creates it in the save file whenever you create and delete files.

It's pretty easy to fix the snapmerge script to deal with the whiteout file IO errors, but the dir_opaque file problem is more complicated. I think the only correct solution is to copy the files up in the init script, and wait to copy them down until shutdown.

Other than that, I think unionfs is working fine with the unlink patch and it's been stable for me in pupmode 12.

Is there a reason you don't do a 'mount -t unionfs -o remount,incgen unionfs /' in the snapmerge script?




Posted on 30 Mar 2010, 23:57 by BarryK
unionfs
"Is there a reason you don't do a 'mount -t unionfs -o remount,incgen unionfs /' in the snapmerge script?"

I can't remember enough details of my thinking at the time I was working on it. Quite awhile ago. Maybe because it was supposed to work for both Aufs and Unionfs.
Or, maybe I didn't think it was necessary.