Found a reason why overlayfs is flakey

I have posted about overlay filesystem many times in the past, for example:

"Overlay filesystem is still a disaster"
https://bkhome.org/news/202402/overlay-filesystem-is-still-a-disaster.html

"OverlayFS is still pathetic"
https://bkhome.org/news/202304/overlayfs-is-still-pathetic.html

In EasyOS 7.0, I got overlay to work, and decided to transition away from aufs.

But why is it working now, yet previous disasters? Today I found out why. I'm running Easy with the session in RAM, which can be flushed to permanent storage in the working-partition when desired, usually at shutdown. However, Easy is also supposed to work with the permanent storage '.session' folder directly mounted on the upperdir (the read-write layer) of overlayfs -- and that fails.

A little test. In the working partition, have folders 'ro1', 'rw1', 'work1' and 'top1':

# mount -t squashfs -o ro,loop,noatime easy.sfs ro1
# rm -rf work1/* 2>/dev/null
# mount -t overlay -o \
xino=on,nfs_export=off,index=off,uuid=null,verity=off,redirect_dir=nofollow,metacopy=off,\
lowerdir=ro1,upperdir=rw1,workdir=work1 overlay top1
# touch top1/opt/file1
#

...the important point above, is that the '/opt' folder is in easy.sfs, layer 'ro1', does not exist in upperdir 'rw1' (which is currently empty). There is no problem, rw1/opt/file1 is created. Now try this:

# umount top1
# rm -rf rw1/* 2>/dev/null
# rm -rf work1/* 2>/dev/null
# mount -t overlay -o \
xino=on,nfs_export=off,index=off,uuid=null,verity=off,redirect_dir=nofollow,metacopy=off,\
lowerdir=ro1,upperdir=easyos/.session,workdir=work1 overlay top1
# touch top1/opt/file1
touch: cannot touch 'top1/opt/file1': Invalid cross-device link
#

...upperdir is now 'easyos/.session', which is the session permanent storage. The thing is, it is an encrypted folder, with ext4 fscrypt. Encrypted, yes, but it has been decrypted and contents are visible. Also looking in 'top1', all contents of '.session' are visible.

However, overlayfs has decided that this is an "Invalid cross-device link". But it isn't. It is just refusing to create the 'opt' folder in the upperdir. Also "mkdir top1/opt" fails. If I was to unmount 'top1', then create rw1/opt folder, then no problem, the "touch top1/opt/file1" works.

Aufs has no problem with this situation, and I do regret having to retire it. If you read back through my blog, the reason for dropping aufs was it was plagued with random hanging at bootup, which went away when moved to overlay.

One solution is to not have encrypted '.session' folder. I don't know if I will be up to it, but could study the kernel overlay driver, see if I can fix it.   

Tags: easy