Overlayfs management in EasyOS

Here are previous posts about the overlay filesystem:

• Finding overlayfs whiteout files — September 04, 2025
• getfattr linked statically with musl — September 04, 2025
• Finally got overlay filesystem to work — September 02, 2025

EasyOS is now working with overlayfs, and it looks good. What follows are technical details how overlayfs is handled in Easy...

Firstly, the Linux kernel, these are the configure choices:

CONFIG_OVERLAY_FS=y
CONFIG_OVERLAY_FS_REDIRECT_DIR=y
# CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW is not set
CONFIG_OVERLAY_FS_INDEX=y
CONFIG_OVERLAY_FS_XINO_AUTO=y
CONFIG_OVERLAY_FS_METACOPY=y
# CONFIG_OVERLAY_FS_DEBUG is not set

...I didn't know much about those options, when I chose them. I just copied one of the other Linux distributions, it might have been Arch Linux. Have done more reading recently, in particular the kernel documentation:

https://docs.kernel.org/filesystems/overlayfs.html

At bootup, the 'init' script in the initrd mounts the overlay filesystem. There is a choice, whether the top read-write layer will be in zram, or direct to permanent storage in the .session folder. Here is the former:

 mount -t overlay -o xino=on,nfs_export=off,index=off,uuid=null,verity=off,redirect_dir=off,\
metacopy=off,lowerdir=${SESSIONHOME}:${EXTRASFS}/easy_ro/easy_sfs,upperdir=/easy_rw/mainrw,\
workdir=/easy_rw/tempwork overlay /easy_new

I figured out all those mount options "xino=on,nfs_export=off,index=off,uuid=null,verity=off,redirect_dir=off,metacopy=off", that you can read about in the kernel docs.

When the read-write layer is saved to the permanent .session folder, usually at shutdown, this is done by script /etc/rc.d/rw-merge. This has been written to handle aufs whiteout files. I posted about this script in 2022, when it was first written:

"A rethink of EasyOS architecture'
https://bkhome.org/news/202205/a-rethink-of-easyos-architecture.html

What I have now done is add support for overlayfs whiteout files and folders, so that the script can handle both. Didn't want to "burn the bridges" in case wanted to go back to aufs. Here is the latest rw-merge script:

#!/bin/ash
#save rw layer to permanent .session folder at shutdown.
#when EOS_TOP_LEVEL_ZRAM='1' in PUPSTATE file, then /dev/zram1 is mounted on
# /mnt/easy_rw which rw top aufs layer.
# underneath is /mnt/${WKG_DEV}/${WKG_DIR}.session mounted ro.
# rc.shutdown will remove .session layer, then call this script.

export LANG=C
. /etc/rc.d/PUPSTATE
endDEST=".session"
if [ $1 ];then
 endDEST="$1" #.session-transit see: ask-save-zram1, save when running, not shutdown.
fi

#20220626
CONSTRAIN=''
if [ $2 ];then
 CONSTRAIN="$2" #exs: mainrw, www
fi

#20220529 src can also be containers...
if [ "$EOS_SUPPORT_CONTAINERS" != "0" ];then
 RWfolders="$(find /mnt/.easy_rw -mindepth 1 -maxdepth 1 -type d | tr '\n' ' ')"
else
 RWfolders='/mnt/.easy_rw/mainrw'
fi

###whiteouts###
for aSRC in $RWfolders #20220529
do
 aSRC="${aSRC##*/}"
 if [ "$CONSTRAIN" ];then #20220626
  if [ "$aSRC" != "$CONSTRAIN" ];then
   continue
  fi
 fi
 cd /mnt/.easy_rw/${aSRC}
 case "$aSRC" in
  mainrw) DEST="/mnt/${WKG_DEV}/${WKG_DIR}${endDEST}" ;;
  *)      DEST="/mnt/${WKG_DEV}/${WKG_DIR}containers/${aSRC}/${endDEST}" ;;
 esac
 if [ "$endDEST" == ".session-transit" -o "$endDEST" == ".session-snapshot" ];then #20220530 20220627
  DEST="${DEST}/${aSRC}" #save into a sub-folder.
 fi
 mkdir -p $DEST #in case of .session-transit
 #find all files and folders in easy_rw, if a matching wh in .session, then delete wh...
 while read F
 do
  [ "$F" == "" ] && continue
  pathF="${F%/*}"     #ex: ./usr/share/doc
  pathF="${pathF#./}" #ex: usr/share/doc
  nameF="${F##*/}"    #ex: zarfy.txt
  if [ -e "${DEST}/${pathF}/.wh.${nameF}" ];then
   rm -f "${DEST}/${pathF}/.wh.${nameF}"
  fi
  
  #20250902 overlayfs...
  if [ -c "${DEST}/${pathF}/${nameF}" ];then
   if [ "$(LANG=C stat -c %t%T "${DEST}/${pathF}/${nameF}")" == "00" ];then
    rm -f "${DEST}/${pathF}/${nameF}"
   fi
  fi
  if [ -d "${F}" ];then
   #see mount options in initrd, hopefully only have to process trusted.overlay.opaque...
   Q1="$(getfattr -d --name='trusted.overlay.opaque' --only-values ${F} 2>/dev/null)"
   if [ "$Q1" == "y" ];then
    rm -rf "${DEST}/${pathF}/${nameF}"
   fi
  fi
  
 done <<_END1
$(find . -mindepth 2 -mount)
_END1
 
 #find wh in easy_rw, fix in .session...
 while read WH
 do
  [ "$WH" == "" ] && continue
  pathWH="${WH%/*}"     #ex: ./lib/firmware
  pathWH="${pathWH#./}" #ex: lib/firmware
  nameWH="${WH##*/}"    #ex: .wh..wh..opq
  
  if [ "$nameWH" == ".wh..wh..opq" ];then
   if [ -h "${DEST}/${pathWH}" -o -f "${DEST}/${pathWH}" ];then
    rm -f "${DEST}/${pathWH}"
   elif [ -d "${DEST}/${pathWH}" ];then
    rm -rf "${DEST}/${pathWH}"
   fi
   continue
  fi
  
  delF="${nameWH#.wh.}" #ex: .wh.ycalc.txt becomes ycalc.txt
  if [ -h "${DEST}/${pathWH}/${delF}" -o -f "${DEST}/${pathWH}/${delF}" ];then
   rm -f "${DEST}/${pathWH}/${delF}"
  elif [ -d "${DEST}/${pathWH}/${delF}" ];then
   rm -rf "${DEST}/${pathWH}/${delF}"
  fi
 done <<_END2
$(find . -mindepth 1 -mount -type f -name '.wh.*')
_END2
 
 #20250902 same for overlayfs...
 while read WH
 do
  #ex: ./usr/bin/afile00
  [ "$WH" == "" ] && continue
  [ "${WH%00}" == "${WH}" ] && continue
  WH="${WH%00}"
  pathWH="${WH%/*}"     #ex: ./lib/firmware
  pathWH="${pathWH#./}" #ex: lib/firmware
  delF="${WH##*/}"    #ex: afile
  if [ -h "${DEST}/${pathWH}/${delF}" -o -f "${DEST}/${pathWH}/${delF}" ];then
   rm -f "${DEST}/${pathWH}/${delF}"
  elif [ -d "${DEST}/${pathWH}/${delF}" ];then
   rm -rf "${DEST}/${pathWH}/${delF}"
  fi
  #remove it from the src, so doesn't get saved...
  rm -f "$WH"
  
 done <<_END2
$(find . -mindepth 1 -mount -type c -exec stat -c %n%t%T {} \;)
_END2
 
 ###merge###
 #20220907
 if [ -h /var ];then
  VAREXC='./var'
  VARREXC=''
 else
  VAREXC='./var/cache'
  VARREXC='--exclude=./var/run  --exclude=./var/lock --exclude=./var/lib/flatpak' #20220918 20250131
 fi
 
 #20241124 save /files if in container...
 if [ "${aSRC}" == "mainrw" ];then
  FILESEXC='--exclude=./files'
 else
  #20250711 redo this section. fix find configuration file and read shared-folder...
  #the folders other than 'mainrw' are all containers...
  FILESEXC=''
  if [ -f /mnt/wkg/containers/${aSRC}/configuration ];then
   PATHecshared="$(grep '^EC_ACCESS_FOLDER_PATH=' /mnt/wkg/containers/${aSRC}/configuration | cut -f 2 -d "'")"
   if [ -n "${PATHecshared}" ];then
    FILESEXC="--exclude=.${PATHecshared}" #ex: ./files
   fi
  fi
 fi
    tar -cpf - ${FILESEXC} --exclude='./dev' --exclude='./mnt' --exclude=${VAREXC} ${VARREXC} --exclude='./run' --exclude='./.*' --exclude='*/.cache' --exclude='./proc' --exclude='./sys' --exclude='./tmp' --exclude='./lost+found' --exclude='./root/.XLOADED' --one-file-system . | tar -xf - -C ${DEST} --overwrite --warning=none
 
 #20241124 ./.control is excluded. need to save *.run-once-*-flag ...
 if [ -d ./.control ];then
  if [ -n "$(ls -A ./.control)" ];then
   mkdir -p ${DEST}/.control
   cp -a -f ./.control/*-flag ${DEST}/.control/ 2>/dev/null
  fi
 fi
done
sync
###end###

There is new code to manage the overlay character devices and xattrs on folders. The script ensures that no 0/0 character devices, nor xattrs, get saved to the .session folder.

Notice that there are two 'tar' operations. The first creates a tar file from the contents of the read-write layer (as well as container read-write layers) and pipes it to another 'tar' that expands it to the permanent storage. The reason that tar was used, is the ability to specify multiple exclusions.

'tar' will not, by default, save xattrs of files and folders. That could be done, with "--xattrs" on the first tar, and "--xattrs --xattrs-include='glob pattern'"; however, Easy works fine without saving xattrs. If we do have any need to save an xattr, it can be added; however, not for the overlay "trusted.overlay.*" xattrs; the logic I have applied requires they never be saved.

EDIT:
Made a modification, so as not to risk compromising the overlay layers:

 #20250902 same for overlayfs...
 echo -n '' > /tmp/delayed-remove #20250904
 while read WH
 do
  #ex: ./usr/bin/afile00
  [ "$WH" == "" ] && continue
  [ "${WH%00}" == "${WH}" ] && continue
  WH="${WH%00}"
  pathWH="${WH%/*}"     #ex: ./lib/firmware
  pathWH="${pathWH#./}" #ex: lib/firmware
  delF="${WH##*/}"    #ex: afile
  if [ -h "${DEST}/${pathWH}/${delF}" -o -f "${DEST}/${pathWH}/${delF}" ];then
   rm -f "${DEST}/${pathWH}/${delF}"
  elif [ -d "${DEST}/${pathWH}/${delF}" ];then
   rm -rf "${DEST}/${pathWH}/${delF}"
  fi
  #remove it from the src, so doesn't get saved...
  #20250904 probably ok because doing it at shutdown, but not good to write directly
  # to rw layer, so delay deletion...
  #rm -f "$WH"
  echo "${DEST}/${pathWH}/${delF}" >> /tmp/delayed-remove
 done <<_END2
$(find . -mindepth 1 -xdev -type c -exec stat -c %n%t%T {} \;)
_END2

Then later, after having saved the session:

 #20250904 delayed remove
 if [ -s /tmp/delayed-remove ];then
  rm -f $(cat /tmp/delayed-remove)
 fi

Probably more changes on the way.      

Tags: easy