Continuing getting APT to work in a layered filesystem

The problem is fundamental; Debian's APT has no concept of a layered filessystem such as overlayfs. Getting it to work in such an environment is a major challenge. The issues started with EasyOS 7.0, that introduced APT as the underlying package manager, with PKGget (PPM) is a GUI frontend.

EasyOS runs in an overlayfs stacked filesystem, with bottom layer the read-only easy.sfs (mounted on a folder), and the read-write layer either in RAM or a folder in the working partition. There may be more read-only layers, in between, for example devx.sfs.

APT maintains, via the underlying dpkg,  /var/lib/dpkg/status file and /var/lib/dpkg/info folder, and these are created in easy.sfs. Any changes, such as install a package, will be written on the read-write layer. At a version upgrade of EasyOS, there will be new APT database information in the easy.sfs read-only layer, making the information in the read-write layer incorrect.

The woofQ2 build system creates the devx.sfs such that when it is layered on top of easy.sfs, the /var/lib/dpkg data in the devx.sfs represents everything, all the installed packages in both the devx.sfs and easy.sfs

This creates a major problem; if the user decides to install the devx.sfs, then at bootup, the database information in the read-write layer will be completely incorrect.

EasyOS 7.0 has basic handling of this, but there were issues, that we have been tackling. Here are past relevant blog posts:

https://bkhome.org/news/202508/dpkg-db-ro-and-rw-layers-synchronized.html

https://bkhome.org/news/202607/fix-for-multiple-sfs-layers-in-main-desktop.html

https://bkhome.org/news/202607/partial-fix-for-devx-sfs-loaded-on-main-desktop.html

It did seem to be working ok in the devx container, but loading devx.sfs on the main desktop, the database information was not correctly updating on the read-write layer. So, have worked some more on it...

Right now, sitting in a tent, at a campsite on the south coast of Western Australia. Very windy, but the tent is holding up and OK inside.

I have introduced this code in /usr/local/easy_containers/start-container:

#20260808 copy debian 'status' file(s) into container...
mkdir -p ${CONTAINER}/root/.packages
cp -a -f /mnt/.easy_ro/easy_sfs/var/lib/dpkg/status ${CONTAINER}/root/.packages/woof-status
grep -q '^EASY_LAYER_RO.*devx\.sfs' /mnt/${WKG_DEV}/${WKG_DIR}containers/${EXE}/configuration
if [ $? -eq 0 ];then
 cp -a -f /mnt/${WKG_DEV}/${WKG_DIR}containers/${EXE}/.devx/var/lib/dpkg/status ${CONTAINER}/root/.packages/devx-woof-status
else
 rm -f ${CONTAINER}/root/.packages/devx-woof-status 2>/dev/null
fi

...purpose is to make the dpkg 'status' file available inside a container. If the devx.sfs is loaded, there will be two of them.

Script /usr/local/petget/alias/sync-dpkg-layers has had an almost complete rewrite:

#!/bin/bash
#bootup new version of easyos, there will be discrepancies between /var/lib/dpkg
#in /mnt/wkg/.session/session1 (and in containers .session) and in easy.sfs
#--new or removed pkgs in easy.sfs, version changes.

export LANG=C
mkdir -p /tmp/petget

###############
#get list of installed ro packages...
if [ -e /root/.packages/devx-woof-status ];then
 #running in a container, with devx loaded.
 Cmode=3
 DEVXloaded=1
 ROstatuspath='/root/.packages/devx-woof-status'
 ROstatusnames="$(grep '^Package: ' /root/.packages/devx-woof-status | cut -f 2 -d ' ' | sort)"
elif [ -e /root/.packages/woof-status ];then
 #running in a container, without devx loaded.
 Cmode=2
 DEVXloaded=0
 ROstatuspath='/root/.packages/woof-status'
 ROstatusnames="$(grep '^Package: ' /root/.packages/woof-status | cut -f 2 -d ' ' | sort)"
elif [ -e /mnt/.easy_ro/devx/usr/bin/gcc ];then
 #running on main desktop, devx loaded.
 Cmode=1
 DEVXloaded=1
 ROstatuspath='/mnt/.easy_ro/devx/var/lib/dpkg/status'
 #devx sfs was constructed to have record of all packages installed, devx+easy.sfs
 ROstatusnames="$(grep '^Package: ' /mnt/.easy_ro/devx/var/lib/dpkg/status | cut -f 2 -d ' ' | sort)"
else
 #running on main desktop, no devx.
 Cmode=0
 DEVXloaded=0
 ROstatuspath='/mnt/.easy_ro/easy_sfs/var/lib/dpkg/status'
 ROstatusnames="$(grep '^Package: ' /mnt/.easy_ro/easy_sfs/var/lib/dpkg/status | cut -f 2 -d ' ' | sort)"
fi
echo -n "${ROstatuspath}" > /tmp/petget/ROstatuspath

#this will contain pkgs installed in ro-layer and user-installed...
# but, will be wrong if devx layer added or removed.
RWstatus="$(grep '^Package: ' /var/lib/dpkg/status | cut -f 2 -d ' ' | sort)"

echo -n "$RWstatus" > /tmp/petget/dpkg-RWstatus
echo -n "$ROstatusnames" > /tmp/petget/dpkg-ROstatusnames

#find pkgs in rw layer, not in ro...
NEWstatus="$(grep -v -F -x -f /tmp/petget/dpkg-ROstatusnames /tmp/petget/dpkg-RWstatus)"
#...some of these are user-installed, some no longer in easy.sfs
echo -n "${NEWstatus}" > /tmp/petget/dpkg-NEWstatus

#find pkgs in ro layer, not in rw...
MISSstatus="$(grep -v -F -x -f /tmp/petget/dpkg-RWstatus /tmp/petget/dpkg-ROstatusnames)"
#...these may be new in easy.sfs or devx.sfs
echo -n "${MISSstatus}" > /tmp/petget/dpkg-MISSstatus

##############
#handle NEWstatus...
for aNEW in ${NEWstatus}
do
 [ -z "$aNEW" ] && continue
 grep -q -F "|${aNEW}|" /root/.packages/user-installed-packages
 [ $? -eq 0 ] && continue
 grep -q -F "|${aNEW}|" /root/.packages/woof-installed-packages
 [ $? -eq 0 ] && continue
 #20250822 careful, devx sfs may be loaded...
 if [ $DEVXloaded -ne 0 ];then
  grep -q -F "|${aNEW}|" /root/.packages/devx-only-installed-packages
  [ $? -eq 0 ] && continue
 fi
 #want to purge it from db, as no longer exists...
 sed -i "/^Package: ${aNEW}$/,/^$/d" /var/lib/dpkg/status
 rm -f /var/lib/dpkg/info/${aNEW}:* 2>/dev/null
 rm -f /var/lib/dpkg/info/${aNEW}.* 2>/dev/null
done

##############
#handle MISSstatus...
for aMISS in ${MISSstatus}
do
 [ -z "$aMISS" ] && continue
 #extract paragraph from ro-layer, append on rw-layer...
 P1="$(sed -n "/^Package: ${aMISS}$/,/^$/p" ${ROstatuspath})"
 [ -z "$P1" ] && continue
 echo "${P1}
" >> /var/lib/dpkg/status
done

#############
#handle pkg version change...
ROpkgs="$(grep -E '^Package: |^Version: |^$' ${ROstatuspath} | cut -f 2 -d ' ' | sed -e 's%^$%ZZZZZ%' | tr '\n' '|' | sed -e 's%|ZZZZZ|%\n%g')"
#...ex: xserver-xorg-video-vmware|1:13.4.0-1 (note, some have epoch prefix, such as "1:"
for aP in ${ROpkgs}
do
 [ -z "$aP" ] && continue
 Pn="${aP%|*}"
 Pv="${aP#*|}"
 RWpara="$(sed -n "/^Package: ${Pn}$/,/^$/p" /var/lib/dpkg/status)"
 grep -q "^Version: ${Pv}$" <<<"$RWpara"
 if [ $? -ne 0 ];then
  #top-level status file has wrong version...
  #delete it...
  sed -i "/^Package: ${Pn}$/,/^$/d" /var/lib/dpkg/status
  ROpara="$(sed -n "/^Package: ${Pn}$/,/^$/p" ${ROstatuspath})"
  echo "${ROpara}
" >> /var/lib/dpkg/status
  #also the info files...
  rm -f /var/lib/dpkg/info/${Pn}:* 2>/dev/null
  rm -f /var/lib/dpkg/info/${Pn}.* 2>/dev/null
  case "$Cmode" in
   3) #running in a container, with devx loaded.
    true
   ;;
   2) #running in a container, without devx loaded.
    true
   ;;
   1)#running on main desktop, devx loaded.
    cp -a -f /mnt/.easy_ro/devx/var/lib/dpkg/info/${Pn}:* /var/lib/dpkg/info/ 2>/dev/null
    cp -a -f /mnt/.easy_ro/devx/var/lib/dpkg/info/${Pn}.* /var/lib/dpkg/info/ 2>/dev/null
   ;;
   0) #running on main desktop, no devx.
    cp -a -f /mnt/.easy_ro/easy_sfs/var/lib/dpkg/info/${Pn}:* /var/lib/dpkg/info/ 2>/dev/null
    cp -a -f /mnt/.easy_ro/easy_sfs/var/lib/dpkg/info/${Pn}.* /var/lib/dpkg/info/ 2>/dev/null
   ;;
  esac
 fi
done
sync
###end###

In a nutshell, what all of the above does, is make sure that the information in the top read-write layer, in particular /var/lib/dpkg/status, agrees with what is actually installed. It seems logical, but will need more testing.   

Tags: easy