Continuing getting APT to work in a layered filesystem

The problem is fundamental; Debian's APT has no concept of a layered filessystem such as overlayfs. Getting it to work in such an environment is a major challenge. The issues started with EasyOS 7.0, that introduced APT as the underlying package manager, with PKGget (PPM) is a GUI frontend.

EasyOS runs in an overlayfs stacked filesystem, with bottom layer the read-only easy.sfs (mounted on a folder), and the read-write layer either in RAM or a folder in the working partition. There may be more read-only layers, in between, for example devx.sfs.

APT maintains, via the underlying dpkg,  /var/lib/dpkg/status file and /var/lib/dpkg/info folder, and these are created in easy.sfs. Any changes, such as install a package, will be written on the read-write layer. At a version upgrade of EasyOS, there will be new APT database information in the easy.sfs read-only layer, making the information in the read-write layer incorrect.

The woofQ2 build system creates the devx.sfs such that when it is layered on top of easy.sfs, the /var/lib/dpkg data in the devx.sfs represents everything, all the installed packages in both the devx.sfs and easy.sfs

This creates a major problem; if the user decides to install the devx.sfs, then at bootup, the database information in the read-write layer will be completely incorrect.

EasyOS 7.0 has basic handling of this, but there were issues, that we have been tackling. Here are past relevant blog posts:

https://bkhome.org/news/202508/dpkg-db-ro-and-rw-layers-synchronized.html

https://bkhome.org/news/202607/fix-for-multiple-sfs-layers-in-main-desktop.html

https://bkhome.org/news/202607/partial-fix-for-devx-sfs-loaded-on-main-desktop.html

It did seem to be working ok in the devx container, but loading devx.sfs on the main desktop, the database information was not correctly updating on the read-write layer. So, have worked some more on it...

Right now, sitting in a tent, at a campsite on the south coast of Western Australia. Very windy, but the tent is holding up and OK inside.

I have introduced this code in /usr/local/easy_containers/start-container:

#20260808 copy debian 'status' file(s) into container...
mkdir -p ${CONTAINER}/root/.packages
cp -a -f /mnt/.easy_ro/easy_sfs/var/lib/dpkg/status ${CONTAINER}/root/.packages/woof-status
grep -q '^EASY_LAYER_RO.*devx\.sfs' /mnt/${WKG_DEV}/${WKG_DIR}containers/${EXE}/configuration
if [ $? -eq 0 ];then
 cp -a -f /mnt/${WKG_DEV}/${WKG_DIR}containers/${EXE}/.devx/var/lib/dpkg/status ${CONTAINER}/root/.packages/devx-woof-status
else
 rm -f ${CONTAINER}/root/.packages/devx-woof-status 2>/dev/null
fi

...purpose is to make the dpkg 'status' file available inside a container. If the devx.sfs is loaded, there will be two of them.

Script /usr/local/petget/alias/sync-dpkg-layers has had an almost complete rewrite:

#!/bin/bash
#bootup new version of easyos, there will be discrepancies between /var/lib/dpkg
#in /mnt/wkg/.session/session1 (and in containers .session) and in easy.sfs
#--new or removed pkgs in easy.sfs, version changes.

export LANG=C
mkdir -p /tmp/petget

###############
#get list of installed ro packages...
if [ -e /root/.packages/devx-woof-status ];then
 #running in a container, with devx loaded.
 Cmode=3
 DEVXloaded=1
 ROstatuspath='/root/.packages/devx-woof-status'
 ROstatusnames="$(grep '^Package: ' /root/.packages/devx-woof-status | cut -f 2 -d ' ' | sort)"
elif [ -e /root/.packages/woof-status ];then
 #running in a container, without devx loaded.
 Cmode=2
 DEVXloaded=0
 ROstatuspath='/root/.packages/woof-status'
 ROstatusnames="$(grep '^Package: ' /root/.packages/woof-status | cut -f 2 -d ' ' | sort)"
elif [ -e /mnt/.easy_ro/devx/usr/bin/gcc ];then
 #running on main desktop, devx loaded.
 Cmode=1
 DEVXloaded=1
 ROstatuspath='/mnt/.easy_ro/devx/var/lib/dpkg/status'
 #devx sfs was constructed to have record of all packages installed, devx+easy.sfs
 ROstatusnames="$(grep '^Package: ' /mnt/.easy_ro/devx/var/lib/dpkg/status | cut -f 2 -d ' ' | sort)"
else
 #running on main desktop, no devx.
 Cmode=0
 DEVXloaded=0
 ROstatuspath='/mnt/.easy_ro/easy_sfs/var/lib/dpkg/status'
 ROstatusnames="$(grep '^Package: ' /mnt/.easy_ro/easy_sfs/var/lib/dpkg/status | cut -f 2 -d ' ' | sort)"
fi
echo -n "${ROstatuspath}" > /tmp/petget/ROstatuspath

#this will contain pkgs installed in ro-layer and user-installed...
# but, will be wrong if devx layer added or removed.
RWstatus="$(grep '^Package: ' /var/lib/dpkg/status | cut -f 2 -d ' ' | sort)"

echo -n "$RWstatus" > /tmp/petget/dpkg-RWstatus
echo -n "$ROstatusnames" > /tmp/petget/dpkg-ROstatusnames

#find pkgs in rw layer, not in ro...
NEWstatus="$(grep -v -F -x -f /tmp/petget/dpkg-ROstatusnames /tmp/petget/dpkg-RWstatus)"
#...some of these are user-installed, some no longer in easy.sfs
echo -n "${NEWstatus}" > /tmp/petget/dpkg-NEWstatus

#find pkgs in ro layer, not in rw...
MISSstatus="$(grep -v -F -x -f /tmp/petget/dpkg-RWstatus /tmp/petget/dpkg-ROstatusnames)"
#...these may be new in easy.sfs or devx.sfs
echo -n "${MISSstatus}" > /tmp/petget/dpkg-MISSstatus

##############
#handle NEWstatus...
for aNEW in ${NEWstatus}
do
 [ -z "$aNEW" ] && continue
 grep -q -F "|${aNEW}|" /root/.packages/user-installed-packages
 [ $? -eq 0 ] && continue
 grep -q -F "|${aNEW}|" /root/.packages/woof-installed-packages
 [ $? -eq 0 ] && continue
 #20250822 careful, devx sfs may be loaded...
 if [ $DEVXloaded -ne 0 ];then
  grep -q -F "|${aNEW}|" /root/.packages/devx-only-installed-packages
  [ $? -eq 0 ] && continue
 fi
 #want to purge it from db, as no longer exists...
 sed -i "/^Package: ${aNEW}$/,/^$/d" /var/lib/dpkg/status
 rm -f /var/lib/dpkg/info/${aNEW}:* 2>/dev/null
 rm -f /var/lib/dpkg/info/${aNEW}.* 2>/dev/null
done

##############
#handle MISSstatus...
for aMISS in ${MISSstatus}
do
 [ -z "$aMISS" ] && continue
 #extract paragraph from ro-layer, append on rw-layer...
 P1="$(sed -n "/^Package: ${aMISS}$/,/^$/p" ${ROstatuspath})"
 [ -z "$P1" ] && continue
 echo "${P1}
" >> /var/lib/dpkg/status
done

#############
#handle pkg version change...
ROpkgs="$(grep -E '^Package: |^Version: |^$' ${ROstatuspath} | cut -f 2 -d ' ' | sed -e 's%^$%ZZZZZ%' | tr '\n' '|' | sed -e 's%|ZZZZZ|%\n%g')"
#...ex: xserver-xorg-video-vmware|1:13.4.0-1 (note, some have epoch prefix, such as "1:"
for aP in ${ROpkgs}
do
 [ -z "$aP" ] && continue
 Pn="${aP%|*}"
 Pv="${aP#*|}"
 RWpara="$(sed -n "/^Package: ${Pn}$/,/^$/p" /var/lib/dpkg/status)"
 grep -q "^Version: ${Pv}$" <<<"$RWpara"
 if [ $? -ne 0 ];then
  #top-level status file has wrong version...
  #delete it...
  sed -i "/^Package: ${Pn}$/,/^$/d" /var/lib/dpkg/status
  ROpara="$(sed -n "/^Package: ${Pn}$/,/^$/p" ${ROstatuspath})"
  echo "${ROpara}
" >> /var/lib/dpkg/status
  #also the info files...
  rm -f /var/lib/dpkg/info/${Pn}:* 2>/dev/null
  rm -f /var/lib/dpkg/info/${Pn}.* 2>/dev/null
  case "$Cmode" in
   3) #running in a container, with devx loaded.
    true
   ;;
   2) #running in a container, without devx loaded.
    true
   ;;
   1)#running on main desktop, devx loaded.
    cp -a -f /mnt/.easy_ro/devx/var/lib/dpkg/info/${Pn}:* /var/lib/dpkg/info/ 2>/dev/null
    cp -a -f /mnt/.easy_ro/devx/var/lib/dpkg/info/${Pn}.* /var/lib/dpkg/info/ 2>/dev/null
   ;;
   0) #running on main desktop, no devx.
    cp -a -f /mnt/.easy_ro/easy_sfs/var/lib/dpkg/info/${Pn}:* /var/lib/dpkg/info/ 2>/dev/null
    cp -a -f /mnt/.easy_ro/easy_sfs/var/lib/dpkg/info/${Pn}.* /var/lib/dpkg/info/ 2>/dev/null
   ;;
  esac
 fi
done
sync
###end###

In a nutshell, what all of the above does, is make sure that the information in the top read-write layer, in particular /var/lib/dpkg/status, agrees with what is actually installed. It seems logical, but will need more testing.   

Tags: easy

YouTube Nomad-series adventure underway

See parts 2, 3 and 4, checking out an old house and inexpensive Shire-run caravan parks:

https://www.youtube.com/@meanderinglight/videos

Right now, I'm at Ravensthorpe, at a free RV overnight stop. Toilet across the road, petrol station that sells food. Nice little place to stopover, maximum allowed 48 hours.

It is not permitted to erect a tent here; anyway, for one night I will just sleep in the car.

So, how do I use my laptop in this tiny little Kia Picanto hatchback? Did consider a tray to hook onto the steering wheel; however, not enough room. Instead, the plywood from the folding table serves as a nice surface, placed on the passenger-side:

...have to twist to the side to use it, but OK.

As my videos are mostly going to be taken while walking, have seen the limitations of the phones. They have EIS (Electronic Image Stabilization), but can see its limitations. Seriously considering buying a DJI Osmo Pocket4, with 3-axis gimbal:

https://www.jbhifi.com.au/products/dji-osmo-pocket-4-standard-combo

...well, that will have to wait, as booked in at the campsite for 4 weeks. Maybe after that will locate the closest JB-HiFi store.

As on-the-go for the last few days, have ignored EasyOS and the forum; however, once settled at the campsite on the coast, will get back to it.    

Tags: nomad

Grey with pink-highlights theme

EasyOS version 7.4.6 has a new theme. I posted about it in the latest YouTube video, Part-24:

"EasyOS Part24: Themes"
https://www.youtube.com/watch?v=6KY1up1JhXM

There have been previous blog posts showing the themes in earlier releases of EasyOS, for example:

• Another look at EasyOS themes — April 27, 2026

Screen snapshot:

...this is the same theme as used in QV (Quantum Vis) -- another experimental Linux distribution, development currently on-hold.

The file in woofQ2, woofq2/configure/x86_64/default-theme, which becomes /root/.packages/default-theme in a running EasyOS, has this for 7.4.6:

#gtk, jwm, wallpaper: names of pet pkgs, which may actually have more than one
# theme inside, hence the : separator with actual theme name.
DEFAULT_THEME_GTK2='gtk_2_3_theme_flat-grey-rounded3:flat-grey-rounded3'
DEFAULT_THEME_GTK3='gtk_2_3_theme_flat-grey-rounded3:flat-grey-rounded3'
DEFAULT_THEME_JWM='jwm3_theme_easygrey:EasyGrey'
DEFAULT_THEME_WALLPAPER='desk_background_galah_pink_grey1:galah-pink-grey1.jpg'
DEFAULT_THEME_DESK_ICONS='desk_icon_theme_treepata_black_pinkish:treepata-black-pinkish'
DEFAULT_THEME_ROX_TEXT_FOREGROUND='#15150b0b0000' #very dark brown
DEFAULT_THEME_ROX_TEXT_SHADOW='0' #0=none
DEFAULT_THEME_ROX_TEXT_FONT='Open Sans Semi-Bold 13'
DEFAULT_THEME_XORG_TEXT_DPI=102
#20240117 3buildeasydistro: write these into /etc/gtk-3.0/settings.ini:
# ${DEFAULT_THEME_GTK3##*:} and
DEFAULT_GTK3_FONT_NAME='Open Sans Semibold 13'
#20240118 set to one of these: blue green grey orange (default orange)
DEFAULT_THEME_BASE_COLOR='grey'
#20260426 text colors in rox window... very dark brown... 7.4.6 go back to default black...
#DEFAULT_THEME_ROX_UNKN_COLOR='#303000000000'
#DEFAULT_THEME_ROX_BDEV_COLOR='#303000000000'
#DEFAULT_THEME_ROX_FILE_COLOR='#303000000000'
#DEFAULT_THEME_ROX_DIR_COLOR='#303000000000'
#DEFAULT_THEME_ROX_CDEV_COLOR='#303000000000'
#DEFAULT_THEME_ROX_PIPE_COLOR='#303000000000'
#20260617 choose chromium theme (default is blue). allowed values: "orange", "yellow", "grey", "green"
DEFAULT_THEME_CHROMIUM='grey'
#20260618 xcursor themes are in /usr/share/icons
# exs: blue_animated_36 blue_animated_48 magenta_animated_36 magenta_animated_48 simple_blue_28 simple_white_28 simple_white_32 simple_yellow_28
DEFAULT_THEME_XCURSOR='magenta_animated_36'

I'll be traveling extensively in the countryside soon, and expect to take photos, one of which may end up as the next wallpaper.

Do you like the grey theme better than the previous orange theme? Or would you prefer a more soothing blue theme? Let me know in the forum.   

Tags: easy

EasyOS Excalibur-series version 7.4.6 released

Changes since 7.4.5:

• RustDesk added to Appi — July 31, 2026
• Fix for multiple sfs layers in main desktop — July 30, 2026
• SFS loading order error — July 30, 2026
• Partial fix for devx sfs loaded on main desktop — July 30, 2026
• Experimenting with grey-pink theme.
  

Note that the graphical installer has not yet been extensively tested, and still to be considered as alpha-quality.

Download:

https://distro.ibiblio.org/easyos/amd64/releases/excalibur/2026/7.4.6/

Feedback welcome at the forum:

https://forum.puppylinux.com/viewtopic.php?p=174736#p174736

Experimenting with a new theme.   

Tags: easy

RustDesk added to Appi

Forum member vtpup asked how to add RustDesk to Appi, the AppImage package manager:

https://forum.puppylinux.com/viewtopic.php?t=17250

Here is the project homepage:

https://rustdesk.com/

Flapi, the Flatpak package manager, has a "Customize" button to add new entries, but Appi doesn't.

In /usr/local/appi, I edited 'appi', inserted this at line 293:

 RustDesk) DIRECTDL='https://github.com/rustdesk/rustdesk/releases/download/1.4.9/rustdesk-1.4.9-x86_64.AppImage'; DIRECTVER='1.4.9'; MENUcat='X-Network'; MENUtop='Network'; MENUname='RustDesk remote access'; ;;

In 'online-info' inserted this at line 95:

 RustDesk) URL='https://rustdesk.com/' ;;

Then in /usr/local/appi/entries/amd64/entries, inserted at line 86:

<item icon-name=\"rustdesk\">RustDesk|${FLGrustdesk}|83M|$(gettext 'Remote access and support')</item>

Then downloaded a 48x48 pixel logo, 'ruskdesk.png' and placed it at /usr/share/pixmaps

Decided to put it into the "System" category in Appi, as it was empty:

Haven't tested it yet.   

Tags: easy

Fix for multiple sfs layers in main desktop

The previous two posts described the problem:

• SFS loading order error — July 30, 2026
• Partial fix for devx sfs loaded on main desktop — July 30, 2026

Nailing it down more, have inserted this into /sbin/fixlayers in the initrd:

#20260730 sync apt layers. see /usr/local/petget/update_dbs.sh, alias/apt, alias/apt-get
if [ -z "$prevVER" ];then #prevVER set if a version update.
 #sfs layers change.
 mkdir -p /mnt/${WKG_DEV}/${WKG_DIR}.session/session1/var/local/petget
 echo -n '0.0.0' > /mnt/${WKG_DEV}/${WKG_DIR}.session/session1/var/local/petget/dpkg-sync-layers-ver
fi

There is code in /usr/local/petget/update_dbs.sh, alias/apt and alias/apt-get, that runs 'sync-dpkg-layers':

#20250821 sync dpkg db layers...
#20260730 see initrd /sbin/fixlayers #20260730
#also run from /usr/local/petget/update_dbs.sh and alias/apt
mkdir -p /var/local/petget
if [ ! -e /var/local/petget/dpkg-sync-layers-ver ];then
 echo -n "${DISTRO_VERSION}" > /var/local/petget/dpkg-sync-layers-ver
else
 V1="$(cat /var/local/petget/dpkg-sync-layers-ver)"
 if [ "${V1}" != "${DISTRO_VERSION}" ];then
  echo -n "${DISTRO_VERSION}" > /var/local/petget/dpkg-sync-layers-ver
  /usr/local/petget/alias/sync-dpkg-layers
 fi
fi

Looks like loading devx sfs on the main desktop is now fixed, but will have to build the next version of EasyOS to verify.   

Tags: easy

SFS loading order error

I am amazed that we keep finding new things to fix, where we think that area is OK. Posted about a problem with loading the devx sfs on the main desktop:

https://bkhome.org/news/202607/partial-fix-for-devx-sfs-loaded-on-main-desktop.html

This lead me to a startling discovery; the easy.sfs and devx.sfs layers were loaded in the overlayfs, but the devx.sfs was on the bottom! That is not supposed to happen; extra sfs's are supposed to be stacked on top of easy.sfs.

They do stack correctly in containers, and that is where my attention has been. Have neglected loading of extra sfs's on the main desktop. The cause goes back to when EasyOS adopted overlayfs, and dropped aufs.

The overlayfs mount has a format like this:

mount -t overlay -olowerdir=/lower1:/lower2:/lower3,upperdir=/upper

...where /lower1 is top of the read-only layers, /lower3 the lowest.

I had it the wrong way round, probably from "aufs thinking". Have fixed these files in the initrd:

/sbin/extrasfs
/inc/layer-lockdown, layer3, layer2, layer1, layer0

In many cases, an sfs stacked below easy.sfs will still work, but in the case of the devx sfs, it mucks up apt. Now on the track to properly fixing the problem that forum member retiredt00 reported.   

Tags: easy

Partial fix for devx sfs loaded on main desktop

Forum member retiredt00 has drawn my attention to this problem:

https://forum.puppylinux.com/viewtopic.php?t=17248

I booted up Easy on a usb-stick and ran SFSget to installed the devx sfs on the main desktop, not in a container. the window has some out-of-date information:

...that is very old information: "NOTE: It is currently recommended to install to the main filesystem, NOT in a container"

I have changed that text to "NOTE: it is not necessary to install it with SFSget; just click on the devx icon on the desktop and it will be downloaded and installed into the devx container -- this is now the preferred method."

Yes, loading the devx sfs in the devx container is now the recommended method. See documentation:

https://easyos.org/dev/how-to-compile-source-code.html

Keeping coding and development separate from the main desktop is a vast improvement. If coming from Puppy Linux, please do adjust to this!

However, if the user does want to load the devx sfs on the main desktop, it should still work. Currently, the packages in the devx sfs are not recognized by APT.

I have modified /usr/local/petget/alias/sync-dpkg-layers, modified at line 106:

##############
#handle NEWstatus...
DEVXloaded=0
#20250822 careful, devx sfs may be loaded...
if [ -f /INSIDE_devx ];then
 DEVXloaded=1
fi
#20260730 devx sfs loaded on main desktop...
losetup --all --output BACK-FILE -n | grep -q -F 'devx.sfs'
if [ $? -eq 0 ];then
 DEVXloaded=1
fi
for aNEW in ${NEWstatus}
do
 [ -z "$aNEW" ] && continue
 grep -q -F "|${aNEW}|" /root/.packages/user-installed-packages
 [ $? -eq 0 ] && continue
 #20250822 careful, devx sfs may be loaded...
 if [ $DEVXloaded -eq 1 ];then
  grep -q -F "|${aNEW}|" /root/.packages/devx-only-installed-packages
  [ $? -eq 0 ] && continue
 fi
 #want to purge it from db, as no longer exists...
 sed -i "/^Package: ${aNEW}$/,/^$/d" /var/lib/dpkg/status
 rm -f /var/lib/dpkg/info/${aNEW}:* 2>/dev/null
 rm -f /var/lib/dpkg/info/${aNEW}.* 2>/dev/null
done

However, that is only part of the story. Need to keep thinking about it...   

Tags: easy