site  contact  subhomenews

Aufs: direct write to Flash

April 26, 2010 — BarryK
When running off a Flash drive, Puppy runs with the top layer of Unionfs or Aufs as a tmpfs in RAM, with the 'pupsave' file on the Flash drive mounted on the layer underneath. Then there are periodic (or manual via a 'save' icon in the desktop) saves from the tmpfs to the pupsave file -- this is done to minimise writes to Flash memory and extend it's life.

An optimisation with Unionfs is that when a package is installed, it is written direct to the 'pupsave' file, however this did not work properly with Aufs -- some files written directly to a lower layer were not visible "on top".

However, I recently discussed this with J. R. Okajima, the founder and main developer of Aufs, reported on my blog:
http://bkhome.org/archive/blog2/201003/quotwhy-unionfs-is-better-than-aufsquot.html

A couple of solutions were suggested. I have implemented one of these and tested it -- it works well. The modified script is /usr/local/petget/installpkg.sh:

#!/bin/sh

#(c) Copyright Barry Kauler 2009, puppylinux.com
#2009 Lesser GPL licence v2 (http://www.fsf.org/licensing/licenses/lgpl.html).
#called from /usr/local/petget/downloadpkgs.sh and petget.
#passed param is the path and name of the downloaded package.
#/tmp/petget_missing_dbentries-Packages-* has database entries for the set of pkgs being downloaded.
#w456 warning: petget may write to /tmp/petget_missing_dbentries-Packages-alien with missing fields.
#w478, w482 fix for pkg menu categories.
#w482 detect zero-byte pet.specs, fix typo.
#100110 add support for T2 .tar.bz2 binary packages.
#100426 aufs can now write direct to save layer.

#information from 'labrador', to expand a .pet directly to '/':
#NAME="a52dec-0.7.4"
#pet2tgz "${NAME}.pet"
#tar -C / --transform 's/^(./)?'"$NAME"'//g' -zxf "${NAME}.tar.gz"
#i found this also works:
#tar -z -x --strip=1 --directory=/ -f bluefish-1.0.7.tar.gz
#v424 .pet pkgs may have post-uninstall script, puninstall.sh

export LANG=C
. /etc/rc.d/PUPSTATE #this has PUPMODE and SAVE_LAYER.
. /etc/DISTRO_SPECS #has DISTRO_BINARY_COMPAT, DISTRO_COMPAT_VERSION

. /etc/xdg/menus/hierarchy #w478 has PUPHIERARCHY variable.

DLPKG="$1"
DLPKG_BASE="`basename $DLPKG`" #ex: scite-1.77-i686-2as.tgz
DLPKG_PATH="`dirname $DLPKG`" #ex: /root

#get the pkg name ex: scite-1.77 ...
dbPATTERN='|'"$DLPKG_BASE"'|'
DLPKG_NAME="`cat /tmp/petget_missing_dbentries-Packages-* | grep "$dbPATTERN" | head -n 1 | cut -f 1 -d '|'`"

#boot from flash: bypass tmpfs top layer, install direct to pup_save file...
DIRECTSAVEPATH=""
if [ $PUPMODE -eq 3 -o $PUPMODE -eq 7 -o $PUPMODE -eq 13 ];then
FLAGNODIRECT=1
[ "`lsmod | grep '^unionfs' `" != "" ] && FLAGNODIRECT=0
#100426 aufs can now write direct to save layer...
if [ "`lsmod | grep '^aufs' `" != "" ];then
#note: fsnotify now preferred not inotify, udba=notify uses whichever is enabled in module...
busybox mount -t aufs -o remount,udba=notify unionfs / #remount aufs with best evaluation mode.
FLAGNODIRECT=$?
[ $FLAGNODIRECT -ne 0 ] && logger -s -t "installpkg.sh" "Failed to remount aufs / with udba=notify"
fi
if [ $FLAGNODIRECT -eq 0 ];then
#note that /sbin/pup_event_frontend_d will not run snapmergepuppy if installpkg.sh or downloadpkgs.sh are running.
#if snapmergepuppy is running, wait until it has finished...
while [ "`pidof snapmergepuppy`" != "" ];do
sleep 1
done
DIRECTSAVEPATH="/initrd${SAVE_LAYER}"
fi
fi

cd $DLPKG_PATH

case $DLPKG_BASE in
*.pet)
DLPKG_MAIN="`basename $DLPKG_BASE .pet`"
pet2tgz $DLPKG_BASE
[ $? -ne 0 ] && exit 1
PETFILES="`tar --list -z -f ${DLPKG_MAIN}.tar.gz`"
#slackware pkg, got a case where passed the above test but failed here...
[ $? -ne 0 ] && exit 1
if [ "`echo "$PETFILES" | grep '^\./'`" != "" ];then
#ttuuxx has created some pets with './' prefix...
pPATTERN="s%^\./${DLPKG_NAME}%%"
echo "$PETFILES" | sed -e "$pPATTERN" > /root/.packages/${DLPKG_NAME}.files
tar -z -x --strip=2 --directory=${DIRECTSAVEPATH}/ -f ${DLPKG_MAIN}.tar.gz
else
#new2dir and tgz2pet creates them this way...
pPATTERN="s%^${DLPKG_NAME}%%"
echo "$PETFILES" | sed -e "$pPATTERN" > /root/.packages/${DLPKG_NAME}.files
tar -z -x --strip=1 --directory=${DIRECTSAVEPATH}/ -f ${DLPKG_MAIN}.tar.gz
fi
;;
*.deb)
DLPKG_MAIN="`basename $DLPKG_BASE .deb`"
PFILES="`dpkg-deb --contents $DLPKG_BASE | tr -s ' ' | cut -f 6 -d ' '`"
[ $? -ne 0 ] && exit 1
echo "$PFILES" > /root/.packages/${DLPKG_NAME}.files
dpkg-deb -x $DLPKG_BASE ${DIRECTSAVEPATH}/
if [ $? -ne 0 ];then
rm -f /root/.packages/${DLPKG_NAME}.files
exit 1
fi
;;
*.tgz)
DLPKG_MAIN="`basename $DLPKG_BASE .tgz`" #ex: scite-1.77-i686-2as
gzip --test $DLPKG_BASE > /dev/null 2>&1
[ $? -ne 0 ] && exit 1
PFILES="`tar --list -z -f $DLPKG_BASE`"
#hmmm, got a case where passed the above test but failed here...
[ $? -ne 0 ] && exit 1
echo "$PFILES" > /root/.packages/${DLPKG_NAME}.files
tar -z -x --directory=${DIRECTSAVEPATH}/ -f $DLPKG_BASE
;;
*.tar.gz)
DLPKG_MAIN="`basename $DLPKG_BASE .tar.gz`" #ex: acl-2.2.47-1-i686.pkg
gzip --test $DLPKG_BASE > /dev/null 2>&1
[ $? -ne 0 ] && exit 1
PFILES="`tar --list -z -f $DLPKG_BASE`"
[ $? -ne 0 ] && exit 1
echo "$PFILES" > /root/.packages/${DLPKG_NAME}.files
tar -z -x --directory=${DIRECTSAVEPATH}/ -f $DLPKG_BASE
;;
*.tar.bz2) #100110
DLPKG_MAIN="`basename $DLPKG_BASE .tar.bz2`"
bzip2 --test $DLPKG_BASE > /dev/null 2>&1
[ $? -ne 0 ] && exit 1
PFILES="`tar --list -j -f $DLPKG_BASE`"
[ $? -ne 0 ] && exit 1
echo "$PFILES" > /root/.packages/${DLPKG_NAME}.files
tar -j -x --directory=${DIRECTSAVEPATH}/ -f $DLPKG_BASE
;;
esac

rm -f $DLPKG_BASE 2>/dev/null
rm -f $DLPKG_MAIN.tar.gz 2>/dev/null

#pkgname.files may need to be fixed...
FIXEDFILES="`cat /root/.packages/${DLPKG_NAME}.files | grep -v '^\./$'| grep -v '^/$' | sed -e 's%^\.%%' -e 's%^%/%' -e 's%^//%/%'`"
echo "$FIXEDFILES" > /root/.packages/${DLPKG_NAME}.files

#flush unionfs cache, so files in pup_save layer will appear "on top"...
if [ "$DIRECTSAVEPATH" != "" ];then
#but first, clean out any bad whiteout files...
find /initrd/pup_rw -mount -type f -name .wh.* |
while read ONEWHITEOUT
do
ONEWHITEOUTFILE="`basename "$ONEWHITEOUT"`"
ONEWHITEOUTPATH="`dirname "$ONEWHITEOUT"`"
if [ "$ONEWHITEOUTFILE" = ".wh.__dir_opaque" ];then
[ "`grep "$ONEWHITEOUTPATH" /root/.packages/${DLPKG_NAME}.files`" != "" ] && rm -f "$ONEWHITEOUT"
continue
fi
ONEPATTERN="`echo -n "$ONEWHITEOUT" | sed -e 's%/\.wh\.%/%'`"'$'
[ "`grep "$ONEPATTERN" /root/.packages/${DLPKG_NAME}.files`" != "" ] && rm -f "$ONEWHITEOUT"
done
#now re-evaluate all the layers...
if [ "`lsmod | grep '^aufs' `" != "" ];then #100426
busybox mount -t aufs -o remount,udba=reval unionfs / #remount with faster evaluation mode.
[ $? -ne 0 ] && logger -s -t "installpkg.sh" "Failed to remount aufs / with udba=reval"
else
mount -t unionfs -o remount,incgen unionfs /
fi
sync
fi

#some .pet pkgs have images at '/'...
mv /*24.xpm /usr/local/lib/X11/pixmaps/ 2>/dev/null
mv /*32.xpm /usr/local/lib/X11/pixmaps/ 2>/dev/null
mv /*32.png /usr/local/lib/X11/pixmaps/ 2>/dev/null
mv /*48.xpm /usr/local/lib/X11/pixmaps/ 2>/dev/null
mv /*48.png /usr/local/lib/X11/pixmaps/ 2>/dev/null
mv /*.xpm /usr/local/lib/X11/mini-icons/ 2>/dev/null
mv /*.png /usr/local/lib/X11/mini-icons/ 2>/dev/null

#post-install script?...
if [ -f /pinstall.sh ];then #pet pkgs.
chmod +x /pinstall.sh
cd /
sh /pinstall.sh
rm -f /pinstall.sh
fi
if [ -f /install/doinst.sh ];then #slackware pkgs.
chmod +x /install/doinst.sh
cd /
sh /install/doinst.sh
rm -rf /install
fi

#v424 .pet pkgs may have a post-uninstall script...
if [ -f /puninstall.sh ];then
mv -f /puninstall.sh /root/.packages/${DLPKG_NAME}.remove
fi

#w465 <pkgname>.pet.specs is in older pet pkgs, just dump it...
#maybe a '$APKGNAME.pet.specs' file created by dir2pet script...
rm -f /*.pet.specs 2>/dev/null
#...note, this has a setting to prevent .files and entry in user-installed-packages, so install not registered.

#add entry to /root/.packages/user-installed-packages...
#w465 a pet pkg may have /pet.specs which has a db entry...
if [ -f /pet.specs -a -s /pet.specs ];then #w482 ignore zero-byte file.
DB_ENTRY="`cat /pet.specs | head -n 1`"
rm -f /pet.specs
else
[ -f /pet.specs ] && rm -f /pet.specs #w482 remove zero-byte file.
dlPATTERN='|'"`echo -n "$DLPKG_BASE" | sed -e 's%\-%\\-%'`"'|'
DB_ENTRY="`cat /tmp/petget_missing_dbentries-Packages-* | grep "$dlPATTERN" | head -n 1`"
fi

#see if a .desktop file was installed, fix category...
ONEDOT=""
DEFICON='Executable.xpm'
CATEGORY="`echo -n "$DB_ENTRY" | cut -f 5 -d '|'`"
#fix category so finds a place in menu... (same code in woof 2createpackages)
#i think allow field5 to specify a sub-category, 'category;subcategory'
#if no sub-cat then it will come into one of these cases:
case $CATEGORY in
Desktop) CATEGORY='Desktop;X-Desktop' ; DEFICON='mini.window3d.xpm' ;;
System) CATEGORY='System' ; DEFICON='mini-term.xpm' ;;
Setup) CATEGORY='Setup;X-SetupEntry' ; DEFICON='so.xpm' ;;
Utility) CATEGORY='Utility' ; DEFICON='mini-hammer.xpm' ;;
Filesystem) CATEGORY='Filesystem;FileSystem' ; DEFICON='mini-filemgr.xpm' ;;
Graphic) CATEGORY='Graphic;Presentation' ; DEFICON='image_2.xpm' ;;
Document) CATEGORY='Document;X-Document' ; DEFICON='mini-doc1.xpm' ;;
Calculate) CATEGORY='Calculate;X-Calculate' ; DEFICON='mini-calc.xpm' ;;
Personal) CATEGORY='Personal;X-Personal' ; DEFICON='mini-book2.xpm' ;;
Network) CATEGORY='Network' ; DEFICON='pc-2x.xpm' ;;
Internet) CATEGORY='Internet;X-Internet' ; DEFICON='pc2www.xpm' ;;
Multimedia) CATEGORY='Multimedia;AudioVideo' ; DEFICON='Animation.xpm' ;;
Fun) CATEGORY='Fun;Game' ; DEFICON='mini-maze.xpm' ;;
Develop) CATEGORY='Utility' ; DEFICON='mini-hex.xpm' ;;
Help) CATEGORY='Utility' ; DEFICON='info16.xpm' ;;
*) CATEGORY='BuildingBlock' ; DEFICON='Executable.xpm' ;; #w482
esac
cPATTERN="s%^Categories=.*%Categories=${CATEGORY}%"
iPATTERN="s%^Icon=.*%Icon=${DEFICON}%"
for ONEDOT in `grep '.desktop$' /root/.packages/${DLPKG_NAME}.files`
do
#w478 find if category is already valid...
CATFOUND="no"
for ONEORIGCAT in `cat $ONEDOT | grep '^Categories=' | head -n 1 | cut -f 2 -d '=' | tr ';' ' '`
do
oocPATTERN=' '"$ONEORIGCAT"' '
[ "`echo "$PUPHIERARCHY" | tr -s ' ' | cut -f 3 -d ' ' | tr ',' ' ' | sed -e 's%^% %' -e 's%$% %' | grep "$oocPATTERN"`" != "" ] && CATFOUND="yes"
done
if [ "$CATFOUND" = "no" ];then
sed -e "$cPATTERN" $ONEDOT > /tmp/petget_category
mv -f /tmp/petget_category $ONEDOT
else
CATEGORY="`echo -n "$ONEORIGCAT" | rev | cut -f 1 -d ' ' | rev`" #w482
fi
#w019 does the icon exist?...
ICON="`grep '^Icon=' $ONEDOT | cut -f 2 -d '='`"
if [ "$ICON" != "" ];then
[ -e $ICON ] && continue #it may have a hardcoded path.
[ "`find /usr/local/lib/X11 /usr/share/icons /usr/share/pixmaps /usr/local/share/pixmaps -name $ICON -o -name $ICON.png -o -name $ICON.xpm -o -name $ICON.jpg 2>/dev/null`" != "" ] && continue
#substitute a default icon...
sed -e "$iPATTERN" $ONEDOT > /tmp/petget-installpkg-tmp
mv -f /tmp/petget-installpkg-tmp $ONEDOT
fi
done

#due to images at / in .pet and post-install script, .files may have some invalid entries...
INSTFILES="`cat /root/.packages/${DLPKG_NAME}.files`"
echo "$INSTFILES" |
while read ONEFILE
do
if [ ! -e "$ONEFILE" ];then
ofPATTERN='^'"$ONEFILE"'$'
grep -v "$ofPATTERN" /root/.packages/${DLPKG_NAME}.files > /tmp/petget_instfiles
mv -f /tmp/petget_instfiles /root/.packages/${DLPKG_NAME}.files
fi
done

#w482 DB_ENTRY may be missing DB_category and DB_description fields...
#pkgname|nameonly|version|pkgrelease|category|size|path|fullfilename|dependencies|description|
#optionally on the end: compileddistro|compiledrelease|repo| (fields 11,12,13)
DESKTOPFILE="`grep '.desktop$' /root/.packages/${DLPKG_NAME}.files | head -n 1`"
if [ "$DESKTOPFILE" != "" ];then
DB_category="`echo -n "$DB_ENTRY" | cut -f 5 -d '|'`"
DB_description="`echo -n "$DB_ENTRY" | cut -f 10 -d '|'`"
CATEGORY="$DB_category"
DESCRIPTION="$DB_description"
if [ "$DB_category" = "" ];then
CATEGORY="`cat $DESKTOPFILE | grep '^Categories=' | cut -f 2 -d '=' | rev | cut -f 1 -d ';' | rev`"
#v424 but want the top-level menu category...
catPATTERN="[ ,]${CATEGORY},|[ ,]${CATEGORY}"'$'
CATEGORY="`grep -E "$catPATTERN" /etc/xdg/menus/hierarchy | grep ':' | cut -f 1 -d ' ' | head -n 1`"
fi
if [ "$DB_description" = "" ];then
DESCRIPTION="`cat $DESKTOPFILE | grep '^Comment=' | cut -f 2 -d '='`"
[ "$DESCRIPTION" = "" ] && DESCRIPTION="`cat $DESKTOPFILE | | grep '^Name=' | cut -f 2 -d '='`"
fi
if [ "$DB_category" = "" -o "$DB_description" = "" ];then
newDB_ENTRY="`echo -n "$DB_ENTRY" | cut -f 1-4 -d '|'`"
newDB_ENTRY="$newDB_ENTRY"'|'"$CATEGORY"'|'
newDB_ENTRY="$newDB_ENTRY""`echo -n "$DB_ENTRY" | cut -f 6-9 -d '|'`"
newDB_ENTRY="$newDB_ENTRY"'|'"$DESCRIPTION"'|'
newDB_ENTRY="$newDB_ENTRY""`echo -n "$DB_ENTRY" | cut -f 11-14 -d '|'`"
#[ "`echo -n "newDB_ENTRY" | rev | cut -c 1`" != "|" ] && newDB_ENTRY="$newDB_ENTRY"'|'
DB_ENTRY="$newDB_ENTRY"
fi
fi

echo "$DB_ENTRY" >> /root/.packages/user-installed-packages

#announcement of successful install...
#announcement is done after all downloads, in downloadpkgs.sh...
CATEGORY="`echo -n "$CATEGORY" | cut -f 1 -d ';'`"
[ "$CATEGORY" = "" ] && CATEGORY="none"
[ "$CATEGORY" = "BuildingBlock" ] && CATEGORY="none"
echo "PACKAGE: $DLPKG_NAME CATEGORY: $CATEGORY" >> /tmp/petget-installed-pkgs-log

###END###


This is good news. It means that if you have limited RAM, installing lots of packages will go direct to the 'pupsave' file and not fill up the RAM. This improvement will be in the next release of Woof.

Comments

Puppy Package Manager
Username: 8-bit
When I install a PET, at the end of installation, fixmenus is run and a restart of the window manager. But... If I use Puppy Package Manager to uninstall/remove a PET, fixmenus and a WM restart is not done. Am I using an old version? Username: 26 Apr 2010, 10:35
"01534"70.56.26.205'trouble"Jemimah"I tried this script in Puppeee - the kernel is built from your 2.6.33.2 source. When I install a pet, new directories will show up, but files in existing directories don't seem to appear until after a reboot. Did I configure AUFS wrong? Do I need to update something? Username: 5 May 2010, 9:29
"01534"71.166.200.104'Aufs"BarryK"When you configure the kernel, Aufs has various options that need to be turned on. There is a 'notify' option, with sub-choice of 'inotify' or 'fsnotify' -- I chose the latter. When Aufs is remounted with 'udba=notify', whichever of those above two that were chosen, will be used. The 'notify' method gives the best re-evaluation when there are direct writes to a layer. For normal use, udba=reval is good enough and is faster. It should all be automatic. You write direct to a layer, Aufs recognises it and will update the "top layer". CONFIG_AUFS_FS=m CONFIG_AUFS_BRANCH_MAX_127=y # CONFIG_AUFS_BRANCH_MAX_511 is not set # CONFIG_AUFS_BRANCH_MAX_1023 is not set # CONFIG_AUFS_BRANCH_MAX_32767 is not set CONFIG_AUFS_HNOTIFY=y CONFIG_AUFS_HFSNOTIFY=y # CONFIG_AUFS_HINOTIFY is not set # CONFIG_AUFS_EXPORT is not set # CONFIG_AUFS_RDU is not set # CONFIG_AUFS_SP_IATTR is not set # CONFIG_AUFS_SHWH is not set CONFIG_AUFS_BR_RAMFS=y CONFIG_AUFS_BR_FUSE=y CONFIG_AUFS_POLL=y CONFIG_AUFS_BR_HFSPLUS=y CONFIG_AUFS_BDEV_LOOP=y # CONFIG_AUFS_DEBUG is not set Username: 5 May 2010, 10:01
"01534"114.129.167.148'symlinks"Jemimah"I will double check the notify option, though my config looks just like yours only I didn't build with HFS support, and I build AUFS into the kernel instead of a module (and adjust the scripts accordingly). On further investigation it appears only the symlink did not show up. I will play with it more."5 May 2010, 10:23"01534"71.166.200.104'eventually it shows up"Jemimah"About 5 or 10 minutes after the pet installs the symlink shows up, though trying to write to it makes it appear sooner. I guess I need to try more pets and see what's going on. I did build with fsnotify, so that's not it."5 May 2010, 10:45"01534"71.166.200.104'nevermind, maybe"Jemimah"Well it seems to be working ok on everything but that one seamonkey pet and now even that is working after wiping my save file. I guess I'm just going mad. :)"5 May 2010, 11:14"01534"72.60.164.187'

Tags: woof