'snapmergepuppy' improved

Script 'snapmergepuppy' is used to save a session running in RAM to the save-file in the physical media, example Flash drive.

Dougal, Jemimah and others have been investigating various fixes and tweaks:
http://murga-linux.com/puppy/viewtopic.php?t=64570

I have applied some of these, mindful that I want the script to also work with unionfs and older "unionfs compatible" aufs. My /usr/sbin/snapmergepuppy in Woof:

#!/bin/ash

#2007 Lesser GPL licence v2 (http://www.fsf.org/licensing/licenses/lgpl.html)
#Barry Kauler www.puppylinux.com
#called from savepuppyd and rc.shutdown, to save tmpfs layer to permanent flash storage.
#updated 13 Sept 2007. copy-down only. flushing is done in petget.
#updated 24 Sept 2007. removed '-u' option when copy-down files.
#8 oct 2007: screen out /root/tmp when saving.
#4 nov 2007: reintroduce '-u' copy option.
#v4.01 19may2008 BK: now called from /sbin/pup_eventd daemon (savepuppyd now history).
#v4.01 19may2008 BK: if called from pup_eventd then X running: graceful exit if X shutdown.
#v409 BK: XRUNNING test should be 'pup_event_frontend_d'.
#v409 BK: save /var dir. previously screened out to save space, but prevents crontab from running.
#v412 /etc/DISTRO_SPECS, renamed pup_xxx.sfs, pup_save.2fs etc.
#w000 pup files renamed to woofr555.sfs, woofsave.2fs.
#w003 bug fix for .wh.__dir_opaque whiteout file.
#w003 screened out some dirs in /dev that should not be saved.
#v424 fix for more layers in unionfs/aufs.
#100222 shinobar: possible timezone problem with BOOTCONFIG. more file exclusions.
#100422 added ^root/ftpd exclusion.
#100429 modify 'trash' exclusion.
#100820 added /var/tmp exclusion (apparently filled by opera crashes).
#101221 yaf-splash fix.
#110206 Dougal: clean up some testing. speedup: LANG=C, also change to /bin/ash.
#110212 recent aufs: .wh.__dir_opaque name changed to .wh..wh..opq.
#110212 Jemimah: files may disappear, more efficient calc of free space, fix i/o err.

export LANG=C #110206 Dougal: I **think** this should not cause problems with filenames

#variables created at bootup by 'init' script in initramfs...
. /etc/rc.d/PUPSTATE
. /etc/DISTRO_SPECS #v412
SAVEPART="`echo -n "$PUPSAVE" | cut -f 1 -d ','`"

#v3.02 first time, do not use '-u' option when saving, save everything...
NEXTPASSTHRU=""
[ -e /tmp/flagnextpassthru ] && NEXTPASSTHRU="yes"

RUNPS="`ps`"
SHUTDOWN="no"
XRUNNING="no"
[ "`echo "$RUNPS" | grep 'rc.shutdown'`" != "" ] && SHUTDOWN="yes"
[ "`echo "$RUNPS" | grep 'pup_event_frontend_d'`" != "" ] && XRUNNING="yes" #v4.01 v409

PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R7/bin"
WD="`pwd`"

SNAP="/initrd/pup_rw"
cd $SNAP || exit 1

if [ "$SAVE_LAYER" ];then #defined in PUPSTATE
BASE="/initrd${SAVE_LAYER}"
else
BASE="/initrd/pup_ro1"
fi

echo "Merging $SNAP onto $BASE..."

SFSPoints=$( grep -o "/initrd/pup_ro[1-9]" /proc/mounts |sort -u ) #110206 Dougal: get a list of the sfs mountpoints

#Handle Whiteouts...
find . -mount \( -regex '.*/\.wh\.[^/]*' -type f \) | sed -e 's/\.\///;s/\.wh\.//' |
while read N
do
BN="`basename "$N"`"
DN="`dirname "$N"`"
[ "$BN" = ".wh.aufs" ] && continue #w003 aufs has file .wh..wh.aufs in /initrd/pup_rw.
[ "$DN" = "." ] && continue
#110212 unionfs and early aufs: '.wh.__dir_opaque' marks ignore all contents in lower layers...
if [ "$BN" = "__dir_opaque" ];then #w003
#'.wh.__dir_opaque' marks ignore all contents in lower layers...
rm -rf "${BASE}/${DN}" 2>/dev/null #wipe anything in save layer. 110212 delete entire dir.
mkdir "${BASE}/${DN}" #jemimah: files sometimes mysteriously reappear if you don't delete and recreate the directory, aufs bug?
#also need to save the whiteout file to block all lower layers (may be readonly)...
touch "${BASE}/${DN}/.wh.__dir_opaque" 2>/dev/null
rm -f "$SNAP/$DN/.wh.__dir_opaque" #should force aufs layer "reval".
continue
fi
#110212 recent aufs: .wh.__dir_opaque name changed to .wh..wh..opq ...
if [ "$BN" = ".wh..opq" ] ; then
rm -rf "${BASE}/${DN}" 2>/dev/null #wipe anything in save layer.
mkdir "${BASE}/${DN}" #jemimah: files sometimes mysteriously reappear if you don't delete and recreate the directory, aufs bug?
#also need to save the whiteout file to block all lower layers (may be readonly)...
touch "${BASE}/${DN}/.wh..wh..opq" 2>/dev/null
rm -f "$SNAP/$DN/.wh..wh..opq" #should force aufs layer "reval".
continue
fi
#comes in here with the '.wh.' prefix stripped off, leaving actual filename...
rm -rf "$BASE/$N"
#if file exists on a lower layer, have to save the whiteout file...
#110206 Dougal: speedup and refine the search...
for P in $SFSPoints
do
if [ -e "$P/$N" ] ; then
[ ! -d "${BASE}/${DN}" ] && mkdir -p "${BASE}/${DN}"
touch "${BASE}/${DN}/.wh.${BN}"
break
fi
done #110206 End Dougal.
rm -f "$SNAP/$DN/.wh.$BN" #remove whiteout file. should force aufs layer "reval".
done

#Directories... v409 remove '^var'. w003 remove aufs .wh. dirs.
#w003 /dev/.udev also needs to be screened out... 100820 added var/tmp
find . -mount -type d | tail +2 | sed -e 's/\.\///' | grep -v -E '^mnt|^initrd|^proc|^sys|^tmp|^root/tmp|^\.wh\.|/\.wh\.|^dev/\.|^dev/fd|^dev/pts|^dev/shm|^var/tmp' |
while read N
do
#v4.01 graceful exit if shutdown X (see /usr/X11R7/bin/restartwm,wmreboot,wmpoweroff)...
[ "$XRUNNING" = "yes" ] && [ -f /tmp/wmexitmode.txt ] && exit
mkdir -p "$BASE/$N"
chmod "$BASE/$N" --reference="$N"
OWNER="`stat --format=%U "$N"`"
chown $OWNER "$BASE/$N"
GRP="`stat --format=%G "$N"`"
chgrp $GRP "$BASE/$N"
touch "$BASE/$N" --reference="$N"
done

#100222 a quick hack: BOOTCONFIG written to in init, before timezone set, can cause trouble...
touch /etc/rc.d/BOOTCONFIG
FREEBASEK=`df -k | grep "$BASE" | tr -s ' ' | cut -f 4 -d ' '` #110212 Jemimah

#Copy Files... v409 remove '^var'. w003 screen out some /dev files. 100222 shinobar: more exclusions. 100422 added ^root/ftpd. 100429 modify 'trash' exclusion. 100820 added var/tmp
find . -mount -not \( -regex '.*/\.wh\.[^/]*' -type f \) -not -type d | sed -e 's/\.\///' | grep -v -E '^mnt|^initrd|^proc|^sys|^tmp|^pup_|^zdrv_|^root/tmp|_zdrv_|^dev/\.|^dev/fd|^dev/pts|^dev/shm|^\.wh\.|^var/run|^root/ftpd|^var/tmp' | grep -v -E -i '\.thumbnails|\.trash|trash/|\.part$' |
while read N
do

#v4.01 graceful exit if shutdown X (see /usr/X11R7/bin/restartwm,wmreboot,wmpoweroff)...
[ "$XRUNNING" = "yes" ] && [ -f /tmp/wmexitmode.txt ] && exit

if [ "$SAVEPART" = "fd0" ];then
#not much space, so screen out more files...
[ "`echo -n "$N" | grep -v -E '^etc|^root'`" != "" ] && continue
[ "`echo -n "$N" | grep -v -E 'mozilla|printjobs|archive|cache'`" = "" ] && continue
fi

#stop saving if not enough room left in ${DISTRO_FILE_PREFIX}save file...
FILESIZEB=`stat --format=%s "/${N}"`
[ ! $FILESIZEB ] && break #w003 precaution.
FILESIZEK=$((FILESIZEB/1024))
FILESIZEK=$((FILESIZEK+200)) #200K slack space.
if [ $FILESIZEK -gt $FREEBASEK ];then
FREEBASEK=`df -k | grep "$BASE" | tr -s ' ' | cut -f 4 -d ' '` #110212 Jemimah: this is very slow; try not to check every iteration
if [ $FILESIZEK -gt $FREEBASEK ];then #110212 Jemimah.
#also the taskbar freemem-applet lets the user know when low.
if [ "$SHUTDOWN" = "no" -a "$XRUNNING" = "yes" ];then
export DISPLAY=':0'
yaf-splash -timeout 4 -bg red -text "WARNING!
Unable to save all files. You need to delete some." &
fi
break
fi
else
FREEBASEK=$((FREEBASEK-FILESIZEK)) #110212 Jemimah: keep track of the worst case
fi

##v2.21 -u causes trouble if timezone malconfigured...
##[ -L "$BASE/$N" ] && rm -f "$BASE/$N"
##cp -a -u -f "$N" "$BASE/$N"
#cp -a --remove-destination "$N" "$BASE/$N"
[ -L "$BASE/$N" ] && rm -f "$BASE/$N"
if [ -L "$N" -o "$NEXTPASSTHRU" = "" ];then
#link, or first run of snapmergepuppy. no '-u' (update) option.
cp -a -f "$N" "$BASE/$N"
else
cp -a -u -f "$N" "$BASE/$N" #v3.02
fi

[ -e "$BASE/$DN/.wh.${BN}" ] && rm "$BASE/$DN/.wh.${BN}" #110212 jemimah bugfix - I/O errors if you don't do this

#if [ "$NEXTPASSTHRU" != "" ];then
# cp -a -u -f "$N" "$BASE/$N"
#else
# cp -a -f "$N" "$BASE/$N" #no update option.
#fi
#flock -x -n "$N" -c rm -f "$N" #remove if file not in use

done

touch /tmp/flagnextpassthru

sync
cd "$WD"
exit 0

###END###


Note that my script calls the "new" yaf-splash, based on gtkdialog. You would need to hardcode the path /usr/X11R7/bin/yaf-splash to use the old one, plus extra commandline parameters are needed.

I have also modified the code in the 'init' script in the initrd to handle the alternate names .wh.__dir_opaque or .wh..wh..opq.


Posted on 12 Feb 2011, 17:51


Comments:

Posted on 13 Feb 2011, 13:52 by shinobar
Much faster!
Testing the improved snapmergepuppy and I feel it much faster!
In these two days,no problem found so far.
Thanks.


Posted on 23 Feb 2011, 13:59 by shinobar
snapmergepuppy patch
1. I wonder why we need to save under /dev. They will be created after boot with udev architech.
I am not sure it was the case, my one laptop failed in sound at a time, but it recovered by removing all /dev and reboot.
I think most of /dev also need not in the main sfs.
2. No problem in wary but serious for some puppy having pup_ro10-19.
3. It is better to give an exit code when saving failed. Also the rc.shutdown needs to modify to use it.
I posted a patch
http://www.murga-linux.com/puppy/viewtopic.php?p=498002#498002