Woof: Making 0setup faster

I reported recently on the super-fast code that Wosh has created:

http://puppylinux.com/blog/?viewDetailed=01167

Wosh sent me version 0.0.1 of PPKG today, and it is nice but not yet quite what I need for building Karmic Puppy.

To fill the immediate need, I thought that I would take another look at the Debian/Ubuntu-to-Puppy database conversion code in the '0setup' script.

Testing the Ubuntu "main" repository db, the script took about 11 minutes. I applied some lateral thinking, and have now got it down to about 2 minutes.

Here is the new code:
v1PATTERN="s%\\-[0-9.]*${DISTRO_BINARY_COMPAT}[0-9.]*$%%"

v2PATTERN="s%\\+[0-9.]*${DISTRO_BINARY_COMPAT}[0-9.]*$%%"
while read DB_ONELINE
do
eval $DB_ONELINE
case $DB_ONELINE in
Description*)
DB_fullfilename="`echo -n "$Filename" | rev | cut -f 1 -d '/' | rev`"
DB_path="`echo -n "$Filename" | rev | cut -f 2-9 -d '/' | rev`"
DB_version="`echo -n "$Version" | sed -e 's%^[0-9]:%%' -e "$v1PATTERN" -e "$v2PATTERN"`"
DB_pkgrelease="`echo -n "$Version" | sed -e 's%^[0-9]:%%' -e "s%${DB_version}%%" -e 's%^\\-%%' -e 's%^\\+%%'`"
DB_pkgname="${Package}_${DB_version}"
DB_category="`${FIND_CAT} $Package "$Description"`"
DB_dependencies="`echo -n "$Depends" | sed -e 's%, %\n%g' | cut -f 1 -d ' ' | tr '\n' ' ' | sed -e 's% $%%' -e 's% %,+%g' -e 's%,$%%'`"
[ "$DB_dependencies" != "" ] && DB_dependencies='+'"$DB_dependencies"
echo "$DB_pkgname|$Package|$DB_version|$DB_pkgrelease|$DB_category|${InstalledSize}K|$DB_path|$DB_fullfilename|$DB_dependencies|$Description|" >> $ONE_PKGLISTS_COMPAT
echo -n "$Package "
;;
esac
done<<_END1
$(grep -E '^Package:|^Installed\-Size:|^Architecture:|^Version:|^Depends:|^Filename:|^Description:' ${ONE_PKGLISTS_COMPAT}pre | tr '[\t"|]' ' ' | tr -s ' ' | sed -e 's%^Installed\-Size%InstalledSize%' -e 's%: %="%' -e 's% $%%' -e 's%$%"%')
_END1


Of course the big one is the "universe" repo. The database file to be processed is 24MB, a lot to churn through. Universe now takes 9 minutes, so all-up the conversion time will be about 12 minutes, apart from the time to download the Debian/Ubuntu db files.
The Debian repo is a bit smaller so conversion will be quicker.

This faster script is just a temporary fix, as I expect to move onto PPKG soon.


Posted on 23 Oct 2009, 23:30


Comments:

Posted on 24 Oct 2009, 5:28 by Sit Heel Speak
Viva Karmic Woof
heh...eagerly awaiting...please do take time to get it right...


Posted on 24 Oct 2009, 16:04 by BarryK
0setup faster
Oh yeah, one other thing. I got those faster speeds by also changing the first line to use the Ash interpreter:

#!/bin/ash

...makes a huge difference.