Faster deb pkg db to pup pkg db

Those running Precise Puppy 5.5 will have noticed that if they choose to update the local package database via the Configure window of the Puppy Package Manager, that it has become rather slow.

There was a huge increase in speed when I rewrote the conversion in BaCon (see /usr/local/petget/debdb2pupdb, or support/debdb2pupdb.bac in Woof), but for Precise 5.5 I introduced some extra checking for invalid dependencies, which slowed the conversion down somewhat.

Running Precise 5.5 on my laptop, with Intel i3 CPU and 4GB RAM, the Ubuntu 'universe' package database took spot-on 2 minutes to convert to Puppy-db-format. My laptop is modest by today's standards, but many of our users have older PCs, and were reporting 4 - 7 minutes conversion time.

Too long. So, I have revisited the code that detects invalid dependencies, and have made it much faster. Now, the 'universe' database converts in 70 seconds on my laptop ('main' in 16 seconds).

The changed files are 'debdb2pupdb.bac', '0setup' and '3builddistro':
http://bkhome.org/fossil/woof2.cgi/info/3b2710862b

For anyone who wants to know, these are the invalid dependencies found:
amd-libopencl1 fglrx-driver libcuda1 mplayer-nogui upstart-job xorg-video-abi-11

Invalid means that a package has a dependency that does not actually exist in the repository. This may be an error, or in some cases the name of the dependency has changed.

This is where packages are that reference these invalid deps:
amd-libopencl1 multiverse
fglrx-driver multiverse
libcuda1 multiverse
mplayer-nogui multiverse
upstart-job main, universe, multiverse
xorg-video-abi-11 main, universe, multiverse


Note, you cannot easily upgrade your Precise 5.5 to this faster conversion. Well, you can, but it involves some work.
You need to create a file /usr/local/petget/invaliddepslist, with a single line in it of the invalid deps, as per the list in italics above.
Then copy the x86 binary 'debdb2pupdb' and the script '0setup' to /usr/local/petget.


Posted on 17 Mar 2013, 8:50


Comments:

Posted on 18 Mar 2013, 3:12 by Terryphi
Manual upgrade failed
I followed instructions above but PPM manual upgrade not working for me. Deb processing produces this error for all Ubuntu repositories:

/usr/local/petget/0setup : line 544: /usr/local/petget/debdb2pupdb: Permission denied


Posted on 18 Mar 2013, 3:44 by Terryphi
PPM works!
Following on from my comment above. Making debdb2pupdb executable solved problem. Processing much faster now.


Posted on 18 Mar 2013, 6:56 by pemasu
findpkgs problem
New duplicate elimination gives headache.

yes|gettext|gettext-base,gettext|exe>dev,dev,doc,nls
yes|gettext|gettext-base,gettext|exe,dev>null,doc>null,nls>null

Both of them are eliminated and so...they are not downloaded or processed.
There are other duplicated rows in DISTRO_SPECS as you know.
This is with DISTRO_PKGS_SPECS-ubuntu-precise

I had to revert back to pre march findpkgs file. I hope it does not give problems with building.


Posted on 18 Mar 2013, 20:11 by BarryK
Duplicate generic names
pemasu,
More than just 'gettext':

./x86/ubuntu/precise/DISTRO_PKGS_SPECS-ubuntu-precise
gettext
gettext
perl-digest-sha1
perl-digest-sha1
perl-html-parser
perl-html-parser
rsync
rsync


I have changed all the package lists to avoid these duplicate names.

Woof commit:
http://bkhome.org/fossil/woof2.cgi/info/a925e7342a



Posted on 18 Mar 2013, 21:34 by darkcity
off topic
OpenSuse has another take on rolling release
http://en.opensuse.org/Portal:Tumbleweed


Posted on 18 Mar 2013, 22:48 by technosaurus
re: duplicate names
that is why a lot of package managers have a "provides" entry. for example libreoffice-gtk provides libreoffice (and others as does libreoffice-kde{4})

... in puppy's case this could extend to the parent distro libreoffice-gtk-debian provides libreoffice-gtk and libreoffice-gtk provides libreoffice,...

this concept could be extend to things like defaultbrowser,... (provided by firefox, seamonkey, opera,...)


Posted on 19 Mar 2013, 22:54 by pemasu
more findpkgs problems
Packages-puppy-*-official duplicate generic name problems. They inhibit the package processing also:

http://www.murga-linux.com/puppy/viewtopic.php?p=693043#693043


Posted on 20 Mar 2013, 4:44 by pemasu
e2fsprogs clash with util-linux
http://www.murga-linux.com/puppy/viewtopic.php?p=693068#693068


Posted on 20 Mar 2013, 8:32 by BarryK
re e2fsprogs vs util-linux
OK, I have commented out that section of code in 3builddistro:

http://bkhome.org/fossil/woof2.cgi/info/5b0b23fcc0



Posted on 20 Mar 2013, 8:47 by BarryK
re missing xdialog
pemasu,
I think that I see what the problem is.

For now, a quick solution is to append "pet:common" to the package-list:

yes|xdialog||exe,dev,doc,nls|pet:common