Very fast deb db to pup db

I reported yesterday about significant speed improvement when converting Ubuntu (or Debian, or Raspbian) package database to Puppy-format:
http://bkhome.org/archive/blog2/201211/fast-conversion-deb-to-pup-db.html

The earlier script took about half an hour, yesterday I got it down to 3 minutes and 40 seconds.

I determined that the call to 'find_cat' from 'debdb2pupdb' is the bottleneck. Actually, the main problem is not 'find_cat' itself, but the BaCon EXEC$() statement.

So, I decided to implement the functionality of 'find_cat' inside 'debdb2pupdb'. I created a separate category data file at /usr/local/petget/categories.dat, so that other to-pupdb converters can share it.
'debdb2pupdb' loads /usr/local/petget/categories.dat.

Well, the proof of the pudding is in the eating... I placed the new 'categories.dat' and 'debdb2pupdb' in /usr/local/petget (readers, please note previous blog post, advising update 0setup also). I then ran the PPM and chose to update the package databases...

All of the Ubuntu databases, 'main', 'universe', 'multiverse', 'updates-main', 'updates-universe' and 'updates-multiverse', took a total of 19 seconds!

Good enough I reckon!

Hmmm, probably the Slacko people would like this go-fast treatment too

Woof commit:
http://bkhome.org/fossil/woof2.cgi/info/4667bbfc88

---

Comments

Awesome. :-)


woof post update problems
Username: scsijon
"Freshly updated woof2 with fossil update this morning (0723ESST). A clean test build has the link in /usr/bin/wmpoweroff to /usr/X11R7/bin/wmpoweroff, however this file is just pointing back to the /usr/bin/wmpoweroff link again. Same thing with wmreboot, the others are ok. This wasn't happening pre this morning's update. Also getting quickset opening over the screen still.http://murga-linux.com/puppy/viewtopic.php?p=664361#664361 and the reply above refers. regards scsijon

Re wmpoweroff
Username: BarryK
"I just checked the Woof Fossil repo, those files are as they should be. It follows that something is wrong in your build system.

debdb2pupdb tweaked
Username: BarryK
"I have improved debdb2pupdb slightly, reading the Ubuntu/Debian "Section" parameter. Also, there was a bug in the PPM Classic-view -- the "Business" category window was empty (no packages listed). Woof commit: http://bkhome.org/fossil/woof2.cgi/info/a8a0826981

debdb2pupdb bug fix
Username: BarryK
"A couple of bugs fixed: Woof commit: http://bkhome.org/fossil/woof2.cgi/info/3f94df5186

wmpoweroff further on
Username: scsijon
"Barry, I have chased this down to the selfmodified /sandbox3/pinstall.sh script. I have pm's on murga-linux with details but it basically relates to a line which states. ln -snf /usr/X11R7/bin/$BASEFILE ./usr/bin/$BASEFILE regards

wmpoweroff further on
Username: scsijon
"just found it's in the pinstall.sh script for zz_t2_fixup. Does this mean it should not be used in a wary with the x_xorg76_mega_package?, even though it's not listed to be turned off?

re zz_t2_fixup
Username: BarryK
"scsijon, You should not be using that package, it is only for the pups that are built from PETs created from T2 packages, where X.org is in /usr/X11R7. All other distros do not put X.org in /usr/X11R7. I just checked the mageia package lists in Woof, that package is not selected. Um, I am thinking that you are building Mageia, which is what you have been doing until now, but I see in your post you refer to Wary. But, if you are building with x_xorg76_mega_package then you are building Racy, not Wary, and there are various other package selection requirements. The package selection for pristine Woof Racy works last time I did a build.

Re: re zz_t2_fixup 2
Username: scsijon
"sorry barry, this is a very cutdown wary build I have used in the past as a 'test' build to make sure everything still works after updating woof, but just with the xorg76 mega package added in lu of the xorg73 set. It's not one of my mageia builds. I will have a go using racy in the merge2out and see if it happens still. I did think after a night's sleep that maybe there needs to be a test to see if a file/link already exists with that name and only create a link if the answer is no. Taking the line out doesn't fix things as there are a number missing including X, I tried it.

woof stucks after these updates
Username: pemasu
"#fix the menus... #111123 ***NOTICE*** cross-build, will have to run fixmenus and indexgen.sh at first bootup. if [ "$WOOF_HOSTARCH" = "$WOOF_TARGETARCH" ];then #111123 echo echo "Constructing configuration files for JWM, Fvwm95, IceWM, openbox..." chroot rootfs-complete /usr/sbin/fixmenus #generate help index... chroot rootfs-complete /usr/sbin/indexgen.sh #...note, rootfs-skeleton/pinstall.sh pre-processes the help files. fi It stucks in above stage and because of that it seems to hang here: Constructing configuration files for JWM, Fvwm95, IceWM, openbox... Generating /root/.icewm/menu... It did that after my first Precise build, when you uploaded the first implementation: http://bkhome.org/fossil/woof2.cgi/info/a8a0826981 But I got it woofed. In the build the fixmenus did hung though. Now even 3builddistro hangs. Is it just me ?

fixmenus...continued...
Username: pemasu
"I killed the chrooted fixmenus and the 3buiddistro didnt stop so I was able to continue the build. I copied those supposedly created and failed files .jwmrc etc.. to the rootfs-complete. Now posting from the build and fixmenus script works in it. Interesting. First it was the buiid which had broken fixmenus, now it was 3builddistro script. I wonder if it has something to do with these 2 woof commits to improve 0setup speed. Yeah, it really improves. Lighting fast, thanks. Yeah, verbose indicator that it does something might be useful....

Re Precise build
Username: BarryK
"I have just been through the whole thing. I ran "getwoof" to download Woof. I then ran "merge2out" and chose Precise pup. Over in the "woofx" directory, I ran 0setup, 1download, 2createpackages and woof-gui. No hanging, it has gone right through OK and built the live-CD. One thing, I have LANG=en_US.UTF-8. Don't know why that would make any difference, but if you have something else and langpack installed, perhaps that could upset something.

Precise build continued....
Username: pemasu
"Thank you Barry of the feedback. I had fi_FI.UTF-8 in the Precise build I was using and en_US.UTF-8 in the 3builddistro choice. This has been my normal setup in both sides. I didnt know which one you meant. Anyway...it seems to be problem in my side. And I got the build created. Seems to work. I will follow and report if this problem give me any insight.

using en_US.UTF-8 it worked
Username: pemasu
"Huoh. Continuing woof building saga. I changed the locale to en_US.UTF-8 in the build which I used to woof. Now 3builddistro executed the chrooted fixmenus without problem. It smells like finnish are disliked, well....at least at locale level. Well...we drink too much and now we cant even woof. Sniff. Seriously if this fixes the woof building I am happy.

zz_t2_fixup -?fixed
Username: scsijon
"I've ?cheated? and switched around the wmreboot and wmshutdown scripts and links in the rootfs-skeleton/usr/bin (now links) and rootfs-skeleton/usr/X11R7/bin (now scripts) as these are the [u]only two in the later[/u] and this part is working now. I did notice the old "links" seemed to be scripts but without a #!bin/sh in front??? Also the /etc/xdg/menus has the puppy-calculate.menu as a text file and not a link to the puppy-business.menu and it was not that way in my previous woof2. Do you want me to start for you a new thread on murga-linux for woof problems, rather than use up your blog? Say with post 01 November updates / download as a pre-requirement to adding an entry. I'm thinking it to be with temporary fixes when found (permanency is up to you of course), but not to contain requests for extras / additional functions and the like. I still have a few others I need to chase through before expanding on, as I suspect some if not all are in the pet package pinstall.sh scripts and also a few other builders are having a few problems I believe that a discussion topic may help to at least find a workaround.

Re Woof problems
Username: BarryK
"pemasu, Sorry, yes, I meant the I am running en_US.UTF-8 in my host system. I am also choosing en_US.UTF-8 when building a pup. scsijon, It looks like you have not followed the instructions properly when checking out Woof from the online Fossil repository. /etc/xdg/menus/puppy-calculate.menu is a symlink. However, if the checkout steps were not followed exactly, then it may be checked out as a file -- the checkout documentation warns against this problem. On the otherhand, use my 'getwoof' script, it does the checkout properly. I know, because that is what I have just used to go through the entire process of downloading a pristine Woof and building a pup. If puppy-calculate.menu is not a symlink, the same would have happened to others that are supposed to be symlinks -- which is going to cause all kinds of problems.

re woof problems
Username: scsijon
"AH! I have been following both your "Woof2: Getting started with Woof" in your wika and mick01's "fossil update - Tutorial" on the smokey01.com site, I thought they were the right directions. I had no idea that 'getwoof' existed, it's not mentioned in either or the Puppylinux wiki, my first source for puppy things. So: I shall archive the existing and clean the partition and try it, and hopefully all will be solved! thanks for the patience.

re woof problems
Username: scsijon
"i've sent you a 'gmail'. I did as you advised, but.... :-( sorry

Re Woof problems
Username: BarryK
"Are you doing the Woof download/checkout in a Linux partition? If you are in a msdos or ntfs partition, that will cause the no-symlinks problem (and others).

PPKG by Wosh
Username: BarryK
"I couldn't recall the guys name, who worked on a C debian-db to pup-db converter. Dougal has reminded me -- it is Forum member Wosh, and here is the relevant post: http://www.murga-linux.com/puppy/viewtopic.php?t=47829 See also: http://www.murga-linux.com/puppy/viewtopic.php?t=47763 Wosh got Debian 'main' db converted in 355 milliseconds on his computer. My 'debdb2pupdb' takes about 3 seconds to convert Precise 'main' on my laptop. My code is doing a bit more, such as versioning of dependencies, also I did not "pull out all the stops" as Wosh did to optimise the speed of the code. Oh yes, another very big item that my code has, the determination of sub-category. For example "Graphic;scanner" -- this requires extra parsing. Ah, another thing. I have just looked at Wosh's C code. It has the equivalent of my 'categories.dat', however the converter does not have the fallback of analysing description keywords to determine category -- this is a sizable chunk of debdb2pupdb's code. Wosh's effort is admirable. However, one problem for me, trying to understand the code, is that trying to extract maximum speed has made it less readable. Wosh disappeared from the Puppy scene soon after posting PPKG. The last post to the Forum was Oct. 24, 2009. I recall, at the time I wanted changes to the code, but that never happened.

Tags: woof