Exploring debootstrap

This is fascinating...

Recently, Caramel has been testing woofQ, the build system for EasyOS, see this forum thread:

https://forum.puppylinux.com/viewtopic.php?t=14493

Caramel commented that the '2createpackages' script is very slow. Yeah, the scripts in woofQ are ancient; I have hacked on them, but really, the entire thing needs a rewrite. Which is what has been done with Woof-CE, in particular due to forum member dimkr.

I actually did do a complete rewrite, substantially anyway, which is woofQV. The decision in that case was to base the build of QV on Void Linux binary packages, also went for a new underlying structure; btrfs and snapshots. The result is not EasyOS, it is a different distribution; QV, "Quantum Vis".

Note that QV still has PKGget, which derives from the Puppy PPM, but substantially rewritten as a frontend to the Void XBPS package manager. It still retained the chameleon-like nature of PPM, able to install many package types, including Debian .deb and .pet packages. It achieves an integration of these disparate repositories.

Thinking about a rewrite of woofQ, I do not want to build from packages of a rolling-release distro, as that is anathema to the principle of building EasyOS with read-only squashfs files; in the case of EasyOS, the main guy is 'easy.sfs'.

Easy Daedalus is built with Devuan Daedalus (Debian Bookworm) .deb packages. Yes, it looks like this is the way to go, moving forward. So, rewrite woofQ, let's call it "woofQ2", based on building with 'dpkg' and 'apt'. So they will be the underlying package management. There could still be a PKGget, but as done with QV, a wrapper around apt.

woofQ2 needs to be able to build easy.sfs fast, and this is where debootstrap comes in. This is a script that builds a rootfs; that is, a folder with apt inside it, that can be chrooted into and any required packages installed.

Forum member fredx181 has used debootstrap for his DebianDog and UbuntuDog distros. I wrote about BookwormDog last year:

• BookwormDog build script — September 23, 2024

A couple of days ago, I downloaded his latest, TrixieDog; the 'mklive-trixie' script. See forum:

https://forum.puppylinux.com/viewtopic.php?t=11771

I ran it in Easy Daedalus. Had to install 'debootstrap', 'dpkg' and 'isolinux' beforehand. Note, Easy has the 'dpkg' applet for busybox, but the full package is required. It went through, created an ISO which I wrote to a USB-stick and booted; got a desktop!

Create rootfs with debootstrap

So, how to go about using debootstrap to create a chrootable rootfs, that could become easy.sfs? I wrote a script to do that...

The script is 'create-devuan-rootfs'. Yeah, decided to go with Devuan. The upcoming Debian Trixie will be Devuan Excalibur. Here is the script:

#!/bin/sh
#created 20250611

#just in case abnormal exit debootstrap...
mount | grep -q '/rootfs-devuan/'
if [ $? -eq 0 ];then
 busybox umount rootfs-devuan/proc 2>/dev/null
 busybox umount rootfs-devuan/sys 2>/dev/null
 busybox umount rootfs-devuan/dev/shm 2>/dev/null
 busybox umount rootfs-devuan/dev/pts 2>/dev/null
 busybox umount rootfs-devuan/dev 2>/dev/null
fi

DPKG="$(which dpkg)"
if [ $? -ne 0 ];then
 echo "Utility 'dpkg' not installed."
 exit 1
else
 if [ -L "$DPKG" ];then
  echo "The busybox 'dpkg' is installed. Need the full 'dpkg' package."
  exit 1
 fi
fi

if [ ! -f Packages ];then
 wget http://packages.devuan.org/devuan/dists/excalibur/main/binary-amd64/Packages.xz
 unxz Packages.xz
fi

which debootstrap >/dev/null
if [ $? -ne 0 ];then
 DEBfnd="$(grep '^Filename: pool/main/d/debootstrap/debootstrap_' Packages)"
 DEBfnd="${DEBfnd##*/}" #ex: debootstrap_1.0.141devuan1_all.deb
 wget --no-check-certificate http://packages.devuan.org/devuan/pool/main/d/debootstrap/${DEBfnd} -O ${DEBfnd}
 dpkg -i ${DEBfnd}
fi
which debootstrap >/dev/null
if [ $? -ne 0 ];then
 echo "Failed to install debootstrap"
 exit 1
fi

if [ ! -f /usr/share/keyrings/devuan-archive-keyring.gpg ];then
 DEBfnd="$(grep '^Filename: pool/main/d/devuan-keyring/devuan-keyring_' Packages)"
 DEBfnd="${DEBfnd##*/}" #ex: devuan-keyring_2023.10.07_all.deb
 wget --no-check-certificate http://packages.devuan.org/devuan/pool/main/d/devuan-keyring/${DEBfnd} -O ${DEBfnd}
 dpkg -i ${DEBfnd}
fi
if [ ! -f /usr/share/keyrings/devuan-archive-keyring.gpg ];then
 echo "failed to install package 'devuan-keyring'"
 exit
fi

#debootstrap 1.0.141devuan1 needs a patch:
#excalibur is a symlink to ceres, not working correctly...
if [ -L /usr/share/debootstrap/scripts/excalibur ];then
 rm -f /usr/share/debootstrap/scripts/excalibur
 cp -a /usr/share/debootstrap/scripts/ceres /usr/share/debootstrap/scripts/excalibur
 PTN1='s%"\$MERGED_USR" = "yes" -o "\$CODENAME" != "daedalus"%"X" == "Y"%'
 sed -i -e "${PTN1}" /usr/share/debootstrap/scripts/excalibur
fi

if [ -d rootfs-devuan ];then rm -rf rootfs-devuan; sync; fi
mkdir rootfs-devuan

mkdir -p rootfs-devuan/etc/dpkg/dpkg.cfg.d
mkdir -p rootfs-devuan/etc/apt/apt.conf.d

echo 'path-exclude=/usr/share/doc/*
path-exclude=/usr/include/*
path-exclude=/usr/share/man/*
path-exclude=/usr/share/bash-completion/*
path-exclude=/usr/share/common-licenses/*
path-exclude=/usr/share/info/*' > rootfs-devuan/etc/dpkg/dpkg.cfg.d/50excldoc

echo 'APT::Install-Recommends "false";
APT::Install-Suggests "false";' > rootfs-devuan/etc/apt/apt.conf.d/50norec


#######
debootstrap --arch=amd64 --variant=minbase --include=busybox-static,locales,dialog,pulseaudio excalibur rootfs-devuan
sync
#######

#why does dpkg in the rootfs ignore dpkg.cfg.d?...
#rm -rf rootfs-devuan/usr/share/doc/*
#rm -rf rootfs-devuan/usr/include/*
#rm -rf rootfs-devuan/usr/share/locale/*
#mkdir rootfs-devuan/usr/share/locale/C
#mkdir rootfs-devuan/usr/share/locale/en
#mkdir rootfs-devuan/usr/share/locale/de
#rm -rf rootfs-devuan/usr/share/man/*
#rm -rf rootfs-devuan/usr/share/bash-completion/*
#rm -rf rootfs-devuan/usr/share/common-licenses/*
#rm -rf rootfs-devuan/usr/share/info/*
#sync
#note: /usr/share/locale will be populated as more pkgs installed.

#tell apt all sources:
#ref: http://packages.devuan.org/devuan/dists/
echo -n 'deb http://deb.devuan.org/merged excalibur main contrib non-free
deb http://deb.devuan.org/merged excalibur-updates main contrib non-free
deb http://deb.devuan.org/merged excalibur-security main
' > rootfs-devuan/etc/apt/sources.list

#/proc has been left mounted. add these, for more chrooting...
busybox mount --rbind /dev rootfs-devuan/dev
busybox mount --bind /sys rootfs-devuan/sys

chroot rootfs-devuan localedef -f UTF-8 -i en_US --no-archive en_US.utf8
#######

echo
echo "Basic rootfs created!"
echo -n "ENTER to install more: "
read WANTMORE

######
echo
chroot rootfs-devuan apt update #coz those extra repos added.

PKGLIST='ca-certificates
cups
file
firefox-esr
firmware-linux-free
geany
geany-plugins
gnupg-utils
gvfs
gxmessage
jwm
kmod
l3afpad
libgdk-pixbuf2.0-bin
librsvg2-bin
mingetty
network-manager
obconf
openbox
pavucontrol
psmisc
sakura
x11-apps
xinit
yad'

for aPKG in ${PKGLIST}
do
 [ "$aPKG" == "" ] && continue
 chroot rootfs-devuan apt install ${aPKG}
 echo
 sync
 echo -n "ENTER to install more: "
 read WANTMORE
done

####
echo
echo "...finished"
echo -n "Press ENTER if you want to umount proc, sys and dev: "
read YESUM
if [ "$YESUM" == "" ];then
 busybox umount rootfs-devuan/proc 2>/dev/null
 busybox umount rootfs-devuan/sys 2>/dev/null
 busybox umount rootfs-devuan/dev/shm 2>/dev/null
 busybox umount rootfs-devuan/dev/pts 2>/dev/null
 busybox umount rootfs-devuan/dev 2>/dev/null
fi
###end###

...not too long to put all of it into this blog post. Have uploaded the script, with false ".gz" appended, here.

Next-up, save the downloaded .deb packages in a local repository, which will be accessed on subsequent builds.

This script builds folder rootfs-devuan from .deb packages, but EasyOS is also built with lots of .pet and OE Kirkstone/Scarthgap .tar.xz packages. The apt management knows nothing about these. In woofQV, these foreign packages get converted to .xbps packages in a local repository, so that the XBPS manager can install them. However, I don't know if want to go to all that trouble again; thinking of doing it a different way.    

Tags: easy