debdb2pupdb.nim cross-compiled in OE

August 28, 2022 — BarryK

This has been a fascinating exercise! I have posted recently about considering rewriting 'debdb2pupdb' in Nim source:

Debian to Puppy package db conversion in Nim — August 26, 2022

Associative arrays in Nim — August 23, 2022

Took a couple of days, but rewrote all of 'debdb2pupdb', and also included the 'find_cat' utility builtin. The source is in package 'pup-tools-20220828.tar.gz':

https://distro.ibiblio.org/easyos/source/alphabetical/p/

A change made to the Puppy-format database, is now including the md5sum. For example:

libastyle3_3.1-2+b1|libastyle3|3.1-2+b1||BuildingBlock|325K|pool/main/a/astyle|libastyle3_3.1-2+b1_amd64.deb|+libc6&ge2.14,+libstdc++6&ge5.2|Shared library for Artistic Style|debian|bookworm|a5ec7a1e14a2641619f0b30860104e78||

Compiling in OpenEmbedded has been improved. Firstly, 'nim-native_1.6.6.bb' now installs 'nim.cfg', with required paths to library modules:

# Recipe created by recipetool
# recipetool create -o nim_1.6.6.bb https://nim-lang.org/download/nim-1.6.6.tar.xz

HOMEPAGE = "https://nim-lang.org/"
DESCRIPTION = "nim compiler"
SECTION = "languages"
SUMMARY = "nim language compiler"

LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://copying.txt;md5=e202ebcd59041b078a1f8cf66709081d"

inherit native

SRC_URI = "https://nim-lang.org/download/nim-${PV}.tar.xz"

SRC_URI[md5sum] = "e316eb7f961071c6aaa7e5376c576f90"
SRC_URI[sha256sum] = "67b111ce6f3861503b9fcc1cae59fc34d0122566d3ecfef3a064a2174121a452"

# ERROR: nim-native-1.6.6-r9 do_populate_sysroot: The recipe nim-native is trying to install files into a shared area when those files already exist. Those files and their manifest location are:
#  /mnt/build/oe-builds/oe-quirky/build-amd64/tmp/sysroots-components/x86_64/nim-native/usr/bin/nim
# It could be the overlapping files detected are harmless in which case adding them to SSTATE_DUPWHITELIST may be the correct solution.
SSTATE_DUPWHITELIST = "/"

do_configure () {
 true
}

do_compile () {
  COMP_FLAGS="${CFLAGS}" LINK_FLAGS="${LDFLAGS}" ./build.sh --os linux --cpu ${BUILD_ARCH}
  bin/nim c -d:release koch
}

do_install () {
 ./install.sh pkg
 install -d ${D}${bindir}
 install -d ${D}${includedir}
 install -d ${D}${libdir}/nim
 install -m 755 pkg/nim/bin/nim ${D}${bindir}/
 install -m 644 pkg/nim/lib/cycle.h ${D}${includedir}/
 install -m 644 pkg/nim/lib/nimbase.h ${D}${includedir}/
 install -m 755 koch ${D}${bindir}/
 (
  cd pkg/nim/lib
  for dir in $(find . -mindepth 1 -type d);do
   install -d ${D}${libdir}/nim/${dir}
  done
  for file in $(find . -mindepth 1 -type f);do
   install -m 755 "${file}" ${D}${libdir}/nim/${file}
  done
 )
 
 echo 'path="$lib/deprecated/core"
path="$lib/deprecated/pure"
path="$lib/pure/collections"
path="$lib/pure/concurrency"
path="$lib/impure"
path="$lib/wrappers"
path="$lib/wrappers/linenoise"
path="$lib/windows"
path="$lib/posix"
path="$lib/js"
path="$lib/pure/unidecode"
path="$lib/arch"
path="$lib/core"
path="$lib/pure"' > nim.cfg
 install -d ${D}${sysconfdir}/nim
 install -m644 nim.cfg ${D}${sysconfdir}/nim/
}

What has got really interesting, is compiling a utility written in Nim. Compiling 'debdb2pupdb.nim' in the x86_64 host OS, got a binary from 51KB to 120KB, depending on various build choices.
In the OE recipe, got unexpected sizes, but eventually got it down to 54KB. Here is part of recipe file 'pup-tools_20220828.bb':

DEPENDS = "gtk+ m4-native bacon-native nim-native flex-native"

inherit gettext pkgconfig

#trying to get a smaller executable...
CC_remove = "-fstack-protector-strong"
CC_remove = "-D_FORTIFY_SOURCE=2"
#might as well get rid of these also...
CC_remove = "-Wformat"
CC_remove = "-Wformat-security"
CC_remove = "-Werror=format-security"

do_compile () {
 
 #new nim app...
 case "${TARGET_ARCH}" in
  x86_64) xTARGET_ARCH=amd64 ;;
  aarch64) xTARGET_ARCH=arm64 ;;
  i686) xTARGET_ARCH=i386 ;;
  *) xTARGET_ARCH="${TARGET_ARCH}" ;;
 esac
 cd nim
  #compile debdb2pupdb...
  export xCC="${CC#* }"
  export CC="${CC/ */}"
  #note: nim will read etc/nim/nim.cfg installed by nim-native
  #nim c --lib:${WORKDIR}/recipe-sysroot-native/usr/lib/nim --opt:speed --mm:orc -d:useMalloc --passC:"-flto ${xCC}" --passL:"-flto ${xCC} -ldl" --cpu:${xTARGET_ARCH} --os:linux -d:release --cc:env --skipUserCfg --skipParentCfg --skipProjCfg --nimcache:nimcache debdb2pupdb
  # ...stripped binary is 82KB
  #nim c --lib:${WORKDIR}/recipe-sysroot-native/usr/lib/nim --opt:size --mm:orc -d:useMalloc --passC:"-flto ${xCC}" --passL:"-flto ${xCC} -ldl" --cpu:${xTARGET_ARCH} --os:linux -d:release --cc:env --skipUserCfg --skipParentCfg --skipProjCfg --nimcache:nimcache debdb2pupdb
  # ...stripped binary is 94KB !!!
  #nim c --lib:${WORKDIR}/recipe-sysroot-native/usr/lib/nim --opt:size --mm:orc --passC:"${xCC}" --passL:"${xCC} -ldl" --cpu:${xTARGET_ARCH} --os:linux -d:release --cc:env --skipUserCfg --skipParentCfg --skipProjCfg --nimcache:nimcache debdb2pupdb
  # ...stripped binary is 122KB
  #nim c --lib:${WORKDIR}/recipe-sysroot-native/usr/lib/nim --opt:none --mm:orc --passC:"${xCC} -Os" --passL:"${xCC} -Os -ldl" --cpu:${xTARGET_ARCH} --os:linux -d:release --cc:env --skipUserCfg --skipParentCfg --skipProjCfg --nimcache:nimcache debdb2pupdb
  # ...stripped binary is 74KB
  nim c --lib:${WORKDIR}/recipe-sysroot-native/usr/lib/nim --opt:none --mm:orc -d:useMalloc --passC:"-flto ${xCC} -Os" --passL:"-flto ${xCC} -Os -ldl" --cpu:${xTARGET_ARCH} --os:linux -d:release --cc:env --skipUserCfg --skipParentCfg --skipProjCfg --nimcache:nimcache debdb2pupdb
  # ...only 54K !!!
 cd ..

Removing some parameters from CC did not make any difference, but very strangely, setting "--opt:none" and passing in my own options, did the trick, and got down to 54KB. Tested it, it works.

The new 'debdb2pupdb' works a little bit differently, so next have to edit '0setup', the script that calls it. That script is found in woofQ and also in /usr/local/petget.

Tags: easy