Fixed compile of libvdpau-va-gl in OE

I posted yesterday about the problem in OpenEmbedded when the compile of a package requires execution of a binary:

https://bkhome.org/news/202101/fixed-compile-of-samba-without-krb5-in-oe.html

This problem does not occur if the build-architecture and target-architectures are the same. The problem occurs with a cross-compile.

Today I had the same problem, with package 'libvdpau-va-gl'. I had previously compiled this in OE, but now the build-arch is x86_64 and the target-arch is aarch64.

So, I used the same trick as with samba, of using pre-compiled binaries, in this case it is just the one binary, 'shader-bundle-tool'. The package uses cmake and ninja, and my knowledge of these is minimal.

This is the recipe meta-quirky/recipes-quirky/libvdpau-va-gl/libvdpau-va-gl_0.4.2.bb:

# Recipe created by recipetool
# recipetool create -o libvdpau-va-gl_0.4.2.bb https://github.com/i-rinat/libvdpau-va-gl/releases/download/v0.4.2/libvdpau-va-gl-0.4.2.tar.gz

# 20210121 PR_NUM is defined in local.conf...
#PR = "r${@int(PR_NUM) + 1}"

LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=5a9126e7f56a0cf3247050de7f10d0f4"

FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

#20210124 wants to run a binary:
#| /bin/sh: /mnt/sda1/nvme/oe-builds/oe-quirky/build-aarch64/tmp/work/aarch64-poky-linux/libvdpau-va-gl/0.4.2-r1/build/glsl/shader-bundle-tool: cannot execute binary file: Exec format error

SRC_URI = "https://github.com/i-rinat/libvdpau-va-gl/releases/download/v${PV}/libvdpau-va-gl-${PV}.tar.gz \
  file://${BUILD_ARCH}/shader-bundle-tool"

SRC_URI[md5sum] = "8db21dcfd5cd14c6ec51b992e20369dc"
SRC_URI[sha256sum] = "7d9121540658eb0244859e63da171ca3869e784afbeaf202f44471275c784af4"

DEPENDS = "libx11 ffmpeg libvdpau libva mesa xserver-xorg"

inherit cmake pkgconfig

EXTRA_OECMAKE = "-DCMAKE_BUILD_TYPE=Release"

do_compile_prepend() {
 mkdir -p ${B}/glsl/
 cp -a -f ${WORKDIR}/${BUILD_ARCH}/shader-bundle-tool ${B}/glsl
 touch -d '+1 hour' ${B}/glsl/shader-bundle-tool
 #fool ninja not to overwrite shader-bundle-tool...
 sed -i -e 's%TARGET_FILE = glsl/shader-bundle-tool%TARGET_FILE = glsl/shader-bundle-toolXXX%' ${B}/build.ninja 
}

FILES_${PN} += "${libdir}/vdpau"

INSANE_SKIP_${PN} += "dev-so"

HOMEPAGE = "https://github.com/i-rinat/libvdpau-va-gl"
SUMMARY = "VDPAU driver with OpenGL/VAAPI backend"

Notice the line "touch -d '+1 hour' ${B}/glsl/shader-bundle-tool" -- I thought that setting the modify date ahead would prevent ninja from recompiling it, but that didn't work. Ninja does not work like make.

Instead, I found the 'build.ninja' file, which specified the name of the compiled file, so I just changed that from 'shader-bundle-tool' to 'shader-bundle-toolXXX'. Great, worked, left the pre-compiled binary intact.

The recipe works. I have added this package to the Pi4 package-list, although I don't know what use it will be. By setting VDPAU_DRIVER=va_gl, it causes VDPAU to use the GL backend, but I don't now much about VDPAU either.

Here is the libvdpau-va-gl project page:

https://github.com/i-rinat/libvdpau-va-gl

Only including this in the next Pi4 build as something extra to play with.

Tags: oe

Fixed compile of Samba without krb5 in OE

Yesterday I posted about improvements compiling in OpenEmbedded:

https://bkhome.org/news/202101/tweaks-for-openembedded-dunfell.html

EasyOS on the Pi4 does not have samba, as compile failed in OE. Yes, I could compile it in a running EasyOS on the Pi4, but would rather fix it in OE.

I have a 'samba_%.bbappend' file, the main objective being to remove the 'pam' and 'krb5' dependencies. I worked on this recipe this morning. The problem is that instead of 'krb5', the internal 'heimdahl' is used, and this compiles two binaries, that are then executed during compile.

The problem is that the binaries are compiled for the target system, in this case aarch64, whereas the build system is x86_64, so the binaries cannot run.

OE does have a mechanism to handle this. It is possible to compile 'samba-native', that is, samba compiled to run on the build-system, and then use the two binaries from that when compile 'samba'.

Fine, except that exactly how to do this is very poorly documented. The official documentation is very vague. A couple of years ago, I bought a book, "Embedded Linux Systems with the Yocto Project", but found that it also said hardly anything about this. I consider this to be an important topic, yet it seems that many OE experts don't know much about it either.

Perhaps it is explained somewhere, and I have just overlooked it.

Anyway, when I hit this problem, I just use pre-compiled binaries, by compiling the package separately on my build-system.

In the case of samba, that is what Uri has done:

http://samba.2283325.n4.nabble.com/Samba4-4-5-cross-compilation-for-PowerPC-td4710972.html

...not on OE, but I can apply the same principle. This is my 'samba_%.bbappend':

#heimdahl compiles these and want to run them during compile...
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI_append = "\
  file://${BUILD_ARCH}/asn1_compile \
  file://${BUILD_ARCH}/compile_et"

DEPENDS = "readline virtual/libiconv zlib popt libtalloc libtdb libtevent libldb libaio libtasn1 jansson libbsd gpgme acl gnutls libarchive openldap"

REQUIRED_DISTRO_FEATURES = ""

SYSTEMD_PACKAGES = ""
SYSTEMD_SERVICE_${PN}-base = ""
SYSTEMD_SERVICE_${PN}-ad-dc = ""
SYSTEMD_SERVICE_winbind = ""
SYSTEMD_AUTO_ENABLE_${PN}-ad-dc = ""

PACKAGECONFIG = ""

EXTRA_OECONF = "--enable-fhs \
                 --with-piddir=/run \
                 --with-sockets-dir=/run/samba \
                 --with-modulesdir=${libdir}/samba \
                 --with-lockdir=${localstatedir}/lib/samba \
                 --with-cachedir=${localstatedir}/lib/samba \
                 --disable-rpath-install \
                 ${@oe.utils.conditional('TARGET_ARCH', 'x86_64', '', '--disable-glusterfs', d)} \
                 --with-cluster-support \
                 --with-profiling-data \
                 --with-libiconv=${STAGING_DIR_HOST}${prefix} \
                 --without-pam --without-sendfile-support \
                 --without-ad-dc  --without-ntvfs-fileserver  --nopyc --nopyo \
                 --without-fam --without-lttng --without-systemd --disable-avahi \
                 --enable-cups --with-acl-support --with-automount --with-quotas \
                 --with-syslog --with-ldap --enable-gnutls --with-gpgme --with-libbsd \
                 --with-libarchive --without-valgrind \
                 --bundled-libraries=!asn1_compile,!compile_et"

RDEPENDS_${PN}-ad-dc = ""

# ref: http://samba.2283325.n4.nabble.com/Samba4-4-5-cross-compilation-for-PowerPC-td4710972.html
export USING_SYSTEM_ASN1_COMPILE = "1"
export ASN1_COMPILE = "${WORKDIR}/${BUILD_ARCH}/asn1_compile"
export USING_SYSTEM_COMPILE_ET = "1"
export COMPILE_ET = "${WORKDIR}/${BUILD_ARCH}/compile_et"

ERROR_QA_remove = "file-rdeps"
WARN_QA_remove = "file-rdeps"

As my build-system is EasyOS x86_64 on a PC, I placed those two 'asn1_compile' and 'compile_et' into meta-quirky/recipes-connectivity/samba/samba/x86_64. I also grabbed the two that were compiled in OE and put them into samba/samba/aarch64. The 'samba_%.bbappend' is in meta-quirky/recipes-connectivity/samba.

Note, 'meta-quirky' is my layer in OE, with all of my customizations and extra packages.

Good, it compiled and installed.

Just a little detail: in OE, HOST_ARCH is "aarch64", that is, the target architecture. Not, as you might think, the build system architecture. We often use the term "host system", but that is misleading with OE, so I use the term "build system".  

Tags: oe

Extra patches applied to bash in OE

I reported about a nasty bug in bash, running in EasyOS Dunfell 0.101 and earlier:

https://bkhome.org/news/202012/easyos-dunfell-update-weird-error.html

I recompiled bash in running Easy Dunfell, with patches from Debian, and created a PET, then released Easy 0.102.

However, it needs to be fixed in OpenEmbedded. The bash recipe in OE applies patches 001 to 016, however, I see from here, that 017 and 018 are available:

https://ftp.gnu.org/gnu/bash/bash-5.0-patches/

And it looks like 017 fixes the bug. So, in my fork of OE, I have created meta-quirky/recipes-extended/bash/bash_5.0.bbappend:

# BK 20201227 bash compiled in OE has a serious bug:
# https://bkhome.org/news/202012/easyos-dunfell-update-weird-error.html
# it looks like patch #17 will fix it. see list of patches here:
# https://ftp.gnu.org/gnu/bash/bash-5.0-patches/

PR = "r1"

SRC_URI += " \
  ${GNU_MIRROR}/bash/bash-${PV}-patches/bash50-017;apply=yes;striplevel=0;name=patch017 \
  ${GNU_MIRROR}/bash/bash-${PV}-patches/bash50-018;apply=yes;striplevel=0;name=patch018"

SRC_URI[patch017.sha256sum] = "4cf3b9fafb8a66d411dd5fc9120032533a4012df1dc6ee024c7833373e2ddc31"
SRC_URI[patch018.sha256sum] = "7c314e375a105a6642e8ed44f3808b9def89d15f7492fe2029a21ba9c0de81d3"

I have recompiled bash in OE, and will use this in the next release of Easy Dunfell, instead of the PET. 

Tags: oe

Fixes for OpenEmbedded Dunfell build

I have a fork of OpenEmbedded (OE), Dunfell release, to build most of the packages required for EasyOS.

Note that EasyOS Buster-series, currently at version 2.5.3, is built with Debian Buster 10.7 DEBs, but also does use some PETs that were compiled in OpenEmbedded Pyro. There is also an experimental EasyOS Dunfell-series built entirely from the packages compiled in OpenEmbedded Dunfell.

OE Dunfell is not in a git repository, only uploaded as tarballs, here:

http://distro.ibiblio.org/easyos/project/oe/dunfell/

Jon (scsijon in the forum) has been testing the tarball, trying to do a complete compile on his computer, but has hit failures:

https://easyos.org/forum/showthread.php?tid=287

So, I tried a build, on a host system running EasyOS 2.5.4 (not yet released), and also hit failures. Then I tried EasyOS Pyro64 1.3, also failures. Hmmm, interesting, considering I had done the build before, many times, and it ran right through.

I don't know why OE is failing to compile or install some packages, that succeeded before. Very odd. Anyway, went through, fixed them, and now have a tarball that works for me, on both EasyOS Buster 2.5.4 and Pyro 1.3:

http://distro.ibiblio.org/easyos/project/oe/dunfell/dunfell-20201219.tar.gz

I have reported to the forum, and Jon will give it a go.

OE Dunfell is using the 5.4 kernel, however, I want to base EasyOS Dunfell-series on the 5.10 kernel, so have now attempted to bump the kernel, and now doing a compile of OE Dunfell.

OE is very complicated, hope that I have got the kernel-bump right. We shall see. Have applied the CAP_SYS_MOUNT patch also. 

Tags: oe

OE gcc target default commandline options

I uploaded Easy Dunfell 0.91 yesterday:

https://bkhome.org/news/202010/easyos-dunfell-series-091-alpha-release.html

Which was a disaster, as most people couldn't get it to boot. I had feedback from Rodney, Feodor and David, and on the forum, banned and average1. I received an email from Alfons confirming that the USB-stick booted on a PC with Intel i7 CPU, not earlier CPUs.

The problem is a bug in OE. I set the target as an Intel Core2 CPU, which is one of the earliest x86 64bit CPUs. That's fine, all of the applications were compiled for a Core2 CPU. However, after building EasyOS, and compiling the Linux kernel, SeaMonkey and Libreoffice in a running EasyOS, then building Easy 0.91 with those, that's when disaster struck.

The gcc compiler in a running EasyOS does not default to Core2, it compiles using extra instructions that are in my i3 CPU but not in earlier CPUs.

On the Internet I found a simple way to discover the default options that gcc uses, and did this in Easy 0.91:

# echo "" | gcc -v -E - 2>&1 | grep cc1
 /usr/libexec/gcc/x86_64-poky-linux/9.3.0/cc1 -E -quiet -v - -march=core2 -mmmx -mno-3dnow \
 -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mno-movbe -mno-aes -mno-sha -mno-pclmul \
 -mno-popcnt -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-sgx -mno-bmi2 -mno-pconfig \
 -mno-wbnoinvd -mno-tbm -mno-avx -mno-avx2 -mno-sse4.2 -msse4.1 -mno-lzcnt -mno-rtm -mno-hle \
 -mno-rdrnd -mno-f16c -mno-fsgsbase -mno-rdseed -mno-prfchw -mno-adx -mfxsr -mno-xsave \
 -mno-xsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 \
 -mno-clflushopt -mno-xsavec -mno-xsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma \
 -mno-avx512vbmi -mno-avx5124fmaps -mno-avx5124vnniw -mno-clwb -mno-mwaitx -mno-clzero -mno-pku \
 -mno-rdpid -mno-gfni -mno-shstk -mno-avx512vbmi2 -mno-avx512vnni -mno-vaes -mno-vpclmulqdq \
 -mno-avx512bitalg -mno-movdiri -mno-movdir64b -mno-waitpkg -mno-cldemote -mno-ptwrite -mtune=generic

 Wow! Complicated, but I would have thought it would be OK with that "-march=core2". Obviously not though.

OE's most generic x86_64 CPU setting is Core2, but back when I ported the Pyro release of OE, I modified it for a Nocona CPU, which is the earliest 64-bit CPU. I attempted to port those Nocona changes into my Dunfell OE derivative, and did a complete recompile today. It did seem to have compiled everything for a Nocona CPU, but now I have used those packages and built Easy 0.92, and applied the same test:

# echo "" | gcc -v -E - 2>&1 | grep cc1
 /usr/libexec/gcc/x86_64-poky-linux/9.3.0/cc1 -E -quiet -v - -march=skylake -mmmx -mno-3dnow -msse \
 -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mmovbe -maes -mno-sha -mpclmul -mpopcnt -mabm \
 -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -msgx -mbmi2 -mno-pconfig -mno-wbnoinvd -mno-tbm -mavx \
 -mavx2 -msse4.2 -msse4.1 -mlzcnt -mno-rtm -mno-hle -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw \
 -madx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf \
 -mno-prefetchwt1 -mclflushopt -mxsavec -mxsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl \
 -mno-avx512ifma -mno-avx512vbmi -mno-avx5124fmaps -mno-avx5124vnniw -mno-clwb -mno-mwaitx \
 -mno-clzero -mno-pku -mno-rdpid -mno-gfni -mno-shstk -mno-avx512vbmi2 -mno-avx512vnni -mno-vaes \
 -mno-vpclmulqdq -mno-avx512bitalg -mno-movdiri -mno-movdir64b -mno-waitpkg -mno-cldemote \
 -mno-ptwrite -mtune=generic

Oh dear, "-march=skylake" is not what I want!

So, I booted up Easy Pyro 1.3:

# echo "" | gcc -v -E - 2>&1 | grep cc1
 /usr/libexec/gcc/x86_64-oe-linux/6.3.0/cc1 -E -quiet -v - -mtune=generic -march=x86-64

Nice and simple!

This is going to take me a bit longer to fix. I think what I want is "-march=nocona -mtune=nocona" and I have to study OE and find out how it has gone wrong, and how all that other stuff has got in.

Changing the subject...

Gtk3 scrollbar

If you have used a GTK-based app, such as SeaMonkey, you might have noticed how annoying the scrollbars have become. In gtk2 if you click above or below the slider, the window will scroll up or down by one window of text. Gtk3 on the other hand, there will be a jump way up or down in the text file, which is hopeless if you want to scroll through the text a window-height at a time.

Some gtk3 themes do restore the gtk2 behaviour, but it is not the default. However, I after a bit of online searching, I found the fix. Add this line under "[Settings]" in /root/.config/gtk-3.0/settings.ini:

gtk-primary-button-warps-slider = false

I have put this into woofQ. 

Tags: oe

OE gatesgarth-pre compile

I am doing a recompile in OE, based on the latest from git master branch. Previously I used the 'dunfell' branch. 'gatesgarth' release of OE is due out this month, but as it is not yet out, I got everything from master branch.

OE is a cross-compiler environment, that can build packages for a complete Linux distribution. The people who use it are mostly targeting embedded hardware, however, it can be used to create packages for a complete Linux distro such as as EasyOS.

OE is composed of layers, each one named 'meta-*', for example 'meta-gnome' and 'meta-networking', and each layer contains recipes for downloading, compiling and installing packages. I have created 'meta-quirky', and meta-quirky/recipes-quirky currently has recipes for 187 packages. These are not provided by OE, they are created by me, packages required for building EasyOS or Quirky Linux. Or any pup.

Usually, when I upgrade to the latest OE, many of my recipes in 'meta-quirky' get broken, and it takes a lot of work to fix them, with some that have to give up on. I decided to download OE "gatesgarth-pre" and see how my 187 recipes hold up.

Not the only reason though. I wanted to remove 'modemmanager' and 'pulseaudio'.

Also wanted to bump to the 5.8.x kernel, as it has enhancements to /proc that look like might improve security in containers.

I am watching it now, compiling on same computer that I am sending this blog post. Going OK.

gcc has bumped from 9.x to 10.x, and I was expecting that might cause failure of some of my recipes, but OK so far.

Note that although OE is very sophisticated and complex, and daunting at first, it can be pretty straightforward to use. I have provided a OE dunfell tarball, with simple instructions, and it is just a matter of typing a few lines and watch it compile 745+ packages.

I intend to provide a OE gatesgarth tarball also, though might do so after 'gatesgarth' is officially released. Coz, the developers are likely finding issues that need fixing in master.

Here is an overview of OE:

https://en.wikipedia.org/wiki/OpenEmbedded

Another overview:

https://elinux.org/Yocto_Project_Introduction

The "Yocto" project tends to get most attention, however, it is really just a layer in OpenEmbedded. 'meta-yocto' contains recipes for building for specific boards. I am using "genericx86-64", which is good for PCs.

EDIT 2020-10-14:
I have decided to abandon the gatesgarth build and stay with dunfell release of OE.

A couple of reasons. One is that lots of my recipes are failing in the gatesgarth build. They are all giving a similar error message when compiling, about symbol redefinition. It would seem that there is some fundamental structural change in OE that is causing this problem. It will probably be in the release-notes when gatesgarth is released, but I don't really need to go down another rabbit hole.

Another reason is that gatesgarth is apparently a "dev" release, whereas dunfell is "lts" -- long term supported. Dunfell obviously has had a lot of effort put into it, given how easy it was to get my recipes to work. Also, layers maintained by other people are likely to become compatible with dunfell sometime, if not already. Many layers are maintained by just one person or a few people, and often lag behind the official releases -- meaning that they might not work with the dunfell release. "meta-debian" is an example, currently only compatible with the warrior release of OE.   

Tags: oe

Celluloid compile fail mystery in OE

"Celluloid" is the new name for Gnome-MPV, the media player, frontend GUI for MPV, the latter being a CLI media player (with a primitive pseudo-GUI). Project home page:

https://celluloid-player.github.io/

I have a "meta-quirky" layer for the Dunfell release of OpenEmbedded, see two previous posts in the saga:

https://bkhome.org/news/202009/pango-version-144-is-a-disaster.html

https://bkhome.org/news/202009/openembeddedyocto-dunfell-meta-quirky-first-release.html

All 755 packages compile, except for one, Celluloid. Or rather, it sometimes succeeds, sometimes not. Early this morning, 1.24am,  I started a complete recompile, then went to bed. I had wanted to time the build, but when I got up at 7.30am, I discovered that the build had stopped on an error -- oh no, Celluloid again.

On previous occasions, I had repeated the build of Celluloid, and maybe after a couple of attempts, it would succeed. Like this:

# bitbake -c cleansstate celluloid
# bitbake -c build celluloid

I am getting different error messages each time. This time, it reported a missing termination of a comment, and I discovered that the source file is truncated. Huh!?

How can that even happen?

Here is the recipe, and I have made a comment about the failure, and put in a possible fix:

# Recipe created by recipetool
# recipetool create -o celluloid_0.18.bb https://github.com/celluloid-player/celluloid/archive/v0.18.tar.gz

LICENSE = "GPLv3"
LIC_FILES_CHKSUM = "file://COPYING;md5=8006d9c814277c1bfc4ca22af94b59ee"

SRC_URI = "https://github.com/celluloid-player/celluloid/archive/v${PV}.tar.gz"
SRC_URI[md5sum] = "3fde6852840798ad61b3e39f1513f8d4"
SRC_URI[sha1sum] = "17c7bcc5f34f003140cad407b6a3ecd4d87ce769"
SRC_URI[sha256sum] = "3ce6158097d94786a62de5f26ab2cb71301e9fd0841ede8381e1535489cf0de8"
SRC_URI[sha384sum] = "1f0eafd8b6542178abee06ce68c978ac3a6a9ebd2b479e85136e8db77e2614a4e70aca622ae4a44a4ff383762ec508a2"
SRC_URI[sha512sum] = "acb76bd216100afbe3b083919e122308def1431845955dd418e7b2d57f37fb1353209d1f77f41790d3e01603ee1137fe2c5a57c7bb0ca9b0b13388278cc291f1"

DEPENDS = "libepoxy mpv gtk+3 glib-2.0 appstream-glib glib-2.0-native"

inherit gettext pkgconfig autotools-brokensep

EXTRA_OECONF = ""

# 20200929 why does build sometimes fail?
# today, error "unterminated comment" line 680 in src/mpris/celluloid-mpris-gdbus.c
# and i see 680 is end of the file -- however, file is supposed to have 8355 lines!
# try this:
PARALLEL_MAKE = "-j 1"

HOMEPAGE = "https://celluloid-player.github.io/"
SUMMARY = "GUI frontend for MPV media player"

Tried again, this time success. Now the build of all packages is continuing.

All 755 packages compile successfully, consistently, except for Celluloid. Why just Celluloid? The source package looks like a pretty ordinary autotools-based setup, can't see anything odd in the makefiles.

I put in that PARALLEL_MAKE = "j 1", but I don't see how that would have anything to do with a source file getting truncated. Is it something to do with ext4? A kernel problem?

Bitbake, the program that runs the OE build, uses all available cores, in my case 4, and will be building 4 packages at once, swapping between them continuously. Actually, it will be building many more than 4, but only 4 at the same time, rotating through them, like a 4-package moving window. 

Tags: oe