How to cross-compile 850+ packages using Yocto/OpenEmbedded

December 12, 2021 — BarryK

Over the years I have received a few enquiries about how to get going with Yocto/OpenEmbedded -- I will call it OE from now on -- and in reply I have mentioned that it has a very steep learning curve. Start reading the documentation and your eyes will glaze over very quickly!

However, you can follow some simple steps and compile 850+ packages without trauma. It should just work. Knowledge of how OE works can be learnt gradually, later on.

One thing to understand is that OE mostly targets embedded systems, and is not really intended to build packages for a desktop PC. However, it is a testimony to the flexibility of OE that it can be made to do this -- my "dunfell" project tarball has many tweaks to achieve this.

Another very important point is that it is a cross-compile environment. This is great, you can be running on a x86_64 Linux host system and compile for a aarch64 target, for example. I am currently supporting x86_64 (nocona) and aarch64 targets. However, cross-compiling is sometimes like trying to sweep dung uphill. Build recipes may have to do all kind of tricky things to perform a successful compile of a package. OE is incredibly sophisticated and despite the cross-compile hurdle can successfully compile large packages, such as Libreoffice.

The current release of OE is Dunfell 3.1.x, now up to 3.1.12. This is an LTS release, supported until April 2024:

https://wiki.yoctoproject.org/wiki/Releases

I am currently using the Dunfell release. I see that the next LTS release will be "Kirkstone", due out in April 2022. I don't know when or if I will upgrade to kirkstone. One thing, there are some syntax changes to recipes in releases after dunfell, so my 340+ recipes will need to be modified.

if you would like to read an overview:

https://docs.yoctoproject.org/3.1.12/overview-manual/overview-manual.html

A getting-started guide:

https://docs.yoctoproject.org/3.1.12/brief-yoctoprojectqs/brief-yoctoprojectqs.html

Documentation is also available in one huge mega-manual:

https://docs.yoctoproject.org/3.1.12/singleindex.html

It might be a good idea to try the "Quick Build" in the getting started guide first, to "kick the tyres" so to speak.

OE-Quirky-Dunfell project

What I have done, basically, is download OE layers, then added my own "meta-quirky" layer. it is very interesting to browse here and see all of the available layers:

http://layers.openembedded.org/layerindex/branch/master/layers/

...you won't find meta-quirky there. My project is currently not online, it is only available as a tarball, here:

https://distro.ibiblio.org/easyos/project/oe/dunfell/

...at time of writing, latest is 'dunfell-20211211.tar.gz'

You need to be running a x86_64 Linux host system, with all compiler tools installed, including gcc. glibc up to version 2.34 is supported. Python3 is required.

EasyOS 3.1.x meets these requirements. The 'devx' SFS has to be loaded, to provide gcc, autotools, etc. Running EasyOS, you are good-to-go.

A ext4 partition with at least 300GB free space is required. My current PC has 32GB RAM, but I have built on a PC with only 8GB RAM. Last night I did a complete recompile, and it took 16½ hours to compile the 850+ packages. The build was on an external usb3 SSD, so the usb would have been a bottleneck.

So, running EasyOS, with 'devx' SFS loaded, you download 'dunfell-20211211.tar.gz', to a ext4 partition with over 300GB free space. You then expand the tarball:

# tar -xf dunfell-20211211.tar.gz
# cd dunfell-20211211

There, you will see a 'readme' file, with instructions what to do next.

Summarizing, you edit the "BUILDSPATH" variable in script 'create-oe-quirky', then run the script. You then change to ${BUILDSPATH}/oe-quirky and start the build:

# cd ${BUILDSPATH}/oe-quirky
# source oe-init-build-env build-amd64
# bitbake-layers show-layers
# bitbake -g core-image-minimal
# sort pn-buildlist > pn-buildlistSORTED1
# bitbake core-image-minimal

...the last one starts the build, and 16+ hours later it should have compiled everything in file 'pn-buildlistSORTED1'.

Note, I selected what packages to build in this file:

oe-builds/oe-quirky/build-amd64/conf

..."core-image-minimal" does specify a basic set of packages, and my list in the 'conf' file adds to that. You should examine what is in the 'conf' file -- it has lots of settings. For example, whether to use pulseaudio or just alsa. In the case of pulseaudio, most recipes do an automatic check if it is enabled, except a few of my recipes do not -- instead I just created two alternative recipes -- do a search for "*bb*ALSA" in folder meta-quirky. It is currently setup to use pulseaudio, no systemd, no avahi, no pam, no polkit. pipewire is also built (which provides jack libraries) -- though in EasyOS only using pulseaudio, pipewire is just built to experiment with.

The 16+ hour build... "Touch wood!" The trickiest part is that it will have to download all the source packages, and it cannot be guaranteed that all sources are available. Have a look at docs/bk-notes for some extra guidelines to help achieve a successful build.

One "problem" is that my build will fail the final "do_rootfs" step. This is where OE attempts to put all of the packages together, into one big chrootable rootfs. That failure does not bother me, as I use scripts in 'woofQ' to extract all of the individual binary packages and import them for creating EasyOS.

When you have completed the build, or if a failure, you can go in and see each package build, or build attempt:

build-amd64/tmp/work/nocona-64-poky-linux

...poke around in there and you can often find out why a package build failed. For example, 'coreutils' build logs are here:

build-amd64/tmp/work/nocona-64-poky-linux/coreutils/8.31-r6/temp

If a recipe needs to be fixed, or you want to create a new recipe, ah, that is where the steep learning curve comes in!

Hopefully though, it should just go right through.

Package extraction

It does build DEB packages; however, I have doubts about those, some will not be packaged correctly -- that is a shortcoming of some of the recipes, not just mine. Instead, I have two scripts in woofQ that extract the individual packages, as .tar.xz packages. Those scripts are '0pre-oe' and '0pre-oe-add'. The second script is for when you modify a package or add a new package.

If you want to create your own collection of .tar.xz packages, you can get those scripts out of the woofQ tarball:

https://distro.ibiblio.org/easyos/project/woofq/2021/

The scripts also create a Puppy-format package database, with full dependencies, but the scripts could be modified for a different package management system.

I know it is not the best that I am only providing these projects as tarballs. Might put them up on a git repository one day. I did put an earlier OE project, "oe-qky-src", based on the "Pyro" release of Yocto/OE, but that is way out of date, now retired:

https://github.com/bkauler/oe-qky-src

A closing note: the above is for someone who doesn't need hand-holding. Especially you need to be competent with bash/ash shell scripting. One very good thing about Yocto/OE is that it is very active. I don't know how many people are involved, must be in the hundreds. So, if you have problems, there is lots of online activity, forum posts, etc., that you can search.

Tags: easy