site  contact  subhomenews

Big build in OE underway

May 06, 2017 — BarryK
The last couple of days, I have been fixing the build of some packages in OpenEmbedded. In particular, I have had to modify the build "recipes" to fit my requirements for dependencies.

Packages that are not in my build include these:
avahi, gtk+3, libpam, pulseaudio, systemd

My core-image-quirky recipe now builds a large collection of packages, including very big ones such as libreoffice, kodi and firefox.

I have just started the build from scratch, the cache deleted so that it is truly a build from scratch, and we shall see how long it takes...

Next, I plan to import the packages into woofQ and build a Quirky. I will put the big apps in, including libreoffice, kodi and firefox, as I am very interested to find out how big the resultant distro will be.

Not every binary package that you find in a typical Quirky or Puppy, is compiled in OE, but that is no problem, will use packages from elsewhere, such as April. Eventually, hope to compile everything in OE.

Tags: oe

New Quirky layer in OpenEmbedded

May 01, 2017 — BarryK
Having lots of fun. Recently posted about getting started with Yocto/OpenEmbedded:
http://bkhome.org/news/201704/part-2-libreoffice-compiled-in-oe.html

The Yocto books from Amazon arrived a few days ago, and I have had my nose down. Complicated, but making progress.

I decided to use OpenEmbedded from git as the basis for building packages for Quirky, not Yocto.

I have created a new layer, named 'meta-quirky', and so far it builds a basic X11 system with development tools.

Kernel also. I have just now looked at how to substitute my own '.config' file, will give that a go.

I will add more packages later. But very carefully, as want to build a very lean-and-mean distro.

Currently, just compiling packages, which are then imported into woofQ. However, in theory, OE could do the whole thing, build an image file ready to be written to a flash stick.
Down the track, might aim to achieve this, if I can master OE sufficiently.

Then, I could provide a single tarball, that someone can download, then run just a couple of commands and it builds the entire Quirky (or Puppy) image file. All it would need is a compatible host system, such as Quirky 8.1.6 or one of the ubuntu-pups or debian-pups (with some small fixes and extra pkgs installed).

The biggest problem right now is how long a build takes. Even a minimal build takes 12 hours, a big one (LibreOffice etc.) more like 24 hours.
That's because it is running one core only on my laptop.
Maybe I should invest in a fast PC/laptop, maybe i5 or i7 CPU, 8GB RAM, 256GB SSD ....don't want to spend the money right now though...

Tags: oe

Part 2, LibreOffice compiled in OE

April 23, 2017 — BarryK
I am documenting how I compiled LibreOffice in OpenEmbedded. Part 1 is the previous blog post:
http://bkhome.org/news/201704/libreoffice-compiled-in-oe.html

One thing about that previous post. I changed the number of parallel processes to one. That is because I have an overheating problem with my laptop, but you might want leave those out, and it will default to using all CPU cores.

Setup OE environment
As explained in the previous post, folder "oe-project" has been created. There needs to be a bit of preliminary work before the actual build can take place...
# cd oe-project

# source ./oe-init-build-env buildQ

...working dir is now in buildQ

buildQ/conf/bblayers.conf, add the layers that we want:
BBLAYERS ?= "
/mnt/sda1/projects/oe/oe-project/meta
/mnt/sda1/projects/oe/oe-project/meta-oe
/mnt/sda1/projects/oe/oe-project/meta-python
/mnt/sda1/projects/oe/oe-project/meta-office
"

confirm layer is picked up:
# bitbake-layers show-layers

keep the downloaded src pkgs outside:
# ln -s ../../downloads downloads

edit buildQ/conf/local.conf:
----------------------------
change these lines:
MACHINE ??= "qemux86-64"
PACKAGE_CLASSES ?= "package_deb"

append:
BB_NUMBER_THREADS = "1"
PARALLEL_MAKE = "-j 1"
BB_NUMBER_PARSE_THREADS = "1"

# in yocto, got an error when building initramfs, default maxsize is too small.
# INITRAMFS_MAXSIZE is set in meta/conf/bitbake.conf (= 131072 kb, 128MB).
# override here, 160MB:
INITRAMFS_MAXSIZE = "163840"

# yocto/poky/meta/conf/bitbake.conf has this line:
# DISTRO_FEATURES_BACKFILL = "pulseaudio sysvinit bluez5 gobject-introspection-data"
# ...this could be edited, or insert this into build/conf/local.conf:
#DISTRO_FEATURES_BACKFILL_CONSIDERED = "pulseaudio"

# want to add libreoffice. have already installed meta-office layer
DISTRO_FEATURES_append = " opengl"
IMAGE_INSTALL_append = " libreoffice"
<

I personally would prefer not to have 'pulseaudio'. Unfortunately, it is required to be there in some recipes in OE. Then there is the problem of the latest Firefox requiring it.
So, I have left 'pulseaudio' in the build for now.

Fix LibreOffice recipes
I am posting fixes to the recipes here, after failures occurred, so hopefully if someone follows my instructions, the build will just sail through.

The Libreoffice recipes specify two dependencies, 'clucene' and 'postgresql', however, these packages failed to build. So, I removed them from the LibreOffice recipes.
These are the two recipe files:

/mnt/sda1/projects/oe/oe-project/meta-office/recipes-libreoffice/libreoffice/libreoffice.bb,
libreoffice-native.bb

In libreoffice.bb I took out these two lines:
clucene-core
--with-system-clucene

In libreoffice-native.bb I took out:
clucene-core-native
--with-system-clucene

To get rid of postgresql, edit libreoffice.bb
remove postgresql line from here, in libreoffice.bb:
PACKAGECONFIG ??= "
gtk
mariadb
postgresql
"
insert this configure option:
--disable-postgresql-sdbc
and comment-out this line:
PACKAGECONFIG[postgresql] = "--enable-postgresql-sdbc --with-system-postgresql, --disable-postgresql-sdbc, postgresql"
<

Not there yet. Unpacking of libreoffice-native failed, reported that 'unzip' does not exist. Then, configuring, failed, reported something about 'neon' package not being in the repository.
To fix both of these:
have added these deps to libreoffice-native.bb:

unzip-native
neon-native
insert this configure option:
--with-system-neon

added dep to libreoffice.bb:
unzip
<

Start the build
Having already run that "source ./oe-init-build-env buildQ" command, "pwd" will show that you are now in the "buildQ" folder, and everything is setup ready to build. Do this:

# bitbake core-image-sato-sdk

If you get a failure somewhere, it may not be a show-stopper, and you can keep going with:

# bitbake --continue core-image-sato-sdk

Be prepared to wait a very long time!

Read more...

LibreOffice compiled in OE

April 23, 2017 — BarryK
My play with OpenEmbedded/Yocto is continuing, with success. For a very long time, I have dreamed of compiling OpenOffice, then LibreOffice, from source. It has been a couple of years since I last tried, but the few times that I had a go, just never got there. Too big and complex, too many failures. This was attempted on various flavours of Puppy and Quirky.

Now, having another go at being able to build a Quirky (or pup) from packages that I have compiled entirely from source, rather than relying on the effort of some other distro, I thought the cherry on the cake would be if I could actually get it to compile LibreOffice.

This blog post is to report success. Well, almost. There was a little fudge that I had to do to get Libreoffice to compile, then the "do_install" function failed. However, it is all there, and I can create a binary package manually.

I am communicating with Andreas, the maintainer of the LibreOffice recipe in OpenEmbedded, so hopefully the "fudge" can get fixed. For now, it would be good to document how I got this far. Remember, I am an almost absolute beginner with OE/Yocto, so there is room for improvement. However, documenting my steps might help others. So, here we go...

The host system
The host operating system has to be setup to be OE-friendly. There are online instructions for the major OS's, but I am running Quirky Linux 8.1.6, x86_64. This is a full installation, not one of those frugal thingies that you can do with Puppy Linux.
Quirky, being built with Ubuntu 16.04 Xenial Xerus DEBs, is pretty close to OK, just needs some tweaks, as documented here:
Running Quirky 8.1.6 x86_64

'devx' PET must be installed.

Needs 'setterm' utility, this is in the 'util-linux' DEB, download and extract
this utility, to /usr/bin:
http://packages.ubuntu.com/xenial/util-linux

check (ok in quirky):
must have "messagebus" and "shutdown" groups in /etc/group
(had to add this to quirky):
must have "crontab" group.

bitbake requires python 3.4.0 or later.
have installed "python3" from the PPM.
this symlink is needed:
# ln -s python3.5 /usr/bin/python3
<

Downloading OE
Previously, I had followed the getting-started page for Yocto, and checked out the "morty" release.
However, I ran into issues when trying to incorporate the "meta-office" layer (which has LibreOffice recipe).
So, I decided to do things at a bit differently. I downloaded from the OpenEmbedded git repository, like this:
create these folders, then cd into:

# cd /mnt/sda1/projects/oe/downloads-oe

# git clone git://git.openembedded.org/meta-openembedded meta-openembedded --depth 1
# git clone git://git.openembedded.org/openembedded-core openembedded-core --depth 1
# git clone git://github.com/schnitzeltony/meta-office.git meta-office --depth 1
# git clone git://git.openembedded.org/bitbake bitbake --depth 1

# cd ..
# mkdir oe-project
# cp -a downloads/openembedded-core/* oe-project/
# cp -a downloads/openembedded-core/.templateconf oe-project/
# cp -a -f --remove-destination downloads/meta-openembedded/* oe-project/
# cp -a downloads/meta-office oe-project/
# cp -a downloads/bitbake oe-project/
<

The "--depth 1" means that no history is being downloaded, so I am working with the very latest only, of master branch.

Folder "oe-project" is now ready for action!

There is just one little hack needed, before can get going. Quirky, like Puppy, we run as the root user. We could probably set this up for a non-root user, but leaving things as they are, a couple of lines need to be commented out:
edit meta/classes/sanity.bbclass, comment-out:


if 0 == os.getuid():
raise_sanity_error("Do not use Bitbake as root.", d)

Like this (keep the indentation):

# if 0 == os.getuid():
# raise_sanity_error("Do not use Bitbake as root.", d)
<

The only downside to doing this, at completion of the build a whole lot of warning messages will be spitted out on the terminal, containing the text "is owned by uid 0, which is the same as the user running bitbake". These are warnings only, not errors.

The next step is to setup the build environment. That will be the next blog post.

Tags: oe

Yocto 2 woofQ 2 Quirky

April 18, 2017 — BarryK
Have completed all of the steps. Compiled from source in Yocto, imported the x86_64 binary packages into woofQ, and built a Quirky distribution. It works, there is a desktop.

So far, have only done a "core-image-sato-dev" build in Yocto, with target-native SDK components included, as described here:
http://bkhome.org/news/201704/oe-native-compiling.html

It is configured to create binary DEB packages, however, I ran into multiple issues with importing them into woofQ.
Instead, I wrote a script named '0pre-yocto' in woofQ, that imports entire un-split binary packages, as .tar.xz files. It also creates the Puppy-format database.

As this is early days with Yocto, to get a reasonable desktop I used many packages from April, for example Gnumeric and Dia. These work fine in the Yocto build.

Installed to a USB stick, booted, got a desktop. Although a "devx" PET has been created, have not yet tested whether there is a sane compiling environment.

'core-image-sato-dev' has some packages that I would rather do without, 'pulseaudio' for example. Next on the to-do list is to have a go at removing pulseaudio, see if it will still compile.

Read more...

Yocto books

April 13, 2017 — BarryK
After learning that it is possible to create target-native compiling tools, such as gcc, my interest in OE/Yocto got revitalised. See previous blog post:
http://bkhome.org/news/201704/oe-native-compiling.html

I have just completed a build with Yocto, and it creates binary DEB packages for the target machine, and yes, that includes 'gcc', 'automake', 'autoconf', 'make', etc. Looks good.

Haven't actually installed it into a partition and tested if can actually compile anything, but intrigued enough to want to learn more about Yocto. It is a very big and complicated project, and the online docs are daunting. So, I have ordered a couple of books:

Embedded Linux Systems with the Yocto Project
https://www.amazon.com/gp/product/0133443248/

Embedded Linux Development with Yocto Project
https://www.amazon.com/gp/product/1783282339/

There are ebooks, but I like to have the paper! Costs a bit more -- total was just on AU$100, about US$75, including shipping to Australia.

Amazon's international shipping is odd. I say this from past experience. This order, that second book came up as does not ship to Australia, so clicked on the "22 New" link, and found that Amazon ships it to Australia, just costs a little more.
Had to go through this rigmarole even though I was logged in and had ticked the checkbox to only show items that can ship to Australia.

Anyway, looking forward to getting them!

Read more...

OE native compiling

April 13, 2017 — BarryK
Native compiling
My misunderstanding of the word "native" in the OpenEmbedded documentation, has been bothering me.

I posted about my latest go at using OE here:
http://bkhome.org/news/201704/openembedded-morty.html

I thought a "native toolchain" would be one that runs in the target machine. But not so. As clarified here:
https://lists.yoctoproject.org/pipermail/yocto/2015-January/023145.html

..."native" refers to the host system.

However, this site is using "native" to refer to Yocto compiling in the target machine:
http://trac.gateworks.com/wiki/Yocto/NativeCompile

...so, it can be done!

Slackware from Scratch
Changing the subject, I was wondering how Slackware is built from scratch. Well, it seems that it isn't, or rather, it is a cobbled-together approach. Apparently, they have used LFS to get going, then their build scripts to build packages.

There is no automated equivalent to LFS, however, some guys have been developing Slackware from Scratch, with automated build scripts, see this thread:
http://www.linuxquestions.org/questions/slackware-14/slackware-from-scratch-and-x11-4175560702/

...interesting!

Tags: oe

OpenEmbedded Morty

April 10, 2017 — BarryK
Last week I spent a few days trying to use OpenADK, tried all kinds of targets and 32-bit and 64-bit hosts, simple basic configurations, always failures. I had the same experience when I tried it early 2016.
Strangely though, the project is active, so some people must be having success. It would have to be for very specific config and host system, details that are not provided.

So, I have moved on to OpenEmbedded. I wrote about this back in June 2016. It is one that does work. In fact, it is the only cross-compiler toolchain/sdk builder thingy that does work. My experience with OpenADK, Crosstool-NG and T2 is that they are all broken.

Here is my June 2016 blog post on OpenEmbedded:
http://bkhome.org/news/201606/playing-with-cross-compile-toolsets.html

And Yocto, which uses OpenEmbedded:
http://bkhome.org/news/201606/yocto-project.html

At that time, the stable release was named "Krogoth". It is now "Morty". See release notes:
https://wiki.yoctoproject.org/wiki/Releases

I want to create a new light-weight pup compiled from source, like I did with Quirky April, and earlier on there was Wary and Racy Puppy. These were built from packages compiled in T2. Unfortunately, T2 has become increasingly broken, and there is only one guy maintaining it.

So, there is only one choice left, OpenEmbedded. I don't know what extra functionality Yocto will bring to the table, so decided to stay with OE.
Right now, doing a target "core-image-sato-dev" build (for machine "qemux86-64", using glibc):
https://layers.openembedded.org/layerindex/recipe/659/

What I want to do is create a bunch of binary packages, that I can then import into woofQ and build a new Quirky.
I suppose it needs a name, so how about "OpenPup"? -- just did a search, can't find that name used anywhere.

Anyway, the OE build is chugging away right now. From prior experience, I have a high confidence that it will succeed.

Read more...