arp-scan, bbe, pnscan added to OE

For Pyro64 0.5, I had compiled these as PETs. have now moved them into OE.

arp-scan:

https://github.com/bkauler/oe-qky-src/commit/fe8329a2b08204ea39092fd692538d1744b504c3

bbe:

https://github.com/bkauler/oe-qky-src/commit/fc72d0a867b14d27702aba6495c4eb7853357dc8

pnscan:

https://github.com/bkauler/oe-qky-src/commit/edac8ce4c70ead37e8a1a5a5a40cabc760fa5a8f

Tags: oe

ntpdate utility compiled in OpenEmbedded

Most of the binary packages used to build Quirky Pyro64 0.5 were compiled in OE, but not all. A few failed to compile, so I compiled them in a running Pyro64, and a few were never attempted in OE -- so these are all PETs.

Utility ntpdate is used by tasmod's Psync app, that syncs the time/date from an NTP server. Quirky/Easy have a rewritten version of Psync, named Qsync. This utility is in the ntp package.

I compiled ntp as a PET, because the recipe in OE is unsuitable. Unfortunately, although systemd is supposed to be an optional feature in OE, some developers are hard-coding it into build recipes. This makes life difficult for me, as I have to write my own recipe for that package.

I will turn this post into a small tutorial, showing how I wrote my own recipe for ntp.

A search with find revealed the recipe here:

meta-networking/recipes-support/ntp/ntp_4.2.8p10.bb

If I could just modify that recipe to suit my needs, I could have created this file:

meta-quirky/recipes-support/ntp/ntp_4.2.8p10.bbappend

...where meta-quirky is my custom layer. See the online github repo:

https://github.com/bkauler/oe-qky-src

However, I wanted to rewrite the recipe from scratch. The thing is, do not modify the original, create a new one. The new recipe, in my custom layer, will override the original:

meta-quirky/recipes-support/ntp/ntp_4.2.8p10.bb

How to write a recipe? Actually, this is made easy (well, easier) by a special script:

# recipetool create -o ntp_4.2.8p10.bb https://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-4.2/ntp-4.2.8p10.tar.gz

...this is neat. You specify the URL of the source package, and required name of the recipe file, and it will create a skeleton recipe. Which most likely will need further tweaking.

Examining the original recipe helps, as does having already compiled the package in a running Pyro64. From previous experience having compiled this package, I knew what configure options are needed, and the dependencies.

Dependencies, ah, this is a bit of a dark art. A few years ago, OE was structured differently. Now, each package is compiled in its own isolated mini-filesystem. You have to know all the required deps, and specify them. Not quite all, as the basic packages such as glibc are provided.

Here is my ntp_4.2.8p10.bb:

LICENSE = "MIT & LGPLv3"
LIC_FILES_CHKSUM = "file://COPYRIGHT;md5=e877a1d567a6a58996d2b66e3e387003 \
                    file://libjsmn/LICENSE;md5=5adc94605a1f7a797a9a834adbe335e3 \
                    file://sntp/COPYRIGHT;md5=e877a1d567a6a58996d2b66e3e387003"

SRC_URI = "https://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-4.2/ntp-${PV}.tar.gz"
SRC_URI[md5sum] = "745384ed0dedb3f66b33fe84d66466f9"
SRC_URI[sha256sum] = "ddd2366e64219b9efa0f7438e06800d0db394ac5c88e13c17b70d0dcdf99b99f"

# openssl has libcrypto...
DEPENDS = "bison-native libevent readline libcap openssl"

inherit perlnative autotools pkgconfig

# --with-yielding_select=yes is required when cross-compiling.
EXTRA_OECONF = "--with-yielding_select=yes --enable-ipv6 --with-crypto --enable-linuxcaps --with-lineeditlibs=readline"

# only want this one executable...
do_install () {
    mkdir -p ${D}/usr/sbin
    install -m 755 ${B}/ntpdate/ntpdate ${D}/usr/sbin
}

HOMEPAGE = "http://www.ntp.org/"
SUMMARY = "NTP is a protocol designed to synchronize the clocks of computers over a network"

I have added a few things to the original skeleton. For example, the original only has "bison-native" as dependency.

As I only want the ntpdate utility, I created a custom install function. Those variables are pre-defined: D=destination, B=build and S=source paths. The build folder is usually different from the source.

To examine the end result, the entire build is here (in the case of my own hard drive):

/mnt/sdb1/projects/oe/oe-quirky/buildPC/tmp-glibc/work/nocona-64-oe-linux/ntp/4.2.8p10-r0

...I did an "ldd ./ntpdate" to check that it had all the expected libs. Yep, ok.

To do the build, I added the ntp package by appending this line "IMAGE_INSTALL_append = " ntp"":

/mnt/sdb1/projects/oe/oe-quirky/buildPC/conf/local.conf

Then, to do the actual build:

# bitbake -c build ntp

If there is an error, and I did get one, I advise cleaning out the entire build. Although you could just run the "build" command again, some packages won't like it. The "clean" command erases everything, all of folder "4.2.8p10-r0" shown above.

# bitbake -c clean ntp

...then run a "build". The final installed result can be seen here:

/mnt/sdb1/projects/oe/oe-quirky/buildPC/tmp-glibc/work/nocona-64-oe-linux/ntp/4.2.8p10-r0/image

...it is the content of this image folder that will be imported into woofQ. However, one final step:

# bitbake -g core-image-quirky

...this creates some package-list and package-dependency files, pn-buildlist, recipe-depends.dot and task-depends.dot. The woofQ import script reads these, which is why they must be updated.

woofQ

Over in woofQ, there are two scripts for importing binary packages from OE, 0pre-oe and 0pre-oe-add. The former was run originally, the latter is for importing additional packages. Just run 0pre-oe-add:

# ./0pre-oe-add

...and it picks up the new ntp package and offers to import it.

It will also pickup packages previously in OE, now removed, and offer to remove them from woofQ.

Tags: oe

Quirky Pyro64 0.5 coming soon

Advance notice, I plan to upload Quirky Pyro64 0.5 in a couple of days. This is Quirky Linux built from packages compiled from source in OpenEmbedded.

Right now, have started a recompile from scratch in OE, then tomorrow should be able to import the binary packages to woofQ, then build Pyro64 0.5.

I intend to upload all of the binary packages, and the source packages, to ibiblio. The binary packages are about 1.2GB, and uploading this lot is not something that I want to do that often, as my only Internet access is via my mobile phone.

I don't have a landline phone, just a mobile. I am on prepaid, not a plan, and pay AU$40 for 6GB, so that's my allocation per month, which is pitiful. I can top it up within a month, AU$10 for 1GB.

These Internet expenses add up, and I am thinking of bringing back a PayPal donate button. I had one years ago, and enough donations trickled in to pay the Internet costs. But that was when I was leading the Puppy Linux project.

Well, I will think about whether to put up a donate button or not. Actually, what prompted me to think about this, was recently there have been a couple of enquiries from people who wanted to donate to my work.

Samba

Note, 0.5 will have Samba, that I have compiled in a running Pyro64, not in OE, so it is a PET package. The Samba in OE wants kerberus dependency, which I did not want, so experimented with configuration options, for the smallest build, and using the inbuilt heimdal instead of kerberus.

Even so, my PET is 11MB, whereas 01micko got his samba PET down to about 8MB -- though he has an older version.

I hope that I haven't configured-out too much functionality, and that it works OK. Well, testers will be welcome when 0.5 comes out.

For the record, here are my configure options:

# ./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc --libdir=/usr/lib --enable-fhs --includedir=/usr/include --bindir=/usr/bin --sbindir=/usr/sbin --with-configdir=/etc/samba --with-piddir=/var/run --with-privatedir=/etc/samba/private --with-privatelibdir=/usr/lib --with-modulesdir=/usr/lib --with-lockdir=/var/cache/samba --with-logfilebase=/var/log/samba --enable-cups --with-acl-support --with-automount --with-quotas --with-syslog --without-winbind --with-ldap --without-pam --without-ads --libexecdir=/usr/libexec --datadir=/usr/share --without-dmapi --without-fam --without-lttng --without-systemd --disable-avahi --enable-gnutls --without-sendfile-support --nopyc --nopyo --extra-python=/usr/bin/python3 --without-ad-dc --without-ntvfs-fileserver

...if you know something about configuring Samba, what do you think? Note that Pyro64 has all 64-bit libs in /lib and /usr/lib, with /lib64 and /usr/lib64 being symlinks.

Feedback can go here. This is where 0.5 will be announced:

http://murga-linux.com/puppy/viewtopic.php?t=110541

Tags: oe, quirky

The return of Pyro64

Every now and again, I get frustrated by the bloatedness and weirdness of Ubuntu and Debian DEBs, that I am using in Quirky and Easy builds. I did play with Pyro64, packages compiled from source in OpenEmbedded, but only briefly.

But have just completed a compile in OE, for the x86_64 architecture (not x32), and built a new Pyro64, and it sure is looking good.

The difference in size of deployed distro is incredible. Quirky Pyro64 (now at version 0.3.3) is 274MB, and that includes LibreOffice, whereas Quirky Xerus 8.3 is 331MB. These have about the same functionality. Furthermore, I will probably be able to reduce the size of the Pyro64 build, as this is an early pre-alpha build.

If Pyro64 is so good, why did I previously go back to building with Ubuntu DEBs?

Well, there were some issues at the time, for example with the video. However, I have tackled some of these, and things are looking very good.

Then there is the Ubuntu DEB repository. I guess that is the big thing, so convenient. With Pyro64, everything has to be compiled. Almost, there is a small repo of packages that I compiled, and we can add to it.

So, 'oe-qky-src' is active again:

https://github.com/bkauler/oe-qky-src/commits/master

Oh, one other thing. SeaMonkey is running without spitting out heaps of complaints to stderr -- that is very reassuring. I am wondering if I am just imagining things, but everything seems faster, including SeaMonkey.

Tags: oe

Intel sna and uxa video acceleration

About four months ago, when I released Pyro64 0.2, built from binary packages compiled in OE, there was feedback about video rendering problems with Intel video hardware.

Blog post announcing Pyro64 0.2:

http://bkhome.org/news/201705/quirky-pyro64-02-alpha.html

Forum feedback:

http://murga-linux.com/puppy/viewtopic.php?t=110541

Today I investigated this, and found that OE has set the video acceleration to "sna", without support for "uxa". sna is intended to replace uxa, and has superior performance, but still has serious issues. The recipe file in OE:

http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-graphics/xorg-driver/xf86-video-intel_git.bb?h=pyro

The interesting parts are:

PACKAGECONFIG ??= "xvmc sna udev ${@bb.utils.contains('DISTRO_FEATURES', 'opengl', 'dri dri1 dri2', '', d)}"

PACKAGECONFIG[sna] = "--enable-sna,--disable-sna"
PACKAGECONFIG[uxa] = "--enable-uxa,--disable-uxa"

I am using the "pyro" release of OE. Looking in the github repo, I found that the recipe has been changed recently:

http://cgit.openembedded.org/openembedded-core/diff/meta/recipes-graphics/xorg-driver/xf86-video-intel_git.bb?h=master-next&id=ce935784cfc53c029c8dfa969d08ee43a8831b60

Which has this:

-PACKAGECONFIG ??= "xvmc sna udev ${@bb.utils.contains('DISTRO_FEATURES', 'opengl', 'dri dri1 dri2', '', d)}"
+PACKAGECONFIG ??= "xvmc uxa udev ${@bb.utils.contains('DISTRO_FEATURES', 'opengl', 'dri dri1 dri2', '', d)}"

The problem is, as far as I can understand how these recipes work, that change will have the effect of enabling uxa but disabling sna. Really, I want both to be enabled.

Linux From Scratch explains that both can be enabled:

http://www.linuxfromscratch.org/blfs/view/cvs/x/x7driver.html

Note, there is also another hardware accelerator called "glamor":

https://www.phoronix.com/scan.php?page=article&item=intel_2dxorg30_ubuntu1404&num=1

glamor is enabled in xorg-server, see meta/recipes-graphics/xorg-xserver/

Anyway, I have created recipes-graphics/xorg-driver/xf86-video-intel_git.bbappend in my OE-Quirky overlay, with this in it:

PACKAGECONFIG_append = " uxa"

...which, again, as far as I understand how these recipes are evaluated, should cause "--enable-uxa", and we will also get "--enable-sna", although LFS explains the latter is the default.

Tags: oe

Change core2 to nocona for x86-64

About four months ago, I released a build of Quirky built with binary packages compiled in OpenEmbedded, for x86_64 CPUs.

It was codenamed "Pyro64" and was version 0.2. Here is the announcement:

http://bkhome.org/news/201705/quirky-pyro64-02-alpha.html

There is a link to a thread on the Puppy Forum. One of the testers reported failure due to a missing CPU instruction.

Yeah, the baseline x86_64 in OpenEmbedded/Yocto is defined as "core2". This refers to an early Intel x86_64 CPU introduced in 2006:

https://en.wikipedia.org/wiki/Intel_Core_2

However, there are earlier x86_64 CPUs from Intel and AMD, and these had less instructions. Quoting from here:

https://en.wikipedia.org/wiki/X86-64

The first processor to implement Intel 64 was the multi-socket processor Xeon code-named Nocona in June 2004.

Quoting some more:

Due to a lack of success with Intel's Itanium and Itanium 2 processors, AMD was able to introduce x86-64, a 64-bit extension to the x86 architecture. Intel followed suit by including Intel 64 (formerly EM64T; it is almost identical to AMD64) in the 90 nm version of the Pentium 4 ("Prescott"), and a Xeon version codenamed "Nocona" with 1 MB L2 cache was released in 2004.

Now, looking at what CPU options are available to configure gcc:

https://gcc.gnu.org/onlinedocs/gcc-6.4.0/gcc/x86-Options.html

This option for '-march' is what I want:

nocona:
Improved version of Intel Pentium 4 CPU with 64-bit extensions, MMX, SSE, SSE2 and SSE3 instruction set support

I have modified my fork of OE to use "-march=nocona" instead of "-march=core2".

Basically, what I have done is create "tune-nocona.inc", based on "tune-core2.inc", see here:

https://github.com/openembedded/openembedded-core/blob/master/meta/conf/machine/include/tune-core2.inc

Curious, you can see in above link, "-march=core2 -mtune=core2 -msse3 -mfpmath=sse", but there is redundency, as those settings for -mtune -msse3 and -mfpmath are the default for "-march=core2". Hmmm, anyway, I have retained that redundancy, and now have "-march=nocona -mtune=nocona -mno-sse3 -mfpmath=sse" -- I have explicitly disabled sse3, as I think there are some early x86_64 CPUs that do not have those instructions.

I have not yet submitted this to my OE github repo, want to see if it works first.

Tags: oe

OpenEmbedded x86_64 x32

I recently received an email from Jorge, asking if I was still interested in x86_64 x32 mode. He has been playing with a x32 port of Debian.

I did briefly look at OE and x32 mid-2016:

http://bkhome.org/news/201606/yocto-project.html

At first I thought, don't really want to get back into that. For a speed gain and smaller binaries, there are a lot of hassles. Some packages will not compile, some will need patches, some might compile then not work properly.

Nevertheless, I got intrigued and decided to see how far a x32 build will get in my latest OE port.

My blog posts on OE:

http://bkhome.org/news/tag_oe.html

My port of OE:

https://github.com/bkauler/oe-qky-src

Debian x32 port:

https://wiki.debian.org/X32Port

http://debian-x32.org/

https://en.wikipedia.org/wiki/X32_ABI

https://sites.google.com/site/x32abi/

Jorge has reported difficulty with compiling SeaMonkey, but he has some patches, in case I want to tackle it.

Don't know yet. Just playing. OE is rocketing along right now, showing 18% complete, no compile failures so far.

EDIT 2017-10-19:

Got up this morning, examined the overnight compile. OE has completed the build, 650 packages successfully compiled (18 of those are no-arch, not needing compilation), only 6 failures.

The failures are:

kodi, hiawatha, xf86-video-sis, notecase, xine-lib, firefox

This is very good. Even the big guy, LibreOffice, succeeded.

EDIT 2017-10-20:

I imported the binary packages into woofQ, then stopped. This is too much of a diversion. It is going to bring with it a heap of issues. Nah, backing off. I am interested in revisiting OpenEmbedded, but only for x86_64, x86 and arm builds.

Tags: oe

No video at first, Pi2 and Pi3

Read more...