site  contact  subhomenews

EasyDD now verifies write

August 27, 2019 — BarryK

Myself and a couple of other people have had issues with faulty USB Flash drives. This is an insidious problem, failing bits may go unnoticed, just causing damage. Or it could be noticed, even causing failure to boot.

I have proposed two ways to tackling this. Firstly, EasyDD can be enhanced to verify a write. Secondly, some checksums can be performed at bootup.

I have implemented the first one. EasyDD is a utility in EasyOS, CLI or GUI, for writing an image file to a drive. An explanation of EasyDD is here:

https://easyos.org/install/how-to-write-easyos-to-a-flash-drive.html

Verification is implemented in two steps. Firstly, the image file is written to the Flash drive with all bits flipped (inverted). It is then read back to verify.

Then, the normal image file is written to the drive, and read back to verify.

This verifies that the bits are functioning, able to flip to 1 or 0, as well as verifies that the image file has been written successfully.

Inverting the image file is an interesting exercise. It has to be done in a pipeline, between uncompressing of the image file, and writing to the drive. There is a method using the 'tr' utility, as described here:

https://unix.stackexchange.com/questions/151003/how-could-i-flip-content-of-a-binary-file-with-bash-commands

...which is very slow. A C utility to do it is here:

https://unix.stackexchange.com/questions/104585/bit-wise-complement-with-dd

I have named this utility 'bitflip' and it will be in the next release of EasyOS. The 'easydd' utility checks for existence of 'bitlfip' and if not existing, uses 'tr'. 

I have uploaded the latest easydd here:

http://bkhome.org/files/easydd.gz

...it lives at /usr/sbin, and will need to have it's executable permissions set after uncompressing.  You will find it to be a bit slow without the 'bitflip' utility. 

Tags: easy

Request to mirror EasyOS from ibiblio.org

August 27, 2019 — BarryK

EasyOS is hosted at distro.ibiblio.org, for which I am extremely grateful. Ibiblio.org has provided a hosting service for Linux distributions for a very long time, I think about 15 years, maybe more.

The hosting on ibiblio.org can be mirrored, using rsync, however, EasyOS currently has no mirrors. To take the strain off the ibiblio servers, it would be great if there were some mirrors.

If you are administrator of a site, and would like to do this, it would be most helpful. Or, you know of a site that would be prepared to do this, and are able to request it.

This is the URL for EasyOS, total size is about 70GB:

http://distro.ibiblio.org/easyos/

Ibiblio.org maintains a forum, with threads on rsync and ftp:

https://answers.ibiblio.org/topics/rsync.html

https://answers.ibiblio.org/topics/ftp.html

Here is how OpenMandriva does rsyncing from ibiblio.org:

https://wiki.openmandriva.org/en/Mirroring 

If you do it, let me know. The alternative download URL can then be added to the PETget package manager (see "petget" icon at top of screen). Note, in EasyOS, the repositories used by PETget are listed in /root/.packages, files 'DISTRO_COMPAT_REPOS' and 'SISTRO_PET_REPOS'.  So, you could edit latter file (the 'PET_REPOS' and 'SFS_REPOS' variables) and test your mirroring.  That last variable is used by "SFSget" (see "sfsget" on desktop). 

Tags: easy

WoofQ 20190827 tarball uploaded

August 27, 2019 — BarryK

WoofQ is the build system used to create EasyOS from a repository of binary packages. In the case of the Pyro series, the binary packages were compiled from source (using another tool, oe-qky-src), for the Buster series, Debian 10 DEBs are used.

WoofQ is not maintained as an online version control system, instead is just uploaded as tarballs. The latest upload is dated August 27, 2019. This is the WoofQ used to build EasyOS Pyro series version 1.2 and Buster series version 2.1.

Well, not quite. There was a minor bug in the initrd, in the 'fixlayers' script, which applies fixes and sanity checking when SFS layers change. It was a typo, "cut =f ..." where it should have been "cut -f ...". That is fixed in the uploaded tarball.

WoofQ tarballs are here:

http://distro.ibiblio.org/easyos/project/woof/ 

Note, if you are using EasyOS 1.2 or 2.1, it is easy to fix that typo yourself. Just click on the 'initrd' file and it can be opened, and you can edit and change anything inside. Then, you click on 'initrd' again to update it. 

Tags: easy

Easy Buster 2.1 released

August 25, 2019 — BarryK

This is the start of EasyOS "Buster" series, versions 2.x. Announcement blurb:

EasyOS versions 1.x are the "Pyro" series, the latest is 1.2. Easy Pyro is built with packages compiled from source using 'oe-qky-src', a fork of OpenEmbedded. Consequently, the builds are small and streamlined and integrated. The Pyro series may have future releases, but it is considered to be in maintenance mode.

The "Buster" series start from version 2.0, and are intended to be where most of the action is, ongoing. Version 2.0 was really a beta-quality build, to allow the testers to report back. The first official release is 2.1.

The main feature of Easy Buster is that it is built from Debian 10 Buster DEBs, using WoofQ (a fork of Woof2. Woof-CE is another fork of Woof2, used to build Puppy Linux).

The advantage of Buster over Pyro is access to the large Debian package repositories. That is a big plus.

On the other hand, DEB packages have many dependencies, and the end result is a release considerably larger than Pyro with similar app selection. For example, the download file of Pyro 1.2 is 418MB, Buster is 504MB -- despite the Buster build having less apps (Pyro has Qt5 and big Qt5-based apps such as Scribus, this is all missing from the Buster build, but can be installed).

Another problem with Buster, potential problem anyway, is the choice of underlying infrastructure. For example, Debian uses systemd, pam, avahi and policykit -- EasyOS does not use these, but as they are dependencies of many apps, they have to be installed, yet inactive.

Finally we have arrived and 2.1 is out. EasyOS is an experimental distribution, and some features are a work-in-progress, and likely to be so for the foreseeable future. So bare that in mind, that the operation might not be "just works" in every respect as you might expect from an official release of one of the major distributions such as Debian or Ubuntu.

Detailed release notes are here:

http://distro.ibiblio.org/easyos/amd64/releases/buster/2.1/release-notes-2.1.htm

It is recommended to also read this "readme" file:

http://distro.ibiblio.org/easyos/amd64/releases/buster/2.1/readme.htm

Download:

http://distro.ibiblio.org/easyos/amd64/releases/buster/2.1/

Feedback is welcome, in this forum thread:

http://murga-linux.com/puppy/viewtopic.php?t=109958

Have fun! 

Tags: easy

Easy Pyro version 1.2 released

August 25, 2019 — BarryK

Version 1.1.1 was supposed to be the EOL (End Of Line) for the Pyro series, superceded by the Buster series. Buster version 2.1 will be announced later today.

I am fond of the Pyro series, and have decided to keep it going, though it will likely only be the occasional maintenance release. Here is an announcement blurb for 1.2:

At the time of writing, Easy Buster 2.1 is being released. Although the Buster series is intended to receive most attention ongoing, Barry decided to keep doing maintenance releases of the Pyro series.

Although the package versions, such as gcc, glibc and libraries, are starting to get a bit "long in the tooth", Barry retains a fondness for them due to the very small size and targeted configuration. That is, the packages were compiled from source, using 'oe-qky-src', a fork of OpenEmbedded, so there are no unwanted dependencies, meaning no bloat.

Consider, the 'easy-1.2-amd64.img.gz' download file is 418MB. This includes the "kitchen sink" of applications, both GTK2, GTK3 and Qt5 based apps. On the other-hand, Buster 'easy-2.1-amd64.img.gz' is 504MB and has less apps -- Qt5 libraries and apps such as Scribus (desktop publisher) were left out, to limit the size.

Yes, Buster works, and has the advantage of access to the huge Debian package respositories (which was the main reason to go for it). However, small size, and the seamless integration off all packages, are the arguments in favour of Pyro. Pyro has a much smaller on-line app repository.

Version 1.2 has a kernel bump to 5.2.9, some bug fixes, infrastructure improvements, and the new "Copy session to RAM & disable drives" boot option.

More detailed release notes are here:

http://distro.ibiblio.org/easyos/amd64/releases/pyro/1.2/release-notes-pyro-1.2.htm

It is recommended also to read the "readme" file here:

http://distro.ibiblio.org/easyos/amd64/releases/pyro/1.2/readme.htm

Download:

http://distro.ibiblio.org/easyos/amd64/releases/pyro/1.2/

...note, there are German, French and English builds.

Feedback:

Feedback is welcome, at this forum thread:

http://murga-linux.com/puppy/viewtopic.php?t=109958

Known bugs and issues:

There is one tiny bug in the initrd, fixing of GTK theming when SFS layers change. Non-critical and will be fixed in the next release.

When testing 1.2, one of my USB Flash sticks decided to become corrupted, resulting in a failure to boot. I intend to address the situation of a faulty Flash drive, as it is not just old drives, even new very cheap sticks can be faulty. Firstly, I propose to put some testing into the EasyDD utility. 

Tags: easy

Cannot write to optical drive

August 22, 2019 — BarryK

Yet another serious problem with Easy Buster. I thought that I would burn radky's Busterpup ISO to a CD, to have a look at his implementation of video playing.

I used 'burniso2cd', but the burn failed. Running the actual execution line directly in a terminal:

# xorrecord -dao -data -eject -v speed=10 dev=/dev/sr0 /mnt/sdc1/downloads2/input532/0-busterpup/buster-8.0-uefi-k4.19.56.iso
xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.

libburn : SORRY : Cannot open busy device '/dev/sr0' : Device or resource busy
libburn : FAILURE : Cannot access '/dev/sr0' as SG_IO CDROM drive
xorriso : FAILURE : Cannot acquire drive '/dev/sr0'
xorriso : NOTE : Giving up for -eject whole -dev ''
xorriso : FAILURE : -as cdrskin: Job could not be performed properly.
xorriso : aborting : -abort_on 'FAILURE' encountered 'FAILURE'
#

I found it works if the CD-R is plugged in before powering-on. However, if plug in the CD after bootup, something is making it "busy". I can't see any process that has claimed it. Weird.

Hmmm, I wonder if udev has something to do with this? 

EDIT 2019-08-22:
Yes, it seemed like a udev rule is the cause. When I plug in a CD, pressing the eject button does nothing, it is locked in. The udev rule /lib/udev/rules.d/60-cdrom_id.rules is responsible. I have renamed that file to '60-cdrom_id.rules-disabled', so it will be ignored, then rebooted, and now the optical drive works properly.

Back in the early days when I introduced the 'udev' package to Puppy, I deleted most of the rules, as most were trivial or useless. This has continued right up to now, only a small subset of the 'eudev' rules are kept.

That '60cdrom_id.rules' is one of the kept rules files. It is in Pyro, and I see also in Radky's BusterPup. However, it is doing something wrong in Easy Buster. It calls the 'cdrom_id' utility, and perhaps that utility is misbehaving in Easy -- well, that is the most likely situation.

Anyway, '60-cdrom_id.rules' doesn't seem to do anything that we really need, so seems OK for it to be disabled. 

EDIT 2019-08-23:
The Linux kernel is to blame! This is the content of /lib/udev/rules.d/60-cdrom_id.rules:

ACTION=="remove", GOTO="cdrom_end"
SUBSYSTEM!="block", GOTO="cdrom_end"
KERNEL!="sr[0-9]*|xvd*", GOTO="cdrom_end"
ENV{DEVTYPE}!="disk", GOTO="cdrom_end"

# unconditionally tag device as CDROM
KERNEL=="sr[0-9]*", ENV{ID_CDROM}="1"

# media eject button pressed
ENV{DISK_EJECT_REQUEST}=="?*", RUN+="cdrom_id --eject-media $devnode", GOTO="cdrom_end"

# import device and media properties and lock tray to
# enable the receiving of media eject button events
IMPORT{program}="cdrom_id --lock-media $devnode"

KERNEL=="sr0", SYMLINK+="cdrom", OPTIONS+="link_priority=-100"

LABEL="cdrom_end"

Back in the pre-udev days, we had to probe the optical drive to detect if a media has been removed. This was done every 4 seconds, in the /usr/local/pup_event/frontend-timeout script. This had to be done, because the kernel is not informed when an optical media is removed.

When I disabled '60-cdrom_id.rules', fine, xorriso worked, however, when the CD was ejected, the desktop icon remained there. There was no information provided to cause it to be removed.

The above udev rule takes care of that. The cdrom_id utility does something to pickup pressing of the eject button on the optical drive. Unfortunately, the 5.2.7 kernel as used in latest Easy Buster, now thinks that the optical drive is busy, and won't let any other app access it.

I verified this by restoring '60-cdrom_id.rules' and booting up with the 5.2.4 kernel, that I was using for earlier builds of Easy Buster. The problem has gone away!

I need to examine the kernel source changes since 5.2.4. If this issue is not identified, I will stay with 5.2.4. 

EDIT 2019-08-23:
I compiled the 5.2.9 kernel, same configure as for 5.2.4 and 5.2.7, this time left out the "cap_sys_mount" patch, just in case that is the cause.

I am able to manually eject a DVD, but gnome-mpv fails to play it, reports lots of read errors. I changed to the 5.2.4 kernel, rebooted, ran gnome-mpv again. it plays the same DVD perfectly.

Something has gone wrong with the kernel. I want to get Easy 2.1 out, so will stay with the 5.2.4 kernel.

Oh, there is one difference between compiling of the 5.2.4 and 5.2.9 kernels. For the former, used Debian's gcc version 7, for the latter used version 8 -- Debian has both in the Buster repos. 

EDIT 2019-08-24:
The saga continues. I compiled the 5.2.9 kernel in Easy Pyro 1.1.1, and CD ejection and playing works fine. I then used the same kernel and modules in easy Buster 2.1pre, and there was trouble -- eject button worked OK, though there is some inconsistency, but gnome-mpv gave read errors when tried to play a DVD movie.

So, the cause of the problem is not the kernel. The actual line in the above udev file that causes the problems:

KERNEL=="sr[0-9]*", ENV{ID_CDROM}="1"

So, I am thinking that this line triggers something else, perhaps another udev rule, that is interfering with usage of the optical drive.  A udev rule that is in Buster, not in Pyro.  

EDIT 2019-08-24:
Bingo! Yes, after going on a meandering path trying to think what the cause of the optical misbehaviour was, I came to think that it was likely to be another udev rule that is conflicting with my intended usage.

So I did a search for the string "cdrom" or "CDROM" in the udev rules, and came up with a few hits. Then realised the likely culprit is /lib/udev/rules.d/80-pktsetup.rules:

# Create and remove packet writing device for each optical block device
ACTION=="add", SUBSYSTEM=="block", ENV{ID_CDROM}=="1", RUN+="/usr/sbin/pktsetup %E{MAJOR}:%E{MINOR}"
ACTION=="remove", SUBSYSTEM=="block", ENV{ID_CDROM}=="1", RUN+="/usr/sbin/pktsetup -d %E{MAJOR}:%E{MINOR}"

Disabled that file, and optical media ejection and DVD movie playing work. Previously, I was getting inconsistent results, sometimes able to eject a CD sometimes not, sometimes able to play a movie sometimes not. There was a fight going on. The above rules caused a process to launch, that did not terminate, 'pktcdvd0', which was doing the fighting.

This rules file is in the 'udftools' DEB, which is described as "tools for UDF filesystems and DVD/CD-RW drives". I only included it in the build as it looked interesting. Well it is now out. 

EDIT 2019-09-03:
Thomas Schmitt, the author of Xorriso, contacted me about this problem, and he requested certain tests. This pretty much nailed it down:

# cdrskin dev=/dev/sr0 -toc
...
cdrskin: SORRY : Cannot open busy device '/dev/sr0'
cdrskin: ( Most recent system error: 16  'Device or resource busy' )
...
# cdrskin --drive_not_o_excl dev=/dev/sr0 -toc
...
cdrskin: status 4 burn_disc_full "There is a disc with data on it in the

Quoting Thomas:

The second attempt to access the drive succeeded.
So it is indeed the Linux specific locking protocol for device files
by flag O_EXCL, which causes the EBUSY error on open(2).

Thomas has analysed the kernel source, and thinks that this exclusive locking behaviour is in recent kernels. He has informed one of the kernel developers, Pali Rohar. 

Tags: easy

Video playing in Easy Buster not so good

August 22, 2019 — BarryK

Another hurdle, just when I was thinking version 2.1 is ready for release. I tried the Xine media player, and it hung the entire desktop, required pressing of the reset button. This is on my HP midi-tower, with Intel graphics. Experimenting...

Running xine with a specific video driver, "# xine -V <driver>":

vdpau
Crashes at startup
xv
Crashes at startup
vaapi
Starts then hangs the desktop
sdl
Works, but cannot resize video or run full-screen
opengl
Works

Trying my collection of test video files, using the opengl driver, plays them, but my Big Buck Bunny MP4 videos play with screen flicker. DVD movie plays OK.

Then I installed VLC from the Debian repository -- after all, this is the main reason for going with binary compatibility with Debian, access to the huge repository.

VLC runs, but all videos play with gross video and audio distortion. Doesn't matter what video driver and video acceleration is chosen in the settings. Oh cr*p. 

Tags: easy

Progress heading toward Buster 2.1

August 20, 2019 — BarryK

In Easy Buster 2.0, the Easy-desktop-in-container has corrupted wallpaper and other corrupted graphics. I fixed that, well, it is a workaround for now. In Pyro, /mnt/wkg/containers/easy/configuration file has "EC_NS_UNSHARE_IPC='true'", however, in Buster it has to be 'false'. Hmmm, anyway, it fixed the graphics, will go with that for now.

However, another problem, video-related again. I built Buster 2.1 German edition, and booted it from USB-stick. For non-English builds, the initrd runs Xorg and full GUI apps to ask for keyboard layout and password. It achieves this by setting up a temporary aufs layered filesystem with easy.sfs as the bottom layer, then chroot into it.

Works great in Pyro, can run gtkdialog-based GUIs, and use translation with gettext. However, in Buster, Xorg fails. It loads the 'fbdev' driver OK, and 'libfbdehw.so' module, but reports "No devices detected" and aborts. But, /dev/fb0 is there, and works.

Very odd. Probably something to do with how Debian have configured Xorg when it was compiled. There is no hint in Xorg.0.log as to why it cannot find /dev/fb0. I am using a custom 'xorg.conf'.

Stumped for now, but will keep at it. Buster 2.1 will be a little bit delayed. 

EDIT 2019-08-21:
I do not know why the Xorg framebuffer driver does not work in the initrd. I have a customised 'xorg.conf' that explicitly selects the 'kbd', 'mouse' and 'fbdev' drivers and specifies device "/dev/fb0". However, when try to launch Xorg in the initrd, /var/log/Xorg.0.log has this in it:

[     7.480] (II) Loading /usr/lib/xorg/modules/libfbdevhw.so
[ 7.481] (II) Module fbdevhw: vendor="X.Org Foundation"
[ 7.481] compiled for 1.20.4, module version = 0.0.2
[ 7.481] ABI class: X.Org Video Driver, version 24.0
[ 7.481] (EE) No devices detected.

That's the Debian Xorg. Now, when I put in Xorg and 'kbd', 'mouse' and 'fbdev' drivers from Pyro, it sees and uses /dev/fb0:

[     7.158] (II) Loading /usr/lib/xorg/modules/libfbdevhw.so
[ 7.158] (II) Module fbdevhw: vendor="X.Org Foundation"
[ 7.158] compiled for 1.19.6, module version = 0.0.2
[ 7.158] ABI class: X.Org Video Driver, version 23.0
[ 7.158] (II) FBDEV(0): using /dev/fb0

I have worked around the problem by creating a PET package with the appropriate Xorg files from Pyro, named 'xserver-fbdev-alt-20190821-pyro64.pet'. The 'init' script in the initrd copies files from it and replaces the Debian ones. This only happens inside the initrd, the replacement is temporary.

Not satisfactory, I would prefer to know why the Debian Xorg is failing.  Anyway, the workaround works.  

Tags: easy