site  news  contact

How Easy works, part 2

October 17, 2017 — BarryK

A fairly technical introduction to how Easy works is part-1, here:

http://bkhome.org/easy/how-easy-works.html

Part-1 examines the code executed at early bootup. Part-2 continues, from after bootup has completed, and this is of greater interest to the user. It is however, informative to at least scan through part-1 and pick up the salient points.

The layered filesystem

Recapitulating, here is a diagram from near the end of part-1, giving a pictorial view of the layered filesystem just after bootup:

after-switch-root

The diagram shows a boot partition, which has the Linux kernel and various files required for bootup. Then there is the working partition, which is where you "are" after bootup.

In the above example, the working partition is /dev/sdb2, and it contains folders .session, repository/easy-0.1.6, home and containers. The example is for running Easy version 0.1.6, and one file is shown in the easy-0.1.6 folder: q.sfs.

You can see how those layers are stacked. q.sfs has all of the files of Easy, it is in fact the entire runtime filesystem. And folder .session has your current working session. Just think of that last one as where all your work occurs -- installed packages, downloaded files, setup, and whatever.

What you see at "/" after bootup is the top of those layers. It is very helpful to have an understanding of this, as after booting up Easy and you look at "/" in the file manager, you will no doubt wonder where are all those files physically located? ROX-Filer (the file manager) will show this:

top

So, what you are seeing above, is not the working partition! It is actually a kind of artificial construct, that you can think of as existing in RAM.

The working partition

Staying with the example of the drive being /dev/sdb, with sdb1 being the boot-partition and sdb2 the working-partition. How do you get at those folders and files in sdb2? Actually, it is quite simple and logical, look at the icons on the desktop:

partitions

This is on my (I am writing this tutorial in the first person!) baby laptop, where /dev/sdb is a USB Flash stick that I have booted Easy off. The orange ball at top-right of the sdb2 icon identifies it as unmountable. Click on sdb1 or sdb2 and you can view the contents.
Note
I need to make a comment about these boot and working partitions. The descriptions so far have been treating them as separate, which is the case with the easy-0.1.6-amd64.img.gz download file (or whatever version you download). However, bear in mind that I have designed Easy so that boot partition and working partition could be the same. in which case it would have to be a Linux partition. This will have to be a topic for another time.

So, if you click on the "sdb2" desktop icon, this is what you will see:

working

I configured ROX-Filer to display hidden folders and files, so that you can see .session and .tempwork -- you can look in those folders, but don't change anything, as the aufs/overlay manager will get upset.

The three folders of interest to the user are containers, home and repository:

containers

This is where applications can be run in isolation. Each container is a layered filesystem, very similar to the main one, with ro layer q.sfs, and a rw folder. Additional SFS files may be loaded as ro layers, for example the "devx" SFS for compiling source packages (Puppy users will know all about that!). Containers serve two purposes, isolation and/or security. There is a lot more to say on this topic!

home

Easy, like Puppy, is a single-user system. With Easy, the home folder has a different meaning to that in Linux/Unix, it is, quite literally, your "home". The basic idea is that this is where you store all your "stuff". Downloaded files, photos, videos, anything. I am attempting to get the apps in Easy to open in the home folder whenever "File -> Open..." menu operation is performed, though this is a work-in-progress.
The home folder is permanent storage, it is not affected by the version control mechanisms, upgrading, downgrading, snapshots.

repository

This is a repository for Easy versions and snapshots. For example, if you upgrade from version 0.1.6 to 0.1.7, all of version 0.1.6 is saved in folder repository/easy-0.1.6, and you can at any time roll back -- the menu "Filesystem - > Easy Version Control" handles this.

Now that you are in a running Easy Linux, we can explore from this new viewpoint...

Easy is now running, and the architecture can be further examined. To start, there has been the example given in part-1 of "How Easy works", of upgrading Easy from version 0.1.6 to 0.1.7. But, how actually do you perform an upgrade?

Upgrade to new version

I have not yet written a GUI app to download and install a new version, but it is so easy to do manually, that it is way down on my to-do list. Say that the new easy-0.1.7-amd64.img.gz is available on the Internet. Download it, click on it, and there will be an offer to uncompress, accept, and you have easy-0.1.7-amd64.img. Click on it again, and a ROX-Filer window will open showing all the contents of the 639MB fat32 partition:

image

Now for the upgrade. Click on the desktop icon for the boot partition. In examples given earlier, that was sdb1. OK, ROX-Filer will open and this is what You will see:

image

Drag the new initrd.q, q.sfs and vmlinuz into the boot partition, replacing the old ones, then click on initrd.q, then reboot, and that's it, you have upgraded.

"click on initrd.q"
Don't overlook that step! Drag the files to the boot-partition, but then you must click on initrd.q. This file has a text file inside it, named BOOT_SPECS, which has some variables that must be set to suit the new location. This is done automatically, simply by clicking on it. This window will popup, and you just choose the 'Yes' button:

image

It can't get any easier than that. At a reboot, the bootup code in the boot-partition will create a new repository/easy-0.1.7 folder in the working partition, and will save all the new files (initrd.q, q.sfs and vmlinuz) into it, and perform some other upgrade operations. All automatic.

Easy Version Control

With the complete history saved in the repository folder, everything is in place for rolling back, or forward. There is even a nice GUI to do do it:

image

Change of theme and version! This snapshot was taken after reaching version 0.4. A complete record of versions 0.3.90 and 0.3.1 is retained, and by ticking one of those checkboxes, I can rollback to that version or snapshot.

You can take a snapshot at any time. All that this does is save the current .session folder, the rw layer, as a file named rw-<date>.sfs, in the repository.

Obviously, saving a complete record of past versions is going to occupy a lot of the working partition. However, the history is saved as compressed SFS files. To avoid the history growing to fill all of the partition, there is a maximum depth setting, set to "3" in the above photo -- meaning that only the current version and two prior are kept.

Another thing to consider, is that the user's "home" is outside of this rollback/snapshot mechanism. /mnt/sdb2/home (for the example of sdb2 being the working partition) is where you can keep all your personal files. Downloads, videos, photos, etc. Try and put as much as you can in here, instead of bulking up the .session folder.

repository sub-folder

All of the files belonging to a particular version of Easy are in a sub-folder inside the repository folder, for example repository/easy-0.4. If you were to look inside one of these sub-folders, this is indicative of what you would see:

image

The configuration files have various configuration variables. SFS file devx-*.sfs is familiar to Puppy users -- it contains everything to turn Easy into a compiler environment, and can be selected to mount as a layer at bootup. Extra SFS files such as devx-*.sfs are optional, and can be selected to be added to the layered filesystem at bootup.

It is very easy to add, or remove, an extra SFS file to/from the layered filesystem: see the menu "Filesystem -> Easy BootManager":

image

...a ticked checkbox will mean it is already loaded.

Easy containers

Easy has been designed as "container friendly". I thought about how containers would fit in, that is, be highly integrated with the rest of Easy, extremely easy to setup and use, and be very efficient.

There are ready-made container solutions out there, such as LXC and Docker, however, for now I am playing with "homebrew" containers. I am far from being a security or container expert, but for my own satisfaction I want to play with containers created from basic principles.

So, Easy Containers are not going to be guaranteed secure isolation, and may have other issues, but it is an on-going learning process, and hopefully they will be refined in the months ahead.

It should also be noted that one of the existing solutions such as LXC or Docker could be made to work in Easy linux.

Starting by looking at containers already created in my currently running Easy. ROX-Filer shows the containers folder in the working partition:

image

Burrowing down, looking inside the seamonkey folder:

images

This is a container for SeaMonkey, the web browser suite. It runs as a layered filesystem, just like the main Easy filesystem, with q.sfs on the bottom (the .ro0 folder), a read-write layer (the .session folder), and the "top" (the container folder).

But, back-tracking for a moment. How did the SeaMonkey container get created? And how to run it?
Well, I have setup Easy so that seamonkey and sh0 are pre-created, however, it is very easy to create a container for any application -- there is a nice GUI, called Easy Containers. Here is a snapshot:

image

An existing container can have its behaviour modified, or be deleted, as shown.

To create a new container, it is just a matter of selecting an app from the drop-down list, choose appropriate security options, then click the "create" button.

There is help available for making the best decisions regarding security, but also there are templates that will preset the checkboxes when you choose an app.

Technical note: these templates are located at /usr/local/easy_containers/templates

Furthermore, a new menu entry is automatically created, just like this one for SeaMonkey:

image

...two SeaMonkey entries. You can choose to run it normally or in the container!

Note the special icon used to denote apps that run in a container.

There isn't anything more to say, that's it, container created and ready to use.

What actually happens if you select that containerized SeaMonkey entry in the menu, is that it executes "ec-chroot seamonkey", instead of just "seamonkey".

ec-chroot is a s/usr/local/easy_containers/cript, found in . In a nutshell, it will "start" the container, then chroot into it and run the app.
By "start", I mean that the layered filesystem will be setup. ec-chroot calls another script to do this, named start-container.

When the app terminates, or the container is in other ways exited, ec-chroot calls stop-container, which kills all processes running in the container and unmounts everything.

This technical description of scripts is only mentioned if you are interested. For someone who is just a user, the GUI does it all, and very simply.

A note about the efficiency of Easy Containers. There is virtually no overhead to add that SeaMonkey container. No files are duplicated. The same q.sfs is used as for the main Easy Linux.
Of course, running SeaMonkey, it will create its own cache files and so on.

The container layered filesystem

What is the point of running an app inside a container? I am thinking about the user, who isn't concerned about the technical details, just what practical benefits the container concept brings, or what problems.

Broadly, there are two reasons, one, to provide isolation from the rest of Easy Linux, and second, security. A diagram helps here, to show the filesystem you have access to while running in a container:

image

The example is for the seamonkey container. q.sfs is mounted on folder .ro0 and this becomes the bottom read-only layer (also, there can be more ro layers). The folder .session is rw, and the user's view in the running container is folder container. This was already explained, however, think about it...

The app running in the container, or any file manager or terminal emulator, will only be able to see the files in folder container. Which is the content of q.sfs and any changes made on the rw layer. That's it, the rest of the system is just not there.

Yet, as folder .session is just a folder in the working partition, the entire storage area of the working partition is available. If the working partition has 14.5GB free space, that's how much free space you will have in your container.

After you exit from the container, if you want to access files created in the container, just look at /mnt/sdb2/containers/seamonkey/.session (the the example of working partition being sdb2). Or, while your container is running, /mnt/sdb2/containers/seamonkey/container.

There are security options, that will impose further restrictions on what can be done, with a view to keeping someone with ill-intent from somehow accessing outside the container. This is neat, but there is a downside. Some apps might not work, or not properly, especially as more of the security options are turned on.

Improving and customizing Easy

This is early days for Easy. It is the latest in the Quirky Linux series of experimental distributions, and a new incarnation of Quirky may not look much like previous releases. Prior to Easy, Quirky was designed as a traditional full-installation only, not using a layered filesystem, no initramfs, and special ramdisk for filesystem repair and system rollback.

Easy was a sudden idea in January 2017, a completely different approach, going back to the layered filesystem and a initramfs, and has now borne fruit.

Easy releases so far have been built with Ubuntu Xenial Xerus 16.04 binary DEB packages, however, that is incidental. The Quirky build system inherits the capability of the Woof build system in Puppy, to use the binary packages of almost any other distribution. It is really a matter of convenience: in the past, we have built puppies from packages that we compiled from source, using the T2-project, however, it is pragmatic to leaverage off the efforts of one of the major distros. In the case of Ubuntu, using their DEBs gives Easy binary compatibility with their package repositories, and we are able to install any packages from their repositories.

So, don't make the mistake of thinking that Easy (or earlier Quirkies, or Puppy) is a clone of Ubuntu. The only common factor is we are using their binary packages. That's it. In almost every respect that you can think of, Easy is different.

Anyway, this is early days for Easy, and input is most welcome. I have only just got started with containers. I have setup a very efficient infrastructure, but of course the devil is in the details. If you would like to look into improving containers, getting apps to run, improving security, etc., go for it!

One thing to note about Easy is that it inherits the run-as-root-only from Puppy. Instead, there is a different approach, to run specific Internet apps as user "spot". So, the user, you, remain always as the root-user, or administrator. You can find some documentation that justifies this locally here:

About root, spot and fido

With containers, the situation changes somewhat. Perhaps spot has become redundant. Running as "root" inside a container, with user namespace isolated from the main system, would seem to be OK.

If you are not a developer or into coding, but a user with an interest in customizing Easy, there are tools that can make this a lot of fun.

As has already been explained on this web page, just by clicking on the downloaded Easy image file, for example file easy-0.1.7-amd64.img.gz, you can open it up and see the files inside (similarly, you click on the file to close it up again).

Inside the image file, a most important file is q.sfs, which has the entire Easy Linux filesystem, from "/" down. Well, there are GUI tools to create your own q.sfs, and just replace the old one, then close up the image file, and you have a customized Easy! Sometimes this is referred to as remastering, though that term really applies to optical media.

If you would like to explore remastering, look at a couple of tools in the Setup menu:

image

The two entries of interest are "Remaster Easy Linux" and "Remove builtin packages". Note, Puppy Linux has these as well, I have rewritten them for Easy.

By removing packages that are already in q.sfs, and installing your own packages, either compiled yourself or from the Ubuntu repositories, you can then create your own q.sfs and put it into the deployable image file. Voila, you have your own customized Easy Linux!

It is not just changes in packages, but bug fixes, user-interface improvements, etc., will also go into your new q.sfs.

Disclaimers, legal, miscellania

Some final notes about Easy Linux. Where not explicitly stated or implied, code, text, artwork and concepts are copyrighted to myself, Barry Kauler, with GPL license version 3.

Regarding responsibility for what I have uploaded for distribution on the Internet, although I have acted in good faith, there is a complete disclaimer of all responsibility. This applies to everything, including code and documentation. You download and use Easy Linux with this understanding.

This How Easy Works web page was written at an early stage of the development of Easy, and sample code and snapshots may differ from the latest release. Though, it should be a matter of detail, and the overall concepts should be the same. This web page is expected to be upgraded incrementally, but there is no guarantee that it matches exactly with the latest release.

Easy is currently considered to be alpha-quality, and in some places pre-alpha. Please bear this in mind when using it.


Have fun! I look forward to interacting with you on the forum.

Regards,
Barry Kauler
October 2017


Tags: easy