Automatic resizing of save-file for Puppy

Just made a discovery that is extraordinary. I didn't know this is possible...

One of the things that I don't like about Puppy is the save-file. When you run out of space, you have to increase the size. And of course, you have to choose an initial size.

A save-file is required on a partition with a non-Linux filesystem, such as FAT or NTFS. With a Linux filesystem, such as ext4, you have the option of having a "save-folder", which does not require resizing -- you just use it as much as you want, until the partition is full.

With EasyOS, I did away with the save-file, and only use the save-folder. So a frugal install of Easy must be to a partition with Linux filesystem, preferably ext4.

However, I have just read this, a post by 'telcoM':

https://unix.stackexchange.com/questions/501500/is-it-possible-to-make-a-dynamically-risizing-filesystem-as-file

telcoM's reply is so good, I have to repeat it here:

A filesystem needs to have a defined maximum size. But as the filesystem-as-file can be sparse, that size can be an arbitrary number which doesn't have much to do with how much space the filesystem-as-file takes up on the underlying filesystem.

If you can accept setting an arbitrary maximum size limit (which can be much greater than the actual size of the underlying filesystem) for the filesystem-as-file, you can create a sparse file and a filesystem on it right now:

/tmp# df -h .
Filesystem                Size  Used Avail Use% Mounted on
<current filesystem>       20G   16G  3.0G  84% /

/tmp# dd if=/dev/null bs=1 seek=1024000000000 of=testdummy
0+0 records in
0+0 records out
0 bytes copied, 0.000159622 s, 0.0 kB/s
/tmp# ll testdummy
-rw-r--r-- 1 root root 1024000000000 Feb 19 08:24 testdummy
/tmp# ll -h testdummy
-rw-r--r-- 1 root root 954G Feb 19 08:24 testdummy

Here, I've created a file that appears to be a whole lot bigger than the filesystem it's stored onto...

/tmp# du -k testdummy
0       testdummy

...but so far it does not actually take any disk space at all (except for the inode and maybe some other metadata).

It would be perfectly possible to losetup it, create a filesystem on it and start using it. Each write operation that actually writes data to the file would cause the file's space requirement to grow. In other words, while the file size as reported by ls -l would stay that arbitrary huge number all the time, the actual space taken by the file on the disk as reported by du would grow.

And if you mount the filesystem-as-file with a discard mount option, shrinking can work automatically too:

/tmp# losetup /dev/loop0 testdummy
/tmp# mkfs.ext4 /dev/loop0
/tmp# mount -o discard /dev/loop0 /mnt
/tmp# du -k testdummy 
1063940 testdummy
/tmp# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      938G   77M  890G   1% /mnt

/tmp# cp /boot/initrd.img /mnt
/tmp# du -k testdummy 
1093732 testdummy

/tmp# rm /mnt/initrd.img
/tmp# du -k testdummy
1063944 testdummy

Automatic shrinking requires:

1.) that the filesystem type of the filesystem-as-file supports the discard mount option (so that the filesystem driver can tell the underlying system which blocks can be deallocated)

2.) and that the filesystem type of the underlying filesystem supports "hole punching", i.e. the fallocate(2) system call with the FALLOC_FL_PUNCH_HOLE option (so that the underlying filesystem can be told to mark some of the previously-allocated blocks of the filesystem-as-file as sparse blocks again)

3.) and that you're using kernel version 3.2 or above, so that the loop device support has the necessary infrastructure for this.

https://outflux.net/blog/archives/2012/02/15/discard-hole-punching-and-trim/

If you're fine with less immediate shrinking, you could just periodically run fstrim on the filesystem-as-file instead of using the discard mount option. If the underlying filesystem is very busy, avoiding immediate shrinking might help minimizing the fragmentation of the underlying filesystem.

The problem with this approach is that if the underlying filesystem becomes full, it won't be handled very gracefully. If there is no longer space in the underlying filesystem, the filesystem-as-file will start receiving errors when it's trying to replace sparse "holes" with actual data, even as the filesystem-as-file would appear to have some unused capacity left. 

...END QUOTE

As noted above, there is a problem when the size of the file grows to fill the underlying partition, so there would still have to be a size-check, the free space in the underlying partition. Then refuse to write anything more to the save-file.

That is just so brilliant. This is making me rethink everything...

EDIT:
I suspected that although this is a new discovery for me, it might be old news for some of the Puppy, Fatdog, etc., developer guys, such 01micko, jamesbond and dimkr. I asked dimkr if he knew, and this is his reply:

Yes, and I even used a sparse file for easy installation.

In https://github.com/puppylinux-woof-CE/woof-CE/pull/2231, I made woof-CE able to produce Chrome OS style images that can be written to a flash drive with dd, or written to the built-in 16 GB SSD to install Puppy persistently, instead of Chrome OS.

To make it easy to flash Puppy to the SSD, I built a 16 GB image (exactly the size of the SSD) and put it inside a small 2 GB image that can be flashed to any flash drive that's at least 2 GB. This way, the user doesn't need a 2+16 GB flash drive and has to boot Puppy, download the 16 GB image, then flash it, because the mostly-empty 16 GB image (a sparse file is already inside the 2 GB image.

See here:

https://github.com/puppylinux-woof-CE/w ... ge.sh#L144 

   

Tags: linux

Puppy moving to Xwayland

Forum members 'dimkr' and '01micko' are doing great things at "woof-CE-testing". I posted about dimkr's elimination of the 'initrd' file:

https://bkhome.org/news/202103/puppy-linux-without-an-initrd.html

There are other developments underway, including a move to using Xwayland instead of Xorg. If you would like to give it a go, see here:

https://forum.puppylinux.com/viewtopic.php?f=40&t=2967

Very early days though, so many things broken. Probably best restricted to people who can fix bugs, rather than just report them.

I haven't tried it myself, but interested as it looks like the way we will have to go, eventually. 'dimkr' posted a screenshot:

I have not studied anything about Wayland or Xwayland, so don't understand why "cage" is used. Cage project:

https://github.com/Hjdskes/cage

...it looks like cage is an easy way to get an X app to run on Xwayland. So dimkr is running jwm, and from that rox, etc. See this:

https://github.com/puppylinux-woof-CE/woof-CE/pull/2265    

Tags: linux

Puppy Linux without an initrd

We know about the 'initrd' file, which is an initramfs that runs first at bootup. EasyOS has this, as do the puppies.

A traditional full installation, occupying an entire partition, may not need an initrd, and can be run directly from the kernel boot parameters. For example, if the full installation is in /dev/sda9, then boot parameters would include root=/dev/sda9, or the PARTUID could be specified.

If an initrd is used, the boot parameters would not have root=, instead would have something like initrd=initrd.gz, where initrd.gz is the name of the file, with perhaps a path.

One of the reasons we have a initrd is to setup the layered filesystem, using overlayfs or aufs, then a switch_root is performed onto the layered filesystem.

However, Dima, forum name 'dimkr' on github and the Puppy Forum, and 'iguleder' on the old Puppy Murga Forum, has come up with a way to load the layered filesystem without requiring an initrd.

It is achieved with a special 'init' executable, written in C and statically linked with musl. The boot parameters would be something like this: root=/dev/sda9 init=/init, which tells the kernel that there is a "full installation" at /dev/sda9 and to run /init.

Except that there isn't really a full installation, there are only SFS files. The init executable does what the initrd used to do, sets up the SFS layers and then switches onto the top of the layers.

Dima has created a new project "frugalify" which has this init executable:

https://github.com/dimkr/frugalify

He has also built a demo Puppy, dpupOS:

https://forum.puppylinux.com/viewtopic.php?f=33&t=2495

It must be emphasized that this is not for end users, only for those who want to play with something new and interesting and provide feedback. I tried it a few days ago, and there are too many issues to be considered as suitable for end users -- but they are details, the guts of it are sound.

A note about iguleder -- he goes way back, about 11 years I think, In the Murga Forum, and has been a fantastic contributor, as well as lots of projects on github. he is active developing woof-CE, which is great. I think that he is a young bloke, compared to me anyway.

If you are interested in compiling it yourself, I uploaded some musl chrootable root filesystems a few years ago, see a blog post here:

https://bkhome.org/news/201704/alpine-x8664-chrootable-rootfs.html

Downloadable from here, x86_64, x86, aarch64, armhf tarballs:

http://distro.ibiblio.org/quirky/alpine/

...I think there are instructions inside the tarballs.     

Tags: linux

puppylinux.com certificate issues

The Puppy Linux Forum, at forum.puppylinux.com currently does not work.

This has been reported by ozsouth at one of the alternative Puppy forums:

https://www.linuxquestions.org/questions/puppy-71/puppylinux-main-forum-having-issues-4175691857/

However, this URL works:

https://puppylinux.rockedge.org/

01micko and others are trying to fix it.

EDIT 2021-03-21:
The forum is fixed, this URL now works:

https://forum.puppylinux.com/

Thanks to Erik and Mick for sorting it out. 

Tags: linux

Iceape-UXP an alternative to SeaMonkey

I received an email from Luke, advising me of Iceape-UXP, an alternative browser suite, that does not require Rust to compile. Quoting from his email:

I was reading your blog the last several days for minimalist inspiration and noticed you disliked Rust as a dependency as well. There is an alternative with UXP: http://thereisonlyxul.org/

Iceape-UXP has no rust dependency, we maintain it here: https://wiki.hyperbola.info/doku.php?id=en:project:iceape-uxp

I got the latest Iceape and Icedove from the git repository, as well as the recommended thunderbird source, and had a go at compiling Icedove. It requires python2 and I used the same PET of python2 that I had used to compile SM. Unfortunately, got a python error when compiling Icedove, and as it was late at night, left it at that. I was running the EasyOS Dunfell build.

Might take another look sometime, perhaps try on EasyOS 2.4.1, the Buster series, which already has python2. Nice to know that we do have a choice. 

Tags: linux

Murga Forum is up, read-only

The Puppy Forum, created and maintained by John Murga until his recent passing away, has been in maintenance-mode for over a month. No one had his login details, not even family members, so the forum stayed down, until today.

The domain murga-linux.com is registered with namecheap.com and the site is hosted with linode.com. I have been in contact with the executor of John's Will and have obtained sufficient information to satisfy Linode tech support, and we were granted login.

We are still waiting on namecheap.com, as we need to be able to renew the registration.

My experience is with conventional shared hosting only (at omnis.com), and Linode's cloud hosting was perplexing when I logged in! I am familiar with having everything setup, nice Control Panel. Linode do things differently. I asked James Budiono (jamesbond in the forums) for help, and wow, he did, got the forum out of maintenance mode, and here it is:

http://murga-linux.com/puppy/

Please treat it as read-only, don't login or post anything.

At this stage, we are uncertain about the future of the Murga Forum. There are so many links to it, that we probably do need to keep it running. I think that it is on a US$5 per month plan with Linode, and then there is the name registration cost. The phpBB forum is running on a very old version of PHP, and maybe it is a bit shaky. We may end up seeking a volunteer who is prepared to look after it. 

Tags: linux

Oh dear, Rust again

Back in April 2020, I made a comment about Rust:

https://bkhome.org/news/201904/easyos-forum-closed.html

Which upset at least one person. Today I received an email suggesting that 'fd' is superior to the 'nnn' file manager. This is my reply:

Thanks for the suggestion.
Unfortunately, fd is written in Rust. It is very difficult to create a small binary when written in Rust -- at least, that is my understanding.
I did a quick search, it is possible to create a fully static rust executable, linking with musl:

https://doc.rust-lang.org/edition-guide/rust-2018/platform-and-target-support/musl-support-for-fully-static-binaries.html

There is another nice console file manager written in python. Same problem. too much overhead to run in the initrd.

I should be careful about making assumptions about Rust, so I found it compiled statically with musl, here:

https://github.com/sharkdp/fd/releases

...the binary is 2.3MB (static, stripped), which would double the size of the initrd!
I tried it, unfortunately it is not a text-mode GUI app like 'nnn', it is only a CLI replacement for 'find'.

Ha ha ha, replacing the tiny 'find' utility with some more features, and bumping the size to 2.3MB, please, no! And it doesn't even provide all of find's capability, quoting from the 'fd' project page:

While it does not seek to mirror all of find's powerful functionality, it provides sensible (opinionated) defaults for 80% of the use cases.

Unfortunately, there are so many people pushing Rust as a systems programming language. 

Tags: linux

EasyDD GUI fixes for Linux Mint

I posted a few days ago about improvements to EasyDD:

https://bkhome.org/news/202006/easydd-works-on-more-distributions.html

I had only tested CLI mode in Linux Mint, however Sebastion has got GUI mode working. He installed 'gtkdialog', which is not available in the default repositories, but is available elsewhere, such as here:

https://repology.org/project/gtkdialog/versions

The "MX-19" DEB is here:

http://mxrepo.com/mx/repo/pool/main/g/gtkdialog/gtkdialog_0.8.3-2mx19_amd64.deb

However, Sebastion discovered a couple of bugs. Drag-and-drop of an image file from the Nemo file manager to EasyDD had some extra text preppended and appended. I reported to the Puppy Forum:

http://murga-linux.com/puppy/viewtopic.php?p=1061174#1061174

...and MochiMoppel responded with a fix. fantastic, thanks for that!

Another problem was the GUI put up a "finished" message, when it wasn't, it was only just started to write to the drive.

I found the cause. EasyDD will use the embedded "vte" terminal support of gtkdialog, however, often gtkdialog is compiled without this, as is the case with the DEB. So then EasyDD looks for 'rxvt', 'urxvt', 'xterm' and 'gnome-terminal', in that order.

Linux Mint has the last one. Unfortunately, "gnome-terminal .... -e easydd" runs as a separate process and returns immediately, hence the premature "finished" window. It requires the "--wait" option, which I have now put into EasyDD.

Haven't tried these fixes in Linux Mint yet, but should be good-to-go. Ditto for any other Debian-based or Ubuntu-based distribution. Just find the gtkdialog package. Gtkdialog is so good, it should be in all official package repositories!

For anyone encountering EasyDD for the first time, see here:

https://bkhome.org/linux/easydd-write-image-file-to-drive.html 

gtkdialog source

Most people are getting the source from here:

https://code.google.com/archive/p/gtkdialog/

However, there are various forks that are still active, especially this one, maintained by Mick Amadio:

https://github.com/01micko/gtkdialog

...this would be where Puppy Linux developers are contributing.

EDIT 2020-06-19:
Michael Amadio has contacted me, advised that he has moved the gtkdialog repository under the umbrella of the Puppy Linux woof-CE repository:

https://github.com/puppylinux-woof-CE/gtkdialog 

Tags: linux