site  contact  subhomenews

Automatic resizing of save-file for Puppy

May 20, 2021 — BarryK

Just made a discovery that is extraordinary. I didn't know this is possible...

One of the things that I don't like about Puppy is the save-file. When you run out of space, you have to increase the size. And of course, you have to choose an initial size.

A save-file is required on a partition with a non-Linux filesystem, such as FAT or NTFS. With a Linux filesystem, such as ext4, you have the option of having a "save-folder", which does not require resizing -- you just use it as much as you want, until the partition is full.

With EasyOS, I did away with the save-file, and only use the save-folder. So a frugal install of Easy must be to a partition with Linux filesystem, preferably ext4.

However, I have just read this, a post by 'telcoM':

telcoM's reply is so good, I have to repeat it here:

A filesystem needs to have a defined maximum size. But as the filesystem-as-file can be sparse, that size can be an arbitrary number which doesn't have much to do with how much space the filesystem-as-file takes up on the underlying filesystem.

If you can accept setting an arbitrary maximum size limit (which can be much greater than the actual size of the underlying filesystem) for the filesystem-as-file, you can create a sparse file and a filesystem on it right now:

/tmp# df -h .
Filesystem                Size  Used Avail Use% Mounted on
<current filesystem>       20G   16G  3.0G  84% /

/tmp# dd if=/dev/null bs=1 seek=1024000000000 of=testdummy
0+0 records in
0+0 records out
0 bytes copied, 0.000159622 s, 0.0 kB/s
/tmp# ll testdummy
-rw-r--r-- 1 root root 1024000000000 Feb 19 08:24 testdummy
/tmp# ll -h testdummy
-rw-r--r-- 1 root root 954G Feb 19 08:24 testdummy

Here, I've created a file that appears to be a whole lot bigger than the filesystem it's stored onto...

/tmp# du -k testdummy
0       testdummy

...but so far it does not actually take any disk space at all (except for the inode and maybe some other metadata).

It would be perfectly possible to losetup it, create a filesystem on it and start using it. Each write operation that actually writes data to the file would cause the file's space requirement to grow. In other words, while the file size as reported by ls -l would stay that arbitrary huge number all the time, the actual space taken by the file on the disk as reported by du would grow.

And if you mount the filesystem-as-file with a discard mount option, shrinking can work automatically too:

/tmp# losetup /dev/loop0 testdummy
/tmp# mkfs.ext4 /dev/loop0
/tmp# mount -o discard /dev/loop0 /mnt
/tmp# du -k testdummy 
1063940 testdummy
/tmp# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      938G   77M  890G   1% /mnt

/tmp# cp /boot/initrd.img /mnt
/tmp# du -k testdummy 
1093732 testdummy

/tmp# rm /mnt/initrd.img
/tmp# du -k testdummy
1063944 testdummy

Automatic shrinking requires:

1.) that the filesystem type of the filesystem-as-file supports the discard mount option (so that the filesystem driver can tell the underlying system which blocks can be deallocated)

2.) and that the filesystem type of the underlying filesystem supports "hole punching", i.e. the fallocate(2) system call with the FALLOC_FL_PUNCH_HOLE option (so that the underlying filesystem can be told to mark some of the previously-allocated blocks of the filesystem-as-file as sparse blocks again)

3.) and that you're using kernel version 3.2 or above, so that the loop device support has the necessary infrastructure for this.

If you're fine with less immediate shrinking, you could just periodically run fstrim on the filesystem-as-file instead of using the discard mount option. If the underlying filesystem is very busy, avoiding immediate shrinking might help minimizing the fragmentation of the underlying filesystem.

The problem with this approach is that if the underlying filesystem becomes full, it won't be handled very gracefully. If there is no longer space in the underlying filesystem, the filesystem-as-file will start receiving errors when it's trying to replace sparse "holes" with actual data, even as the filesystem-as-file would appear to have some unused capacity left. 


As noted above, there is a problem when the size of the file grows to fill the underlying partition, so there would still have to be a size-check, the free space in the underlying partition. Then refuse to write anything more to the save-file.

That is just so brilliant. This is making me rethink everything...

I suspected that although this is a new discovery for me, it might be old news for some of the Puppy, Fatdog, etc., developer guys, such 01micko, jamesbond and dimkr. I asked dimkr if he knew, and this is his reply:

Yes, and I even used a sparse file for easy installation.

In, I made woof-CE able to produce Chrome OS style images that can be written to a flash drive with dd, or written to the built-in 16 GB SSD to install Puppy persistently, instead of Chrome OS.

To make it easy to flash Puppy to the SSD, I built a 16 GB image (exactly the size of the SSD) and put it inside a small 2 GB image that can be flashed to any flash drive that's at least 2 GB. This way, the user doesn't need a 2+16 GB flash drive and has to boot Puppy, download the 16 GB image, then flash it, because the mostly-empty 16 GB image (a sparse file is already inside the 2 GB image.

See here: ... 


Tags: linux