site
contact
newsAutomatic resizing of save-file for Puppy
Just made a discovery that is extraordinary. I didn't know this is possible...
One of the things that I don't like about Puppy is the save-file.
When you run out of space, you have to increase the size. And of course,
you have to choose an initial size.
A save-file is required on a partition with a non-Linux filesystem,
such as FAT or NTFS. With a Linux filesystem, such as ext4, you have the
option of having a "save-folder", which does not require resizing --
you just use it as much as you want, until the partition is full.
With EasyOS, I did away with the save-file, and only use the
save-folder. So a frugal install of Easy must be to a partition with
Linux filesystem, preferably ext4.
However, I have just read this, a post by 'telcoM':
telcoM's reply is so good, I have to repeat it here:
A filesystem needs to have a defined maximum size. But as the filesystem-as-file can be sparse, that size can be an arbitrary number which doesn't have much to do with how much space the filesystem-as-file takes up on the underlying filesystem.
If you can accept setting an arbitrary maximum size limit (which can be much greater than the actual size of the underlying filesystem) for the filesystem-as-file, you can create a sparse file and a filesystem on it right now:
/tmp# df -h .
Filesystem Size Used Avail Use% Mounted on
<current filesystem> 20G 16G 3.0G 84% /
/tmp# dd if=/dev/null bs=1 seek=1024000000000 of=testdummy
0+0 records in
0+0 records out
0 bytes copied, 0.000159622 s, 0.0 kB/s
/tmp# ll testdummy
-rw-r--r-- 1 root root 1024000000000 Feb 19 08:24 testdummy
/tmp# ll -h testdummy
-rw-r--r-- 1 root root 954G Feb 19 08:24 testdummy
Here, I've created a file that appears to be a whole lot bigger than the filesystem it's stored onto...
/tmp# du -k testdummy
0 testdummy
...but so far it does not actually take any disk space at all (except for the inode and maybe some other metadata).
It would be perfectly possible to losetup it, create a
filesystem on it and start using it. Each write operation that actually
writes data to the file would cause the file's space requirement to
grow. In other words, while the file size as reported by ls -l would stay that arbitrary huge number all the time, the actual space taken by the file on the disk as reported by du would grow.
And if you mount the filesystem-as-file with a discard mount option, shrinking can work automatically too:
/tmp# losetup /dev/loop0 testdummy
/tmp# mkfs.ext4 /dev/loop0
/tmp# mount -o discard /dev/loop0 /mnt
/tmp# du -k testdummy
1063940 testdummy
/tmp# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/loop0 938G 77M 890G 1% /mnt
/tmp# cp /boot/initrd.img /mnt
/tmp# du -k testdummy
1093732 testdummy
/tmp# rm /mnt/initrd.img
/tmp# du -k testdummy
1063944 testdummy
Automatic shrinking requires:
1.) that the filesystem type of the filesystem-as-file supports the discard mount option (so that the filesystem driver can tell the underlying system which blocks can be deallocated)
2.) and that the filesystem type of the underlying filesystem supports "hole punching", i.e. the fallocate(2) system call with the FALLOC_FL_PUNCH_HOLE
option (so that the underlying filesystem can be told to mark some of
the previously-allocated blocks of the filesystem-as-file as sparse
blocks again)
3.) and that you're using kernel version 3.2 or above, so that the loop device support has the necessary infrastructure for this.
https://outflux.net/blog/archives/2012/02/15/discard-hole-punching-and-trim/
If you're fine with less immediate shrinking, you could just periodically run fstrim on the filesystem-as-file instead of using the discard
mount option. If the underlying filesystem is very busy, avoiding
immediate shrinking might help minimizing the fragmentation of the
underlying filesystem.
The problem with this approach is that if the underlying filesystem
becomes full, it won't be handled very gracefully. If there is no longer
space in the underlying filesystem, the filesystem-as-file will start
receiving errors when it's trying to replace sparse "holes" with actual
data, even as the filesystem-as-file would appear to have some unused
capacity left.
...END QUOTE
As noted above, there is a problem when the size of the file grows to
fill the underlying partition, so there would still have to be a
size-check, the free space in the underlying partition. Then refuse to
write anything more to the save-file.
That is just so brilliant. This is making me rethink everything...
EDIT:
I suspected that although this is a new discovery for me, it might be
old news for some of the Puppy, Fatdog, etc., developer guys, such
01micko, jamesbond and dimkr. I asked dimkr if he knew, and this is his
reply:
Yes, and I even used a sparse file for easy installation.
In https://github.com/puppylinux-woof-CE/woof-CE/pull/2231, I made woof-CE able to produce Chrome OS style images that can be written to a flash drive with dd, or written to the built-in 16 GB SSD to install Puppy persistently, instead of Chrome OS.
To make it easy to flash Puppy to the SSD, I built a 16 GB image (exactly the size of the SSD) and put it inside a small 2 GB image that can be flashed to any flash drive that's at least 2 GB. This way, the user doesn't need a 2+16 GB flash drive and has to boot Puppy, download the 16 GB image, then flash it, because the mostly-empty 16 GB image (a sparse file is already inside the 2 GB image.
See here:
https://github.com/puppylinux-woof-CE/w ... ge.sh#L144
Tags: linux