Update delta files for smaller downloads

When you click the "update" icon on the desktop, Easy looks online to see if there is a later version. If so, it downloads it. Say that you are running 4.0 and the new version is 'easy-4.1-amd64.img'; the update script will download this to /mnt/wkg/easy-4.1-amd64.img (or, if you want the full path; /mnt/${WKG_DEV}/${WKG_DIR}easy-4.1-amd64.img).

In earlier versions of Easy, prior to 4.1, the update script used to use 'rsync'. If you had, say /mnt/wkg/easy-4.0-amd64.img' from the previous update, the 'rsync' utility is able to compare that with the online 'easy-4.1-amd64.img' and only download differences, then construct the full 'easy-4.1-amd64.img'.

The download size saving is enormous; unfortunately, the rsync server at ibiblio.org has been unreliable, disconnecting seemingly randomly, causing download failure.

The latest version of Easy, 4.1, does not use rsync, so the entire new file has to be downloaded. Version 4.1 is 801MiB, a very big download. Many people have limited data allocation, but apart from that, downloading such a big file increases the risk of download failure or file corruption.

I have now introduced delta files, using the 'xdelta3' utility. What this utility does is compare two files and generate a difference file, like this:

# xdelta3 -e -s OLDFILE NEWFILE DIFFERENCEFILE

Then, if you have the old-file and the difference-file, you can perform a reverse operation:

# xdelta3 -d -s OLDFILE DIFFERENCEFILE NEWFILE

These difference files were used in Puppy Linux a very long time ago, back in the landline dialup days, when data allocation and download speeds were very low.

The pups, most of them I think, have kept this utility, as has EasyOS. Inherited from the pup-days is a little GUI wrapper for xdelta3, that you can find in the "Utility" menu. You can use this GUI to create a difference file and do the reverse.

The difference-files back then were named like "OLDFILE___NEWFILE.delta", and I have kept this same naming scheme.

In woofQ, the '3buildeasydistro' script now looks to see what prior easy*.img files it has created, and generates difference-files.

If the update script has found a later version online, say 4.3, it will now check for online existence of a difference-file. If you are running version 4.2, as long as you kept that /mnt/wkg/easy-4.2-amd64.img, the update script will look for an online 'easy-4.2-amd64.img___easy-4.3-amd64.img.delta' file, and if it exists, will download and perform that reverse operation, rebuilding 'easy-4.3-amd64.img'.

This is all coded, but is still theory. Not yet tested. This is only going to be supported from Easy 4.2 onwards. In fact, there are so many changes from 4.1, I am wondering whether should recommend current users do a new install. Not sure how an update from 4.1 to 4.2 will work.

For anyone who is interested, this is the project page:

https://github.com/jmacd/xdelta   

Tags: easy