File downloader

I have written a "heavy duty" file downloader, /usr/sbin/download_file, in Woof.

This utility checks that the file exists online before downloading it, downloads, offers to continue if previous partial download. Reports everything, so you know exactly what has happened.

I am now calling this utility from the Video Upgrade Wizard, file /usr/local/video_upgrade/video_upgrade_wizard, instead of just running wget.

Next, I will modify PPM to do the same.


Posted on 16 Nov 2010, 22:07


Comments:

Posted on 16 Nov 2010, 22:31 by BarryK
PPM uses download_file
Ok, the Puppy Package Manager now uses download_file.



Posted on 17 Nov 2010, 15:21 by 01micko
code link?
Hi Barry,

I figure I can ditch all the rigorous code in quickpet for lucid puppy, if only I had a link to the code!

I gather it is built into the next woof, but a snapshot of the code would give me some time to update my stuff.

Cheers


Posted on 17 Nov 2010, 17:30 by BarryK
download_file posted
01micko,
Ok, it is here:

http://murga-linux.com/puppy/viewtopic.php?p=468143#468143



Posted on 18 Nov 2010, 7:47 by BarryK
download_file size fix
The 'download_file' script checks first that the remote file exists, and obtains its size.

This worked for an http protocol site, however for an ftp site the file-size information is returned differently. I have modified download_file to correctly obtain the remote file size for an ftp URL.



Posted on 18 Nov 2010, 8:04 by 01micko
thanks
Yes I just discovered that myself! (actually I knew it but they have changed it recently)
Now they list the whole directory and sizes, so I'm grepping the exact size with the exact file name, I guess you do similar.
Cheers
(and aren't some of those wget header files wild? .. turtle power? hehe!)


Posted on 18 Nov 2010, 8:09 by 01micko
typo
forgot to mention the typo at line 50, 'abiwotd'


Posted on 20 Nov 2010, 8:49 by 01micko
re download_file
Ok, part of quickpet is an sfs grabber, it didn't use the rigorous code. Someone complained. http://murga-linux.com/puppy/viewtopic.php?p=468996#468996

I rewrote it to use 'download_file. Worked great for me, lets hope it works great for all.
Cheers


Posted on 20 Nov 2010, 9:53 by 01micko
re Downlod_file
Ok, doesn't matter.

I put code in to detect download_file and if it's there it uses it, Otherwise the old wget (with some params) is run from rxvt -e
Cheers


Posted on 20 Nov 2010, 21:33 by 01micko
wget woes
Hi Barry

One particular user is reporting failure with PPM and my sfs_grab script (which uses 'download_file') but success with quickpet, http://www.murga-linux.com/puppy/viewtopic.php?p=469115#469115
It just dawned on me what it is... the "-4" option with wget. (ipv4 vs ipv6, if that makes sense)
aarf (I'm sure you remember him/her) was considerably vocal and helpful on this issue in quickpet development, but what perplexes me is aarf's lack of problems with PPM at the time, which was some months ago.
So.. for some it seems the '-4' option is needed, but I bet for others it may bork their download. Incidentally, in all of quickpet's testing, since the '-4' option was added to the wget call, there have been no problems reported, that doesn't mean they are not there!
Cheers


Posted on 4 Jan 2011, 9:57 by 01micko
sfs issue, download_file
jpeps reported an sfs issue downloading the devx for lupu using my sfs_grabber
http://www.murga-linux.com/puppy/viewtopic.php?p=483290#483290
I narrowed it down to some code in download_file that needs 3X the space to allow for installation of the file.

Here's my proposal At line 93 I added a case for pupmodes 6 and 12 conditional upon the file being an sfs,
case $PUPMODE in

3|7|13|77) #tmpfs on top
#when install pkg, it writes direct to save-layer.
SIZEK_REQUEST=`expr $SIZEK_ONLINE + 6000` #6MB slack.
;;
6|12)
FILE_SFS=`echo $FILENAME|grep .sfs$` #01micko incase file is an sfs 20110104, pwd may not be inside savefile
if [ "$FILE_SFS" = "" ];then
SIZEK_REQUEST=`expr $SIZEK_ONLINE \* 3` #3MB slack, no, need install space too.
else
SIZEK_REQUEST=`expr $SIZEK_ONLINE + 3000` #3MB slack.
fi
;;
*)
SIZEK_REQUEST=`expr $SIZEK_ONLINE \* 3`
;;
esac


Shouldn't do any harm
Thoughts?