SFSget now much faster

Those who have used SFSget, see "sfsget" icon on the desktop, will have experienced a big delay while the online SFS repository is probed. This was done with 'wget' "spider" mode, searching for all files under the "sfs" folder. This is very slow.

Now, the "spider" search is eliminated, as I am uploading a file, named 'sfs-repo-file-list', that lists all the files under the "sfs" folder. All that SFSget has to do is download that file. Here it is:

http://distro.ibiblio.org/easyos/amd64/packages/sfs/sfs-repo-file-list

Two files have been edited, /usr/local/sfsget/sfsget and /usr/local/sfsget/filterpkgs. The latter has a fix for properly handling different SFS repositories.

If anyone wants to setup their own SFS repository, it needs to be added to the 'SFS_REPOS' variable in /root/.packages/DISTRO_PET_REPOS. This is the current assignment:

SFS_REPOS='ibiblio.org|http://distro.ibiblio.org/easyos/amd64/packages/sfs| nluug.nl|https://ftp.nluug.nl/os/Linux/distr/easyos/amd64/packages/sfs|'

You would need to have an upload script that creates file 'sfs-repo-file-list'. This is my script:

#!/bin/sh
#200331 upload a list of files.

[ ! $1 ] && exit 1
PATH1="$1"
[ ! -d "$PATH1" ] && exit 1

echo
echo -n 'Enter the password for easyos@login.ibiblio.org: '
read SSHPASS
[ "$SSHPASS" == "" ] && exit
echo " ...ok, ${SSHPASS}"
export SSHPASS

#create a date file for sfs uploads
DATE="$(date +%Y%m%d%H%M)" #ex: 201908121739
case "$PATH1" in
 */sfs/*)
  #200331 have to keep this file for older versions of easyos...
  echo "$DATE" > ${PATH1%/sfs/*}/sfs/date-last-upload
  #seems have to upload this separately...
  echo "Uploading 'date-last-upload' file..."
  rsync -cpt -e "sshpass -e ssh -p 22" ${PATH1%/sfs/*}/sfs/date-last-upload easyos@login.ibiblio.org:/public/distributions/easyos/${PATH1%/sfs/*}/sfs
  #200331 upload file-list...
  find ${PATH1%/sfs/*}/sfs -mindepth 2 -type f > ${PATH1%/sfs/*}/sfs/sfs-repo-file-list
  PTN="s%^${PATH1%/sfs/*}/sfs/%%" #ex: easyos/oe/pyro/kernel_5.2.21-p1_src.sfs
  sed -i -e "$PTN" ${PATH1%/sfs/*}/sfs/sfs-repo-file-list
  echo "Uploading 'sfs-repo-file-list' file..."
  rsync -cpt -e "sshpass -e ssh -p 22" ${PATH1%/sfs/*}/sfs/sfs-repo-file-list easyos@login.ibiblio.org:/public/distributions/easyos/${PATH1%/sfs/*}/sfs
 ;;
esac

rsync --exclude-from=rsync-exclusions --exclude=synchronize --delete -rltzvc -e "sshpass -e ssh -p 22" ${PATH1}/ easyos@login.ibiblio.org:/public/distributions/easyos/${PATH1}

SFS files need to be in an appropriate path. for example:

http://distro.ibiblio.org/easyos/amd64/packages/sfs/easyos/debian/buster/

Locally, you would have something like "www/amd64/packages/sfs/easyos/debian/buster" with the SFS files (and .specs and .png files, see the 'dir2sfs' script). The above script would be at "www/remote-sync", and you would sync the buster SFS files like this:

# ./remote-sync amd64/packages/sfs/easyos/debian/buster

Some more reading:

https://easyos.org/tech/how-easy-works-part-2.html

https://easyos.org/user/using-easy-containers.html

https://bkhome.org/news/201810/dir2sfs-utility-for-sfs-package-manager.html

https://bkhome.org/news/201811/dir2sfs-take-2.html

https://easyos.org/dev/coding-for-easyos.html 

Tags: easy