ibiblio sync scripts

December 21, 2013 — BarryK

Hmmm, when is my "hiatus" going to start?

Anyway, I am making things easier. For years I have manually uploaded to ibiblio.org, but today decided to finally automate it. I like to share whatever I do, so here it is...

This might interest other guys maintaining the 'puppylinux' repo at ibiblio.org, or 'fatdog'. But my script suits just one guy having upload access, I don't know what you do with lots of different people having ftp access, unless they are disciplined to only upload to their own directories.

My first problem, I construct PETs in one place, I have the 'Packages-*' db files somewhere else (in Woof actually), and I have a local mirror directory of what is going to go onto ibiblio.org.
This means that I have to go through two big steps, update my local mirror directory from the scattered working directories, secondly upload and sync with the remote ibiblio.org site.

I have written two scripts to automate this. The first, 'synchronise', syncs the local working directories with my local mirror-directory. The second, 'upload', syncs local mirror dir with ibiblio.

Note that it deletes old PETs from ibiblio, as I don't want the PETs to keep accumulating in ibiblio like they used to do. Be aware of this, if you use it, it will delete all PETs not in the current db file.

I will have to ask efiabruni about that double-spacing of lines...

synchronise

#!/bin/sh

#Barry Kauler, 20 dec 2013

#sync the place where i keep all files to upload to ibiblio, with working places.

#ex: i create pets in /mnt/sda10/woofproject2/pet-construction/x86/pet_packages-quirky6

#    and i maintain the package db file at /mnt/sda10/woofproject2/woof2/woof-distro/x86/t2/quirky6/Packages-pet-quirky6-official

#    this script is running in the dir where i will be uploading to ibiblio, for me that is: /mnt/sda10/www/ibiblio/quirky6/x86/packages

#this is local sync. there is a separate 'upload' script for remote sync.



MNTPT="/mnt/sda10"

echo -n "" > /tmp/delete-pet-list



#do the quirky pets...

echo

echo "Updating pet_packages-quirky6..."

ALLPETS="$(find ${MNTPT}/woofproject2/pet-construction/x86/pet_packages-quirky6 -mindepth 1 -maxdepth 1 -name '*.pet')"

WWWPETS="$(find pet_packages-quirky6 -mindepth 1 -maxdepth 1 -name '*.pet')"

#weed out old ones...

for APET in `echo -n "$WWWPETS" | tr '\n' ' '`

do

 BASEPET="`basename $APET`"

 pPTN="/${BASEPET}$"

 if [ "`echo "$ALLPETS" | grep "$pPTN"`" = "" ];then

  echo "Deleting: ${BASEPET}"

  rm -f ./${APET}

  echo >> /tmp/delete-pet-list

 fi

done

#add new pets...

for APET in `echo -n "$ALLPETS" | tr '\n' ' '`

do

 BASEPET="`basename $APET`"

 if [ ! -f pet_packages-quirky6/${BASEPET} ];then

  echo "Adding new PET: ${BASEPET}"

  cp -f $APET pet_packages-quirky6/

 fi

done

echo "Grabbing latest Packages-pet-quirky6-official..."

cp -f ${MNTPT}/woofproject2/woof2/woof-distro/x86/t2/quirky6/Packages-pet-quirky6-official ./

echo '...done'

upload

#!/bin/sh

#Barry Kauler, 20 Dec 2013

#sync local with ibiblio.org



#returns a list of all pets in remote directory...

echo "Fetching list of PETs in remote directory 'pet_packages-quirky6'..."

#test url... -t 2 means try 2 times, -T 20 means timeout in 20 seconds. removed:  --recursive

WWWPETS="$(wget -4 -t 2 -T 20 --waitretry=20 --spider -S --no-parent --no-directories -A '*.pet' ftp://distro.ibiblio.org/quirky/quirky6/x86/packages/pet_packages-quirky6/ 2>&1 | grep -o '.*\.pet' | rev | cut -f 1 -d ' ' | rev)"



ALLPETS="$(find pet_packages-quirky6 -mindepth 1 -maxdepth 1 -name '*.pet')"



echo -n "" > /tmp/delete-pet-list

echo -n "" > /tmp/add-pet-list



#weed out old ones...

for APET in `echo -n "$WWWPETS" | tr '\n' ' '`

do

 BASEPET="$APET"

 pPTN="/${BASEPET}$"

 if [ "`echo "$ALLPETS" | grep "$pPTN"`" = "" ];then

  echo "Deleting: ${BASEPET}"

  echo "${BASEPET}" >> /tmp/delete-pet-list

 fi

done

#add new pets...

for APET in `echo -n "$ALLPETS" | tr '\n' ' '`

do

 BASEPET="`basename $APET`"

 pPTN="^${BASEPET}$"

 if [ "`echo "$WWWPETS" | grep "$pPTN"`" = "" ];then

  echo "Adding new PET: ${BASEPET}"

  echo "${BASEPET}" >> /tmp/add-pet-list

 fi

done



#ssh -e none -l quirky -p 22 login1.ibiblio.org -s sftp

#precaution...

chmod 644 pet_packages-quirky6/*

chmod 644 Packages-pet-quirky6-official



echo

echo "Synchronizing remote ibiblio.org PETs in pet_packages-quirky6..."

REMOP="$(cat /tmp/delete-pet-list | sed -e 's%^%rm %' -e '/^rm $/d')"

ADDOP="$(cat /tmp/add-pet-list | sed -e 's%^%put %' -e '/^put $/d')"

#this will ask for password...

echo "cd /public/distributions/quirky/quirky6/x86/packages/pet_packages-quirky6

${REMOP}

lcd pet_packages-quirky6

${ADDOP}

lcd ..

cd ..

rm Packages-pet-quirky6-official

put Packages-pet-quirky6-official

quit" | sftp USERNAMEGOESHERE@login1.ibiblio.org

...you will need to change the paths to suit your situation. And put in the correct username.

Tags: linux