Script to update names (Hey aru !!)))

aru · March 28, 2004

I tried that, but same results :(

I don't know man. Every test I do here works fine. For example your test:

Testing this values:

LOCAL_DIR=~/TEST/updates/

RSYNC_DIR='rsync://carroll.cac.psu.edu/mandrake/updates/9.2'

where ~/TEST/updates/ emulates a mirror for 9.1 as shown above and 9.2 to test your example.

If I change the version of two files:

tcpdump-3.7.2-2.1.92mdk.i586.rpm renamed to tcpdump-3.7.2-2.1.92mdk.i586.rpm

XFree86-Xvfb-4.3-24.4.92mdk.i586.rpm renamed to XFree86-Xvfb-4.3-24.3.92mdk.i586.rpm

and run the script:

~$ ~/rsyncOptimum
mv 9.2/RPMS/tcpdump-3.7.2-2.0.92mdk.i586.rpm 9.2/RPMS/tcpdump-3.7.2-2.1.92mdk.i586.rpm
mv 9.2/RPMS/XFree86-Xvfb-4.3-24.3.92mdk.i586.rpm 9.2/RPMS/XFree86-Xvfb-4.3-24.4.92mdk.i586.rpm
~$

It works quite well, though the DIRS variables syntax MUST be the rigth one since this script is just a prototype. On the production version this will be handled better

March 28, 2004

Maybe I'm doing something stupid here then?? I'm no good at scripts anyway.

Well, instead of running it from a consol, i named the script rsyncOptimum.sh made it executable and then did

 ./rsyncOptimum.sh

But then i got no output at all. :wall: So i tried:

./rsyncOptimum.sh | tee myscreen_1.log

But the log was empty :wall:

aru · March 28, 2004

.... So i tried:
./rsyncOptimum.sh | tee myscreen_1.log
But the log was empty

To see the log run this (exactly as is written):

 ~$ bash -x rsyncOptimum &> mylog.txt
~$ less mylog.txt

Well, ofcourse if your script is called rsyncOptimum.sh, put that name instead of plain rsyncOptimum :P IMHO adding surnames (extensions) for the scripts it's a bad habit

Take special look to the values of LOCAL_LISTING= and RSYNC_LISTING= ; each file of each of the listings should start with exactly the same path otherwise the rest of the script fails. That's why I'm putting extreme attention to the LOCAL_DIR path syntax.

I won't have free time untill wednesday, but I promise you that then I'll work in this. If rsync supports your trick this is going to save me loads of bandwidth (even may make possible to keep a cooker mirror updated with my modem :cheesy: )

March 28, 2004

OK, well that got me some output, but not sure what to make of it. The log file was 2.4MB :o

+ alias 'rm=rm -i'
+ alias 'cp=cp -i'
+ alias 'mv=mv -i'
+ '[' -f /etc/bashrc ']'
+ . /etc/bashrc
+++ id -gn
+++ id -un
+++ id -u
++ '[' root = root -a 0 -gt 99 ']'
++ umask 022
++ '[' '' ']'
+ LOCAL_DIR=/var/ftp/pub/Mandrake/Updates/
+ RSYNC_DIR=rsync://carroll.cac.psu.edu/mandrake/updates/9.2
++ find /var/ftp/pub/Mandrake/Updates/ -name '*'
++ sort
++ sed -n s@/var/ftp/pub/Mandrake/Updates/@@gp
+ LOCAL_LISTING=
9.2
9.2/apache-1.3.28-3.1.92mdk.i686.rpm>>>>>THIS IS THE FILE NAME I CHANGED
9.2/apache2-2.0.47-6.3.92mdk.i586.rpm
9.2/apache2-common-2.0.47-6.3.92mdk.i586.rpm
9.2/apache2-devel-2.0.47-6.3.92mdk.i586.rpm
9.2/apache2-manual-2.0.47-6.3.92mdk.i586.rpm

This continues until:

++ rsync -av rsync://carroll.cac.psu.edu/mandrake/updates/9.2
++ sort
++ awk '{print $NF}'
++ sed -n '/^receiving/,/^wrote / {/^receiving\|^wrote /d;p;}'
+ RSYNC_LISTING=9.2
9.2/base
9.2/base/hdlist.cz
9.2/base/list
9.2/base/pubkey
9.2/base/synthesis.hdlist.cz
9.2/base/timestamp

End of log reads:

+ local_file=
+ '[' '' '!=' '' ']

aru, if you get this to work, hundreds of sites around the world will save millions of GBs of bandwith, you might want to think about selling the script :D :D

aru · March 29, 2004

aru, if you get this to work, hundreds of sites around the world will save millions of GBs of bandwith, you might want to think about selling the script :D :D

:lol2: :lol2: :lol2:

This is about free software isn't it :P

Well, tomorrow I'll see if I can handle why yours isn't working. Are you sure you are using the same DIRS (local and remote) you'd have used when launching your 'normal' rsync command. It is quite strange the fact that the remote dir has a directory "/base/" that the local mirror lacks, both paths aren't coherent one to eachother

March 29, 2004

Well the live one i use is;

rsync -au --delete ftp.sunet.se::pub/Linux/distributions/mandrake/updates/9.2/RPMS/ /var/ftp/pub/Mandrake/Updates/9.2

Others jobs are;

rsync -au --partial --delete --exclude '.*'

etc etc.

aru · March 31, 2004

I did a minimal update yesterday to show debug errors and to handle (in a bizarre way) the command line. Now you can call the script as if you where calling rsync, so using your example, if you'd call rsync this way:

rsync -au --delete ftp.sunet.se::pub/Linux/distributions/mandrake/updates/9.2/RPMS/ /var/ftp/pub/Mandrake/Updates/9.2

now you can call rsyncOptimum as:

rsyncOptimum -au --delete ftp.sunet.se::pub/Linux/distributions/mandrake/updates/9.2/RPMS/ /var/ftp/pub/Mandrake/Updates/9.2

This is an example (obviously my destination dir differs a little bit):

~/anon_project$ ./rsyncOptimum -au --delete ftp.sunet.se::pub/Linux/distributions/mandrake/updates/9.2/RPMS/ ~/anon_project/TEST/var/ftp/pub/Mandrake/Updates/9.2
mv ethereal-0.10.0a-0.1.92mdk.i586.rpm ethereal-0.10.3-0.1.92mdk.i586.rpm
mv squid-2.5.STABLE3-3.0.92mdk.i586.rpm squid-2.5.STABLE3-3.1.92mdk.i586.rpm

~/anon_project$

as you see the lastnight updates are reflected today (almost as I just changed names as I mirrored it today).

Hope your tests are as succesfull as mine

the new script version is attached.

Edit: NOW it is attached

rsyncOptimum

Edited March 31, 2004 by aru

March 31, 2004

It works !!!!! :headbang: :thumbs:

:g-balloon: :b-balloon: :r-balloon:

Well done aru, your a genius!

./rsyncOptimum -au --delete ftp.sunet.se::pub/Linux/distributions/mandrake/updates/9.2/RPMS/ /var/ftp/pub/Mandrake/Updates/9.2
mv ethereal-0.10.0a-0.1.92mdk.i586.rpm ethereal-0.10.3-0.1.92mdk.i586.rpm

Whats next?

aru · April 1, 2004

Coooool!!! I'm glad to see that at last it works!!

Now, the next step is do a full and robust rewrite of the script and start doing some testings with a non critical local mirror. I'll work on it next weekend

:thumbs:

April 1, 2004

Coooool!!! I'm glad to see that at last it works!!

Now, the next step is do a full and robust rewrite of the script and start doing some testings with a non critical local mirror. I'll work on it next weekend

OK, cool.

We can use the >n< option with rsync for testing. I will set up a test dir on the server, and do some tests with PLF-8.2 versus PLF-9.2 as i already have all those files for both versions.

aru · April 2, 2004

<WARNING>THIS POST IS ALMOST OFFTOIC</WARNING>

Last night I was reading about "unit tests" and I feel that it was something really powerfull for developing, and since I'm an ill minded person this evening I decided to apply some of what I understood that was 'unit testing' to this 'project' ;)

Please, people that do serious development, don't read this post or at least don't laught. This is just a side approach of someone that hadn't heard about "unit testing" until last night. This post is thought for those like me that are ill :screwy:

Going into matter: I've choosed the function 'getArgs' from the prototype of rsyncOptimum which is in charge of proccessing the command line arguments. See above for the downloadable 1.2 version.

First I created the Unittest script for checking that the yet to be written function accomplishes the following requisites:

returns the RSYNC_DIR variable which has the rsync mirror address
returns the LOCAL_DIR variable which has the local rsync mirror path
returns the rsync command line arguments (see previous posts)
returns also the rsyncOptimum own cmdline arguments (these will be used for testing, verbose, .... modes)
must do some tricks, ie: add always an '/' to the LOCAL_DIR to avoid syntax problems.
must be robust with rsync arguments such as quoting.

To test all or most of the posible compromised cases I wrote the following UnitTest script:

#! /bin/bash
# ./TEST_function_getArgs_devel
# unittest getArgs::rsyncOptimum

. function_getArgs_devel # sourcing getArgs()

SOURCE=rsync://carroll.cac.psu.edu/mandrake/updates/9.2 # remote dir
DEST=~/__anon_project/TEST/updates   # local dir (final slash added in testcases)
rsync_args1='--this are some --rysnc -opts'        # first scenario
rsync_args2="--this are some '.*' --rysnc -opts"   # second scenario
rsync_args3='--this are some ".*" --rysnc -opts'   # third scenario
script_args="-v -t -c"

# test cases:
cmdline1='$rsync_args $SOURCE $DEST -- $script_args'
cmdline2='$rsync_args $SOURCE ${DEST}/ -- $script_args'
cmdline3='$rsync_args $SOURCE ${DEST}'
cmdline4='$rsync_args $SOURCE ${DEST}/'
ALL=$(echo cmdline{1,2,3,4})

# output:
output () { printf "Test %s::%-15s.................. %s\n" "${1}" "${2}" "${3}"; }

# RUN ALL THE TESTCASES -----------------------------------------------
for scenario in $(echo rsync_args{1,2,3}); do # checking quotation protection
   rsync_args=$(eval echo \$$scenario)
   echo "------------"
   echo "SCENARIO ${scenario##*s}"
   echo "------------"
for testcase in $ALL; do
   LOCAL_DIR=
   RSYNC_DIR=
   RSYNC_ARGS=
   SCRIPT_ARGS=
   eval $(eval "getArgs \$$testcase") # CALLING the FUNCTION as if cmdlineargs
   # first test, check local dir:
   first_EVENT="local dir"
   if [ "${LOCAL_DIR}" ==  "${DEST%%/}/" ]; then
       first_UNITTEST="OK"
   else    
       first_UNITTEST="FAIL"
   fi
   output $testcase "$first_EVENT" $first_UNITTEST

   # second test, check rsync dir:
   second_EVENT="rsync dir"
   if [ "${RSYNC_DIR}" ==  "${SOURCE}" ]; then
       second_UNITTEST="OK"
   else
       second_UNITTEST="FAIL"
   fi
   output $testcase "$second_EVENT" $second_UNITTEST

   # third test, check the rsync args:
   third_EVENT="rsync args"
   if [ "${RSYNC_ARGS}" ==  "${rsync_args} $SOURCE ${DEST%%/}/" ]; then
       third_UNITTEST="OK"
   else
       third_UNITTEST="FAIL"
   fi
   output $testcase "$third_EVENT" $third_UNITTEST

   # fourth test, check script args:
   fourth_EVENT="script args"
   old_script_args="${script_args}"
   eval "echo \$$testcase" | grep -q ' -- ' || script_args= 
   if [ "${SCRIPT_ARGS}" ==  "${script_args}" ]; then
       fourth_UNITTEST="OK"
   else
       fourth_UNITTEST="FAIL"
   fi
   script_args="$old_script_args"
   output $testcase "$fourth_EVENT" $fourth_UNITTEST

   # end tests
   echo
done
done

which after some error/test runs, has yielded this function:

#! /bin/bash
# ./function_getArgs_devel
# function getArgs::rsyncOptimum
                                                                                                                                                 
getArgs () {
   # returns a string with the variables used in the main script
   # input: command line arguments
   # a '--' delimits rsync args from the script args
   local CMDL_ARGS="${@}"
   # check if there are script args:
   if echo ${CMDL_ARGS} | grep -q ' -- '; then
       local RSYNC_ARGS="${CMDL_ARGS%% -- *}"
       local SCRIPT_ARGS="${CMDL_ARGS##* -- }"
   else
       local RSYNC_ARGS="${CMDL_ARGS}"
       local SCRIPT_ARGS=
   fi
   local DEST=$(echo $RSYNC_ARGS| awk '{print $NF}')
   local SRC=$(echo $RSYNC_ARGS | awk '{print $(NF-1)}')
   # return string, this values will be global in the main script
   STRING="LOCAL_DIR=\"${DEST%%/}/\"; RSYNC_DIR=\"${SRC}\";
           RSYNC_ARGS=\"${RSYNC_ARGS%%/}/\"; SCRIPT_ARGS=\"${SCRIPT_ARGS}\""
   # echo $STRING >&2  # For debug
   echo ${STRING}
  }

And now it is a pleasure to see the run of the UnitTest script:

~/__anon_project$ bash TEST_function_getArgs_devel
------------
SCENARIO 1
------------
Test cmdline1::local dir      .................. OK
Test cmdline1::rsync dir      .................. OK
Test cmdline1::rsync args     .................. OK
Test cmdline1::script args    .................. OK

Test cmdline2::local dir      .................. OK
Test cmdline2::rsync dir      .................. OK
Test cmdline2::rsync args     .................. OK
Test cmdline2::script args    .................. OK

Test cmdline3::local dir      .................. OK
Test cmdline3::rsync dir      .................. OK
Test cmdline3::rsync args     .................. OK
Test cmdline3::script args    .................. OK

Test cmdline4::local dir      .................. OK
Test cmdline4::rsync dir      .................. OK
Test cmdline4::rsync args     .................. OK
Test cmdline4::script args    .................. OK

------------
SCENARIO 2
------------
Test cmdline1::local dir      .................. OK

.......


Test cmdline4::local dir      .................. OK
Test cmdline4::rsync dir      .................. OK
Test cmdline4::rsync args     .................. OK
Test cmdline4::script args    .................. OK

~/__anon_project$

COOL isn't it :juggle:

I'll be glad to comment this whit anyone interested

April 3, 2004

<WARNING>THIS POST IS ALMOST OFFTOIC</WARNING>

Its not off topic aru, this kind of thing is just what this forum is for B)

This is just a side approach of someone that hadn't heard about "unit testing" until last night.

Never heard of it either, and i have to say i am none the wiser. What you have written above is totally over my head :woops:

But don't stop :D :D

aru · April 6, 2004

Version 1.3 is OUT!!!! ready for real tests!!!

I sadly have to say that I couldn't do a full rewrite as was my intention because I had no free time during past weekend. So today I've made some changes to make the script operative and I've added a testing mode too. The script own options are passed by adding at the end of the command line '-- <script options>' (at pressent time the -t tag for testing is the only option handled by the script). Better explained here:

~/__anon_project$ head rsyncOptimum
#! /bin/bash

# $Id: rsyncOptimum,v 1.3 2004/04/06 16:16:44 aru Exp aru $
# PROTOTYPE :: FOR TESTING ONLY PURPOSES
# You may pass scripts options throug the command line, by adding -- at the end
# of normal rsync options. Available options are:
#   -t     Testing; just show what would be done, but don't run it.
# Examples of usage:
# rsyncOptimum <rsync options> SOURCE DEST
# rsyncOptimum <rsync options> SOURCE DEST -- -t  (TESTING MODE!!!)

~/__anon_project$

Now you can run the script either in testing mode or in real mode.

The script is attached below.

A couple of examples of usage:

Testing mode (adding at the end of command line '-- -t')

~/__anon_project$ ./rsyncOptimum -av --partial --delete --exclude '/.*' rsync://carroll.cac.psu.edu/mandrake/updates/9.2 /home/aru/__anon_project/TEST/updates/ -- -t

TESTMODE:: mv /home/aru/__anon_project/TEST/updates/9.2/RPMS/XFree86-Xvfb-4.3-24.3.92mdk.i586.rpm /home/aru/__anon_project/TEST/updates/9.2/RPMS/XFree86-Xvfb-4.3-24.4.92mdk.i586.rpm
TESTMODE:: mv /home/aru/__anon_project/TEST/updates/9.2/RPMS/ethereal-0.10.0a-0.1.92mdk.i586.rpm /home/aru/__anon_project/TEST/updates/9.2/RPMS/ethereal-0.10.3-0.1.92mdk.i586.rpm
TESTMODE:: mv /home/aru/__anon_project/TEST/updates/9.2/RPMS/libdha0.1-0.91-7.1.92mdk.i586.rpm /home/aru/__anon_project/TEST/updates/9.2/RPMS/libdha0.1-0.91-8.2.92mdk.i586.rpm
TESTMODE:: mv /home/aru/__anon_project/TEST/updates/9.2/RPMS/libpostproc0-0.91-7.1.92mdk.i586.rpm /home/aru/__anon_project/TEST/updates/9.2/RPMS/libpostproc0-0.91-8.2.92mdk.i586.rpm
TESTMODE:: mv /home/aru/__anon_project/TEST/updates/9.2/RPMS/libpostproc0-devel-0.91-7.1.92mdk.i586.rpm /home/aru/__anon_project/TEST/updates/9.2/RPMS/libpostproc0-devel-0.91-8.2.92mdk.i586.rpm
TESTMODE:: mv /home/aru/__anon_project/TEST/updates/9.2/RPMS/mencoder-0.91-7.1.92mdk.i586.rpm /home/aru/__anon_project/TEST/updates/9.2/RPMS/mencoder-0.91-8.2.92mdk.i586.rpm
TESTMODE:: mv /home/aru/__anon_project/TEST/updates/9.2/RPMS/mplayer-0.91-7.1.92mdk.i586.rpm /home/aru/__anon_project/TEST/updates/9.2/RPMS/mplayer-0.91-8.2.92mdk.i586.rpm
TESTMODE:: mv /home/aru/__anon_project/TEST/updates/9.2/RPMS/mplayer-gui-0.91-7.1.92mdk.i586.rpm /home/aru/__anon_project/TEST/updates/9.2/RPMS/mplayer-gui-0.91-8.2.92mdk.i586.rpm
TESTMODE:: mv /home/aru/__anon_project/TEST/updates/9.2/RPMS/tcpdump-3.7.2-2.0.92mdk.i586.rpm /home/aru/__anon_project/TEST/updates/9.2/RPMS/tcpdump-3.7.2-2.1.92mdk.i586.rpm
TESTMODE:: mv /home/aru/__anon_project/TEST/updates/9.2/SRPMS/ethereal-0.10.0a-0.1.92mdk.src.rpm /home/aru/__anon_project/TEST/updates/9.2/SRPMS/ethereal-0.10.3-0.1.92mdk.src.rpm
TESTMODE:: mv /home/aru/__anon_project/TEST/updates/9.2/SRPMS/mplayer-0.91-7.1.92mdk.src.rpm /home/aru/__anon_project/TEST/updates/9.2/SRPMS/mplayer-0.91-8.2.92mdk.src.rpm
TESTMODE:: exec rsync -av --partial --delete --exclude /.\* rsync://carroll.cac.psu.edu/mandrake/updates/9.2 /home/aru/__anon_project/TEST/updates/

~/__anon_project$

Real mode (w/o the -- part):

~/__anon_project$ ./rsyncOptimum -av --partial --delete --exclude '/.*' rsync://carroll.cac.psu.edu/mandrake/updates/9.2 ~/__anon_project/TEST/updates
«/home/aru/__anon_project/TEST/updates/9.2/RPMS/XFree86-Xvfb-4.3-24.3.92mdk.i586.rpm» -> «/home/aru/__anon_project/TEST/updates/9.2/RPMS/XFree86-Xvfb-4.3-24.4.92mdk.i586.rpm»
«/home/aru/__anon_project/TEST/updates/9.2/RPMS/ethereal-0.10.0a-0.1.92mdk.i586.rpm» -> «/home/aru/__anon_project/TEST/updates/9.2/RPMS/ethereal-0.10.3-0.1.92mdk.i586.rpm»
«/home/aru/__anon_project/TEST/updates/9.2/RPMS/libdha0.1-0.91-7.1.92mdk.i586.rpm» -> «/home/aru/__anon_project/TEST/updates/9.2/RPMS/libdha0.1-0.91-8.2.92mdk.i586.rpm»
«/home/aru/__anon_project/TEST/updates/9.2/RPMS/libpostproc0-0.91-7.1.92mdk.i586.rpm» -> «/home/aru/__anon_project/TEST/updates/9.2/RPMS/libpostproc0-0.91-8.2.92mdk.i586.rpm»
«/home/aru/__anon_project/TEST/updates/9.2/RPMS/libpostproc0-devel-0.91-7.1.92mdk.i586.rpm» -> «/home/aru/__anon_project/TEST/updates/9.2/RPMS/libpostproc0-devel-0.91-8.2.92mdk.i586.rpm»
«/home/aru/__anon_project/TEST/updates/9.2/RPMS/mencoder-0.91-7.1.92mdk.i586.rpm» -> «/home/aru/__anon_project/TEST/updates/9.2/RPMS/mencoder-0.91-8.2.92mdk.i586.rpm»
«/home/aru/__anon_project/TEST/updates/9.2/RPMS/mplayer-0.91-7.1.92mdk.i586.rpm» -> «/home/aru/__anon_project/TEST/updates/9.2/RPMS/mplayer-0.91-8.2.92mdk.i586.rpm»
«/home/aru/__anon_project/TEST/updates/9.2/RPMS/mplayer-gui-0.91-7.1.92mdk.i586.rpm» -> «/home/aru/__anon_project/TEST/updates/9.2/RPMS/mplayer-gui-0.91-8.2.92mdk.i586.rpm»
«/home/aru/__anon_project/TEST/updates/9.2/RPMS/tcpdump-3.7.2-2.0.92mdk.i586.rpm» -> «/home/aru/__anon_project/TEST/updates/9.2/RPMS/tcpdump-3.7.2-2.1.92mdk.i586.rpm»
«/home/aru/__anon_project/TEST/updates/9.2/SRPMS/ethereal-0.10.0a-0.1.92mdk.src.rpm» -> «/home/aru/__anon_project/TEST/updates/9.2/SRPMS/ethereal-0.10.3-0.1.92mdk.src.rpm»
«/home/aru/__anon_project/TEST/updates/9.2/SRPMS/mplayer-0.91-7.1.92mdk.src.rpm» -> «/home/aru/__anon_project/TEST/updates/9.2/SRPMS/mplayer-0.91-8.2.92mdk.src.rpm»
exec rsync -av --partial --delete --exclude /.\* rsync://carroll.cac.psu.edu/mandrake/updates/9.2 /home/aru/__anon_project/TEST/updates/

 
                             Welcome to
 The Pennsylvania State University / Center for Academic Computing
                FTP Archives at carroll.cac.psu.edu
 
Due to U.S. Exports Regulations, all cryptographic software on this
site is subject to the following legal notice:
 
    This site includes publicly available encryption source code
    which, together with object code resulting from the compiling of
    publicly available source code, may be exported from the United
    States under License Exception "TSU" pursuant to 15 C.F.R. Section
    740.13(e).
 
This legal notice applies to cryptographic software only. Please see
the Bureau of Export Administration, http://www.bxa.doc.gov/ for more
information about current U.S. regulations.
 
Public mirror sites who use our rsyncd may want to contact us for access
to a restricted-access rsyncd to better avoid being blocked out.

Please send suggestions/comments to ftp@carroll.cac.psu.edu.
 

receiving file list ...
<...>

~/__anon_project$

Waiting for your input after your tests :P

rsyncOptimum

April 6, 2004

Great! will let you know the results soon B)

Hey! you posted this 5 minutes ago, and six people have down loaded it already??? I think were being watched, :D

aru · April 6, 2004

:lol2:

Oh! I've noticed that I've forgotten to add a disclaimer in the script. Lads, don't do tests on critical mirrors! :deal:

btw, names are still accepted

Script to update names (Hey aru !!)))

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Guest anon

Link to comment

Share on other sites

Link to comment

Share on other sites

Guest anon

Link to comment

Share on other sites

Link to comment

Share on other sites

Guest anon

Link to comment

Share on other sites

Link to comment

Share on other sites

Guest anon

Link to comment

Share on other sites

Link to comment

Share on other sites

Guest anon

Link to comment

Share on other sites

Link to comment

Share on other sites

Guest anon

Link to comment

Share on other sites

Link to comment

Share on other sites

Guest anon

Link to comment

Share on other sites

Link to comment

Share on other sites

Join the conversation

Announcements