rsync vs wget

[ramfree17]

hi,

 

once i read somewhere about a question regarding mirroring packages. i suggested rsync but somebody else gave a reply that wget is better because rsync treats progressive package updates as separate entities (meaning it will redownload a package that has been updated) while wget can handle it. or something like that.

 

anyway, i have a couple or rsync commands at home for whenever i connect to the internet and i noticed that i do have multiple rpms for certain packages and i deleted manually the old versions. i tried reading the manpage of wget but i dont recognize an option for progressive downloads of updated packages. i saw something like if X bytes of old_package has been downloaded and new_package in the server has been uploaded (and has a bigger file size Y) then wget will only downloaded X - Y bytes. that doesnt make much sense as new packages might have changed in the area of the X downloaded bytes.

 

anybody here have an idea on how to get wget handle the job of rsync more efficiently? or am i still confused out of my wits?

 

ciao!

[Guest anon]

Only reason you have duplicate RPMs is,

1) you set up rsync wrongly :P

2) The server you retrieved them from carries old copies, unlikely but it does happen.

The following command will Archive, update, and give you verbose output. It will also delete files on your machine that do not exist on the sending side.

rsync -auv --delete

There are all kinds of options you can add, here's one i use for our server:

 rsync -auv --partial --delete --exclude '.*' ftp.sunet.se::pub/Linux/distributions/mandrake/iso/ /var/ftp/pub/Mandrake/stable

Hre's a few commands from the rsync man.

OPTIONS SUMMARY

      Here is a short summary of the options available in rsync. Please refer

      to the detailed description below for a complete description.

 

       -v, --verbose               increase verbosity

       -q, --quiet                 decrease verbosity

       -c, --checksum              always checksum

       -a, --archive               archive mode, equivalent to -rlptgoD

       -r, --recursive             recurse into directories

       -R, --relative              use relative path names

       -b, --backup                make backups (default ~ suffix)

           --backup-dir            make backups into this directory

           --suffix=SUFFIX         define backup suffix

       -u, --update                update only (don´t overwrite newer files)

       -l, --links                 copy symlinks as symlinks

       -L, --copy-links            copy the referent of symlinks

           --copy-unsafe-links     copy links outside the source tree

           --safe-links            ignore links outside the destination tree

       -H, --hard-links            preserve hard links

       -p, --perms                 preserve permissions

       -o, --owner                 preserve owner (root only)

       -g, --group                 preserve group

       -D, --devices               preserve devices (root only)

       -t, --times                 preserve times

       -S, --sparse                handle sparse files efficiently

       -n, --dry-run               show what would have been transferred

       -W, --whole-file            copy whole files, no incremental checks

           --no-whole-file         turn off --whole-file

       -x, --one-file-system       don´t cross filesystem boundaries

       -B, --block-size=SIZE       checksum blocking size (default 700)

       -e, --rsh=COMMAND           specify the remote shell to use

           --rsync-path=PATH       specify path to rsync on the remote machine

       -C, --cvs-exclude           auto ignore files in the same way CVS does

           --existing              only update files that already exist

           --ignore-existing       ignore files that already exist on the recei

ving side

--delete                delete files that don´t exist on the sending side

           --delete-excluded       also delete excluded files on the receiving side

           --delete-after          delete after transferring, not before

           --ignore-errors         delete even if there are IO errors

           --max-delete=NUM        don´t delete more than NUM files

           --partial               keep partially transferred files

           --force                 force deletion of directories even if not empty

           --numeric-ids           don´t map uid/gid values by user/group name

           --timeout=TIME          set IO timeout in seconds

       -I, --ignore-times          don´t exclude files that match length and time

           --size-only             only use file size when determining if a file should be tr

ansferred

           --modify-window=NUM     Timestamp window (seconds) for file match (default=0)

       -T  --temp-dir=DIR          create temporary files in directory DIR

           --compare-dest=DIR      also compare destination files relative to DIR

           --link-dest=DIR         create hardlinks to DIR for unchanged files

       -P                          equivalent to --partial --progress

       -z, --compress              compress file data

           --exclude=PATTERN       exclude files matching PATTERN

           --exclude-from=FILE     exclude patterns listed in FILE

           --include=PATTERN       don´t exclude files matching PATTERN

           --include-from=FILE     don´t exclude patterns listed in FILE

           --version               print version number

           --daemon                run as a rsync daemon

           --no-detach             do not detach from the parent

           --address=ADDRESS       bind to the specified address

           --config=FILE           specify alternate rsyncd.conf file

           --port=PORT             specify alternate rsyncd port number

           --blocking-io           use blocking IO for the remote shell

           --no-blocking-io        turn off --blocking-io

           --stats                 give some file transfer stats

           --progress              show progress during transfer

           --log-format=FORMAT     log file transfers using specified format

           --password-file=FILE    get password from FILE

           --bwlimit=KBPS          limit I/O bandwidth, KBytes per second

           --read-batch=PREFIX     read batch fileset starting with PREFIX

           --write-batch=PREFIX    write batch fileset starting with PREFIX

       -h, --help                  show this help screen

[/code]

[ramfree17]

thanks gramps. im trying it now but i dont think i will be seeing the result pretty soon on my sucky dial-up. :#:

 

btw, when does the delete take place? after all the packages have been downloaded? here are my old and new commands for synching...

OLD

rsync -Pavz  carroll.cac.psu.edu::mandrake/updates/9.1/RPMS/* /vault/rsync/updates/9.1/RPMS/

NEW

rsync -Pauvz --delete carroll.cac.psu.edu::mandrake/updates/9.1/RPMS/* /vault/rsync/updates/9.1/RPMS/

 

hope it works...

 

ciao!

[Guest anon]

New one should work fine. :)

 

Files are deleted before any packages are downloaded.

 

EDIT

Why have you typed "a" twice ?

EDIT, I see you corrected it :lol: