Tips&Tricks WGET RULZ! Use it to download

[aru]

 

illogic-al

Senior user

Joined: 01 May 2002

Posts: 358

Location: Smack dab in the middle of it all.

 

Post Posted: Sat Sep 21, 2002 5:54 am Post subject: WGET RULZ! Use it to download

_________________________________________________________________

 

 

If you want to download a really large file like one of the Mandrake RCs then use wget. all you have to do is type wget followed by a space and then the url at a console. Kinda like this:

 

Code:

wget  ftp://mirrors.secsup.org/pub/linux/mandrake/Mandrake-iso/i586/MandrakeLinux-9.0rc3-inst-1.i586.iso  

 

then hit enter. It was also be a good thing to find out which of the ftp sites is closer to you (and therefore usually faster). to do this type tracroute at a console follow by the home page of the ftp site (not the whole url to the file) eg

 

Code:

traceroute mirrors.secsup.org  

 

notice that I didn put the ftp:// part in there or the /pub/linux/mandrake/Mandrake-iso/i586/MandrakeLinux-9.0rc3-inst-1.i586.iso.

 

 

arusabal

Moderator

Joined: 17 Apr 2002

Posts: 836

Location: Spain

Post Posted: Sat Sep 21, 2002 10:23 am Post subject: Re: WGET RULZ! Use it to download

_________________________________________________________________

 

 

NEVER forget to use the -c flag in wget (especially if you have a low/unestable connection) because with that flag the download will be resumed at the point it stopped (in case it did) instead of start a completely new download from 0.

 

Also I want to add that wget is fundamental in bash scripts to automate downloads. Here is a little example extracted from a script of mine that attempts to download a new kernel version from "www.kernel.org" (it is used by cron):

 

Code:

#!/bin/bash

....

#variable declarations and functions, not shown as I just want to show a couple of examples of wget;)



KERNEL_URL=$(/usr/bin/wget -O - http://www.kernel.org 2>/dev/null 

| egrep 'linux-2.4.[0-9]+.tar.gz'

| sed 's/^.*http://(.*)".*/1/')

KERNEL_WEB=$(basename ${KERNEL_URL})



#Where $KERNEL_URL will return the full url to current stable kernel as:

#  www.kernel.org/pub/linux/kernel/v2.4/linux-2.4.19.tar.gz

#and $KERNEL_WEB is just the name of that kernel (well the tarball name), as:

# linux-2.4.19.tar.gz

....

# Then my script do a comparision with a config file where it says which

# is the last kernel version that I have (it can be done also through 'uname') and if the

#kernel at kernel.org ($KERNEL_WEB) is newer the script will attempt to download it:

....



/usr/bin/wget -c http://${KERNEL_URL} &> /dev/null



....

# then rewrites the config file changing the name of the last kernel and

# emails me (also updates a log file... and many other stuff (like in case that it is

# not completely download resume next time)... but that's

# not the point now;) )

....



#End

 

 

arusabal

Moderator

Joined: 17 Apr 2002

Posts: 836

Location: Spain

Post Posted: Mon Sep 23, 2002 6:14 pm Post subject: Just a stupid

example

_________________________________________________________________

 

 

today I'm completely bored, so I've made this stupid script:

who_is_on_mub

 

I was not sure at all which was the propper place to post it, my first idea was to post it here, as it is just another example of wget usage within scripts, but I also thought that it could perfectly fit at MUB discussion Twisted Evil

 

here is the script:

 

Code:

[user@host]$ cat who_is_on_mub

#!/bin/bash

# Shows the users logged-in at MUB (those not invisible;) )

# Usage:

#       who_is_on_mub [mub]  ---> Will show you who is loged on the Official Board

#       who_is_on_mub mubs   ---> Will tell who is loged in the 'renew' board



if [ "${1:-"mub"}" == "mubs" ]; then

  URL="http://www.mandrakeusers.org/index.php"

else

  URL="http://www.club-nihil.net/mub/index.php"

fi



echo ""Visible" users logged-in at ${URL%"index.php"}:"



wget -q -O - $URL |

 grep "Registered Users:" |

 sed 's/.*(Registered Users:*.)/2/' |

 tr ',' 'n' |

 sed 's/^.*">?/2/; s/</.*>//; s/<b>(.*)/1 (mod)/'

 

A couple of examples of execution:

 

Code:

[user@host]$ ./who_is_on_mub mubs

"Visible" users loged at http://www.mandrakeusers.org/:

MoonChild

thayne (mod)



[user@host]$ ./who_is_on_mub

"Visible" users loged at http://www.club-nihil.net/mub/:

arusabal (mod)

darksun

lakelover



 

 

stupid, isn't it? Rolling Eyes

 

I'm very bad with regular expressions, so I'll be very glad if any of you could make a better use of sed in my script, or for example include a line or two to show the total number of users online (hidden and not logged-in). please tell me!

 

mmm, I think I'm going to include this script in my Bash Tips&tricks thread because of that couple of 'parameter expansion' that I've used on it (I still remember that I promise a series of posts about shell parameter expansion Wink )

 

***Sorry for my english***

 

 

Ludootje

Newbie

Joined: 12 Jan 2003

Posts: 2

Post Posted: Sun Jan 12, 2003 3:26 pm Post subject: Re: WGET RULZ!

Use it to download

_________________________________________________________________

 

 

 

illogic-al wrote:

#variable declarations and functions' date=' not shown as I just want to show

a couple of examples of wget Wink

 

[/i']

 

Could you please post the entire script? This can be very useful to me

- not especially to d/l the kernel, but to learn more about bash.

Thanks!

 

 

 

Editor's note: This thread was originally posted at the old MUB (Mandrake User Board at club-nihil). This post is the result of a 99% automatic backup, so due to its nature some text may be lost (improbable but possible).