Help with a shell script to download and parse webpage

[Steve Scrimpshire]

I'm trying to archive on my computer two particular threads from a WebCrossing bulletin board. Each thread contains 1000s of messages and hundreds of pages, so it is not feasible to do it by hand. I've tried with links, but it dies with some error before it downloads another page. Here is how I want to do:

 

I download the first page of the thread with wget, like this:

 

wget http://forums.desmoinesregister.com/webx/c...iH^2@.ee81626/0

 

Then there is a line on the page that contains a link to the next page that looks like this:

 

<a href="/webx/cgi-bin/WebX?14@105.rbZqaUujeiH^4@.ee81626/20"><img hspace=0 align=top alt="More Messages" border=0 src="/Images/smmore.gif" WIDTH=68 HEIGHT=15></a>

 

Regexps always kick my butt, but I'm sure what I need to do is real simple. I need to find that line (there are lots of other links on the page, too) and extract only the /webx/cgi-bin/WebX?14@105.rbZqaUujeiH^4@.ee81626/20 part and append the http://forums.desmoinesregister.com/ to the front of it and download it with wget, then do the process again, until there is no line that looks like that (no 'More Messages' button). If I download the first page and run grep on it like this:

 

grep "More Messages" WebX?14@105.rbZqaUujeiH%5E2@.ee81626%2F0

 

I get:

<hr width=200 align=left><a href="/webx/cgi-bin/WebX?14@105.rbZqaUujeiH^4@.ee81626/20"><img hspace=0 align=top alt="More Messages" border=0 src="/Images/smmore.gif" WIDTH=68 HEIGHT=15></a><a href="/webx/cgi-bin/WebX?14@105.rbZqaUujeiH^4@.ee81626"><img hspace=0 align=top alt="Recent Messages" border=0 src="/Images/smrecent.gif" WIDTH=68 HEIGHT=15></a><a href="/webx/cgi-bin/WebX?128@105.rbZqaUujeiH^4@.ee81626"><img hspace=0 align=top alt="All Messages" border=0 src="/Images/smallmsg.gif" WIDTH=68 HEIGHT=15></a><a href="/webx/cgi-bin/WebX?230@105.rbZqaUujeiH^4@.ee81626"><img hspace=0 align=top alt="Outline" border=0 src="/Images/smoutline.gif" WIDTH=68 HEIGHT=15></a>  (3884 following messages)<p>

 

Any help would be appreciated.

 

Edit: In case it's not obvious, this part: /webx/cgi-bin/WebX?14@105.rbZqaUujeiH^4@.ee81626/20 is going to be different each time and I can't guarantee any part of it, except up to /webx/cgi-bin/WebX?

[Steve Scrimpshire]

Never mind. I did it in Perl.