Jump to content
Sign in to follow this  
rescue

A good mirroring tool

Recommended Posts

I need a mirroring tool that can mirror ftp directories (I want to mirror Mandrivalinux 2005 LE). The triky part here is that I am behind a firewall and the only port in and out is port 80 for web browing. Wget doesn't work, I can set HTTP_PROXY but with -m key I only get index.http with directory listing.

Edited by rescue

Share this post


Link to post
Share on other sites
wget -e ftp_proxy=http://your.proxy:port --proxy-user=username --proxy-passwd=passwd --mirror ftp://some.url/some/dir/

Share this post


Link to post
Share on other sites
I need a mirroring tool that can mirror ftp directories (I want to mirror Mandrivalinux 2005 LE). The triky part here is that I am behind a firewall and the only port in and out is port 80 for web browing. Wget doesn't work, I can set HTTP_PROXY but with -m key I only get index.http with directory listing.

 

[anz@andres ~]$ wget -e ftp_proxy=http://10.129.1.1:8080 --mirror ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586

--11:18:27-- ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586

=> `ftp.aso.ee/pub/Mandrake/official/2005/i586'

Loon ühendust serveriga 10.129.1.1:8080... ühendus loodud.

Proxy päring saadetud, ootan vastust... 200 OK

Pikkus: määramata [text/html]

 

[ <=> ] 5,821 --.--K/s

 

Last-modified päist pole -- ei kasuta ajatempleid.

11:18:28 (50.30 KB/s) - `ftp.aso.ee/pub/Mandrake/official/2005/i586' saved [5,821]

 

 

LÕPETATUD --11:18:28--

Alla laetud: 5,821 baiti kokku 1 failis

[anz@andres ~]$

 

as I said I can not use wget ... I get index.html with directory listing with it ...

Share this post


Link to post
Share on other sites
ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/

 

Look at the ``/'' at the end of the url. And please don't say such thing like "wget can't do it" if you don't even know what you are doing.

 

I am sure wget can where is no firewall:

 

[anz@andres ~]$ wget -e ftp_proxy=http://10.129.1.1:8080 --mirror ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/

--11:42:55-- ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/

=> `ftp.aso.ee/pub/Mandrake/official/2005/i586/index.html'

Loon ühendust serveriga 10.129.1.1:8080... ühendus loodud.

Proxy päring saadetud, ootan vastust... 200 OK

Pikkus: määramata [text/html]

 

[ <=> ] 5,757 --.--K/s

 

Last-modified päist pole -- ei kasuta ajatempleid.

11:42:56 (152.96 KB/s) - `ftp.aso.ee/pub/Mandrake/official/2005/i586/index.html' saved [5,757]

 

 

LÕPETATUD --11:42:56--

Alla laetud: 5,757 baiti kokku 1 failis

[anz@andres ~]$

Share this post


Link to post
Share on other sites
Guest anon

The proxy has little to do with it, your only getting the index.html file because you need to add some switches to wget.

If you want to mirror everything at that exact url then do;

wget -r  ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/

 

If you just want the files for 2005 then do :

wget -r  ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/media/main/

Share this post


Link to post
Share on other sites
The proxy has little to do with it, your only getting the index.html file because you need to add some switches to wget.

If you want to mirror everything at that exact url then do;

wget -r  ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/

 

If you just want the files for 2005 then do :

wget -r  ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/media/main/

 

ftp_proxy and http_proxy are exported

 

[anz@andres ~]$ wget -r ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/

--13:08:19-- ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/

=> `ftp.aso.ee/pub/Mandrake/official/2005/i586/index.html'

Loon ühendust serveriga 10.129.1.1:8080... ühendus loodud.

Proxy päring saadetud, ootan vastust... 200 OK

Pikkus: määramata [text/html]

 

[ <=> ] 5,757 --.--K/s

 

13:08:19 (158.84 KB/s) - `ftp.aso.ee/pub/Mandrake/official/2005/i586/index.html' saved [5,757]

 

 

LÕPETATUD --13:08:19--

Alla laetud: 5,757 baiti kokku 1 failis

[anz@andres ~]$

 

I am pretty sure it's the firewall ... cause at home where is no firewall (but where I get like max 20K/s) wget works with -m key

Edited by rescue

Share this post


Link to post
Share on other sites

Try this:

wget -r  ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/*

 

It will search for .listing not index.html.

Share this post


Link to post
Share on other sites
Try this:

wget -r  ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/*

 

It will search for .listing not index.html.

 

What will I do with .listing fail? Can it be uesd with wget to retrive files?

 

[anz@andres ~]$ wget -r ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/*

Hoiatus: HTTP ei toeta jokkereid. <- what this means? HTTP doesn't allow jockers?

--08:23:38-- ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/*

=> `ftp.aso.ee/pub/Mandrake/official/2005/i586/*'

Loon ühendust serveriga 10.129.1.1:8080... ühendus loodud.

Proxy päring saadetud, ootan vastust... 404 Not Found

08:23:38 VIGA 404: Not Found.

 

 

LÕPETATUD --08:23:38--

Alla laetud: 0 baiti kokku 0 failis

[anz@andres ~]$

Share this post


Link to post
Share on other sites

Wget will recursively lists all files under the directory that you wish to download, and save it to .listing. But wildcards is not allowed in HTTP. So I think the problem is your http_proxy, CMIIW. Can you just unexport the http_proxy and use only ftp_proxy? So wget will try to connect to the server's ftp port. Or maybe you can add the servers ftp port number, such as ftp://ftp.aso.ee:21/pub/Mandrake/official/2005/i586/*.

 

Btw, can you translate the error messages to English? I don't understand Estonian.

Share this post


Link to post
Share on other sites

If I use it like this

wget -e ftp_proxy=http://10.129.1.1:8080 -r ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/*

 

will it use the ftp_proxy or my exported http proxy? ... but this way it didn't work:

 

[anz@andres ~]$ wget -e ftp_proxy=10.129.1.1:8080 -r ftp://ftp.aso.ee:21/pub/Mandrake/official/2005/i586/*

Hoiatus: HTTP ei toeta jokkereid.

--08:57:54-- ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/*

=> `ftp.aso.ee/pub/Mandrake/official/2005/i586/*'

Loon ühendust serveriga 10.129.1.1:8080... ühendus loodud.

Proxy päring saadetud, ootan vastust... 404 Not Found

08:57:54 VIGA 404: Not Found.

 

 

LÕPETATUD --08:57:54--

Alla laetud: 0 baiti kokku 0 failis

[anz@andres ~]$

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
Sign in to follow this  

×
×
  • Create New...