rescue Posted April 25, 2005 Report Share Posted April 25, 2005 (edited) I need a mirroring tool that can mirror ftp directories (I want to mirror Mandrivalinux 2005 LE). The triky part here is that I am behind a firewall and the only port in and out is port 80 for web browing. Wget doesn't work, I can set HTTP_PROXY but with -m key I only get index.http with directory listing. Edited April 25, 2005 by rescue Quote Link to comment Share on other sites More sharing options...
idud Posted April 25, 2005 Report Share Posted April 25, 2005 wget -e ftp_proxy=http://your.proxy:port --proxy-user=username --proxy-passwd=passwd --mirror ftp://some.url/some/dir/ Quote Link to comment Share on other sites More sharing options...
rescue Posted April 25, 2005 Author Report Share Posted April 25, 2005 I need a mirroring tool that can mirror ftp directories (I want to mirror Mandrivalinux 2005 LE). The triky part here is that I am behind a firewall and the only port in and out is port 80 for web browing. Wget doesn't work, I can set HTTP_PROXY but with -m key I only get index.http with directory listing. <{POST_SNAPBACK}> [anz@andres ~]$ wget -e ftp_proxy=http://10.129.1.1:8080 --mirror ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586 --11:18:27-- ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586 => `ftp.aso.ee/pub/Mandrake/official/2005/i586' Loon ühendust serveriga 10.129.1.1:8080... ühendus loodud. Proxy päring saadetud, ootan vastust... 200 OK Pikkus: määramata [text/html] [ <=> ] 5,821 --.--K/s Last-modified päist pole -- ei kasuta ajatempleid. 11:18:28 (50.30 KB/s) - `ftp.aso.ee/pub/Mandrake/official/2005/i586' saved [5,821] LÕPETATUD --11:18:28-- Alla laetud: 5,821 baiti kokku 1 failis [anz@andres ~]$ as I said I can not use wget ... I get index.html with directory listing with it ... Quote Link to comment Share on other sites More sharing options...
idud Posted April 25, 2005 Report Share Posted April 25, 2005 (edited) ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/ Look at the ``/'' at the end of the url. And please don't say such thing like "wget can't do it" if you don't even know what you are doing. Edited April 25, 2005 by idud Quote Link to comment Share on other sites More sharing options...
rescue Posted April 25, 2005 Author Report Share Posted April 25, 2005 ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/ Look at the ``/'' at the end of the url. And please don't say such thing like "wget can't do it" if you don't even know what you are doing. <{POST_SNAPBACK}> I am sure wget can where is no firewall: [anz@andres ~]$ wget -e ftp_proxy=http://10.129.1.1:8080 --mirror ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/ --11:42:55-- ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/ => `ftp.aso.ee/pub/Mandrake/official/2005/i586/index.html' Loon ühendust serveriga 10.129.1.1:8080... ühendus loodud. Proxy päring saadetud, ootan vastust... 200 OK Pikkus: määramata [text/html] [ <=> ] 5,757 --.--K/s Last-modified päist pole -- ei kasuta ajatempleid. 11:42:56 (152.96 KB/s) - `ftp.aso.ee/pub/Mandrake/official/2005/i586/index.html' saved [5,757] LÕPETATUD --11:42:56-- Alla laetud: 5,757 baiti kokku 1 failis [anz@andres ~]$ Quote Link to comment Share on other sites More sharing options...
Guest anon Posted April 25, 2005 Report Share Posted April 25, 2005 The proxy has little to do with it, your only getting the index.html file because you need to add some switches to wget. If you want to mirror everything at that exact url then do; wget -r ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/ If you just want the files for 2005 then do : wget -r ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/media/main/ Quote Link to comment Share on other sites More sharing options...
rescue Posted April 25, 2005 Author Report Share Posted April 25, 2005 (edited) The proxy has little to do with it, your only getting the index.html file because you need to add some switches to wget.If you want to mirror everything at that exact url then do; wget -r ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/ If you just want the files for 2005 then do : wget -r ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/media/main/ <{POST_SNAPBACK}> ftp_proxy and http_proxy are exported [anz@andres ~]$ wget -r ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/ --13:08:19-- ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/ => `ftp.aso.ee/pub/Mandrake/official/2005/i586/index.html' Loon ühendust serveriga 10.129.1.1:8080... ühendus loodud. Proxy päring saadetud, ootan vastust... 200 OK Pikkus: määramata [text/html] [ <=> ] 5,757 --.--K/s 13:08:19 (158.84 KB/s) - `ftp.aso.ee/pub/Mandrake/official/2005/i586/index.html' saved [5,757] LÕPETATUD --13:08:19-- Alla laetud: 5,757 baiti kokku 1 failis [anz@andres ~]$ I am pretty sure it's the firewall ... cause at home where is no firewall (but where I get like max 20K/s) wget works with -m key Edited April 25, 2005 by rescue Quote Link to comment Share on other sites More sharing options...
idud Posted April 26, 2005 Report Share Posted April 26, 2005 Try this: wget -r ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/* It will search for .listing not index.html. Quote Link to comment Share on other sites More sharing options...
rescue Posted April 26, 2005 Author Report Share Posted April 26, 2005 Try this: wget -r ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/* It will search for .listing not index.html. <{POST_SNAPBACK}> What will I do with .listing fail? Can it be uesd with wget to retrive files? [anz@andres ~]$ wget -r ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/* Hoiatus: HTTP ei toeta jokkereid. <- what this means? HTTP doesn't allow jockers? --08:23:38-- ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/* => `ftp.aso.ee/pub/Mandrake/official/2005/i586/*' Loon ühendust serveriga 10.129.1.1:8080... ühendus loodud. Proxy päring saadetud, ootan vastust... 404 Not Found 08:23:38 VIGA 404: Not Found. LÕPETATUD --08:23:38-- Alla laetud: 0 baiti kokku 0 failis [anz@andres ~]$ Quote Link to comment Share on other sites More sharing options...
idud Posted April 26, 2005 Report Share Posted April 26, 2005 Wget will recursively lists all files under the directory that you wish to download, and save it to .listing. But wildcards is not allowed in HTTP. So I think the problem is your http_proxy, CMIIW. Can you just unexport the http_proxy and use only ftp_proxy? So wget will try to connect to the server's ftp port. Or maybe you can add the servers ftp port number, such as ftp://ftp.aso.ee:21/pub/Mandrake/official/2005/i586/*. Btw, can you translate the error messages to English? I don't understand Estonian. Quote Link to comment Share on other sites More sharing options...
rescue Posted April 26, 2005 Author Report Share Posted April 26, 2005 If I use it like this wget -e ftp_proxy=http://10.129.1.1:8080 -r ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/* will it use the ftp_proxy or my exported http proxy? ... but this way it didn't work: [anz@andres ~]$ wget -e ftp_proxy=10.129.1.1:8080 -r ftp://ftp.aso.ee:21/pub/Mandrake/official/2005/i586/* Hoiatus: HTTP ei toeta jokkereid. --08:57:54-- ftp://ftp.aso.ee/pub/Mandrake/official/2005/i586/* => `ftp.aso.ee/pub/Mandrake/official/2005/i586/*' Loon ühendust serveriga 10.129.1.1:8080... ühendus loodud. Proxy päring saadetud, ootan vastust... 404 Not Found 08:57:54 VIGA 404: Not Found. LÕPETATUD --08:57:54-- Alla laetud: 0 baiti kokku 0 failis [anz@andres ~]$ Quote Link to comment Share on other sites More sharing options...
idud Posted April 26, 2005 Report Share Posted April 26, 2005 Proxy user? Proxy passwd? Quote Link to comment Share on other sites More sharing options...
rescue Posted April 26, 2005 Author Report Share Posted April 26, 2005 Proxy user? Proxy passwd? <{POST_SNAPBACK}> no, I don't need to use them ... but I wget failes is thtere something else to use? Quote Link to comment Share on other sites More sharing options...
idud Posted April 27, 2005 Report Share Posted April 27, 2005 I don't have any other ``gun''. Wget is my best weapon. Sorry. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.