Jump to content

Session-to-Session Segmented Downloads


skyhawk
 Share

Recommended Posts

I have dial-up Internet access and I want to find a way to download games or other applications (usually archived in zip format) in segments that can be saved to disk one day, then resumed another day, and so on until the complete archive has been downloaded. Segments will generally not exceed 15 MB in size, corresponding to about one hour of download time per session.

 

I see from the Wget man page that Wget will allow such behavior using the "c" (continue) command-line parameter. However, the downloads I am primarily seeking are protected by layers of security to deter "link stealers." Thus, if a URL for a download can be resolved and can be seen on-screen, it cannot be used after the session has ended. The URL changes each time a new session is started.

 

I also see from the Curl man page that it accepts the "--referer" command-line parameter, which might make it a more likely candidate for segmented downloads than Wget.

 

I would guess that I could use my web browser to work through the layers of security, resolve a download's URL, then enter the URL into the command-line for starting Wget, but this creates another problem. Some sites will not allow simultaneous downloading of muliple files, nor will they allow download managers that use multiple connections to accelerate downloads. Using a web browser and Wget to simultaneously download the same archive, even for a fraction of a second, would probably get me banned from some websites.

 

So, does anyone know a good way, if any, to do what I want to do? My primary objective is to be able to resume a partial file download that has been saved to hard disk during a previous session. My secondary concern is the best way to get around the "multiple-file-download" ban, by not accidently triggering the ban.

 

The following is taken from the HOTU FAQs page, for those who want to experiment:

 

"If you've got the referer setting in your download manager set up correctly, but you can't convince it to accept the link, make a bookmark with this link:

 

java script:document.writeln(document.forms[document.forms.length - 1].action + '?firewall=' + (document.getElementsByName('firewall')[0].checked ? 1 : 0) + '&code=' + document.getElementsByName('code')[0].value);

 

Then, when you're ready to download, fill in the form as usual, but instead of clicking the "Go" button, select this bookmark. Then copy-and-paste the URL it gives you into your download manager. I repeat: this will not work unless your referer setting is correctly set up and you fill out the form on the download page completely."

 

I have not yet been able to get a URL to copy-and-paste, as per the above instructions.

 

My one-and-only operating system is Mandriva 2007.0, kernel 2.6.17-5mdv. I primarily use Firefox 1.5.0.7, which came with my Mandriva installation discs, but I also have Firefox 3.0 and Opera 9.50 installed. A download manager that allows session-to-session segmented downloads, that allows web browser integration, and that allows referer entry, would be ideal.

Link to comment
Share on other sites

Have you tried kget?

 

If I am not mistaken, kget is a GUI front-end for Wget. Correct me if I'm wrong. I do have kget installed, by the way.

 

I found the home page for Retriever Download Manager and it looks like it offers all the features I want. It requires JRE 1.5. Retriever is now in version 1.3.

Link to comment
Share on other sites

  • 2 weeks later...

This follow-up to my initial post on this topic should be of interest to those who use dial-up for Internet access and who want a means to save multi-segment downloads session-to-session.

 

cURL is the application you will want to use. Assuming you have cURL installed, open Konsole, <cd> to where you want the download to be saved, and, as an example, type the following command-line:

 

curl -m 3600 -o FILE -v -# URL

 

The remote file (URL) will be saved in the pwd as local file, named FILE (use any local filename you wish, as long as it is not the same as the remote filename). cURL will automatically time-out at 3600 seconds (one hour) in this example. Omit or modify the "-m" option as you wish. The "-v" option tells cURL to function in verbose mode (recommended), and the "-#" option uses the # character for the progress meter. View the page source for each download page to view the desired download link, then copy-and-paste it into the command-line above.

 

Note that if your chosen URL reads something like:

 

http://www.FreeForTheTaking.com/Really Big File.zip

 

where there are spaces in the remote filename, you will need to insert %20 for every space in the remote filename. Thus, the URL in the command-line will read:

 

http://www.FreeForTheTaking.com/Really%20Big%20File.zip

 

Let's assume that you used the "-m 3600" option, cURL automatically timed-out at 3600 seconds, and you have only downloaded 15 megabytes of the 28 megabytes remote file. For the next session, do as above, but use the command-line:

 

curl -C - -m 3600 -o FILE -v -# URL

 

This will cause cURL to continue downloading local FILE at the correct offset of the remote file. Do not forget the " -" after "-C".

 

In this example, two sessions were sufficient to download the complete remote file, but for even larger remote files repeat the procedure as needed. I am forced to restrict my download sessions to 3600 seconds, due to limitations imposed by my ISP. I choose to use a local filename that is different from the remote filename because, if this is not done, I found that in certain instances where the path on the server to the remote file changes every session (security precautions), each download segment will be saved locally as separate files, not appended to one another as they should be.

 

Read the man page for cURL for information on all options available. cURL is a great tool for saving multi-segment downloads. It works flawlessly, if used properly.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...