Vdubjunkie Posted August 29, 2003 Report Share Posted August 29, 2003 Latest dillema, Be it doze or be it Linux I have never had good luck moving tons of data in many nested folders from one drive (or set of drives as the case may be) to another. My latest endeavor involves three 120GB drives set up with approximately 75GB of raid5 space, and the remaining partitions set in raid0 to create about 210GB. I have painstakingly created equivalent folders on the new partitions, and tried my best to not move too much data in any one mv command so as not to lock up the entire box as I did several times in the beginning, but it is getting tiresome, and now, even with those efforts, I have managed to not totally lock the box, but the mv command is hung, I cannot seem to kill it even with a -9, and I've now even tried to umount -l the /mnt/raid5 involved in the process. I guess my query is two pronged. Is there a better way to move massive amounts of data internally in a single machine (much less across a network), and what the heck can I do to kill this hung process and set this back to normal to continue my migration of data? anybody who knows me knows without me saying, but... as always, any help is greatly appreciated :) Quote Link to comment Share on other sites More sharing options...
Cannonfodder Posted August 29, 2003 Report Share Posted August 29, 2003 cp -pax <source> <Destination> Quote Link to comment Share on other sites More sharing options...
Vdubjunkie Posted August 29, 2003 Author Report Share Posted August 29, 2003 cp -pax <source> <Destination> Very nice. Thanks ..fodder. This option for "staying on the same filesystem".. is that referring to ext2, ext3, or as in locally to that machine? I've always wondered about that switch. Also... if anybody has any thoughts on how to kill those jobs without rebooting. I won't be able to boot the machine (nor do I want to at all) until the morning when I return home. And darnit, I need a file off there! :roll: Quote Link to comment Share on other sites More sharing options...
mtweidmann Posted August 29, 2003 Report Share Posted August 29, 2003 Regardless of OS you really shouldn't be getting data corruption moving from HD/partion to another. As your talking about large transferes it could be that your system is getting too hot after long periods of continous HD activity. If so you may want invert in some case fans or think about re-arranging your system for better air flow. Quote Link to comment Share on other sites More sharing options...
theYinYeti Posted August 29, 2003 Report Share Posted August 29, 2003 This is the most common Unix way of moving many files: (cd /source/folder && tar cf - .) | (cd /destination/folder && tar xpf -) Note that the files are copied this way, not moved. If you want, there's also an option to "unreference" links, that is replace symbolic links with the real files (use with extreme caution, and if you know what you're doing). This option is 'h' if I remember correctly, so 'tar cf' would become 'tar chf'. See tar's man page. There's also an option to know what's going on. It is 'v'. So 'tar xpf' would become 'tar xpvf'. In that case, if you want to keep the log output in a file, just append this after the closing parenthesis at the end: >/your/log/file 2>&1 Quote Link to comment Share on other sites More sharing options...
Gowator Posted August 29, 2003 Report Share Posted August 29, 2003 A lot of this depends on the architecture of your PC. One thing that limits this is you are moving all the data across the PCI bus. The bandwidth is limited ... When you say hung, it depends how long ... Personally I wouldn't do a move, if it goes wrong you can loose data. And like mtweidman says continuous disk activity will generate heat. You have a few options, You could nice the job and lower its priority. From O'Reilly Unix power tools tar cf - . | cat | (cd /mnt/newdisk/ && tar xbf 1 -) Using the tar will actually give better integrity and use some CPU. Using some CPU might be a good thing to slow down the process slightly ... I'd modify the above tar so that it preserves permissions as well. hints: See http://linux.oreillynet.com/pub/a/linux/lpt/18_15.html for the pitfalls of cp -r See http://linux.oreillynet.com/pub/a/linux/lpt/18_16.html for an explaination when tar is better :D Quote Link to comment Share on other sites More sharing options...
Vdubjunkie Posted August 30, 2003 Author Report Share Posted August 30, 2003 First, I like to say thank you to everyone who provided information. I have already used the cp -pax, and it worked quite well. In the future when I have another need to transfer a large number of files, I will try some of the others. When you say hung, it depends how long ... ...uh.. hours and hours. I don't know linux in and out yet, but I am no computer noob :lol: As far as heat goes, that wasn't really a factor here. I have experienced the same issue on IBM servers made to get hammered all day for years on end. My box is also quite well suited. It is a good point though. I would never try this in the box I built for my kids :) oh yeah, as far as killing those jobs that wouldn't die.. well, by the time I got home the system was totally locked anyway, so... Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.