Help with details of cp -u [solved]

[banjo]

I have an interesting problem with using cp -u.

 

I have an external USB drive with 40 Gig FAT32 (for portability).

I plan on using the external drive to keep some backups.

I am doing my backups by simply copying selected directories

to the external drive.... nothing fancy.

 

I am writing the files using "cp -u" to make the copies a bit faster

(the -u switch causes it to copy only new files and files that have

newer timestamps than the destination file).

 

I wrote a bunch of my directories to it last night to see how it would work.

This went very well for the most part, and the files were copied OK.

But there were problems with files that had the same names and only different

capitalizations (FAT32 is case insensitive). So I changed the names of some

of those files to avoid the problem and did the "cp -u" again.

 

The "new" files were not copied over. I tried this a couple of times with

the same result. The last time I tried it, cp returned immediately without

really accessing the drive, as though the information had been cached

or something. The renamed files did not show up on the external drive.

 

Questions:

How does cp decide what is a "new" file?

Does changing the name leave the modification time untouched?

Do I also need to touch the files?

Does cp cache file information?

How can it know the timestamp of the destination without reading the

external drive?

 

I am running Mandy 9.1 on a ext3 copying files to an external USB with

a 40 Gig FAT32.

 

Thanks in advance

Banjo

(_)=='=~

 

[moved from Software by spinynorman]

[banjo]

Ah, me. I fgured this one out.

 

:blush:

 

I started using Unix in about 1975, and cp was just about the second command that I

learned after learning cd....... so you would think that I would have it under control

after 30 years, but............

 

Did I mention..... :blush: ?

 

Here is what happened in the Case of the Missing Files.

 

If you use a command such as

 

cp -dpRu /home/user/stuff /mnt/storage/home/user/stuff

 

to copy a directory to an external drive, it copies the entire directory "stuff" to the

destination.... including the directory file itself........ OKfine. That

makes perfect sense.

 

But the first time I executed the command the destination file, /mnt/storage/home/user/stuff

did not exist yet.

 

It appears that if the target directory does not exist, cp is happy enough to copy the

entire directory one level up... into.... /mnt/storage/home/user, thus creating the

target directory itself. So I ended up with the intended result, /mnt/storage/home/user/stuff

with the rest of the directory tree beneath it.

 

The second time I executed the same command, it did find the directory called

/mnt/storage/home/user/stuff, so it copied the source directory /home/user/stuff

into /mnt/storage/home/user/stuff thus giving me /mnt/storage/home/user/stuff/stuff

where the entire directory tree was duplicated.... including the "missing" new files.

 

I simply did not notice that the new duplicate had been created and therefore I

could not find the new files.

 

The solution to this embarrasing problem is to issue the correct command to

the cp program, which is:

 

cp -dpRu /home/user/stuff /mnt/storage/home/user

 

with no "stuff" on the end. This works with the intended results first time and every

time thereafter since all it needs to do is to copy the source directory into

the destination directory.

 

Thanks to the MUB for your patience with this 30 year, veteran n00b as

he stumbles through the use of basic commands.

 

Linux rocks!

 

:blush:

 

Banjo

(_)=='=~

 

P.S. I also put this into the wrong forum since it is a software issue anyway.

:blush:

Edited June 18, 2005 by banjo

[banjo]

Not to beat a dead horse or anything, but never ever ever never ever try to back up

a Linux system onto a FAT32 file system. It works great for transferring files, but

there are so many problems with file incompatibilites that it is not worth the effort.

 

1). Links are not copied since they cannot be created on the FAT32.

2). Mail cannot be backed up because the file names have invalid characters in them.

3). File ownership is not preserved during the copy, and chown on the FAT32 is not allowed.

4). The FAT32 uses more space to store the files than ext3 (because of cluster issues, I suppose)

 

I am just about to give up on this and wipe the disk and start over. Or maybe I will give this

one to the kids for their fnWindows laptops and get a newer, bigger one for me.

:D

 

Anyway, if you are thinking of using a FAT32 system as a backup disk, think again.

 

Banjo

(_)=='=~

[banjo]

I thought that I would add an update to this topic since I am discovering

things that I did not know before.

 

When talking to a FAT32 partition, there is no facility in the file system to store

Linux style file attributes. The only ones that it can save are read/write

and perhaps the create and/or modification dates.

 

This makes perfect sense if I had just thought for a minute about how

the FAT32 has worked in the past. It is simply a really lame file system.

 

What is confusing at the surface is that Linux fakes up the ownership

fields by using the mount properties of the device and displays them

for all the files. Hence, the output of ls presents file ownership for the

user who mounted the external FAT32. That ownership information is not

preserved anywhere.

 

I have confirmed this by logging in as another user, mounting the external

drive, and then seeing that all the files on the disk now belong to that other user.

So, of course, chown will not work. D'uh.

 

More discoveries to come, until somebody tells me to shut up.

 

Banjo

(_)=='=~

[AussieJohn]

I think you are going great guns mate. I am following your posts with great interest and I am learning a lot from you.

 

Cheers. John.

[theYinYeti]

What I often do with "limited" filesystem (ISO-9660, FAT...) is to use them to store tar files. So the backup goes something like this:

/place/to/backup# tar cjf /mnt/fat32/BCKP0001.TBZ .

Everything is kept as it should (Unix-like) in the tar file.

 

Yves.

[banjo]

Thanks for the tips. I am considering how to use the tar files effectively

to do these backups. I am still trying to keep my backups portable.

I also like having the files visible on the drive instead of buried in

a tar. The other people using this computer can cope with a copy,

but might not want to deal with extraction from a tar. There is

only one Linux guru in our family....

 

The files in my Mail directory are being rejected by the FAT32 because of

invalid characters in their names - I think it is a colon. But, if I tar them off,

I will only be able to extract them back to a Linux box.

But that is OK because they would be useless on a WinBox anyway.

So I might do that.

 

I now have a script that I use to do the copies for me. It asks me where

the USB disk is mounted and then it reads a list of directories from a file,

which I call ~/.backupdirs, and copies them -R over to the external

drive. Other than the glitches I mentioned before, it seems to work fairly well.

I might add some code to look for the Mail directory and tar.gz it

just in case. I will never go back to fnWindows permanently anyway, so

that should do the job.

 

I still have some permission glitches, but I might be able to get around

those by running the script as root. Since ownership is not stored on the

external FAT32 that should not be detrimental for any user's ability to retrieve

the files.

 

The search goes on.

 

Banjo

(_)=='=~

[theYinYeti]

If you want a really Cool! backup with files readily available, then look at this:

http://mikerubel.org/computers/rsync_snapshots/index.html

 

Yves.

[banjo]

That certainly is a unique and interesting method for backups. Unfortunately I am

making my backup on a FAT32 system, and there is no facility for any

kind of links at all, much less hard links. It doesn't even store ownership or

file permissions.

 

Banjo

(_)=='=~