External USB Storage Device Issues

[MikeH30]

I am running SUSE LINUX ENTERPRISE SERVER 10 - SP2

I recently completed some server maintenance and unplugged a 300 GB Western Digital USB External Storage device from my server. I noticed that there was a directory in my file system called "media/disk". It had a copy of the files on my external USB drive even though the USB external storage device was unplugged from the server. I do not understand why the files were also on my server file system. This is the cause of a problem I have had before where my server hard disk became full unexpectedly.

A few posts back I had a problem with my server hard disk becoming full as one of my database web applications stopped working. I had the following error "Got error 28 from storage engine".

My intention with the external hard drive is to back up my web root and file server files. I do not intend to keep a copy of the files on my server.

Can someone tell me the proper way to set up a external USB storage device?
I thought it was just plug and play but apparently there is more happening behind the scenes. I would like to run scheduled back up jobs to the external USB device without worrying that a copy of the files will fill the hard drive on my server again.

Thanks.

-Mike

[gogalthorp]

Did you unmount it before you jerked the plug?

[MikeH30]

Yes, I right clicked and chose "unmount". On my distro(SLES10-SP2), I use the GUI the majority of the time.

[Cool01]

There seems to be bugs regarding the use of USB hard drives. Over the last couple of days, I tried to use an ACOMDATA 320 GB hard drive to backup files on an OpenSuSE 10.2 installation. My first attempts were to take big folders like "/root" and simply "copy"/"paste" them onto FAT32 partitions I created on the external drive. This appeared to work, but when I went through the folders to check the contents, I found that only part of the copy was made and the rest was never done -- without any indication of an error.

I then tried manually copying smaller contained folders and checking the results. This got most of the work done. Lastly, I used "tar" to create some tar files and moved them across, and then verified the tar file copies from the external drive, comparing them to the originals on the main hard drive. I could not make a single big tar file of the particularly large hard drive. I might have been able to make a single big tar file on the external drive, (created directly through "tar") but because of the earlier failure, I did not trust the USB drive connection to receive a big (over 10 GB) single target file.

I would advise you to verify your external copies. It sounds like they might not have been properly created.

Also, be sure to make a bug report. Unfortunately, I will probably NOT be making a bug report because I just replaced 10.2 with 10.3, so I cannot help them solve the problem.

[MikeH30]

Yes, I also noticed that copying large directories ( by right clicking and choosing 'copy' then 'paste' with the mouse) to the USB device does not result in a 100% copy of all the files.

I noticed that in order for me to have 100% copy of all my files to the USB device I must use the command line: "cp -r /some/directory/ /destination/directory.

It was my understanding that the mouse and GUI are using the same command line commands underneath the surface. Does someone know why the copy commands on the mouse would not yield the same results as the command line?


-Mike
(SLES10-SP2, GNOME)

[bigtomrodney]

Be careful copying to FAT32. There are many restrictions on it, from the character present in the filenames/paths to the maximum allowed depth to the size of files. FAT32 is quite conservative in the characters it accepts in filenames.

Also if you want to backup your root directory be aware that this should be done offline and more importantly that FAT32 has no support for file permissions and as such would not be suitable for restoring a system from. The best option is to create a tarball of the directory being backed up while preserving permissions.

[MikeH30]

Thanks for the explanation. I never had to format the external hard drive so I do not know if it is FAT32. I just plugged it in right out of the box and the OS detected it. What is the default format of a Western Digital external storage device?

-Mike

[Cool01]

Be careful copying to FAT32. There are many restrictions on it, from the character present in the filenames/paths to the maximum allowed depth to the size of files. FAT32 is quite conservative in the characters it accepts in filenames.

Also if you want to backup your root directory be aware that this should be done offline and more importantly that FAT32 has no support for file permissions and as such would not be suitable for restoring a system from. The best option is to create a tarball of the directory being backed up while preserving permissions.

I am aware of the issues. I have used dual-boot Linux with Windows as far back as 10 years ago, and have experience with Unix System V and MS-DOS in my history. I keep my file names portable within directories, and the permissions do not affect my data folders on most of my computers, since I am usually not on multi-user installations.

It is possible to write the copy program to cover naming problems. But even is you just let it fail there is NO excuse for not reporting the failure to the user in a dialog window on a windowing system, or an error return to the standard output (or the standard error output) in a text shell program. It is either incredibly sloppy programming, or a bug. In this case I would hope it was just the latter.

[Cool01]

Actually, I was going to say that really, you should get to know "tar" and use it for this kind of job. Because of the first failure, I decided to be very cautious in what I was doing, but if "tar" and the "usb" drivers are working properly, you can make a "tar" file on the external drive, do all your backups retaining all the permissions and proper file names, and then verify that the copies are correct on the external drive.

On OpenSuSE 10.3 I found the the "--verify" did not work while I was making the "tar" file, but the "--compare" was definitely working. The only problem is that you cannot use any compression if you want to use the "--compare" option. But that was no problem for me.

This is really the best way to know that you have a good copy and not trash.

Either that or get one of the commercial backup utilities.

[oldcpu]

Thanks for the explanation. I never had to format the external hard drive so I do not know if it is FAT32. I just plugged it in right out of the box and the OS detected it. What is the default format of a Western Digital external storage device?

External USB drives tend to be formatted either as FAT32 or as NTFS. 300GB is close to the dividing line (smaller are FAT32 and larger NTFS) so you could flip a coin. Not very scientific.

Instead, with your USB drive plugged in, I recommend you type in a gnome-terminal / konsole:

df -h
su -c 'fdisk -l'

(that is a small "L" and not a "one")

Correlating the output of those two commands should enable you to figure out if your external drive is FAT32 or NTFS (assuming the drive shows up).

If it is NTFS, then openSUSE by default tends to have difficulty in providing hot-plug read/write to that drive. A work around for that hotplug automounting of NTFS is documented here:
NTFS external hard drive hotplug - openSUSE

this also works in openSUSE-10.3 in addition to 11.0. To re-iterate the work around is to remove your external hard drive, create a symbolic link in /sbin:

cd /sbin
su -c 'ln -s mount.ntfs-3g mount.ntfs'

and then plug your external hard drive back in. You only need to do this once. This way the removable media will be hotplugged. If for any reason you need to get back to the old behavior, simply remove the symlink.