Why can't I unmount my 2nd SATA drive?

[websissy]

I'm having a problem with my secondary (backup) hard drive. The file system on my PRIMARY drive is damaged. So, I'm trying to run an FSCK on the secondary drive before I flip the two drives and make backup my primary drive. The problem is even though sdb1 is NOT listed in /etc/fstab and I have NOT explicitly mounted it, Linux insists sdb1 is mounted when I try to run even a read-only e2fsck or a fsck on that partition.

In short, when I type:

e2fsck -nf /dev/sdb1

I get:

Warning! /dev/sdb1 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.

Yet, when I try to umount the drive, Linux insists it isn't there!

umount /dev/sdb1

umount replies,

umount: /dev/sdb1: not mounted

There is a directory named /sdb1 in the main file system directory; but it shows as completely empty.

This has been going on since my server host's techs reinstalled the OS on the primary boot drive in mid-August. It was NOT this way in the original system setup. But something happened during the drive rebuild and it is this way now.

There seems to be NOTHING I can do from here to unmount that drive.


I noticed someone in this thread:

http://www.linuxforums.org/forum/deb...ring-boot.html

had the exact opposite problem (couldn't mount their sata drive) and it turned out to be driver related. Could I be suffering from a similar issue?

Can anyone here tell me why linux reports /dev/sdb1 can’t be unmounted because it isn't mounted; but e2fsck insists it IS mounted?

Thanks!

[HROAdmin26]

What do mtab, /proc/scsi/scsi, and fdisk show?

cat /etc/mtab

cat /proc/scsi/scsi

fdisk -l

* Edit: Can your issue be driver as in the post you linked? No, if the drives are both SATA, the same driver is used - if sda is mounted, the driver is active/available.

[websissy]

Hold on while I run those commands and capture their output...

Sorry it took a while. Capturing this output turned out to be a bear to do! The KVM device I'm using runs under java control so there's no cut and paste of all that complicated output from the screen to the forums and with no server using images was out too. So I had to capture the output as screen shots then OCR the text in the image in order to paste text here! What a PIA!

Here you go...

---------
cat /etc/mtab
---------

myserver:~# cat /etc/mtab
/dev/sda1 / ext3 rw,errors=remount-ro 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,mode=0755 8 0
proc /proc proc rw,noexec,nosuid,nodev 0 0
sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0
proc/bus/usb /proc/bus/usb usbfs rw 0 0
udev /dev tmpfs rw,mode=0755 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0

---------
cat /proc/scsi/scsi
---------

myserver:~# cat /proc/scsi/scsi 

Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00 
	Vendor: ATA	Model: ST3500630AS      Rev: 3.AA
	Type:	Direct-Access                   ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 00 Lun: 00
	Vendor: ATA	Model: ST3500630AS      Rev: 3.AA
	Type:	Direct-Access                   ANSI SCSI revision: 05

---------
fdisk -l
---------

myserver:-# fdisk -l

Disk /dev/sda: 500.1 GB, 500107862016 bytes 
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device 		Boot	Start	End	Blocks		Id System
/dev/sda1	*	1	59327	476544096  	83 Linux
/dev/sda2		59328	60801	11839905	 5 Extended
/dev/sda5		59328	60801	11839873+ 	82 Linux swap / Solaris

Disk /dev/sdb: 500.1 GB, 500107862016 bytes 
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16865 * 512 = 8225280 bytes

Device 		Boot	Start	End	Blocks		Id System
/dev/sdb1	*	1	59327	476544096	83 Linux
/dev/sdb2		59328	60801	11839905	5  Extended
/dev/sdb5		59328	60801	11839873+ 	82 Linux swap / Solaris

myserver:~#

Does that help?

[HROAdmin26]

I can't tell directly what the issue is...

Try using lsof to find any open files? Try running it against that /sdb1 directory...

Does dmesg show anything?

What was keeping these drives "sync'ed" as a backup? Is that process still active?

[websissy]

lsof shows nothing on /sdb1

However, dmesg does show THIS related to sdb1:

Kernel command line: root=/dev/sdb1 ro single

The trouble is I'm not sure what exactly dmesg contains (are these boot time messages?) or what this message means. One thing I DO remember however, is that sdb1 is definitely being referenced somewhere in the /etc/init.d boot process. I remember messages about it flash by on the screen.

There's also this tasty morsel in dmesg:

sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSA device sdb: 976993168 512-byte hdwr sectors (500108 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
 sdb: sdb1 sdb2 < sdb5 >
sd 1:0:0:0: Attached scsi disk sdb
kjournald startng. Commit interval 5 minutes
EXT3-fs: mounted filesystem with ordered data mode.

[websissy]

Well, after sleeping on this a few hours, I managed to figure out the reason /dev/sdb1 can't be dismounted is because grub somehow has it specified as the primary boot device. In short, it appears the system is actually taking its boot stuff from the second drive and its data and everything else from the "primary" (first) drive.

I managed to prove this by (just for the helluvit) deliberately trying to unmount sda1 (umount /dev/sda1) while the system was running in single user mode. When it reported the unmount was successful, I KNEW there was something screwy going on. After that, I did a test run of e2fsck in "do-no-harm" mode:

e2fsck -nf /dev/sda1

and discovered Linux did not complain /dev/sda1 was mounted -- even though it was originally the primary boot device in this system. After that test ran and reported no errors, I tried:

e2fsck -p /dev/sda1

Again I got no complaints and the process ran to completion without errors or problems.

On the other hand, when I try to umount /dev/sdb1, umount STILL insists the device isn't mounted but when I try to:

e2fsck -nf /dev/sdb1

it says:

Warning!  /dev/sdb1 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.

So, I did some further digging and eventually realized there's some sort of trick being played in the grub setup that's boots the system in read-only mode from the second drive, sdb1 (thus protecting the OS and all of its components from being deleted) but then remaps everything but the system areas (/usr, /var, /boot and probably /etc) back into the drive image from partition sda1.

This 'boot trick' effectively protects the system areas of the sda1 drive from being wiped out accidentally -- which happened on this box back in August (while we were still in sandbox mode) when an overlooked synonym and a mount of sdb1 in a temporary work directory I removed with rm -rf accidentally wiped out BOTH drives at once.

Bottom line: I now know the issue is being caused by grub. But even after hours of carefully studying the grub docs and trying to force the boot to occur from sda1 again, I've been unable to figure out how to do that.

Are there any Grub experts around ANYWHERE who can offer some advice? That program is completely hopeless!

Thanks.

[websissy]

Okay, I figured it out. The way grub was set up on my server it's primary boot device was set as /dev/sdb1 (a.k.a. hd1) in the grub startup menu defaults.

So even though everything ELSE for the system was drawn from /dev/sda1 (e.g. all of the web content) the boot utilities and linux itself were coming from /dev/sdb1. In short, the system portion of sda1 wasn't being used. This seems to have served the DUAL purpose of protecting the primary drive's bootstrap from accidental deletion and ensuring that there was always a good un-modified bootstrap on the system no matter what happened. I guess it might also be a way to protect the main system bootstrap against rootkits, for example.

Here's an example of what those 'AutoMagic' parameters in /boot/grub/menu.lst look like:

---------
BEFORE
---------

## ## End Default Options ##

title           Debian GNU/Linux, kernel 2.6.18-6-amd64
root            (hd1,0)       <=========== for sda1, make this hd0 
kernel          /boot/vmlinuz-2.6.18-6-amd64 root=/dev/sdb1 ro    <=========== change sdb1 to sda1
initrd          /boot/initrd.img-2.6.18-6-amd64
savedefault

title           Debian GNU/Linux, kernel 2.6.18-6-amd64 (single-user mode)
root            (hd1,0)       <=========== for sda1, make this hd0
kernel          /boot/vmlinuz-2.6.18-6-amd64 root=/dev/sdb1 ro single    <=========== change sdb1 to sda1
initrd          /boot/initrd.img-2.6.18-6-amd64
savedefault

### END DEBIAN AUTOMAGIC KERNELS LIST

---------
AFTER
---------

## ## End Default Options ##

title           Debian GNU/Linux, kernel 2.6.18-6-amd64
root            (hd0,0)       <=========== note change!
kernel          /boot/vmlinuz-2.6.18-6-amd64 root=/dev/sda1 ro    <=========== note change!
initrd          /boot/initrd.img-2.6.18-6-amd64
savedefault

title           Debian GNU/Linux, kernel 2.6.18-6-amd64 (single-user mode)
root            (hd0,0)       <=========== note change!
kernel          /boot/vmlinuz-2.6.18-6-amd64 root=/dev/sda1 ro single    <=========== note change!
initrd          /boot/initrd.img-2.6.18-6-amd64
savedefault

### END DEBIAN AUTOMAGIC KERNELS LIST

This "AutoMagic" change of boot drives was all being handled behind the scenes at boot time by Grub. It apparently grabs its startup parameters from /boot/menu.lst on the first partition of the first drive where it finds a bootstrap and then uses them to control where to go from there. In our case all four of the "AutoMagic" parameters there for both boot methods (multi-user and single-user [aka 'maintenance mode']) were set to use /dev/sdb1 and NOT /dev/sda1

Grub's boot parameters can be changed by the admin by going to the system console (or accessing the system via KVM) and waiting for the grub boot options menu to appear (assuming of course that grub was configured during installation to display this menu). The arrow keys can be used there to select the boot method you want to use and THEN the admin can press "e" to edit and change the start-up parameters for that boot method. Once those parameters have been changed if they include the "savedefault" option, the parameters should be saved for the next reboot. However, in our case I had to tinker with it a bit to make sure the new startup changes got saved.

That explains why e2fsck and fsck both detected /dev/sdb1 as "in use". Because they were designed to protect the filesystem, those utilities were written to be smart enough not to be fooled by boot tricks played by grub. However, the Linux mount utilities and unmount utilities ARE fooled by grub's trickery.

I have a hunch this technique may be in some ways unique to the Debian Linux distro. However, the rules and methods described here for grub should probably apply to most other distribtuions as well.

So, there you have it. I hope this technique and what I've learned here eventually helps someone else.

Thanks a bunch for your efforts to help!