Multipath/device mapper error caused read-only root filesystem

[gnerd]

I've gotten myself into a nice pickle. I added a SAN disk to a machine so I could make a filesystem and mount it. I scanned the bus, and the new disk showed up in multipath -ll:

mpath16 (360060e80056468000000646800000053) dm-28 HITACHI,OPEN-V
[size=34G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 1:0:0:14 sdp 65:224 [active][ready]

I partitioned /dev/sdp with fdisk (the whole disk as one partition). However, when tried to make a filesystem, the OS said it was in use by another program. I ran fuser against sdp, and it showed multipath had hold of it. I was looking for a way to make multipath let go, and I thought if I gave the wwn a static name and then blacklisted the name in /etc/multipath.conf, I could get control of it. I found this article for assigning static device names to wwns:

[can't post the url, new user, but it's easy to find googling for "assign a specific device name to a wwn in linux"]

My 25-names.rules looked like this:

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="360060e80056468000000646800000053", SYMLINK+="dbdump%n"

I ran udevtest and it looked Ok, so I ran start_udev. udev created the dbdump1 link, so I thought I was home free. I added the following to multipath.conf and restarted multipathd:

devnode_blacklist {
# devnode "*"
devnode "^{ram|raw|loop|fd|md|dm-|sr|scd|st)[0-15]*"
devnode "^VxDMP[0-15]*"
devnode "^VxDM[0-15]*"
devnode "^dmpconfig[0-15]*"
devnode "^dbdump[0-15]*"
devnode "^cciss*"
}

I restarted multipathd but multipath -ll still showed the device. I modified my blacklist regex with: "^mpath16", "^sdp[0-15]*" and "^dm-28". Restarting multipath after each edit, the device still showed up in multipath -ll. However, after adding the "^dm-28" pattern and restarting multipath, I got the following error on multipath stop:

dm_task_set_name: Device /dev/cciss/c0d0p3 not found
Command failed

/dev/cciss/c0d0p3 is the root partition on the internal drive. I discovered that running multipath -f mpath16 would remove sdp from the list, and running dmsetup remove mpath16 removed it from device mapper. I was then able to create a filesystem and mount it up.

HOWEVER, when I rebooted to test my fstab entry for the new disk, the system errored on boot, insisting that there was an invalid entry in fstab for sdp, and put me into a single user shell. The root filesystem is in read-only with (REPAIR) before the root prompt.

All I can figure is that the system somehow had /dev/cciss/c0d0p3 mapped to dm-28. But I don't know how that could be. Can anyone save me? Thanks in advance.

ADDENDUM: I was able to remount the root fs in read-write and the system came up. However, sdp didn't mount, and crs can't start (it's an oracle system).

[moader]

Fixed by putting _netdev in /etc/fstab in the fourth column.