Random SATA Drive is always busy/in use (RAID 5 / mdadm)

[Dakhara]

I have 4 SATA's in a RAID 5 array using mdadm. Yesterday when I started the computer the RAID did not build/mount.

When trying to load the array manually I get the message
"mdadm: cannot open device /dev/sd(a,b,c,d)1: Device or resource busy"
The drives should not be mounted or in use.

The output of the affected drive in mdadm (mdadm --examine /dev/sd_1) looks normal.

The weirdest part is that rebooting often changes which drive is marked as busy, it can be any of the 4 SATA drives.

Any suggestions on how to figure out why/what is being used and how to disable it? I have tried searching for similar threads here and in google and haven't found anything similar or that worked.

Let me know if I should post any outputs as well.

Thanks!

[Manko10]

Could you post the content of /proc/mdstat here? Also the output of ls -l /dev/md/ could be helpful.

[Dakhara]

I have the array up right now with just 3 of 4 drives using the following command.

mdadm --assemble /dev/md0 --run /dev/sdb1 /dev/sdc1 /dev/sde1


/proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdb1[0] sde1[3] sdc1[1]
2930279808 blocks level 5, 64k chunk, algorithm 2 [4/3] [UU_U]

md_d0 : inactive sdd1[2](S)
976759936 blocks

unused devices: <none>



ls -l /dev/md/

total 0
brw------- 1 root root 254, 0 2010-12-03 15:34 d0
brw------- 1 root root 254, 1 2010-12-03 15:34 d0p1
brw------- 1 root root 254, 2 2010-12-03 15:34 d0p2
brw------- 1 root root 254, 3 2010-12-03 15:34 d0p3
brw------- 1 root root 254, 4 2010-12-03 15:34 d0p4

[Manko10]

Hmm, strange. Looks as if there were some issues with the superblocks. The only idea I have, is zeroing them and recreating the array. Even though mdadm would do its best not destroying your striped data you should make a backup before.
But before you do anything you should evaluate the contents of /proc/mdadm the next time this issue occurs. It could be that Linux has assembled the devices to another array according to their superblocks. Also running mdadm -D /dev/mdX could be useful to get some more information about the status of your RAID.
If Linux has actually set up an array you don't want, you can stop it with mdadm --stop. Then you should be able to assemble it manually.
But of course that's not a real solution. If this problem continues to happen you should consider recreating the array. Also make sure your disks are still alive and not damaged.

Note that RAID5 is also often a danger for your data, particularly when you have more than 12TB but here you have only 4 disks so RAID5 should be relatively safe.

[Dakhara]

I figured I was going to have to do something like that.

mdadm -D /dev/md0

/dev/md0:
Version : 00.90
Creation Time : Sat Aug 21 18:14:54 2010
Raid Level : raid5
Array Size : 2930279808 (2794.53 GiB 3000.61 GB)
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Fri Dec 3 16:34:25 2010
State : clean, degraded
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

UUID : cf5236c3:6f08b462:65a04b8d:362531ae (local to host SERVER)
Events : 0.5198

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
2 0 0 2 removed
3 8 65 3 active sync /dev/sde1

[Dakhara]

I guess what I am the most confused about is how it can be a superblock issue when the RAID starts correctly with any of the 3 drives.

For example I just restarted and now it is sdb that is "busy" and has the superblock error. The drives have always historically booted up in the same order.


/proc/mdstat Have not yet started the array with the --run flag

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md_d0 : inactive sdb1[0](S)
976759936 blocks

unused devices: <none>

ls -l /dev/md/
total 0
brw------- 1 root root 254, 0 2010-12-03 17:58 d0
brw------- 1 root root 254, 1 2010-12-03 17:58 d0p1
brw------- 1 root root 254, 2 2010-12-03 17:58 d0p2
brw------- 1 root root 254, 3 2010-12-03 17:58 d0p3
brw------- 1 root root 254, 4 2010-12-03 17:58 d0p4

mdadm -D /dev/md0
mdadm: md device /dev/md0 does not appear to be active.

Would this mean Linux isn't setting up an array that I don't want (you mentioned in your last post).



Could I try formating the busy disk and adding back into array? Or would I not even be able to format since it's "busy"

Thank you very much for all your help! Going to start backing everything up completely as you suggested.

[Manko10]

That says your array is downgraded because there is one disk disk missing which is /dev/sdd1. Add it with mdadm -a.
But I think that doesn't solve your problem in general, does it? Well, maybe it can but that wouldn't make sense because you said it could be any disk that isn't auto-added.

EDIT:
Too late...
I haven't said you should format the disks. I just suggested recreating the array with mdadm --zero-superblock and then mdadm --create as usual. But you should make a backup before.

[Dakhara]

Ah alright. Yeah I can't even add the disk back because it's "busy"

I'll try the zero-superblock in the morning once I know everything is backup. By format I meant that if this all fails I'll have to start from scratch no?

[Manko10]

You shouldn't need to start from scratch. Anway, recreating the array could be necessary.
What you could also try is to stop the running array, remove all disks from it with mdadm -r and then add them with mdadm -a.

[Dakhara]

One problem that I have had in this past on this system, which may have caused this superblock error in the long run is this.

Occasionally my IDE hard drive, which has the OS installation on it. Ends up shifting from sda to sdc.

It seems to usualy be recognized first, but sometimes moves into the middle of the SATA drives, this makes the RAID array not assemble unless I run mdadm -E /dev/sd_1 on the 4 raid disks and use --force to assemble the array.

However eventually the order switches with the IDE first again I just have to run the same mdadm -E again and force assembly.

It happens maybe once every 30-40 boots. Is there anyway to lock down these device assignments? I haven't had any luck previously.