[Loothin]

I have a server (crappy workstation converted to server - mostly file storage only) at my office and I'm having an issue with one of my hard-drives switching to read-only. I have a 1TB harddrive that I mounted as my /home. There is a cluster that connects to this computer to upload files on a regular basis (currently using rsync w/ ssh though it used to use scp). Every couple of days or sometimes a couple times in a single day, that device switches to read-only. This causes a HUGE issue because the files that are being uploaded to it are very important for me to do my job (and they aren't small). I recently changed the fstab to ...defaults,errors=continue... though that appears to have made no difference given it was read-only again this morning. When I restart the machine, everything is back to normal.

Edit: I'm currently running F8

Two questions:
1) Ignoring the root cause for a moment, how can I prevent the file-system from switching to read-only. Even if there are errors, I NEED to keep it writable so files can keep coming in, regardless of the potential issues in doing so.

2) What is the root cause. Here is output from the /var/log/messages. If someone can please help me fix the underlying cause I would be very thankful.

***********************
Jul 18 00:29:37 avrserver kernel: ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Jul 18 00:29:37 avrserver kernel: ata6.00: BMDMA2 stat 0x6c0001
Jul 18 00:29:37 avrserver kernel: ata6.00: cmd 35/00:00:a7:23:79/00:04:63:00:00/e0 tag 0 cdb 0x0 data 524288 out
Jul 18 00:29:37 avrserver kernel: res 51/10:cf:d8:23:79/00:01:63:00:00/e3 Emask 0x81 (invalid argument)
Jul 18 00:29:37 avrserver kernel: ata6.00: configured for UDMA/100
Jul 18 00:29:37 avrserver kernel: sd 5:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
Jul 18 00:29:37 avrserver kernel: sd 5:0:0:0: [sdb] Sense Key : Aborted Command [current] [descriptor]
Jul 18 00:29:37 avrserver kernel: Descriptor sense data with sense descriptors (in hex):
Jul 18 00:29:37 avrserver kernel: 72 0b 14 00 00 00 00 0c 00 0a 80 00 00 00 00 00
Jul 18 00:29:37 avrserver kernel: 63 79 23 d8
Jul 18 00:29:37 avrserver kernel: sd 5:0:0:0: [sdb] Add. Sense: Recorded entity not found
Jul 18 00:29:37 avrserver kernel: end_request: I/O error, dev sdb, sector 1668883367
Jul 18 00:29:37 avrserver kernel: Buffer I/O error on device sdb1, logical block 208610413
Jul 18 00:29:37 avrserver kernel: lost page write due to I/O error on sdb1
Jul 18 00:29:37 avrserver kernel: Buffer I/O error on device sdb1, logical block 208610414
Jul 18 00:29:37 avrserver kernel: lost page write due to I/O error on sdb1
Jul 18 00:29:37 avrserver kernel: Buffer I/O error on device sdb1, logical block 208610415
Jul 18 00:29:37 avrserver kernel: lost page write due to I/O error on sdb1
Jul 18 00:29:37 avrserver kernel: Buffer I/O error on device sdb1, logical block 208610416
Jul 18 00:29:37 avrserver kernel: lost page write due to I/O error on sdb1
Jul 18 00:29:37 avrserver kernel: Buffer I/O error on device sdb1, logical block 208610417
Jul 18 00:29:37 avrserver kernel: lost page write due to I/O error on sdb1
Jul 18 00:29:37 avrserver kernel: Buffer I/O error on device sdb1, logical block 208610418
Jul 18 00:29:37 avrserver kernel: lost page write due to I/O error on sdb1
Jul 18 00:29:37 avrserver kernel: Buffer I/O error on device sdb1, logical block 208610419
Jul 18 00:29:37 avrserver kernel: lost page write due to I/O error on sdb1
Jul 18 00:29:37 avrserver kernel: Buffer I/O error on device sdb1, logical block 208610420
Jul 18 00:29:37 avrserver kernel: lost page write due to I/O error on sdb1
Jul 18 00:29:37 avrserver kernel: Buffer I/O error on device sdb1, logical block 208610421
Jul 18 00:29:37 avrserver kernel: lost page write due to I/O error on sdb1
Jul 18 00:29:37 avrserver kernel: Buffer I/O error on device sdb1, logical block 208610422
Jul 18 00:29:37 avrserver kernel: lost page write due to I/O error on sdb1
Jul 18 00:29:37 avrserver kernel: ata6: EH complete
Jul 18 00:29:37 avrserver kernel: Aborting journal on device sdb1.
Jul 18 00:29:37 avrserver kernel: sd 5:0:0:0: [sdb] 1953525168 512-byte hardware sectors (1000205 MB)
Jul 18 00:29:37 avrserver kernel: sd 5:0:0:0: [sdb] Write Protect is off
Jul 18 00:29:37 avrserver kernel: sd 5:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jul 18 00:29:39 avrserver kernel: ext3_abort called.
Jul 18 00:29:39 avrserver kernel: EXT3-fs error (device sdb1): ext3_journal_start_sb: Detected aborted journal
Jul 18 00:29:39 avrserver kernel: Remounting filesystem read-only

***************

[HROAdmin26]

You do not want to ignore this error. You're getting HW I/O errors writing to the HDD. You are potentially writing bad data, so continuing to use the drive is pretty pointless. You also have a journal error, which means the journal may have been dropped and the filesystem has dropped back to "ext2" mode where there is no journal. Without a journal, it's very easy to introduce filesystem corruption.

If you care about this data, you need to fix the HW. Anything written to the drive is suspect at this point.

[Loothin]

So you think it is the hard-drive itself? That is a shame, it's a pretty new harddrive. Bummer...

I think I maybe should have been more clear, I wasn't implying I want to ignore the error, however, I do want to know how to ignore the error WHILE I fix the issue. So if I do need to order a new hard-drive, it would still be nice to prevent this from happening in the mean time. If some thing gets corrupted while writing to this hard drive it isn't a huge deal because all those files are still on the remote cluster. I'll know as soon as I open it if it is corrupted or not. I would rather risk that and have to redownload one or two files instead of having 20 to download the next day because it switched to read-only on me during the night.

Is there any way for me to prevent it switching to read-only? It would be a huge help, especially over this upcoming weekend.

[HROAdmin26]

The issue could also be the IDE controller or cabling, but is likely to be the drive itself.

You have an engine that's "catching on fire" and turning itself off. You're asking for a way to keep the engine running while it's on fire. So I think the short answer is no, there is not a built in way that will circumvent the safety net.

You could create a script that checks the drive status and remounts it, but that still seems like a bad idea because corrupted data is not always so easy to detect.

[Loothin]

I like your analogy. However, I don't like that you think it could possibly be the controller as well. That is a real bummer, because it would be equally as likely I think. I had to get a PCI sata controller to get that hard-drive to work, so it isn't built into the motherboard. I guess that could be a good thing in the sense that I maybe wouldn't have to buy a new motherboard if that was the issue, but still.

The good thing is that if my data is corrupted even in the slightest degree, I'll know. They are all tecplot files and tecplot freaks out if there is even the SLIGHTEST thing wrong with the file and just flat out won't open.

Thanks for the help, I guess it just sounds like I'm screwed till I can get a budget to start fiddling with the hardware. Bummer.

[HROAdmin26]

From the many times I've seen this type of HW error, it's *very* rarely a controller issue. Except for parallel SCSI/terminator issues, it's also rarely the cabling.

BUT, now that you mentioned it's an add-on card - I have seen many *cheaper* add-on cards that fail quickly. You may want to try the drive in another machine/another controller. If you don't care about the data, you can just write zero's to the whole drive and see if you have any I/O errors.

Also, if the drive was a retail box, you should have at least a 3 year warranty on it. 1TB's haven't been out nearly that long. If not, you now know how to weigh that extra $15 price to get a retail boxed vs. OEM drive.