Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 13
I've been trying to get a cold backup of a 1TB database this weekend, started the whole process Friday and still have yet to get a single device backed up. ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    May 2005
    Posts
    17

    rsync and external usb drive, mpath blacklist


    I've been trying to get a cold backup of a 1TB database this weekend, started the whole process Friday and still have yet to get a single device backed up. I'm using rsync to copy files from my /u17 thru /u29 mounts, and the usb is formatted ext3. Each time the rsync would start off fine but after about 30 minutes it would fail with any number of errors but the most prevalent is "Read only file system", "broken pipe". Here are samples:

    rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown" [sender]: Broken pipe (32)

    rsync: write failed on "<path to one of my .dbf files" failed: Read-only file system (30)
    rsync: chown "<path>" failed: Read-only file system (30)
    rsync: rename "<path of .dbf> -> <rename attempt>": Read-only file system (30)
    rsync error: error in file IO (code 11) at receiver.c(305)
    rsync: connection unexpectantly closed (16787 bytes received so far) [generator]
    rsync error: error in rsync protocol data stream (code 12) at io.c (359)

    I've unmounted and remounted a number of times and kicked off the rsync again and it goes about 30 minutes and I get the same errors. This was all as user 'root', so I tried to do the rsync as user 'oracle' and I get the same thing.

    After looking into the device as it is recognized, it is being picked up by multipath. Would the fact that a usb device is being managed by multipath be a problem? Currently it is mpath15. How would I add usb devices to the mpath blacklist? The usb is being assigned /dev/sdbj but I'm worried that it would change at a reboot. I've searched the web for all of these errors and still no answer.

    Note: I've also just tried to do a copy using 'cp' and got the same "Read only file system" errors. I can sometimes touch a file and sometimes I can't. Any help appreciated, I want to try and get this backup done this weekend.

  2. #2
    Just Joined!
    Join Date
    May 2005
    Posts
    17

    Thumbs down

    Wow. Lots of activity in this forum, I'll be sure to check it regularly.

  3. #3
    Linux Guru Lazydog's Avatar
    Join Date
    Jun 2004
    Location
    The Keystone State
    Posts
    2,677
    Sorry you feel this way. We try to help when we can and this is the first I am seeing this.

    Since you are getting the error failed: Read-only file system. I would think this is because of the way the drive is mounted.
    What does mount show you? Can you post the output from mount and the rsync command you are using?

    Regards
    Robert

    Linux
    The adventure of a life time.

    Linux User #296285
    Get Counted

  4. $spacer_open
    $spacer_close
  5. #4
    Just Joined!
    Join Date
    May 2005
    Posts
    17
    The rsync command I use: "nohup rsync -avSCP /u17/* /media/usb/u17/ &"
    Then that is repeated for /u18 thru /u29, totaling ~1TB. I used to do it all at once in a command with each rsync ending with && and starting the next but with this problem that is a waste of time.

    I spent ten hours yesterday babysitting the backup (have to try again this weekend) and have managed to get about 80% done. I added the usb drive to the multipath.conf file blacklist and am mounting the usb with
    "mount /dev/sdbj1 /media/usb" instead of
    "mount /dev/mpath/mpath15p1 /media/usb".
    The backup lasts much longer before errors occur. As soon as the error occurs, I umount the usb, and run "e2fsck /dev/sdbj1". It finds errors and I have to approve "y" the repairs. Once that is done (takes 30 minutes to and hour each time), I start the backup again where it left off. This occurs on both servers. I split the rsyncs between two different usbs to cut down the total time so I can get it done over a weekend. I get the same errors and have to do the same thing with e2fsck on each server.

    I will post the output from 'mount' and the actual error in next post.

  6. #5
    Just Joined!
    Join Date
    May 2005
    Posts
    17
    Output from "mount": /dev/sdbj1 on /media/usb type ext3 (rw)

    I get the same output even if the usb has errored out with "read only" error listed in first post.

    The first thing that happens is:
    rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown" [sender]: Broken pipe (32)

    Then you get the read only file system errors. These are Dell 2850 servers, RHEL 4 with latest update.

  7. #6
    Linux Guru Lazydog's Avatar
    Join Date
    Jun 2004
    Location
    The Keystone State
    Posts
    2,677
    Are you trying to write more then what the USB drive can handle at one time? It sure sounds like it. There is a limit on how much data can pass over the USB port. Just because you can doesn't mean you are not going to take a performance hit for it.

    Have you tried to run one backup at a time?

    If you are looking to send a lot of data then you should be looking at something that can handle the amount of data you want to transfer.
    Maybe a Hotswap drive bay or an external SATA connection.

    I am sure this is not what you want to hear, but in the game of life the rules cannot be changed without an upgrade.

    Regards
    Robert

    Linux
    The adventure of a life time.

    Linux User #296285
    Get Counted

  8. #7
    Just Joined!
    Join Date
    May 2005
    Posts
    17
    This is a 4-node Oracle RAC environment, so there are 4 servers clustered together, sharing the same datafiles. A 2TB external USB drive is plugged into node1 and another 2TB USB drive is plugged into node3. Once the database is cleanly shutdown, /u17 thru /u23 are rsync'ed to the USB drive on node1 and /u24 thru /u29 plus oracle_home are rsync'ed to the USB drive on node3. There is no contention going.

  9. #8
    Linux Guru Lazydog's Avatar
    Join Date
    Jun 2004
    Location
    The Keystone State
    Posts
    2,677
    Quote Originally Posted by wolfee View Post
    Once the database is cleanly shutdown, /u17 thru /u23 are rsync'ed to the USB drive on node1 and /u24 thru /u29 plus oracle_home are rsync'ed to the USB drive on node3. There is no contention going.
    Are you sure about that? Looking at what you posted above about the script you use I can only think differently,

    The rsync command I use: "nohup rsync -avSCP /u17/* /media/usb/u17/ &"
    That '&' at the end starts the rsync then move onto the next line in your script. If you are listing them one right after the other then they are all starting and fighting for the same bandwidth to the USB port.

    If you want then to run one at a time you need to remove the '&'.

    Regards
    Robert

    Linux
    The adventure of a life time.

    Linux User #296285
    Get Counted

  10. #9
    Just Joined!
    Join Date
    May 2005
    Posts
    17
    ?? The "&" tells nohup to run in the background, "&&" tells nohup to move to the next command. They don't all start at the same time, I've done it that way numerous times and I've seen the logfile output. It processes the first command then moves to the next. That being said, I'm not even doing it that way this time because of the errors I'm getting.

  11. #10
    Linux Guru Lazydog's Avatar
    Join Date
    Jun 2004
    Location
    The Keystone State
    Posts
    2,677
    I would suggest you try it and find out for yourself.
    You may be surprised.

    Think of it this way, how many processes can you have running in the background? Once the process is placed into the background the next process is kick off.

    Don't believe me use top to check.

    Regards
    Robert

    Linux
    The adventure of a life time.

    Linux User #296285
    Get Counted

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •