Find the answer to your Linux question:
Results 1 to 3 of 3
So yesterday and this night everythings works fine. Then This morning everybody tries to start the database application and then it appears that our database is gone. After some confusion, ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Mar 2010
    Posts
    9

    recovery corrupts database


    So yesterday and this night everythings works fine.

    Then This morning everybody tries to start the database application and then it appears that our database is gone.

    After some confusion, frustration and other things your probably donīt want to know I discovered the following:

    Jun 30 06:25:02 vkodb syslogd 1.5.0#5: restart.
    Jun 30 06:45:56 vkodb -- MARK --
    Jun 30 07:05:56 vkodb -- MARK --
    Jun 30 07:17:01 vkodb CRON[25020]: (root) CMD ( cd && run-parts --report cron.hourly)
    Jun 30 07:45:48 vkodb syslogd 1.5.0#5: restart.
    Jun 30 07:45:48 vkodb kernel: klogd 1.5.0#5, log source = kmsg started.
    Jun 30 07:45:48 vkodb kernel: Linux version 2.6.18.8.xs5.0.0.10.439 (pondo-2) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Wed Aug 6 23:55:12 UTC 2008
    Jun 30 07:45:48 vkodb kernel: BIOS-provided physical RAM map:
    Jun 30 07:45:48 vkodb kernel: Xen: 0000000000000000 - 0000000177000000 (usable)
    Jun 30 07:45:48 vkodb kernel: 5272MB HIGHMEM available.
    Jun 30 07:45:48 vkodb kernel: 727MB LOWMEM available.
    Jun 30 07:45:48 vkodb kernel: On node 0 totalpages: 1536000
    Jun 30 07:45:48 vkodb kernel: DMA zone: 4096 pages, LIFO batch:0
    Jun 30 07:45:48 vkodb kernel: Normal zone: 182270 pages, LIFO batch:31
    Jun 30 07:45:48 vkodb kernel: HighMem zone: 1349634 pages, LIFO batch:31
    Jun 30 07:45:48 vkodb kernel: Allocating PCI resources starting at 10100000 (gap: 10000000:00400000)
    Jun 30 07:45:48 vkodb kernel: Detected 2500.143 MHz processor.
    Jun 30 07:45:48 vkodb kernel: Built 1 zonelists. Total pages: 1536000
    Jun 30 07:45:48 vkodb kernel: Kernel command line: root=xvda1 ro
    Jun 30 07:45:48 vkodb kernel: Enabling fast FPU save and restore... done.
    Jun 30 07:45:48 vkodb kernel: Enabling unmasked SIMD FPU exception support... done.
    Jun 30 07:45:48 vkodb kernel: Initializing CPU#0
    Jun 30 07:45:48 vkodb kernel: PID hash table entries: 4096 (order: 12, 16384 bytes)
    Jun 30 07:45:48 vkodb kernel: Xen reported: 2500.088 MHz processor.
    Jun 30 07:45:48 vkodb kernel: Console: colour dummy device 80x25
    Jun 30 07:45:48 vkodb kernel: Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
    Jun 30 07:45:48 vkodb kernel: Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
    Jun 30 07:45:48 vkodb kernel: Software IO TLB disabled
    Jun 30 07:45:48 vkodb kernel: vmalloc area: ee000000-f51fe000, maxmem 2d7fe000
    Jun 30 07:45:48 vkodb kernel: Memory: 6084488k/6144000k available (1384k kernel code, 58604k reserved, 466k data, 112k init, 5398536k highmem)
    Jun 30 07:45:48 vkodb kernel: Checking if this processor honours the WP bit even in supervisor mode... Ok.
    Jun 30 07:45:48 vkodb kernel: Calibrating delay using timer specific routine.. 5007.52 BogoMIPS (lpj=25037626)
    Jun 30 07:45:48 vkodb kernel: Security Framework v1.0.0 initialized
    Jun 30 07:45:48 vkodb kernel: SELinux: Disabled at boot.
    Jun 30 07:45:48 vkodb kernel: Capability LSM initialized
    Jun 30 07:45:48 vkodb kernel: Mount-cache hash table entries: 512
    Jun 30 07:45:48 vkodb kernel: CPU: After generic identify, caps: 1fc98b75 00000000 00000000 00000000 04080221 00000000 00000000
    Jun 30 07:45:48 vkodb kernel: CPU: After vendor identify, caps: 1fc98b75 00000000 00000000 00000000 04080221 00000000 00000000
    Jun 30 07:45:48 vkodb kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
    Jun 30 07:45:48 vkodb kernel: CPU: L2 cache: 6144K
    Jun 30 07:45:48 vkodb kernel: CPU: After all inits, caps: 1fc98b75 00000000 00000000 00000140 04080221 00000000 00000000
    Jun 30 07:45:48 vkodb kernel: Checking 'hlt' instruction... OK.
    Jun 30 07:45:48 vkodb kernel: SMP alternatives: switching to UP code
    Jun 30 07:45:48 vkodb kernel: Brought up 1 CPUs
    Jun 30 07:45:48 vkodb kernel: migration_cost=0
    Jun 30 07:45:48 vkodb kernel: NET: Registered protocol family 16
    Jun 30 07:45:48 vkodb kernel: SMP alternatives: switching to SMP code
    Jun 30 07:45:48 vkodb kernel: Initializing CPU#1
    Jun 30 07:45:48 vkodb kernel: CPU: After generic identify, caps: 1fc98b75 00000000 00000000 00000000 04080221 00000000 00000000
    Jun 30 07:45:48 vkodb kernel: CPU: After vendor identify, caps: 1fc98b75 00000000 00000000 00000000 04080221 00000000 00000000
    Jun 30 07:45:48 vkodb kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
    Jun 30 07:45:48 vkodb kernel: CPU: L2 cache: 6144K
    Jun 30 07:45:48 vkodb kernel: CPU: After all inits, caps: 1fc98b75 00000000 00000000 00000140 04080221 00000000 00000000
    Jun 30 07:45:48 vkodb kernel: migration_cost=303
    Jun 30 07:45:48 vkodb kernel: Brought up 2 CPUs
    Jun 30 07:45:48 vkodb kernel: xen_mem: Initialising balloon driver.
    Jun 30 07:45:48 vkodb kernel: NET: Registered protocol family 2
    Jun 30 07:45:48 vkodb kernel: IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
    Jun 30 07:45:48 vkodb kernel: TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
    Jun 30 07:45:48 vkodb kernel: TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
    Jun 30 07:45:48 vkodb kernel: TCP: Hash tables configured (established 131072 bind 65536)
    Jun 30 07:45:48 vkodb kernel: TCP reno registered
    Jun 30 07:45:48 vkodb kernel: audit: initializing netlink socket (disabled)
    Jun 30 07:45:48 vkodb kernel: audit(1277876739.621:1): initialized
    Jun 30 07:45:48 vkodb kernel: highmem bounce pool size: 64 pages
    Jun 30 07:45:48 vkodb kernel: VFS: Disk quotas dquot_6.5.1
    Jun 30 07:45:48 vkodb kernel: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
    Jun 30 07:45:48 vkodb kernel: Initializing Cryptographic API
    Jun 30 07:45:48 vkodb kernel: io scheduler noop registered
    Jun 30 07:45:48 vkodb kernel: io scheduler anticipatory registered (default)
    Jun 30 07:45:48 vkodb kernel: io scheduler deadline registered
    Jun 30 07:45:48 vkodb kernel: io scheduler cfq registered
    Jun 30 07:45:48 vkodb kernel: RAMDISK driver initialized: 16 RAM disks of 8192K size 1024 blocksize
    Jun 30 07:45:48 vkodb kernel: Xen virtual console successfully installed as tty1
    Jun 30 07:45:48 vkodb kernel: Event-channel device installed.
    Jun 30 07:45:48 vkodb kernel: netfront: Initialising virtual ethernet driver.
    Jun 30 07:45:48 vkodb kernel: xen-vbd: registered block device major 202
    Jun 30 07:45:48 vkodb kernel: blkfront: xvda: barriers enabled
    Jun 30 07:45:48 vkodb kernel: xvda: xvda1
    Jun 30 07:45:48 vkodb kernel: blkfront: xvdb: barriers enabled
    Jun 30 07:45:48 vkodb kernel: xvdb: xvdb1
    Jun 30 07:45:48 vkodb kernel: i8042.c: No controller found.
    Jun 30 07:45:48 vkodb kernel: TCP bic registered
    Jun 30 07:45:48 vkodb kernel: NET: Registered protocol family 1
    Jun 30 07:45:48 vkodb kernel: NET: Registered protocol family 17
    Jun 30 07:45:48 vkodb kernel: Using IPI No-Shortcut mode
    Jun 30 07:45:48 vkodb kernel: XENBUS: Device not ready: device/vbd/51760
    Jun 30 07:45:48 vkodb kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
    Jun 30 07:45:48 vkodb kernel: EXT3-fs: write access will be enabled during recovery.
    Jun 30 07:45:48 vkodb kernel: kjournald starting. Commit interval 5 seconds
    Jun 30 07:45:48 vkodb kernel: EXT3-fs: recovery complete.
    Jun 30 07:45:48 vkodb kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Jun 30 07:45:48 vkodb kernel: VFS: Mounted root (ext3 filesystem) readonly.
    Jun 30 07:45:48 vkodb kernel: Freeing unused kernel memory: 112k freed
    Jun 30 07:45:48 vkodb kernel: Adding 522072k swap on /dev/xvdb1. Priority:-1 extents:1 across:522072k
    Jun 30 07:45:48 vkodb kernel: EXT3 FS on xvda1, internal journal
    Jun 30 07:45:48 vkodb kernel: device-mapper: ioctl: 4.7.0-ioctl (2006-06-24) initialised: Jun 30 07:45:49 vkodb kernel: NET: Registered protocol family 10
    Jun 30 07:45:49 vkodb kernel: lo: Disabled Privacy Extensions
    Jun 30 07:45:49 vkodb kernel: IPv6 over IPv4 tunneling driver
    Jun 30 07:45:52 vkodb cron[2121]: (CRON) INFO (pidfile fd = 3)
    Jun 30 07:45:52 vkodb cron[2123]: (CRON) STARTUP (fork ok)
    Jun 30 07:45:52 vkodb cron[2123]: (CRON) INFO (Running jobs)
    Jun 30 07:46:00 vkodb kernel: eth0: no IPv6 routers present
    Jun 30 08:05:49 vkodb -- MARK --
    Jun 30 08:17:01 vkodb CRON[5071]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
    Jun 30 08:45:49 vkodb -- MARK --
    Jun 30 09:05:50 vkodb -- MARK --
    Jun 30 09:07:01 vkodb CRON[7071]: (root) CMD (cron/tempdelete.pl #Delete Webmin temporary files)
    Jun 30 09:17:01 vkodb CRON[7566]: (root) CMD ( cd / && run-parts --report cron.hourly)
    Jun 30 09:45:51 vkodb -- MARK --
    After this, our database was renamed to something like gber29062010.GDB and no client was therefore able to connect.

    Running Linux debian Lenny. How do I protect my database against corruption due, logrotation, planned system recoveries, filesystems checks. Please advise.

    On our database: read, write and execute settings are enabled for every user.

    Regards
    Robert
    B.t.w.: It really sucks when you want to post a logfragment and your are not allowed (yet) to post a post with URLīs

  2. #2
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,380
    What kind of database is this?

    What I can see is a regular boot process.
    A filesystem was not unmounted cleanly, so the machine was shutdown hard before for whatever reason.
    You might want to know that reason (someone tripping over powercable, broken hardware, etc) and reassure a safe environment again.

    ext3 restored the last good commit from its journal.
    The problem: ext3 can of course only judge from file IO actions, if a filesystem is consistent or not.

    That is *not* neccessarily eqal to what a database (or any other app for that matter) declares consistent.

    So, if things go well, you might find a tool from the dbīs package to verify and restore db consistency of that file.

    If not..
    .. you will need the last db dump and restore from there.


    Best practices for DBs start with live replication to another machine *combined* with daily db dumps that then are written to external storage (tapes, etc) by a backup system.


    btw,
    I think the CODE tags will provide nice scrollbars, so it is possible to post longer texts
    You must always face the curtain with a bow.

  3. #3
    Just Joined!
    Join Date
    Mar 2010
    Posts
    9
    Quote Originally Posted by Irithori View Post
    What kind of database is this?

    What I can see is a regular boot process.
    A filesystem was not unmounted cleanly, so the machine was shutdown hard before for whatever reason.
    You might want to know that reason (someone tripping over powercable, broken hardware, etc) and reassure a safe environment again.

    ext3 restored the last good commit from its journal.
    The problem: ext3 can of course only judge from file IO actions, if a filesystem is consistent or not.

    That is *not* necessarily eqal to what a database (or any other app for that matter) declares consistent.

    So, if things go well, you might find a tool from the dbīs package to verify and restore db consistency of that file.

    If not..
    .. you will need the last db dump and restore from there.


    Best practices for DBs start with live replication to another machine *combined* with daily db dumps that then are written to external storage (tapes, etc) by a backup system.


    btw,
    I think the CODE tags will provide nice scrollbars, so it is possible to post longer texts
    Thanks for your reply.
    De database is firebird.
    It appears that the whole xen server with 3 VM was booted at 7:45...
    Maybe a bug in Xen server?
    its complaning about some kink of add-on during te reboot.
    I think I will update my xenserver and wait for the next malfunction.
    Maybe a diagnostic live CD for analysing the hardware.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •