boot stalls with assertion 'j->installed' failed at src/manager.c:1205

[jez_noake]

To the point:
F15 2.6.40.3-0.f15.i686.PAE (from grub) running on a Lattitude D620
Encrypted lvm
The last things I did before restarting, was to install ext2fuse, VMWare Player and then yum update - installed 3 updates.
I can mount (on another F14 machine) both the boot partition and the LVM \, fsck -f does not find anything on /boot or /.

Escaping the plymouth screen during boot, I see:

...
Starting Initialize storage subsystems (RAID, LVM, etc.)...
Starting Huge Pages File System...
Started Huge Pages File System.
Started Initialize storage subsystems (RAID, LVM, etc.).
(then there is a pause of about 30 seconds - no disk activity)
[ 99.595479] systemd[1]: Assertion 'j->installed' failed at src/manager.c:1205, function transaction_apply(). Aborting.
Starting /boot aborted because a dependency failed.
[ 99.614373] system[1]: Caught <ABRT>, dumped core as pid 881.
Starting Mark the need to relabel after reboot aborted because a dependency failed.
Starting Relabel all filesystems, if necessary aborted because a dependency failed.
Starting Cryptography Setup for luks-76bb572e-e697-4332-8b4b-1b34e0af8474 aborted because a dependency failed.
[ 123.662581] systemd-cgroups-agent[882]: Failed to get D-Bus connection: Failed to connect to socket /org/freedesktop/systemd1/private: Connection refused

... and there it stops.


google searches yield limited suggestions, there was a hint that fstab might have a problem, but I have checked that against a working OS with a very similar config and all is well.

I posted this on Fedora forums, but there has been no reply.

I can usually sort stuff out just by reading others posts here, but this one has got me stumped.

Thanks for reading.

[Rubberman]

How did you build/install this kernel? Also, what errors/warnings did you get when you built it?

[jez_noake]

Hey, Rubberman.

The OS was installed from a downloaded F15 DVD and then updated almost entirely with further yum packages. There are just a couple of RPMs that were not from yum and then ext2fuse was compiled and 'make installed' and then VMWare Player was downloaded and the install run from their binary.

I am not aware that I have specifically compiled the kernel at any point.
I had some trouble compiling the ext2fuse (I think this is stand-alone though and should not come into play during boot), did not have a compiler installed at first, and then was missing some libraries, but finally succeeded without error.

Never had any warnings about kernel problems.

I had a bit of bother configuring for NVidia display (disabling nouveau, installing akmod-nvidia and changing the path through the glx libraries so it finds the nvidia ones before the xorg ones) other than that it has been well stable up to this point.

[jez_noake]

I Found this while looking for files that changed on the day that the problem started:
in /var/log/vmware-installer

make[1]: Entering directory `/usr/src/kernels/2.6.40.3-0.fc15.i686.PAE'
  CC [M]  /tmp/vmware-root/modules/vmmon-only/linux/driverLog.o
  CC [M]  /tmp/vmware-root/modules/vmmon-only/linux/iommu.o
make[1]: Leaving directory `/usr/src/kernels/2.6.40.3-0.fc15.i686.PAE'
make: Leaving directory `/tmp/vmware-root/modules/vmmon-only'

[2011-08-28 00:16:32,529] Using 2.6.x kernel build system.
/tmp/vmware-root/modules/vmmon-only/linux/iommu.c:47:44: error: â~@~XSPIN_LOCK_UNLOCKEDâ~@~Y undeclared here (not in a function)
/tmp/vmware-root/modules/vmmon-only/linux/iommu.c: In function â~@~XIOMMU_SetupMMUâ~@~Y:
/tmp/vmware-root/modules/vmmon-only/linux/iommu.c:103:28: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
/tmp/vmware-root/modules/vmmon-only/linux/iommu.c:113:17: warning: format â~@~X%lxâ~@~Y expects argument of type â~@~Xlong unsigned intâ~@~Y, but argument 3 has type â~@~Xlong long unsigned intâ~@~Y [-Wformat]
/tmp/vmware-root/modules/vmmon-only/linux/iommu.c:117:35: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
/tmp/vmware-root/modules/vmmon-only/linux/iommu.c:120:20: warning: format â~@~X%luâ~@~Y expects argument of type â~@~Xlong unsigned intâ~@~Y, but argument 3 has type â~@~Xuint64â~@~Y [-Wformat]
make[2]: *** [/tmp/vmware-root/modules/vmmon-only/linux/iommu.o] Error 1
make[1]: *** [_module_/tmp/vmware-root/modules/vmmon-only] Error 2
make: *** [vmmon.ko] Error 2
Unable to install vmmon

[2011-08-28 00:16:32,532] Unable to install kernel modules

Is this significant?

[Rubberman]

So, since ext2 file system support is native to Linux, why did you install the ext2fuse (user space) driver? As for disabling nouveau to install the native nVidia drivers, yes that is a PITA. Also, I have had better luck with installing the nVidia drivers downloaded directly from nVidia than I have had installing them from the Yum repositories.

So, my suggestions are these:

1. Uninstall the ext2fuse driver unless there is some pressing need for it.
2. Download latest nVidia driver for your card and system from nVidia - it is a run-able shell script.
3. Go to runlevel 3 (modify /etc/inittab and reboot) and uninstall the yum packaged nVidia driver, then reboot back into runlevel 3
4. Install the downloaded nVidia driver. It will also properly install the nVidia-specific OpenGL libraries for you.
5. Switch back to runlevel 5 (restore /etc/inittab and reboot) and see how things go now.

Note that you may need to uninstall/reboot/reinstall the VMware packages. I've tested VMware on my systems in the past, but have stuck with VirtualBox for my VM needs. It has worked well for me, even though I am generally unhappy with Oracle these days - for other reasons.

[jez_noake]

Thanks for your response Rubberman,
ext2fuse was installed to take advantage of the 16k blocksize capability for retrieving lost data from a Netgear ReadyNAS Duo disk:

h-t-t-p://w-w-w.readynas.com/forum/viewtopic.php?f=25&t=24861&hilit=16+blocksize&star t=15
(sorry, forum blocks links!)

(this problem was solved by installing vmware player on a windows machine and the data has now been recovered)

I take your point about NVidia, but I got it to work well and I dont have to re-install it each time there is a kernel update, so no worries there.

I agree on VirtualBox - my preferred option, however, Netgear retrieval tools were pre-installed in a VMWare image so I was heading for that when it all went pear shaped.

My real problem is that I cannot boot at all, I can't get to runlevel 3 ( by adding init 3 on the grub loader line) and so cannot install or uninstall anything.
The only access I can get to the filesystem is mounting it on another machine and I have no idea how to uninstall packages from filesystem mounted in this way.
I have tried removing the init.d scripts for VMWare player from each of the rc(x).d folders.
The scripts that are left seem reasonable compared t the F14 machine that the files system is mounted on.

Any ideas on how to get past this boot failure would be very welcome

[Rubberman]

Can you boot with a live CD/DVD and mount/access the boot logs on the base operating system?

[jez_noake]

Yep,
although it is easier and faster to use the F14 (full install on external USB hard disk)
I can see all the logs, the only one which changes modification date is boot.log, since it fails very early in the boot process.

The end of the boot.log is included in the first post.

Last night I tried booting with the install DVD and invoked the recovery option.
I was able to chroot the filesystem and therefore could use the likes of yum or RPM to install/uninstall stuff. Unfortunately VMWare player is not an RPM and does not show up as a removable item.

I will have to see if VMWare Player has a proper uninstall method.
I still don't know where to start the investigation. I am just assuming that VMware is the culprit because it was one of last things to change.

[jez_noake]

I booted using the DVD, started recovery mode, chroot the filesystem and ran the vmware-install -u vmware-player

This uninstalled VMWareplayer
Rebooted and the problem is still there - so it is not VMWare, something else.

Here is the latest boot.log:

luks-76bb572e-e967-4332-8b4b-1b34e0af8474

Failed to load SELinux policy.



Welcome to Fedora release 15 (Lovelock)!



Starting Bootup hack...

Started Load legacy module configuration.

Started Load Kernel Modules.

Starting Set Up Additional Binary Formats...

Starting Apply Kernel Variables...

Starting Syslog Kernel Log Buffer Bridge...

Started Syslog Kernel Log Buffer Bridge.

Started File System Check on Root Device.

Started Lock Directory.

Started Runtime Directory.

Starting Remount Root FS...

Starting Remount API VFS...

Starting Media Directory...

Starting Setup Virtual Console...

Starting Apply System Clock UTC Offset...

Started Bootup hack.

Started Apply Kernel Variables.

Started Remount Root FS.

Started Remount API VFS.

Started Media Directory.

Started Apply System Clock UTC Offset.

Starting udev Kernel Device Manager...

Starting Configure read-only root support...

Starting Stdio Syslog Bridge...

Started Stdio Syslog Bridge.

Starting Arbitrary Executable File Formats File System...

Started Arbitrary Executable File Formats File System.

Started Set Up Additional Binary Formats.

Started udev Kernel Device Manager.

Starting udev Coldplug all Devices...

Started Setup Virtual Console.

Started udev Coldplug all Devices.

Starting udev Wait for Complete Device Initialization...

Started Configure read-only root support.

Starting /dev/mapper/vg_jezd620linux-LogVol00...

Started /dev/mapper/vg_jezd620linux-LogVol00.

Started udev Wait for Complete Device Initialization.

Starting Wait for storage scan...

Started Show Plymouth Boot Screen.

Started Wait for storage scan.

Starting Initialize storage subsystems (RAID, LVM, etc.)...

Starting Huge Pages File System...

Started Huge Pages File System.

Starting POSIX Message Queue File System...

Started POSIX Message Queue File System.

Started Initialize storage subsystems (RAID, LVM, etc.).

Starting Show Plymouth Reboot Screen...

Stopped /dev/disk/by-id/dm-uuid-LVM-xZYE72QF1oYro4uEOKzsyA3U4VOtBZReMKM9YmpWgykAmf3FkDZBUlkJLjZo7s2L.

Stopped /dev/disk/by-id/dm-name-vg_jezd620linux-LogVol00.

Stopped /dev/vg_jezd620linux/LogVol00.

Stopped Reconfigure the system on administrator request.

Stopped Restore Sound Card State.

Stopping Configure read-only root support...

Stopped Configure read-only root support.

Stopped LSB: Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.

Stopped LSB: processor frequency scaling support.

Stopped Rescue Shell.

Stopping Bootup hack...

Stopped Bootup hack.

Starting Console System Startup Logging...

Stopped Relabel all filesystems, if necessary.

Stopped Mark the need to relabel after reboot.

Stopping Apply Kernel Variables...

Stopped Apply Kernel Variables.

Stopping Setup Virtual Console...

Stopped Setup Virtual Console.

Stopping Set Up Additional Binary Formats...

Stopped Set Up Additional Binary Formats.

Stopped /dev/mapper/vg_jezd620linux-LogVol00.

Stopped Enable all detected swap partitions.

Stopping Apply System Clock UTC Offset...

Stopped Apply System Clock UTC Offset.

Starting Update RTC With System Clock...

Starting Store Sound Card State...

Stopped Load Random Seed.

Starting Tell Plymouth To Write Out Runtime Data...

Starting Recreate Volatile Files and Directories...

Stopped /boot.

Stopped Initialize storage subsystems (RAID, LVM, etc.).

Stopped Cryptography Setup for luks-76bb572e-e967-4332-8b4b-1b34e0af8474.

Stopping Remount Root FS...

Stopped Remount Root FS.

Stopping Initialize storage subsystems (RAID, LVM, etc.)...

Stopped Initialize storage subsystems (RAID, LVM, etc.).

Stopping Wait for storage scan...

Stopped Wait for storage scan.

Stopping Remount API VFS...

Stopped Remount API VFS.

Stopping Media Directory...

Starting Save Random Seed...

Starting Show Plymouth Reboot Screen failed, see 'systemctl status plymouth-reboot.service' for details.

Started Console System Startup Logging.

Started Store Sound Card State.

Started Recreate Volatile Files and Directories.

Stopped Media Directory.

Started Save Random Seed.

Starting Notify Audit System and Update UTMP about System Runlevel Changes...

Starting Console System Reboot Logging...

Started Console System Reboot Logging.

Started Notify Audit System and Update UTMP about System Runlevel Changes.

Starting Notify Audit System and Update UTMP about System Shutdown...

Started Notify Audit System and Update UTMP about System Shutdown.

Started Tell Plymouth To Write Out Runtime Data.

Started Update RTC With System Clock.

Starting File System Check on /dev/disk/by-uuid/3c7d3c93-727f-4487-9391-db82c84bdd45 aborted because a dependency failed.

[jez_noake]

These are the last yum updates in yum.log before the fatal reboot.

Aug 25 20:59:51 Installed: libstdc++-devel-4.6.0-10.fc15.i686
Aug 25 20:59:52 Installed: gcc-c++-4.6.0-10.fc15.i686
Aug 25 23:15:30 Installed: fuse-devel-2.8.5-5.fc15.i686
Aug 27 22:30:10 Installed: automake16-1.6.3-18.fc13.1.noarch
Aug 27 22:30:10 Installed: automake14-1.4p6-21.fc15.noarch
Aug 27 22:30:10 Installed: automake17-1.7.9-14.fc15.1.noarch
Aug 27 22:30:11 Installed: automake15-1.5-29.fc13.1.noarch
Aug 28 00:29:20 Updated: 12:dhcp-libs-4.2.1-10.P1.fc15.i686
Aug 28 00:29:20 Updated: 12:dhcp-common-4.2.1-10.P1.fc15.i686
Aug 28 00:29:20 Updated: xz-libs-5.0.3-1.fc15.i686
Aug 28 00:29:21 Updated: ksysguardd-4.6.5-5.fc15.i686
Aug 28 00:29:23 Updated: kdebase-workspace-libs-4.6.5-5.fc15.i686
Aug 28 00:29:30 Updated: kdebase-workspace-4.6.5-5.fc15.i686
Aug 28 00:29:32 Updated: 1:cups-libs-1.4.8-2.fc15.i686
Aug 28 00:29:33 Updated: plasma-scriptengine-python-4.6.5-5.fc15.i686
Aug 28 00:29:34 Updated: 12:dhclient-4.2.1-10.P1.fc15.i686
Aug 28 00:29:37 Updated: yum-3.2.29-9.fc15.noarch
Aug 28 00:29:40 Updated: 1:cups-1.4.8-2.fc15.i686
Aug 28 00:29:42 Updated: kdm-4.6.5-5.fc15.i686
Aug 28 00:29:42 Updated: xz-5.0.3-1.fc15.i686
Aug 28 00:29:44 Updated: flash-plugin-10.3.183.7-release.i386
Aug 28 00:29:45 Updated: libmx-1.2.1-1.fc15.i686
Aug 28 00:29:46 Updated: libgtop2-2.28.3-2.fc15.i686

Any of these significant?