PCI hotplug resource allocation problem
I'm still kind of new to the linux kernel, but I am trying to port an application from VxWorks to Linux. The application interfaces with hotpluggable CompactPCI peripheral cards. The problem I am encountering is with resoruce allocation at the time of hot insertion when no resources had previously been set up by the BIOS.
But first, some background and details:
The system controller that is running linux is the ADLINK cPCI6910 using a PLX6540 PCI-to-PCI bridge as the bridge to the compactPCI bus. I use this as the example here but the app/driver will need to work on other boards as well. The compactPCI bus itself is divided into two segments. The first segment (bus 5 with the 6910) holds up to 3 periperal cards and the bridge to the second segment. The bridge between the segments is a Texas Instruments PCI2050 PCI-to-PCI bridge. The second segment (bus 6) will hold up to 7 peripheral cards.
The peripheral device uses the PLX 9054 PCI chip which initializes with defaults. The (custom) driver handling the device shows up in dmesg as cdcdrv.
For reference, we made this work in VxWorks in a very manual fashion by reserving an address range in sysHwInit using sysMmuMapAdd and flag VM_STATE_FOR_PCI. After the app starts, that window is opened on all the bridges between bus 0 and the compactPCI bus, and then half of that range on the TI bridge between the two cPCI segments (so the peripherals on the upstream side of the TI bridge do not get configured with addresses which the bridge also has in its window). BAR registers are then set with nailup addresses in that range as cards are detected on a periodic bus poll.
The hotplug module being used on the linux system in conjuction with cpci-hotplug is similar to cpcihp-generic but instead of polling the state of ENUM# in an IO port it just reads the HS_CSR register at each known bus/dev location where the peripheral cards can appear and checks if the insertion or extraction bits are set and returns 1 if they are (without clearing them). The hotplug module shows up as abshp in dmesg.
Both the hotplug module and the cdcdrv module are loaded with insmod after the system is running. The system itself is running OpenSUSE 12.1 (Kernel 3.1.0). The kernel has not been specially recompiled, however I am passing the parameter memmap=16m$2032m at boot in order to reserve a segment of RAM for some of the driver functionality.
Upon insertion, the hotplug module detects the card and linux attempts to allocate resources requested by the BARs. The dmesg output is attached. What appears to happen is that it begins with BAR2, requesting 1MB, and configures it. However, the size of the mem window on the bridges is 1MB, so when it attempts to do any other allocations it just says "can't allocate mem". The driver then fails to initialize the device because the resources have not been allocated.
Attached is an archive with the output of dmesg, /proc/iomem, and lspci -vvv before and after hotplugging as well as the same information before and after driver insertion on a system where the card was present at boot (no hotplugging). (Tried linking to individual text files on google sites but forum told me I didn't post enough to do that).
So finally the actual questions... Is Linux supposed to handle this? It seems like hotplug logic isn't terribly useful if it is incapable of resizing the memory windows on the bridges. If Linux should be able to do that, what might be going wrong here?
If Linux cannot resize the windows automagically, is it possible to reserve address space at boot for PCI MMIO later on? I assume something special would have to happen to it in order for it to be enabled in the MMU for PCI.