Reloading Module Stops MSI Interrupts

[misterhw]

I'm running a 2.6.30.9 kernel on a SuperMicro/Intel chipset, and I'm writing a module for a PCIe device that uses MSI. The first time I load the module using insmod, interrupts work correctly, and the ISR is run as it should. When I do a rmmod and another insmod, interrupts are not caught by my module.

I've included the code from the init and exit functions below, but I have followed the directions in the MSI HOW-TO.txt file to call pci_enable_msi then request_irq in the init function. In the exit function, I call free_irq, then pci_disable_msi.

When interrupts are working (first insmod after reboot), /proc/interrupts shows the following after running a test program:

53:      33289     566723          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      RMA Interrupt

When interrupts aren't working after reloading the module, /proc/interrupts show the following after running the same test program:

53:      0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      RMA Interrupt

I know that the problem isn't with the device because a reboot will fix the problem without restarting the device. Additionally, with a signal tap on the FPGA, the IRQ request signal remains high for the time when the first interrupt should come. The acknowledge signal isn't coming back to the device because the module isn't receiving the interrupts and going through the ISR.

Both when interrupts are working, and when they aren't, lspci -vv show the following:

05:00.0 Class ff00: Altera Corporation Unknown device 0004 (rev 01)
        Subsystem: Altera Corporation Unknown device 0004
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 53
        Region 0: Memory at 680000000 (64-bit, prefetchable) [size=1024M]
        Capabilities: [50] Message Signalled Interrupts: 64bit+ Queue=0/2 Enable+
                Address: 00000000fee00000  Data: 40ba
        Capabilities: [68] MSI-X: Enable- Mask- TabSize=2
                Vector table: BAR=0 offset=00000400
                PBA: BAR=0 offset=00000500
        Capabilities: [78] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [80] Express Endpoint IRQ 0
                Device: Supported: MaxPayload 2048 bytes, PhantFunc 0, ExtTag-
                Device: Latency L0s <64ns, L1 <1us
                Device: AtnBtn- AtnInd- PwrInd-
                Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
                Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                Device: MaxPayload 256 bytes, MaxReadReq 512 bytes
                Link: Supported Speed unknown, Width x4, ASPM L0s, Port 1
                Link: Latency L0s unlimited, L1 unlimited
                Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
                Link: Speed unknown, Width x4

Here is my source code for the init and exit functions for loading and unloading the module:

static int __init ssrma_init(void){
  int retval;
  ssrma_DEV_T *rma_dev;
  irq_handler_t irq_handler = &ssrma_pciIsr;
  rma_dev = &install_device;

  /* Register the driver with the PCI core */
  retval = pci_register_driver(&pci_driver);
  if(retval) return retval;

  /* Setup the cdev portion of the rma_device struct */
  retval = rma_setup_cdev(rma_dev);
  if(retval){
    goto registered;
  }

  /* Verify the device is found on a bus */
  rma_dev->pci_dev = pci_get_device(ssrma_VENDOR_ID, ssrma_DEVICE_ID, NULL);
  if(!rma_dev->pci_dev){
        retval = -ENODEV; // No such device
    goto cleanup;
  }
  /* Add the device to the device structure */
  pci_dev_put(rma_dev->pci_dev);

  /* Enable the MSI interrupts */
  if(pci_enable_msi(rma_dev->pci_dev)){
    retval = -1;
    goto cleanup;
  }
 /* Setup the interrupt ISR */
  if(request_irq(rma_dev->pci_dev->irq, irq_handler, IRQF_DISABLED, "RMA Interrupt", rma_dev)){
       retval = -1;
    goto msi_cleanup;
  }

  return 0;

  ioremap_cleanup:
    /* Unmap the mapped memory space */
    iounmap(rma_dev->virt_start_addr);
    /* Free the requested IRQ */
    free_irq(rma_dev->pci_dev->irq, rma_dev);
  
msi_cleanup:
    /* Disable the MSI interrupts */
    pci_disable_msi(rma_dev->pci_dev);
  
cleanup:
    /* Remove the device */
    cdev_del(&rma_dev->cdev);
 
 registered:
    /* Unregister the driver */
    pci_unregister_driver(&pci_driver);
  return retval;
} /* end rma_init */



static void __exit ssrma_exit(void){
  ssrma_DEV_T *rma_dev;
  rma_dev = &install_device;

  /* Unmap the mapped space */
  iounmap(rma_dev->virt_start_addr);

  /* Free the requested IRQ */
  free_irq(rma_dev->pci_dev->irq, rma_dev);

  /* Disable the MSI interrupts */
  pci_disable_msi(rma_dev->pci_dev);

  /* Remove the device */
  cdev_del(&rma_dev->cdev);

  /* Unregister the driver */
  pci_unregister_driver(&pci_driver);

} /* end rma_exit */

If you have had a similar problem, or know what I'm doing wrong, please let me know. If you need more information, I would be happy to provide it. It is frustrating to have to reboot everytime I change the module code. Thank you.

[ths]

Hi,

I have the same problem here that I don't get any MSI interrupts after reloading my driver module.
Did you find any solution for this problem or anyone else?

Thank you

[ths]

I found my problem here:
Calling pci_disable_device removes the busmaster flag from the PCI device.
But pci_enable_device doesn't set it again when loading the module.
And for MSI the busmaster flag is needed because the interrupt is a data package
over the data bus, not an IRQ line.

ths

[hentolinux]

Hi,

I am on my way to implement MSI:
On the Hardware side all it's ok but now I am looking for a guidance (piece of code)
on testing MSI Capabilities on the Software side(Driver)
For that I would appreciate any help to move forward

Regards,