I have written a PCI driver for my device. The device is an FPGA which is configured with a design that allows it to have direct access to memory of a host computer to which the fpga board is connected. My device driver is responsible for translating virtual addresses to physical addresses and sending these to the FPGA so the DMA unit in the FPGA can directly access pages in memory.

I have a C program which uses the driver to open this PCI device and then sends a command to the device, so it can begin accessing memory(for some calculations done in the fpga). This is done via IOCTL. The FPGA sends an interrupt when its done. However my C program does not wait for the FPGA but instead resumes execution immmediately after the IOCTL call returns. If the FPGA has a virtual address which it needs to translate, it interrupts my PCI driver and I want to translate this address in the interrupt handler and write back the translated address to the device using memory mapped I/O.

The issue is that when my driver gets the virtual address, it attempts to lock the user space pages first(those of the running C program) in memory using get_user_pages() and then translates the address using pci_map_page(). Pinning the pages in memory is needed as the fpga will access them later using DMA. Here is the code I use :

//get page
result = get_user_pages(htex_dev->tsk, htex_dev->tsk->mm, virt_addr, 1, 1, 0, &page, NULL);
if (result <= 0)
ERROR_MSG("unable to get page\n");
return 0;
//get buss address of page
translated = pci_map_page(htex_dev->dev, page, 0, PAGE_SIZE, DMA_BIDIRECTIONAL);

update_htx_tlb(index, tag, translated, htex_dev);

However the call fails with -14 (bad address) everytime. It appears that perhaps the running C program's process pages cannot be accessed by the ISR to lock them in memory.

Yet I need to allow the C program to run while the driver services interrupts in the background. This is for performance reasons as halting the program would waste time.

If I do put the C program to sleep when it makes the IOCTL call using wait_event_interruptible() :

static int htex_ioctl(struct inode *inode, struct file *filp, unsigned int cmd, unsigned long arg)
struct htex_dev_t *htex_dev = filp->private_data;
int result;
switch (cmd)
//reset the tlb and ccu
iowrite32(HTEX_RESET_C, htex_dev->bar2->bar);
*(((u64 *)htex_dev->bar2->bar) + 1) = arg;
u32 temp;
temp = ioread32(htex_dev->bar2->bar+16);
iowrite32(HTEX_SET_C, htex_dev->bar2->bar);
result = wait_event_interruptible(wait_queue, htex_dev->done != 0); // <------------------

then it does work, but this is probably due to the fact that the process is sleeping and its pages can be locked in memory.

So how should I be doing the address translation with the running user space process?