Results 1 to 2 of 2
Enjoy an ad free experience by logging in. Not a member yet? Register.
- Join Date
- Aug 2008
PCIe Driver - need high performance
We are developing a spin-off of one of our PCI-X devices which runs on custom hardware under VxWorks. Our new device will be a PCIe device that will plug into a Dell server (1950 and the likes) which runs a 2.6 Linux Kernel flavor (Possibly, Red hat enterprise 5.0). The main job of this PCIe device is to push externally gathered data at a high-rate into memory for an application to consume. The rates reach over 2 Gbps, so avoiding memory copies is desired. Our PCIe device is a bus master which DMAs data into a 256 MB circular buffer. When this device receives external data, it pushes this data onto the circular buffer and then updates a write pointer to indicate data is ready to read.
I have read Linux Device Drivers, third edition, but Iím stuck on one sticky point. How can I map a huge user space buffer (256Mb, or higher) into kernel space such that a PCIe device can freely write to it with out the need for constant mitigation by a device driver (marking memory as Dirty, etc). Do I even need to map this user space buffer into kernel space, or can I just pin this buffer in memory, translate the virtual address to a physical addresses and tell our PCIe device this physical address it can write to.
I was thinking of developing a char driver, which initializes our PCIe device and implements mmap(). We want to write a user space program which controls and configures this device. Part of the configuration is to write into its registers the base address of the 256MB buffer and the address of the write pointer. We would like to allocate this 256MB in user space, lock it into memory such that is can not get swapped, translate these addresses into physical addresses that our PCIe device can write freely to them. We want to lock this buffer into physical memory for the lifetime of the driver/application; its unlikely the application/driver will ever get unloaded, the server will have at least 4GB of RAM. We also want to use cache, but are unsure if the hardware supports cache coherency or if there is a way to tell the kernel to uphold cache coherecny for this buffer.
Our kernel space driver will basically map our PCIe device registers into user space, where a user space driver/application then configures this device to transfer all external data received into a user space buffer. Our user space application will then consume and work on the data in the buffer.
Any help is appreciated, Iím quite new to Linux kernel hacking.
- Join Date
- Aug 2008
I tracked down a linux expert, here is the answer:
Normally, you would allocate memory with something like
pci_alloc_consistent() or dma_alloc_coherent() when you want a PCI
device to be able to DMA into it. You are correct that using these (or
even kmalloc()) for any largish size will likely fail (due to inevitable
Allocating the buffer from userspace won't really work, because you
won't have any way to get a chunk of _physically contiguous_ memory from
there. It would be the same as using something like vmalloc() in the
kernel -- virtually contiguous but physically fragmented.
Unfortunately there is no clean solution to this in the standard kernel
that I know of. I think this type of situation is fairly unique --
usually drivers/devices will have smaller DMA regions or utilize a
scatter/gather streaming DMA method.
In order to get a large chunk of physically contiguous memory, you'll
have to reserve it during boot time. One solution is to use the
"bigphysarea" patch that's been floating around, which is intended for
exactly this situation. You can find fairly recent forward-ports of
this patch here: