Experimental Physics and
Industrial Control System

<[email protected]> · Fri, 23 Sep 2016 07:19:50 +0000

From: [email protected] [mailto:tech-talk-
> This all works well, and I have mmap() access to registers on the FPGA for
> low speed applications.  Now I need to send megabytes of data from the
> FPGA into EPICS PVs, as quickly as I can.  I have a FIFO-based solution that
> does around ~100 kB/s (as long as EPICS is quick enough to keep the FIFO
> from filling up).  But this isn't nearly fast or robust enough for some
> applications we'd like to support.  I played with the AXI DMA, but I don't
> know how to tell Linux to allocate physical memory that the DMA can write
> to.  (Dan Paskvan mentioned that you were using the MIG to talk to
> memory.  Is this better or easier than DMA?)

Driver support for physical memory requires a bit of a dance, but it's not too hard.  I follow this recipe for my drivers, and it works well:

1. Use 

	void *buf = (void *) __get_free_pages(GFP_KERNEL, N);

 to allocate 2^N pages of contiguous physical memory.  This returns an unsigned long, but you'll want to immediately cast to to a void* -- this is the pointer to the memory in kernel space.

2. Create a "DMA mapping" for this block of pages:

	size_t buf_size = 1 << (N+PAGE_SHIFT);
	dma_addr_t dma_addr = dma_map_single(dev, pages, buf_size, DMA_FROM_DEVICE)

This returns the physical address of the allocated buffer.

3. Hand the dma physical address over to the device (write dma_addr to a device register).

Now you're neally ready to roll, but you need to watch your step with cache management.  The following recipe seems to work.  For each transfer from hardware to memory follow this procedure:

1. Call

	dma_sync_single_for_device(dev, dma_addr, buf_size, DMA_FROM_DEVICE);

I'm not sure if this call is necessary.

2. Trigger DMA transfer from your device, wait until all transfer is complete before moving to the next step.

3. Call

	dma_sync_single_for_cpu(dev, dma_addr, buf_size, DMA_FROM_DEVICE);

This is the crucial call.  Without this when you come to read from buf you'll risk getting stale data.

4. Copy your data from the buffer, for example with

	copy_to_user(user_buf, buf + offset, copy_count);

This will be in the .read method of your driver.

If you want to skip the extra copy of step 4, the procedure is *much* harder, and I've never done it.  I think you either have to figure out how to hand a contiguous block of pages over to user space, or else support scatter gather and mapping of user allocated pages.  Don't bother.

-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System