[Click] User-level polling driver

[Beyers Cronje]

Hi all,

To anyone interested, the attached UDevE1000 element implements a userlevel
polling element for Intel E1000 adapters. It uses the LibE1000 user-space
library to receive and transmit packets from/to Intel E1000 adapters.
LibE1000 implements "OS bypass" mechanism, where fast path processing and
RX/TX data transfers are done without kernel assistance (i.e. no system
calls). See http://sourceforge.net/projects/libe1000/ for more information
on LibE1000.

UDevE1000 manipulates RX and TX descriptors directly through the LibE1000
API. Data is transfered via DMA, which is mapped only once and therefor does
not require expensive remapping for each transfer, all provided through
LibE1000 library. This results in great performance and throughput in
userlevel Click. My informal test results indicates that UDevE1000
outperforms Click's userlevel FromDevice/ToDevice (using either PCAP or
packet socket). Using a simple configuration I was able to consistently push
receive throughput to over 900k pps, without changing Click process
priorities/nice level nor using click-devirtualize. I think full line-rate
performance is not out of the question given the right hardware.

UDevE1000 can run in two modes:
(1) Polling mode where interrupts are disabled at the adapter and UDevE1000
will continuously poll for packets. This mode operates similar to
PollDevice.
(2) Interrupt mode where interrupts are enable at the adapter and UDevE1000
will receive interrupt notification via select(). This mode operates similar
to FromDevice()

UDevE1000 does not copy packet data into Click packet structures, instead it
uses LibE1000 pre-allocated packet buffers, similar to how PollDevice uses
kernel socket buffers for Click packet data. Unfortunately this requires a
minor change to Click's Packet structure, see attached udeve1000-click.patch

Have a look at udeve1000.hh for Click configuration arguments and examples.

Installation steps:

1) Download and unpack LibE1000 library from
http://sourceforge.net/projects/libe1000/  This version of UDevE1000 was
tested on LibE1000 3.0

2) Apply udeve1000-libe1000.patch to LibE1000, ie change into
libe1000-3.0dir and apply patch with 'patch -p1 -b <
udeve1000-libe1000.patch'. This patch is needed for now to fix a bug where
Click select() does not always receive the adapter interrupt when operating
in interrupt mode. If you plan to only use UDevE1000 in polling mode then
this patch is optional.

3) Install and configure LibE1000 library. Follow the installation and
configuration instructions in the README file of LibE1000. This version of
UDevE1000 was tested on LibE1000 3.0 and Linux 2.6.24 using the LibE1000
bootmem-patch. It is highly recommended to apply and use the bootmem-patch
supplied by LibE1000 to allocate buffers at bootup, see LibE1000 README for
more details.

4) IMPORTANT - Make sure you have a working installation of LibE1000 by
running the rx test found in libe1000-3.0/tests.

5) Get a clean git version of Click

6) Apply UDevE1000 Click patch to extend Click's Packet structure to enable
support for LibE1000 packet buffers. Ie change into click dir and issue
'patch -p1 -b < udeve1000-click.patch'

7) Copy udeve1000.{cc,hh} and fifoqueue.{cc,hh} files to click/userlevel, or
click/local

8) Configure Click with --enable-local if you chose click/local for the
source files

9) 'make elemlist' and then 'make install' under click/


Let me know if you need any additional information or installation help.

Cheers

Beyers Cronje
-------------- next part --------------
A non-text attachment was scrubbed...
Name: udeve1000.hh
Type: text/x-c++hdr
Size: 9894 bytes
Desc: not available
Url : https://pdos.csail.mit.edu/pipermail/click/attachments/20080305/2bed9d43/attachment-0002.hh 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: udeve1000.cc
Type: text/x-c++src
Size: 19935 bytes
Desc: not available
Url : https://pdos.csail.mit.edu/pipermail/click/attachments/20080305/2bed9d43/attachment-0002.cc 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fifoqueue.hh
Type: text/x-c++hdr
Size: 2972 bytes
Desc: not available
Url : https://pdos.csail.mit.edu/pipermail/click/attachments/20080305/2bed9d43/attachment-0003.hh 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fifoqueue.cc
Type: text/x-c++src
Size: 1140 bytes
Desc: not available
Url : https://pdos.csail.mit.edu/pipermail/click/attachments/20080305/2bed9d43/attachment-0003.cc 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: udeve1000-click.patch
Type: text/x-patch
Size: 3285 bytes
Desc: not available
Url : https://pdos.csail.mit.edu/pipermail/click/attachments/20080305/2bed9d43/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: udeve1000-libe1000.patch
Type: text/x-patch
Size: 434 bytes
Desc: not available
Url : https://pdos.csail.mit.edu/pipermail/click/attachments/20080305/2bed9d43/attachment-0003.bin