Intel(R) I/O Acceleration Technology Software Guide for Linux* 
=============================================================

January 19, 2007


Contents
========
- Overview
- Integration
- Configuration
- Relevant System Variables
- Enabling and Disabling Intel I/OAT
- Intel I/OAT sysfs Interface
- Known Issues
- Support
- License


Overview
========

This document contains instructions on how to integrate and configure 
the ioatdma engine for Linux.  For additional information on Intel(R) 
I/O Acceleration Technology (Intel(R) I/OAT), see the following web 
page: http://www.intel.com/technology/ioacceleration/index.htm. 

Where applicable, please consult the End User Guide for specific instructions
to enable and configure I/OAT.

These instructions are intended for the following platform configuration:

- Mainboard chipset: Intel(R) 5000 Series Chipsets Integrated Device - 
  1A38

- BIOS: A BIOS that supports Intel I/OAT.

- Linux OS: the supported kernel version number is embedded in the patch 
  file name (e.g., cb-2.6.xx.diff, where "2.6.xx" is the supported kernel 
  version). 

- Drivers: 
  - e1000 LAN driver, version 7.0.xx or higher
  - ioatdma driver (in this tarball)

I/OAT is focused on optimizing performance - throughput & CPU utilization - 
for applications that 1) use TCP protocol 2) have a significant portion
of their traffic that is Receive-based 3) uses application buffer sizes 
that are larger than 4k. (Applications with larger packets typically see 
greater gains with I/OAT.)
 
For optimal benefits from the Intel I/OAT network solution, system memory 
channels should be fully populated. Customers with limited system memory 
may not receive the full benefits of I/OAT. Typically, there are two
memory slots per channel, i.e., slots 0 & 1 are associated memory channel 0.
Please refer to your system guide to specific information re memory channels.

When adding network adapters to a system, where possible please use slots
that are directly connected to the MCH. Please refer to your system guide
to specific information on slots that are attached to the MCH.

Users are encouraged to work with their applications & systems to determine
how they can best take advantage of I/OAT.


Integration
===========

There are two IOATDMA tar balls - the ioatdma-x.x.tar.gz and the 
ioatdma-x.x-full.tar.gz. The first tar ball does NOT include the kernel
patch required to enable I/OAT and the second tar ball does.

Starting with the 2.6.18 kernel, support for IOAT is included with the 
kernel. Some distributions - Novell SLES 10 - have also integrated the 
path into their 
kernels. In these cases, integrating the kernel patch is
not required for IOAT support. Please use the ioatdma-x.x.tar.gz.

NOTE: If the kernel does support I/OAT, skip to the Configuration section.


1) Start with an unaltered kernel source tree.  
   - Make sure it compiles and boots.  
   - Verify that lspci shows the Intel I/OAT compatible adapter.  
       (bus 0, device 8, unknown device 1a38).  
     If the Intel I/OAT capable adapter does not appear, make sure it is 
     enabled in the BIOS menu: 
     -> Device Manager -> Blackford Configuration -> Crystal Beach/DMA 
     Configuration

2) Extract all files from this ioatdma tarball to a new directory in the 
   kernel tree, named drivers/dma.  

3) Move drivers/dma/dmaengine.h to include/linux.  

4) Apply drivers/dma/cb-{kernel version}.diff using patch.  
   Example:  patch -p1 < drivers/dma/cb-2.6.xx.diff

5) Verify that the kernel has CONFIG_PCI_MSI enabled.  

6) Enable the DMA Subsystem from the menuconfig menu.  Select:
      Device Driver > DMA Engine Support
   In the DMA Engine screen, select/enable all of the following:
     - Support for DMA Engines
     - Network: TCP receive copy offload
     - Test DMA Client
     - Intel I/OAT DMA support

7) Compile the kernel and reboot.  If compiled as a module, the DMA 
   engine driver will be named ioatdma.ko.



Configuration
=============

The ioatdma driver is a loadable module. Listed below are the steps
required to make and install the module.


1) Apply the SLES10 specific patch.
	patch p1 <patches/sles10.patch

2) Make and install the ioatdma module
	make install 

3) Ensure that the updated module is used.  
	depmod a

4) Optionally, you should rename the old module to minimize the potential for 
   confusion.
 
	mv /lib/modules/`uname r`/kernel/drivers/dma/ioatdma 
	/lib/modules/`uname r`/kernel/drivers/dma/ioatdma_old.ko


NOTE: Make sure that I/OAT support is configured into the kernel.


Relevant Syscntl Settings
-------------------------

The tcp_dma_copybreak variable represents the lower limit of the size of 
the socket reads that will be offloaded to the DMA copy engine when 
CONFIG_NET_DMA is enabled. The default value is 4096 bytes. Lowering this 
limit reduces the thresold when offloading will be enabled.

The tcp_low_latency system variable (/proc/sys/net/ipv4/tcp_low_latency) 
ensures data is forwarded through the TCP stack to the application buffer. 
When set to a value of 1, it disables tcp_prequeue in the Linux ipv4 TCP 
stack and no offloading will occur.


Enabling and Disabling Intel I/OAT
----------------------------------

The Intel I/OAT network accelerations are enabled by loading the Intel 
I/OATDMA engine driver at runtime.  The driver module filename is ioatdma.ko.  

To enable Intel I/OAT, load the ioatdma driver module:

    modprobe ioatdma

Removing the ioatdma module once it has been loaded is not recommended 
since TCP receives hold a reference to the ioatdma driver when offloading
traffic. However, if the "Forced module unloading" option is enabled in 
the kernel, the module may be unloaded with:

    rmmod -f ioatdma

WARNING: This command above may hang the system.


Intel I/OAT sysfs Interface
---------------------------

When the Intel I/OAT driver is properly loaded, there will be directories 
created in sysfs, under /sys/class/dma, named dma0chanX, where X is 0-3.  

Channel Entries:

in_use
------
1 if the DMA channel is allocated to a client, such as the network stack.  

bytes_transferred
-----------------
The total number of bytes transferred by either the DMA engine or CPU copy.  

memcpy_count
------------
The total number of copy operations initiated.  


Known Issues
============

Using I/OAT with IPv6 may degrade performance.


Support
=======

For general information, go to the Intel support website at:

    http://support.intel.com

If an issue is identified with the released source code on the supported
kernel with a supported adapter, email the specific information related 
to the issue to linux.nics@intel.com.  


License
=======

This software program is released under the terms of a license agreement 
between you ('Licensee') and Intel.  Do not use or load this software or 
any associated materials (collectively, the 'Software') until you have 
carefully read the full terms and conditions of the LICENSE located in 
this software package.  By loading or using the Software, you agree to 
the terms of this Agreement.  If you do not agree with the terms of this 
Agreement, do not install or use the Software.  

* Other names and brands may be claimed as the property of others.  