OpenOnload 201109-u1 Release Notes
==================================

 Major new features of this release are identified below, along with
 limitations that we are aware of.


----------------------------------------
Per-thread control of spinning
------------------------------

 This release adds a new extension API:

   onload_thread_set_spin(api, enable);

 This can be used to control spinning on a per-thread as well as per-API
 basis.  The existing spin-related configuration options set the default
 behaviour for threads, and this API overrides the default.


----------------------------------------
Scalable packet buffer mode
---------------------------

 Previous versions of Onload were only able to support a limited number of
 buffers for storing packet data, due to these buffers consuming a limited
 hardware resource.  The limit was typically around 120,000 packet buffers
 in aggregate, shared amongst all Onload stacks.

 The openonload-201109 release introduced a new feature that removes that
 limit.  The openonload-201109-u1 includes some significant improvements to
 this feature.  Please see the user guide for the details, but briefly the
 prerequisites are:

 - The system hardware must have an IOMMU and it must be enabled in the
   BIOS [1].

 - The kernel must be compiled with support for IOMMUs [2], and kernel
   command line options are needed to select the IOMMU mode used [3].

 - SR-IOV must be enabled on the network adapter via the "sfboot" utility.

 - The kernel must be compiled with support for SR-IOV APIs
   (CONFIG_PCI_IOV).

   NB. you can often find the config for the running kernel in one of:
      /boot/config-$(uname -r)
      /lib/modules/$(uname -r)/build/.config
      /proc/config.gz

 - In order to use more than 6 VFs (virtual functions) the system hardware
   and kernel must support PCIe Alternative Requester ID (ARI) which is a
   PCIe genII feature.  Current Solarflare adapters support up-to 127 VFs.

 - Onload option EF_PACKET_BUFFER_MODE=1 must be in the environment.

 When "scalable packet buffer mode" is enabled there is no hardware-imposed
 limit on the number of packet buffers that can be allocated, but the
 per-stack limit enforced by EF_MAX_PACKETS still applies.

 There were several bugs in the Linux kernel's early support for SR-IOV.
 We recommend you use a kernel at least as recent as 2.6.35, or an RHEL
 kernel at least as recent as 2.6.32-131.0.15.  For other distributions we
 suggest (a) to use the latest stable kernel available for that
 distribution and (b) to perform extensive testing with scalable packet
 buffer mode before deploying in production environments.

 [1] To detect if the BIOS has IOMMU support enabled please see the
     messages printed as the kernel boots.

     The BIOS presents ACPI tables to the kernel. For Intel machines the
     ACPI table "DMAR" represents the IOMMU so expect a line in the syslog
     starting:

        kernel: ACPI: DMAR

     For AMD machines the ACPI table to expect is IVRS:

        kernel: ACPI: IVRS

     The BIOS may call this option "IO virtualization" or for Intel systems
     "VT-d" or it may enable the IOMMU with a more generically named
     virtualization option.  Please consult your system documentation.

 [2] Most recent kernels are compiled with support for IOMMUs by default,
     but unfortunately the realtime (-rt) kernel patches are not currently
     compatible with IOMMUs.

     It is possible to use scalable packet buffer mode on some systems
     without IOMMU support, but in an insecure mode.  In this configuration
     the IOMMU is bypassed, and there is no checking of DMA addresses
     provided by Onload in user-space.  Bugs or misbehaviour of user-space
     code can cause system compromise.  To enable this insecure mode set
     unsafe_sriov_without_iommu=1 for the sfc_resource kernel module.

     RedHat MRG kernels are compiled with CONFIG_PCI_IOV disabled.  It
     should be possible to recompile the kernel with CONFIG_PCI_IOV
     enabled, but note that Solarflare does not currently test this
     configuration.

 [3] We recommend you configure the kernel to use the IOMMU in pass-through
     mode.  Set the following kernel command-line options in your
     bootloader config:

     For an Intel system:
       intel_iommu=on iommu=on,pt

     or for an AMD system:
       amd_iommu=on iommu=on,pt

     In pass-through mode the IOMMU is bypassed for regular devices.  This
     gives the best possible performance, and also avoids kernel bugs that
     are exhibited on a variety of kernel versions when the IOMMU is fully
     enabled for all devices.


David Riddoch
2011/12/23