I/O Virtualization (IOV) & its uses in the Network Infrastructure: Part 3

It offers up to 20 Gb/s of accelerated network processing for a variety of applications including IDS/IPS systems, L2-L7 processing in network infrastructure equipment, and network security appliances. The NFP is highly programmable, including many aspects of its PCIe x8 (v2.0) host interface.

This programmability allows greater flexibility in how aspects of the device are presented to the host system. The NFP supports up 256 queues to the host, which are grouped to form endpoints. The queues are generic in the sense that they can carry arbitrary descriptors, thus it is possible to create endpoints of different endpoint types.

Example endpoint types are standard NIC style interface with RX and TX queues, optimized packet capture interfaces, or look-a-side crypto interfaces. The details of these interfaces are under software control running on the NFP, which defines, amongst others, the purpose of the assigned queues, the descriptor formats used, and how and when DMA is initiated. Endpoints of these different types can be created dynamically at runtime.

Furthermore, for a variety of applications both in the data center as well as in network infrastructure equipment it is necessary for these different endpoints to be accessible by different Guest VMs executing on the host with low overhead --- a IOV solution is required. However, as pointed out earlier in this series, SR-IOV is not designed to support this type of highly dynamic and highly flexible devices.

A more flexible IOV solution
The key insight is that SR-IOV relies on a number of PCI standards and essentially only adds a device enumeration and resource discovery mechanism. It is this mechanism, which is the primary reason for the limitations of SR-IOV.

With the Netronome IOV architecture, device enumeration is delegated to a driver running on the host. This driver is specific to the NFP and is capable of managing endpoints on the NFP: creation and removal of endpoints of arbitrary types.

To the host OS (or hypervisor) the host driver acts as PCI bus driver and enumerates NFP endpoints as standard PCI devices. It essentially implements a virtual PCI bus. All configuration space access for devices on this virtual PCI bus are passed to the host driver which either emulates them or translates them to accesses on the NFP.

This solution is not dissimilar to the SR-IOV approach. The host driver performs the same function as the PF driver for a SR-IOV device (management of VFs) and the PCIM (translating SR-IOV VFs to PCI devices to the host OS or hypervisor).

However, the Netronome architecture does not require VFs to also be enumerated in hardware. We therefore refer to the NFP endpoints as Software Configurable Virtual Functions (SCVFs). Because the Netronome host driver is not restricted by the SR-IOV device enumeration limitations it can enumerate arbitrary types of functions on its virtual PCI bus.

Figure 4: SR-IOV (left) and Netronome's IOV (right) compared. The primary difference is that with Netronome's IOV solution different types of devices are easily supported.

Figure 4 above depicts the commonality and difference between SR-IOV and Netromone's IOV solution. With the Netronome solution, different types of endpoints (indicated by the different colors) are easily supported while SR-IOV mandates that all VFs associated with a PF are of the same device type.

With Netronome's IOV solution the PF driver in combination with a virtual PCI implementation performs the same function as the PF driver and the PCIM for SR-IOV.

For both SR-IOV and for Netronome's IOV solution virtual functions are presented to the host OS or the hypervisor as standard PCI devices, thus they both leverage the mechanism provided by modern hypervisors to assign PCI devices to Guest VMs.

During initialization, each SCFV gets assigned a unique PCI function ID, which the NFP uses to tag DMA requests originating from the corresponding endpoint. Thus IO MMU page tables can be set up appropriately to provide DMA protection between different SCVFs. Each SCVF also gets assigned one or more MSI vectors from the PFs set of MSI-X vectors, thus they can directly notify the Guest VMs they are assigned to.

Network virtualization with NFP
The host side interactions of the NFP are identical to a SR-IOV solution. SCVFs can be assigned to Guest VMs just like VFs or PCI devices. This provides Guest VMs with low overhead and low latency access to network devices.

Guest VMs run standard device drivers to talk to the SCVF and interrupts can be delivered directly to the cores on which a Guest VM is executing.

However, since the NFP is a programmable network device, the multiplexing and de-multiplexing of packets from and to SCVFs is not limited to some fixed function hardware implementation as in most SR-IOV or MQ NICs.

Instead extensive packet processing can be performed including flow-based classification, load balancing or filtering. This provides a significantly more flexible solution than other hardware based IOV approaches.