Creating a NIC Driver

In this section, we introduce the NIC driver and explain how to create a project and implement its functionalities.

Topics:

 

Creating a Project

To create a project in Visual Studio:

  1. To create a project:
  1. Create a new project in Visual Studio.
  2. Select C or C++ as the template type.
  3. Select Finish to create the project.
  1. Add #include <Rtnd.h> to the header file.

Note: The declarations and definitions needed by the driver's source code are in Rtnd.h. A driver must NOT include Rtnl2Api.h nor be linked with Rtnl2Api.edll.

Using the Sample Driver

eRTOS SDK provides source code and project files for a NIC driver sample (see NL2NICDriverSample). You can use this sample as a reference when implementing driver functions into your project. This sample provides a basic structure for a NIC driver. Please access this sample from the directory %public%\Public Documents\IntervalZero\MaxRT\eRTOS SDK\1.0\Samples or through the Start Menu.

General Information about NIC Drivers

NIC drivers are RTDLL modules loaded by the Network Link Layer (NL2) at startup.

The NL2 uses the NIC driver's services by calling the functions exported by the driver. These functions are described in the Real-Time Network Driver API Functions section.

After loading a driver, the first function called by the NL2 is RtndInitDriver. This function initializes the driver and passes pointers to NL2 callback functions that the driver can call.

Some of the functions exported by drivers are always called from the main thread of the NL2 process, while others may be called from any thread of the NL2 process. or from the application process. This information is specified in the Remarks section of each API function.

To prevent driver and hardware state corruption, the NL2 prevents an application process from terminating while it's executing a driver’s exported function.

Real-Time Requirements

NL2 drivers are designed for real-time applications, but not all driver functions require real-time performance. This section outlines the meaningful real-time requirements for a driver and identifies the critical parts of a driver that must adhere to them.

Topics in this section:

Concepts

Real-time is a multifaceted concept that varies based on context.

For clarity, we define the following real-time behavior:

Note: Being deterministic depends not only on the software design of the driver but also on the underlying hardware. If the code for sending a frame needs to wait on some shared NIC hardware resources, then that code can't be deterministic.

Requirements

We distinguish two categories of functions in an NL2 driver:

The requirement is to avoid preventing an NL2 application from achieving deterministic behavior simply because it needs to send/receive Ethernet frames. Driver developers must try to fulfill the above requirement, unless the hardware doesn't allow it.

Note: For drivers that implement the EnableInterruptFromSpecifiedMessage function (in MSI-X multi-vector mode), the requirements on ISR/IST also apply to EnableInterruptFromSpecifiedMessage.

Note: Drivers that control a virtual (software) device follow the same requirements as above, but the affected chain of functions starts when the Transmit/Receive event occurs.

Memory Allocation and Deallocation Restrictions

The NL2 imposes strict restrictions on when a NIC driver is allowed to allocate memory.

Drivers can dynamically allocate/deallocate two types of memory:

Memory type

Description

Local memory

This type of memory contains the following:

  • Kernel external MSpace: This is used when the driver creates a Shared Memory region.
  • Kernel internal MSpace: This is used when the driver creates a kernel object.
  • eRTOS Process external MSpace: This is used when the driver makes an explicit memory allocation with RtAllocateLocalMemory.
  • eRTOS Process internal MSpace: This is used when the driver opens a handle to a kernel object.

Contiguous physical memory

This memory type is used when the driver allocates memory that needs to be shared with the NIC hardware, typically for Tx/Rx buffers and DMA descriptors rings.

Note: Each Real-time Network Driver (RTND) entry point function has different memory allocation requirements.

Implementing the NIC Driver Functionality

In this section, we will introduce the different features that a NIC Driver must implement.

Topics in this section:

Initialize the Driver and Register the Callback Function Pointers

After the NL2 loads the NIC driver, it calls RtndInitDriver to provide the driver with a structure containing function pointers to the callbacks exposed by the NL2.

Note: This set of exported callback functions is not specific to an interface or a driver. The NL2 provides the same functions to all drivers.

The RtndInitDriver function also allows the driver to perform any required global initialization.

Probe an Interface on the PCI Bus

The Probe operation involves searching for the interface on the PCI Bus and reading the PCI Configuration Space to identify the device and ensure that it is supported.

The NL2 calls RtndManageInterface to request a Probe operation. If the driver finds a supported device, it must allocate a context for that interface, record the information found in the PCI Configuration Space, and return a number or a pointer (ULONG_PTR type) to identify that interface.

Note: The driver is not allowed to write any register of the NIC in this function. It must NOT start the interface.

Return the Capabilities of an Interface and Its Queues

After a successful call to RtndManageInterface, the NL2 calls RtndQueryInterfaceCapability to retrieve the interface's capabilities and queues. The NL2 calls RtndQueryInterfaceCapability once for each capability type it requires.

Configure an Interface and Its Queues

After retrieving the interface's capabilities and queues, the NL2 configures them by calling RtndSetInterface. The NL2 calls RtndSetInterface once for each type of setting it requires to configure.

This operation is always done by the NL2 before calling RtndStartInterface to start the interface.

Note: If the NL2 does not explicitly configure a setting, the driver must use the default value.

Start an Interface

After configuring the interface and its queues, the NL2 calls RtndStartInterface to start the hardware. The driver must use the settings configured in the previous calls to RtndSetInterface.

In the RtndStartInterface function, the driver is expected to allocate receive buffers for all of the interface's enabled receive queues. When done this way, the NIC is ready to accept incoming frames when the function returns. The driver must use RTND_CALLBACKS.CreateRxBuffers to allocate the receive buffers. See Allocate Frame Buffers and Receive Frames.

The driver is also expected to allocate Transmit Buffers for all enabled Transmit Queues of the interface. This is done so that when the function returns, the NL2 can call RtndGetTxBuffers to get Transmit Buffers and frames. The driver must use RTND_CALLBACKS.CreateTxBuffers to allocate the Transmit Buffers. See Allocate Frame Buffers and Get Buffers for Transmit.

Return the MAC Address of an Interface

After the interface is started, the NL2 calls RtndQueryMacAddress to query its MAC Address.

Return the Features Supported by an Interface

In addition to capabilities, the driver reports features to the NL2.

Capabilities describe the static options and ranges that the NL2 can set before starting an interface, while features describe the functionalities that can be used by the NL2 after the interface is started.

The NL2 calls RtndQueryInterfaceFeature to retrieve the features supported by an interface. This function is called once for each feature type defined by the NL2.

Monitor the Link Status of an Interface

After the interface is started, the driver is responsible for monitoring its link status.

In general, NICs use hardware interrupts to inform the driver of a link status change, but the driver can also implement a polling operation to keep track of the link status.

When the driver detects a link status change, it must call RTND_CALLBACKS.NotifyLinkStatusChange to inform the NL2. The driver is not expected to immediately read the new link status (see below).

Note: The callback function must be called from within the NL2 process.

Inside the callback, the NL2 notes that a link status change happened and wakes up its main thread. When the main thread wakes up, the NL2 calls RtndQueryLinkStatus. At this point, the driver is expected to read the NIC registers to retrieve and return the interface's current link status.

Monitor the Egress Timestamps Captured by an Interface

When an interface supports Egress Timestamping, the driver is responsible for monitoring the availability of those Egress Timestamps.

In general, NICs use hardware interrupts to inform the driver when a new Egress time stamp is available, but the driver can also implement a polling operation to monitor Egress Timestamps.

When the driver detects the availability of a new Egress Timestamp, it must call RTND_CALLBACKS.NotifyEgressTimestamp to inform the NL2. The driver is not expected to extract the Egress Timestamp immediately (see below).

Note: The callback function must be called from within the NL2 process.

Inside the callback, the NL2 notes that a new Egress Timestamp is available and wakes up its main thread. When the main thread wakes up, the NL2 calls RtndExtractLastTxTimestamp. At this point, the driver is expected to read the NIC registers to extract the last Egress Timestamp for a given Transmit Queue.

Extracting an Egress Timestamp has 2 side effects:

Configure the Operating Mode of an Interface

Some NICs can alter their nominal operating mode, which is described by a set of TRUE or FALSE settings.

The NL2 supports two of these settings:

By default, at startup, the driver must set the interface to operate in nominal mode. All the above settings must be turned off.

Note: Support for the above settings is optional. The NL2 will never try to modify a setting not supported by the underlying driver. See RTND_FEATURE_INTERFACE_MODES.

Configure the Multicast Hash Filter of an Interface

Every NIC hardware has a hash table to filter out undesired Multicast frames early on the reception path.

The NL2 maintains an interface-specific linked list of RTND_MULTICAST_ENTRY structures which contain the Multicast MAC addresses that should pass the Multicast Hash Filter. Every time the content of this list is modified, the NL2 calls the RtndSetMulticastFilter function and provides a pointer to the head of the new linked list of RTND_MULTICAST_ENTRY structures. The driver is responsible for updating the hardware hash table accordingly.

The NL2 can also request the driver to let ALL Multicast frames pass. Instead of providing a list of RTND_MULTICAST_ENTRY structures, the NL2 sets the bPassAllMulticast parameter of the RtndSetMulticastFilter function to TRUE.

By default, at startup, the driver must clear the Multicast Hash Filter to prohibit the reception of all Multicast frames.

Note:  If the hardware lacks a Multicast Hash Filter, the driver should allow all Multicast addresses to pass (same result as a theoretical Multicast Hash Filter of size 0).

Configure the Interrupt Moderation

Some NICs support a feature to slow down the rate of interrupts generated by the hardware. This is done by specifying an interval in nanoseconds. When the feature is enabled, no more than one interrupt is generated per interval.

In MSI-X mode, there is one interval value per MSI-X Message. In Line-Based and MSI mode, there is a single interval value for the whole interface.

The NL2 configures interrupt moderation by calling RtndSetInterruptModeration.

By default, at startup, interrupt moderation must be turned off by the driver.

Note: Support for interrupt moderation is optional. When it’s supported, the interval value can't exceed a given limit. The NL2 will never try to use a non-supported feature or a non-supported value. See RTND_FEATURE_INTERRUPT.

Configure the EtherType Hardware Dispatcher

The NL2 views the EtherType Hardware Dispatcher as a fixed-size table. Each entry within this table has the following structure:

When Enabled is FALSE, the entry is inactive. When Enabled is TRUE, the entry is active and the hardware forwards all incoming frames with the EtherType into the Receive Queue specified by RxQueueIndex.

Incoming frames with an EtherType that doesn’t belong to any active entry are forwarded to Receive Queue 0.

The NL2 guarantees that it never enables different entries with the same EtherType value. It also guarantees that the RxQueueIndex value in each enabled entry corresponds to an existing and enabled Receive Queue.

By default, at startup, the driver must disable all the entries of the EtherType Hardware Dispatcher. This means that all received frames go to Receive Queue 0.

The NL2 uses two functions provided by the driver to configure the EtherType Hardware Dispatcher:

Note: The driver is responsible for maintaining a copy of the EtherType Hardware Dispatcher table in memory. The RtndGetDispatcherEtherTypeEntry function must NOT access any hardware register; it must read and return the content saved in memory.

Allocate Frame Buffers

Frame buffers are buffers that hold the content of an Ethernet frame and can be accessed by the hardware. Transmit Frame Buffers are meant to be used by transmit queues, while receive frame buffers are meant to be used by receive queues.

As specific hardware may have specific requirements in terms of memory alignment and memory location used by the frame buffers, the driver allocates and frees those buffers.

All buffers are allocated at the driver's initiative in the RtndStartInterface function. The workflow is as follows:

  1. Within the RtndStartInterface function, the driver calls RTND_CALLBACKS.CreateTxBuffers or RTND_CALLBACKS.CreateRxBuffers.
  2. Within the CreateTxBuffers or CreateRxBuffers function, the NL2 allocates the buffer headers and calls RtndAllocateTxFrameDataBuffers or RtndAllocateRxFrameDataBuffers to allocate the frame buffers.
  3. When the driver returns, the NL2 completes its buffer header management.
  4. When the NL2 returns, the driver can use the allocated NL2 Buffers.

Note: The NL2 will never call RtndAllocateTxFrameDataBuffers or RtndAllocateRxFrameDataBuffers outside the CreateTxBuffers or CreateRxBuffers functions.

All Buffers are freed at the driver's initiative, either by RtndStartInterface or RtndStopInterface. The workflow is as follows:

  1. Within RtndStartInterface or RtndStopInterface, the driver calls RTND_CALLBACKS.DestroyTxBuffers or RTND_CALLBACKS.DestroyRxBuffers.
  2. Within the DestroyTxBuffers or DestroyRxBuffers function, the NL2 marks the buffer as pending free. If all buffers of the same set are marked as pending free, the NL2 calls RtndFreeTxFrameDataBuffers or RtndFreeRxFrameDataBuffers to free all the frame buffers of the set in one shot.
  3. When the driver returns, the NL2 frees the associated buffer headers.
  4. When the NL2 returns, it indicates to the driver that the resources associated with the NL2 Buffers have been freed. It should not reference them again.

Note: The NL2 will never call RtndFreeTxFrameDataBuffers or RtndFreeRxFrameDataBuffers outside the DestroyTxBuffers or DestroyRxBuffers functions.

Get Buffers for Transmit

At startup, in the RtndStartInterface function, the driver must allocate Transmit Buffers for its Transmit Queues and manage them in pools of available Transmit Buffers. There is one pool for each enabled Transmit Queue.

When RtndStartInterface returns, the driver owns all the allocated Transmit Buffers. Before being able to transmit (see Transmit Frames), the NL2 calls RtndGetTxBuffers to get one or more Transmit Buffers from the driver. Ownership of those buffers is then transferred to the NL2, and the NL2 can use those buffers to prepare the content of Ethernet frames and submit them to the driver for transmission. Whenever a Transmit Buffer is submitted, its ownership is transferred back to the driver. Once a Transmit Buffer is extracted, its ownership is transferred to the NL2.

The NL2 transfers all the Transmit Buffers it owns back to the driver before the RtndStopInterface function is called. This is done either by submitting the buffers for transmission with RtndSubmitTxBuffer, or by returning them with RtndReturnTxBuffers. The difference is that RtndReturnTxBuffers does not perform transmission.

Transmit Frames

Transmit operations may be executed in the context of the application process to meet the user applications' high throughput and low-latency requirements. The NL2 provides the required locking mechanisms to prevent race conditions (see below).

For each enabled Transmit Queue of each interface, the driver must support the following operations:

Although a given Transmit Queue can be used by multiple processes simultaneously, the NL2 guarantees the following workflow is always followed for a given process:

  1. Call RtndAttachTxQueue.
  2. Call RtndSubmitTxBuffer, RtndApplyTxBuffers, and RtndExtractTxBuffer as many times as needed, in any order. The NL2 guarantees these three functions are never simultaneously called by two different threads of any attached processes.
  3. Either call RtndDetachTxQueue or get terminated. After calling RtndDetachTxQueue, the calling process is no longer allowed to call RtndSubmitTxBuffer, RtndApplyTxBuffers, and RtndExtractTxBuffer until it calls RtndAttachTxQueue again (thus returning to step 1 above).

Based on the above workflow, the following rules are derived:

Important: Although the driver must allow calls to RtndSubmitTxBuffer, RtndApplyTxBuffers, and RtndExtractTxBuffer from multiple processes for the same queue, real-time applications optimized for low latency will call those functions from a single thread only, always running on the same processor. The driver should be implemented so that the best performances are attained when RtndSubmitTxBuffer, RtndApplyTxBuffers, and RtndExtractTxBuffer are always called from the same processor.

Restart a Transmit Queue

Some NICs can stop a Transmit Queue and restart it later without affecting other queues. This is useful, as it allows for all buffers submitted to the queue to be canceled without waiting for them to be consumed, which can take time or, when the link is down, never complete.

NICs that can stop and restart and Transmit Queue should set RTND_CAPABILITY_TX.DynamicStopStartTxQueueSupported to TRUE and implement these functions:

A call to RtndStopTxQueue is expected to stop the hardware queue and cancel any buffer that was already submitted but not yet consumed. A call to RtndStartTxQueue is expected to restore the hardware queue in the state it was just after the interface was started.

The NL2 guarantees that the following functions are called only between a call to RtndStartTxQueue and RtndStopTxQueue:

The NL2 also guarantees that it returns to the driver, by calling RtndReturnTxBuffers, all the Transmit Buffers that it owns before calling RtndStopTxQueue.

Process Transmit Interrupts

By default, at startup, the driver must disable all Transmit interrupts on all Transmit Queues of all interfaces.

After startup, the NL2 may dynamically request the driver to enable Transmit interrupts on some of its Transmit Queues. This is done by the RtndEnableTxInterrupts function, which enables or disables the Transmit interrupt. This function is always called in the context of the NL2 process, but the NL2 doesn’t guarantee it is always called from the same thread.

Note: The driver MUST be prepared to receive a call to RtndEnableTxInterrupts from the NL2 process while it is executing RtndSubmitTxBuffer, RtndApplyTxBuffers, or RtndExtractTxBuffer from a user process, for the same Transmit Queue.

A Transmit interrupt eventually triggers the execution of the associated IST in the context of the NL2 process. Depending on the hardware and the configured interrupt mode (MSI, MSI-X), the driver may be unable to distinguish which Transmit Queue caused the interrupt. In that case, the driver must call RTND_CALLBACKS.NotifyTxInterrupt once for each Transmit Queue that has Transmit interrupts enabled and that may be the cause of the interrupt. If the driver knows which Transmit Queue caused the interrupt, it must call RTND_CALLBACKS.NotifyTxInterrupt for that queue only.

Note: The NL2 tolerates calls to RTND_CALLBACKS.NotifyTxInterrupt for a Transmit Queue which it requested to disable Transmit interrupts. The driver should avoid doing this as it unnecessarily consumes CPU resources.

Even though the NL2 does NOT have any locking mechanism to prevent an IST from being executed at the same time as another function of the driver, this should not be an issue because the IST should be designed to be as simple as:

  1. Read the interrupt status register.
  2. Acknowledge the interrupts.
  3. If it's a Transmit interrupt:

For each Transmit Queue which may have caused this interrupt, and which has Transmit interrupts enabled, call RTND_CALLBACKS.NotifyTxInterrupt.

  1. If it's a Receive interrupt:

For each Receive Queue which may have caused this interrupt, and which has Receive interrupts enabled, call RTND_CALLBACKS.NotifyRxInterrupt.

  1. If it's a Link Status Change interrupt:

Call RTND_CALLBACKS.NotifyLinkStatusChange.

  1. If it's an Egress Timestamp Available interrupt:

Call RTND_CALLBACKS.NotifyEgressTimestamp.

If a particular driver needs to access queue-specific registers during the IST, it must have locking mechanisms to avoid race conditions with other driver functions, especially RtndSubmitTxBuffer, RtndApplyTxBuffers, and RtndExtractTxBuffer.

Receive Frames

Like transmit operations, the receive operations presented in this section need to be very fast to reach the high throughput and/or low-latency requirements of the user applications. To achieve this, they may be executed in the user application context. The NL2 provides the required locking mechanisms to avoid race conditions (see below).

For each enabled Receive Queue of each interface, the driver must support the following operations:

For each process (NL2 or user application) using the Receive Queue, the NL2 guarantees the following workflow is always followed:

  1. Call RtndAttachRxQueue.
  2. Call RtndSubmitRxBuffer, RtndApplyRxBuffers, and RtndExtractRxBuffer as many times as needed, in any order. The NL2 guarantees that these three functions are never called simultaneously by two different threads of any attached processes.
  3. Either call RtndDetachRxQueue or get terminated. After calling RtndDetachRxQueue, the calling process is no longer allowed to call RtndSubmitRxBuffer, RtndApplyRxBuffers, and RtndExtractRxBuffer until it calls RtndAttachRxQueue again (thus returning to step 1 above).

From the driver's perspective, when a user application process is terminated, the effect is the same as calling RtndDetachRxQueue.

Based on the above workflow, the following rules are derived:

Note: Although the above guarantees should be sufficient, the driver may use additional custom locks to protect its data from race conditions. However, this should be avoided as much as possible, as it degrades the overall application performance.

Important: Although the driver must allow calls to RtndSubmitRxBuffer, RtndApplyRxBuffers, and RtndExtractRxBuffer from multiple processes for the same queue, real-time applications optimized for low latency will call those functions from a single thread only, always running on the same processor. The driver should be implemented so that the best performances are attained when RtndSubmitRxBuffer, RtndApplyRxBuffers, and RtndExtractRxBuffer are always called from the same processor.

Restart a Receive Queue

Some NICs can stop and restart a Receive Queue without affecting other queues. NICs that support this feature should set RTND_CAPABILITY_RX.DynamicStopStartRxQueueSupported to TRUE and implement these functions:

A call to RtndStopRxQueue is expected to stop the hardware queue and prevent it from filling a buffer. A call to RtndStartRxQueue is expected to restore the hardware queue to its state just after the interface was started.

The NL2 guarantees that the following functions are called only between a call to RtndStartRxQueue and RtndStopRxQueue:

The NL2 also guarantees that it submits to the driver, by calling RtndSubmitRxBuffer, all the Receive Buffers it owns before calling RtndStopRxQueue.

Process Receive Interrupts

By default, at startup, the driver must disable all Receive interrupts, on all Receive Queues of all interfaces.

After startup, the NL2 may dynamically request the driver to enable Receive interrupts on some of its Receive Queues. This is done by the RtndEnableRxInterrupts function, which requests both enabling and disabling the Receive interrupt. This function is always called in the context of the NL2 process, but the NL2 doesn’t guarantee that it is always called from the same thread.

Note: The driver MUST be prepared to receive a call to RtndEnableRxInterrupts from the NL2 process while it is executing RtndSubmitRxBuffer, RtndApplyRxBuffers, or RtndExtractRxBuffer from a user process, for the same Receive Queue.

The occurrence of a Receive interrupt eventually triggers the execution of the associated IST in the context of the NL2 process. Depending on the hardware and the configured interrupt mode (MSI, MSI-X), the driver may not be able to distinguish which Receive Queue caused the interrupt. In that case, the driver must call RTND_CALLBACKS.NotifyRxInterrupt once for each Receive Queue that has Receive interrupts enabled and that may be the cause of the interrupt. If the driver knows which Receive Queue caused the interrupt, it must call RTND_CALLBACKS.NotifyRxInterrupt for that queue only.

Note: The NL2 tolerates calls to RTND_CALLBACKS.NotifyRxInterrupt for a Receive Queue which it requested to disable Receive interrupts. The driver should avoid doing this as it unnecessarily consumes CPU resources.

Though the NL2 does NOT have any locking mechanism to prevent an IST from being executed at the same time as another function of the driver, this should not be an issue because the IST should be designed to be as simple as:

  1. Read the interrupt status register.
  2. Acknowledge the interrupts.
  3. If it's a Transmit interrupt:

For each Transmit Queue which may have caused this interrupt, and which has Transmit interrupts enabled, call RTND_CALLBACKS.NotifyTxInterrupt.

  1. If it's a Receive interrupt:

For each Receive Queue which may have caused this interrupt, and which has Receive interrupts enabled, call RTND_CALLBACKS.NotifyRxInterrupt.

  1. If it's a Link Status Change interrupt:

Call RTND_CALLBACKS.NotifyLinkStatusChange.

  1. If it's an Egress Timestamp Available interrupt:

Call RTND_CALLBACKS.NotifyEgressTimestamp.

If a driver needs to access queue-specific registers during the IST, it must have locking mechanisms to avoid race conditions with other driver functions, especially RtndSubmitRxBuffer, RtndApplyRxBuffers, and RtndExtractRxBuffer.