Network Abstraction Layer (NAL) Architecture

The RTX64 Network Abstraction Layer (NAL) was created to provide a flexible and clean interface to the Network Interface Cards (NIC) and their drivers. This allows for greater system flexibility with regard to how applications interact with network assets, while also providing increased stability. The image below shows a high-level layout of the NAL architecture.

Key Components

Network Abstraction Layer (NAL)

The RTX64 Network Abstraction Layer serves a dual purpose. First, it abstracts away details of the NIC devices from the upper-level applications. It does this by providing a RtNal interface for Real-time applications to query what NIC devices, and queues are available for use along with what device configuration and capabilities are available.

Based on this information, an application can choose the network resources that are best for meeting its resource requirements. Once this determination is made, the application can then use the RtNal interface to reserve and attach to the appropriate device interface(s).

The second purpose of the NAL is to host the actual network interface drivers (NIC drivers). This allows for a much more stable system since the hardware drivers are completely decoupled from user applications.

NOTE: The RTX64 NAL and current RTX64 3.4 RT-TCP/IP Stack can run in parallel independent from each other if they access different physical network interfaces.

As an abstraction layer, the NAL is a real-time process that loads all configured drivers at launch and provides the information to real-time applications via the RtNal interface.

Network Interface Drivers

Network interface drivers are RTDLLS that are loaded and initialized by the NAL. Driver functions (RTN and RTND API) execute in the context of the NAL and Real-time applications attached to the NAL.

Understanding the Calling Sequence

RtNal NIC drivers loaded by the RTX64NAL process have a defined interface between the NAL and the driver. The sequences of initialization, transmit, and receive are shown below. The following examples use the RtNalIGB.rtdll driver.

NOTE: In the following diagram, the color red indicates the NAL context. The color gray indicates the context of a NAL application.

Initialization

The RTX64NAL process loads the RTDLL (driver) that corresponds to the NIC. In the below diagram, the RtNal loads and initializes RtNalIGB.rtdll.

1. The NAL must make the correct calls to the NIC driver, such that the NIC hardware is properly initialized, beginning with a call to RtndInitializeEx. Legacy drivers use RtndInitialize.
2. When the driver receives this call, it must find the NIC on the PCI/PCIe bus. To do this, the driver calls the Real-Time Network library function RtnEnumPciCards, in the context of RtndInitializeEx/RTX64NAL process. RtndInitializeEx returns configuration information for the driver to the NAL. This can be useful to NAL applications. Note that RtndInitialize lacks this ability.
3. Next, the RTX64 NAL process calls RtndConfigure and the hardware is initialized and configured.

The below diagram focuses on a network application. It shows the simplest use of a network application in RTX64. At a minimum, the network application queries the NAL for its queues and selects one for use. It is then up to the network application to release those queues once it is done with them. Note that the NAL releases unreleased application queues when the network application terminates. The image below depicts this interaction on the network application with the NAL and, ultimately, the specific network adapter hardware associated with its queues.

Transmitting a Packet

A network application sends Ethernet packets at the data link layer. The data link layer, or layer 2, is the second layer of the seven-layer OSI model of computer networking.

A NAL application can allocate memory for a packet in its own application format that can store some housekeeping data in addition to the packet data buffer. A NAL application can pass the pointer to this packet memory to a RtNalTransmit call, which calls the driver’s RtndTransmit API. The driver then calls RtnDecodePacket which in turn invokes the application decode packet callback fpRtnDecodePacket to get the pointer to the data buffer for the Ethernet packet to transmit, as well as its size. fpRtnDecodePacket can be configured via the RtNalConfigureQueue API. The driver then copies the data to a buffer previously allocated by the driver for the DMA transaction. It then communicates the new data to the NIC and returns.

Transmitting Multiple Packets

RtNalTransmitEx transmits multiple Ethernet frames in a single call. An application must pre-allocate NAL frames by calling RtNalAllocateFrame and pass the array of pointers to NAL frames to RtNalTransmitEx. RtNalAllocateFrame returns the pointer to a RTNAL_FRAME structure, which contains virtual and physical addresses of the Ethernet frame. The application must not change these addresses. An application can also provide callbacks to be called with each of the frames from a transmit complete thread. This callback pointer is written to the RTNAL_FRAME structure.

NOTE: To use transmit complete callbacks, the transmit complete notifications must be enabled for the queue. RtNalTransmitEx calls RtndTransmitEx in the driver to submit frames to the driver.

The NAL takes ownership of all the frames submitted to the driver. It relinquishes ownership of each frame after the transmission completes in a transmit complete thread. An application should call the RtNalIsApplicationFrame function to query a frame’s ownership.

NOTE: The application must not modify the RTNAL_FRAME structure or the frame data buffer when the NAL frame is owned by NAL.

When a queue is acquired with transmit notifications, the NAL runs an additional transmit complete thread per queue. This thread runs on the ideal processor of the process calling RtNalAcquireQueue and with the priority configured for the interface. This thread calls the RtndServiceTransmitQueue API exported by the driver.

The function RtndServiceTransmitQueue calls the RtnTransmitComplete callback to transfer the ownership of packets, transmitted by RtNalTransmitEx, back to the application. It then calls an application transmit complete callback.

If an application has not configured transmit complete notifications for the queue, transmitted packets are serviced in the driver’s transmit complete thread. This thread also transfers the ownership of packets, transmitted by RtNalTransmitEx, back to the application but it does not call an application callback.

NOTE: In the following diagram, the color red indicates the NAL context. The color gray indicates the context of a NAL application.

Receiving Packets

A network application can also receive Ethernet packets at the data link layer (Layer 2). To do this, it creates a thread that waits for a signal indicating receipt of multiple packets. When the NIC generates an interrupt to indicate the receipt of multiple packets, the driver’s Interrupt Service Thread (IST) tells the network application by calling RtnNotifyReceiveQueue, which in turn sets the signal that the thread is waiting on. Legacy drivers may use RtnNotifyReceive to run a single receive queue. The application receive thread should call RtNalGetReceivePacketCount to know how many packets are waiting and then call RtNalReceivePacket in a loop., RtNalReceivePacket calls the driver’s RtndReceive function to retrieve the waiting packet. The driver calls RtnGetPacket, which in turn invokes an application registered get packet callback to retrieve a packet from the application. The driver then calls RtnDecodePacket, which in turn invokes the application registered decode packet callback to get the pointer to the application data area. Lastly, the driver copies the data from its DMA buffer to the application data buffer and returns.

Pointers to the application get packet and decode packet callbacks are supplied in the capabilities structure in the RtNalConfigureQueue function.