EP2176773B1

EP2176773B1 - Data packet processing method for a multi core processor

Info

Publication number: EP2176773B1
Application number: EP08808136.9A
Authority: EP
Inventors: Nilakantan Mahadevan; Ananth Yelthimar Shenoy; Srikanth Lakshminarayan
Original assignee: Hewlett Packard Development Co LP
Current assignee: Hewlett Packard Development Co LP
Priority date: 2007-07-09
Filing date: 2008-07-07
Publication date: 2015-09-02
Anticipated expiration: 2028-07-07
Also published as: WO2009008007A2; CN101689158A; US20100241831A1; EP2176773A2; EP2176773A4; US8799547B2; CN101689158B; WO2009008007A3

Description

Introduction:

Known multi core processor platforms comprise one or more central processor units (CPU) which can each have multiple processing cores which share memory and I/O (input and output) resources. The multiple core architecture enables simultaneous execution of processes or "threads" on more than one core. Such multi core processors have been developed to increase platform processing capacity over that of single core processors.
In multi core symmetric multiprocessing processing (SMP) architecture the system memory is shared so any processor core can work on any task regardless of where the data for that task resides in system memory. It is also known for multi core systems to utilise a cache memory shared between multiple cores of the one CPU.
Multi core processor platforms may be used as servers in a network, such as an Ethernet local area network (LAN). Server architecture can comprise a plurality of processors, however, are physically connected to the network using only one LAN interface or network interface card (NIC) connected to one processor core. This network interface can only interrupt one processor core when packets are received from the network. The received packet is then processed through the network stack in the interrupted core until it reaches the application. Thus, the network connected processor core must be used for all data received from the network. In the case of high speed TCP/IP applications or any network stack applications and high bandwidth networks, such as Gigabit and ten Gigabit Ethernet networks, the full server processing capacity may not be realised due to throughput of data between the network and the server being limited by saturation of the network connected processor core.
Current solutions to overcome this problem require the server to be provided with further network interfaces connected to further processors in order to distribute the network interface processing load across multiple processors. This solution can improve throughput between the server and the network, but is inefficient as it requires additional network interface cards. This solution also introduces an additional overhead for coordination of the multiple network interfaces.
Another proposed solution to this problem is to redistribute network stack operations for inbound data to various cores by the network interface. This solution requires interoperability between the LAN or network interface and the processor operating system. Thus, modification of the network interface and operating system is required to implement such a solution, so this solution cannot be implemented with standard currently available network interfaces.
Patent application US2005/0060462 A1 (Eiji Ota ) discloses a method and system where, upon detecting an interrupt received from an I/O device directed to one or more processors, a policy is invoked and the interrupt is redirected to another processor for efficient interrupt handling.
There is a need for server architecture to improve throughput between a network and a server having multi core processor architecture.

Brief description of the drawings:

Embodiments of the invention will now be described by way of example only with reference to the accompanying drawings in which:

Figure 1: illustrates an example of a multi core processor architecture;
Figure 2: illustrates an example of processing packets through the network stack of a multi core processor according to an embodiment of the present invention;
Figure 3: is a flow chart illustrating a kernel process of selecting a processor core and handing over packet processing to the selected core;
Figure 4: illustrates a conventional multi core processor network stack architecture test model used in comparative test Example 1;
Figure 5: illustrates a multi core processor network stack architecture in accordance with an embodiment of the present invention used in comparative test Example 1; and
Figure 6: is a graph illustrating the results of comparative test Example 1.

Detailed description:

According to an embodiment there is provided a method of processing a data packet received from a network in a multi core processor platform comprising the steps of:

receiving an interrupt in a designated central processor unit (CPU) core from a network interface for processing a data packet received from the network;
associating the data packet with an application connection established in a central processor unit (CPU) core selected based on processor load; and
creating an interrupt thread on the CPU core associated with the application connection for processing the received data packet.

According to an embodiment there is provided a method of processing a data packet being sent to a network by an application executing in one central processor unit (CPU) core of a multi core processor platform, the method comprising the steps of:

determining whether to perform outgoing processing of the data packet using the CPU core executing the application or using an alternative central processor unit (CPU) core;
and where it is determined to use an alternative CPU core:
selecting an alternative CPU core for outgoing packet processing based on processor load;
establishing a application connection in the selected alternative CPU core;
associating the data packet with the application connection; and
creating an interrupt thread on selected alternative CPU core for processing the data packet to the network interface for transmission over the network.

According to an embodiment there is provided a network server system comprising at least one central processor unit (CPU) having a plurality of cores and a network interface for connecting to a network, and adapted to create an interrupt in a designated CPU core for processing each data packet received from the network. Each data packet received from the network is associated with an application connection established in a selected CPU core selected based on processor load. An interrupt thread is created on the CPU core associated with the application connection for processing the data packet.
According to an embodiment there is provided a network server system comprising at least one central processor unit (CPU) having a plurality of cores and a network interface for connecting to a network, and adapted to create an interrupt in a designated CPU core for each data packet received from the network, and further adapted such that each CPU core can access the network interface for transmitting data packets to the network. Each data packet being sent to the network is associated with an application connection established either in the CPU core in which the application is executing or an alternative CPU core selected based on processor load. Where the application connection is established in an alternative CPU core, an interrupt thread is created on the CPU core associated with the application connection for processing the data packet.
Embodiments of the present invention provide a method and system for processing packets sent to and received from a network 110 in a multi core processor platform 100 to reduce limitations to throughput of data to and from the network 110 due to the network interface interrupting one CPU core for all received packets. Received packets are associated with an application connection established in a selected central processor unit (CPU) core, wherein the CPU core is selected based on processor load. An interrupt thread is then created on the CPU core associated with the application connection for the received packet. This enables the packet processing to be handed off to a core, other than the network interface interrupted CPU core, in which an application connection exists or is established to reduce the processing capacity being used in the network interface connected core for packet processing.
An example of a network server system for implementing this method is illustrated in Figure 1. The system 100 includes at least one central processor unit CPU 120 122 each having a plurality of cores 130 131 132 133, optionally the cores 130 131 within one processor 120 have a shared cache 145 as well as each core 130 131 having its own respective cache 140 141. For symmetric multiprocessing (SMP) the system memory 160 is shared such that any processor can access any data in system memory 160. A network interface 150 is provided for forming a connection between the server 100 and the network 110. The network interface 150 is connected to a predetermined CPU core 130 such that an interrupt will be created in the predetermined CPU core 130 for each data packet received from the network. The predetermined CPU core 130 distributes each data packet received from the network to a selected one of the plurality of processor cores 130 131 132 133 based on processor load where the packet is processed until it reaches the destination application 170 171 172 .
Advances in high speed networks and interfaces has lead to a situation where the network interface driver is capable of delivering packets to a processor faster than the processor has the capacity to process the packets through the network stack and deliver the data to the destination applications 170 171 172 . This is a problem in SMP network architecture where the network interface 150 is programmed to interrupt only one processor core 130 on receipt of a packet and all further processing of that packet occurs on the same core 130 until the data reaches the application. In this situation the CPU core 130 which receives the interrupt from the network interface 150 acts as a bottleneck, limiting the throughput of data to the applications.
Embodiment of the present invention alleviate this problem by relieving the CPU core 130 interrupted by the network interface 150 from packet processing as the packet is received by the network stack and by parallization of network stack operations. Thus the processing of received packets can be distributed to other CPU cores 131 132 133 in order for the network interface core 130 to be freed to receive more data from network interface 150.
The process for distributing each received packet 230 231 232 is illustrated in Figure 2. Each packet received from the network is initially handled in the network stack 210 of the network interface 150 connected CPU core 130. The received packet is then associated with an application connection 240 241 242 for a destination application 220 221 222 in a selected CPU core 130 131 132. An interrupt is created on the CPU core associated with the application connection for processing the received packet through the network stack 210 211 212 of the respective core 130 131 132.
When each packet is received from the network 110 by the network interface 150, an interrupt is generated to start the processing of the received packet through the network stack 210 of the network interface connected CPU core 130. The packet is associated with an application connection in a selected CPU core and the processing of the packet handed over to the selected CPU core as soon as possible, at the bottom of the network stack, after the packet is received.
The process of associating a packet with an application in a selected CPU core and handing over packet processing is illustrated in the flowchart of Figure 3. For example, the process of figure 3 is executed for data transmission, data reception, a new outgoing connection request or a new incoming connection request, however, the processing occurs at different levels in the network stack depending whether the an incoming or outgoing request is being processed. For incoming requests this process occurs at the bottom of the network stack for example in the data link layer or a layer just above, whereas for outgoing requests this process occurs at the top of the network stack for example in the session layer. The incoming or outgoing request 310 is received in the relevant layer of the network stack, it is then determined whether a new application connection is required 320 for the request.
Where a connection already exists, and the CPU core for the connection is known 330 then this known CPU is interrupted 350 and the processing handed over to this CPU core, such that a packet is associated with the connection and all further processing continues using this connection in the interrupted CPU core 360.
Applications connections may exist and can be handled in more than one core and in this situation the preferred CPU core for processing of data packets may not be known 330. If the preferred CPU core is not known then a CPU core is selected 340, from those where an application connection is established, based on the processor load and other preferred criteria. For example, for a send request it is preferable to establish the application connection in the CPU core being used for the application execution. If a preferred CPU core is not known, then a CPU core can be selected 340 based on processor load alone. An interrupt to associate data processing with the connection in the selected core is then created 350 and processing continues 360 in the selected core.
Where no connection is already established 320, for example for an outgoing request for a new connection, a CPU core for the connection is selected 340 based on processor load. An interrupt is then created 350 to establish the connection in the selected core and associate data processing with this connection then processing continues 360 in the selected core.
Once the selected core has been interrupted and processing handed over to this core in accordance with the established connection 360, then the processing of the incoming or outgoing request is completed 370 for the original core, which frees the original core to continue processing the next operation.
In a preferred embodiment a separate interrupt thread is created for each outgoing or incoming request for a connection. In this embodiment a processor core per connection technique is adopted and once the received packet is associated with the connection all further work is done by the software interrupt thread that is created on the selected CPU core for that incoming or outgoing request. This interrupt thread will do all further processing on the received or transmitted data until it reaches application or is transmitted to the network. The interrupt thread is utilized as early as possible so as to free up the network interface connected CPU core during an incoming request.
In the case of received packets, selection of a processor core for a connection occurs at the bottom of the network stack, for example in the data link layer or a layer just above during the connection establishment. The core is selected based on load, however preference can be given to select the core on which the destination application for the packet is running. This can provide advantages by minimising future interrupts and network stack processing handoffs, for example for data transmission and improve cache utilisation or minimise cache misses. Further, in a preferred embodiment affinity of the application is set with the selected core so that application is not rescheduled on another core.
A further advantage of the above method is minimisation of locking and associated wasted CPU cycles waiting for a lock. Where data is transmitted over the network by the application in conventional server architecture, the network stack has to acquire a spin lock in order to synchronize access to data structures and avoid contention between processing of incoming and outgoing packets. This can result in wasted CPU cycles spinning while waiting to acquire the lock.
Embodiments of the present invention alleviate the requirement for network stack spin locks, and the associated overhead, by the approach of selecting a processor core per connection. The selection of the CPU core and establishment of the connection for data transmission is performed at the top of the network stack. As is illustrated in Figure 2, once the processor core for a connection is identified, the packet processing for the outbound and inbound data for that connection occurs at the same interrupt priority level (IPL) on that core. This provides a natural synchronization for each connection through the network stack of the processor core. Thus relieving the network stack from locking which avoids wasting CPU cycles due to CPU spinning.
Variations in the core selection process and algorithm may be made depending on the server architecture. For example, in a hyper threading environment, a plurality of co-threads may be established for parallel processing though each CPU core, appearing as each co-thread being executed in a separate virtual CPU emulated in the CPU core. In this example, an embodiment can be implemented where the possibility of CPU contention by co-threads is minimised by keeping the co-thread CPU core free as much as possible. In this embodiment, during connection establishment a core other than the co-thread CPU core is preferentially selected.
Alternatively, in an embodiment suitable for architectures having a cache shared between partner cores, a partner core can be selected to leverage the benefits of the shared cache. For example, where the cache is shared then by handing off the received packet processing, as described above, to the partner core can minimise cache updates.
The selection of the processor core can be implemented using an algorithm integrated with the operating system (OS) scheduler to select the processor core based on software interrupt load on all CPU cores. As the number of network connections increases, thus it is desirable to evenly distribute stack processing across all the available processors cores.
The operating system schedules processes but does not schedule software interrupts. In an embodiment a software interrupt load detection algorithm is used to evenly distribute the load by identifying the CPU core with least software interrupt load which is coupled with OS scheduler and is used by the network stack to select a CPU core in which to establish a connection. The established connection is maintained on one CPU core or is migrated to a different CPU as the interrupt load changes. Thus, the total load is distributed among the plurality of cores enabling more data to be processed.
The decision of selecting a particular core for a connection depends on the least heavily loaded CPU core, which can also be identified as the CPU which spends the least amount of time in and above the software interrupt at which network stack is executed. An example of a mathematical model which can be used to find the optimal CPU core for each connection is described below.
The time spent at and above the software interrupt priority level (IPL) in which the network stack executes for sample n is represented as ∂_n.
Δ∂ _n represents the total time spent by the CPU core at and above software IPL during the time interval between n and n-1. $Δ \partial_{n} = \partial_{n} - \partial_{n - 1}$
The average of the Δ∂ over a period of time is represented as λ i.e. it is the average time the CPU core has spent in the required software IPL and above over a period of time. ${λavg}_{n} {cpu}_{x} = \frac{Δ \partial_{n} + {λavg}_{n - 1} {cpu}_{x} * (n - 1)}{N}$
The next kernel job will be triggered on cpu_x with the lowest λavg_ncpu_x . This means that next kernel job, to establish an application connection, is triggered on a CPU core which has spent the least amount of time on and above the software interrupt at which network stack is executed.
The deviation of the average time for each CPU core from the previous value is represented as Γcpu_x where ${[{λavg}_{n} {cpu}_{x} - {λavg}_{n - 1} {cpu}_{x}]}^{\land} 2 < Γ {cpu}_{x}$
This represents the deviation from the previous calculation. In order to minimize the overhead an embodiment uses deviation as a measure to decide whether the CPUs average is recalculated during the next sampling period. If the previous difference in λ is less than a threshold r, then the average calculation for that CPU is not done for twice the previous time interval $σ_{cpux} = 2 * σ_{cpux}$

where σ _cpux is the sampling frequency for CPUx.
Further, the total number of processors can be divided into an equal number of partitions and for each sample one of the set can be used for the average calculation. The λavg_ncpu_x is maintained in a sorted array representing the average interrupt load value for each CPU and is accessible for use by the network stack to determine the optimal CPU for a particular connection.

Example 1:

This example uses the results of proof of concept testing which compares the performance of an embodiment of the present invention implemented as a test model using SMP architecture as illustrated in Figure 5 with a test model implemented using conventional multi core server architecture comprising three CPU cores 430 431 432 433 as illustrated in Figure 4.
In the conventional SMP server architecture of Figure 4, all received packets 460 - 463 are processed through the network stack 410 of the LAN card 450 connected core 431, whereas sent packets (not shown) are processed through the respective network stack of the core 430 431 432 433 on which the application 420 421 422 423 is executing, requiring spin locking. In the server of Figure 4 received packets 460 461 462 463 are sequentially processed 440 441 442 443 through the network stack 410 of the LAN connected CPU core 431 until they reach their respective destination applications 420 421 422 423 and the data is used by the application on the core in which it is scheduled.
In the proof of concept distributed stack architecture of Figure 5, received packets 560 - 563 are associated with respective connections 540 541 542 543 and processing handed off, where necessary, from network stack 511 in the network interface 550 connected core 531 to the respective network stacks 510 511 512 513 for processing until reaching the respective destination application 520 521 522 523. Sent packets are associated with the application connections 540 541 542 543 and processed through the respective network stack 510 511 512 513 of the respective CPU core 530 531 532 533 and are sent via the network interface 550.

Table 1 provides experimental results for the distributed network stack of Figure 2 compared with that of Figure 4.

Table 1: Effective distribution of software interrupts among all CPUs, Spin wait and CPU utilization

Stack	CPU Spinwait %	% Spinwait in Transport layer	Average Throughput %	system utlization %	Total Interrupt Event/Sec	CPU	0	CPU 1	CPU 2	CPU 3
						(IPL event)	LAN CPU	IPL events
							IPL event		IPL events
Conventional Stack	33	40	43	15	1162	36.4	1121	.1	5.3
Distrbuted network stack	23	0	46	17	1457	105	1146	106	100

As can be appreciated by a person skilled in the relevant art the distributed stack architecture of figure 5 achieves concurrency in the transport layer by placing a connection in each CPU core 530 - 533. The network interface connected CPU core 531 is been relieved from packet processing, as is demonstrated by the results of Table 1 by distribution of the interrupts among all CPU cores 530 - 531. The results show a 10% reduction in CPU spin wait state.
The throughput achieved for the distributed stack architecture is at least comparable to the conventional network stack architecture and, as illustrated in Figure 6, the distributed stack architecture 610 starts performing better than the conventional architecture 620 as the number of connections and data volume increases.
The CPU utilization of the distributed stack architecture is marginally higher than that of the conventional architecture due to overhead involved in creating the interrupt thread in the CPU associated with each connection. The distributed stack architecture test model foreshadows more potential once concurrency is achieved in all layers.

Claims

A method of processing a data packet received from a network (110) in a multi core processor platform (100), the multi core processor platform (100) comprising a network interface (150) connected to a first central processor unit (CPU) core (130), the method comprising the steps of:
receiving an interrupt in the first CPU core (130) from the network interface for processing a data packet (230) received from the network (110);

selecting a second central processor unit (CPU) core (131, 132) based on processor load and establishing an application connection in said second CPU core (131, 132), the selecting and establishing steps being performed at a data link layer level of a network stack (210) of the first CPU core (130);

associating the data packet (230) with the application connection established in said second CPU core (131, 132);

creating a separate interrupt thread on said second CPU core (131, 132) associated with the application connection for processing the received data packet (230);

handing off handling of the data packet to a network stack (211, 212) of said second CPU core (131, 132); and

handling the data packet at levels above the data link layer level in the network stack (211, 212) of said second CPU core (131, 132).
A method of processing a data packet being sent to a network by an application executing in one central processor unit (CPU) core of a multi core processor platform, the multi core processor platform comprising a network interface accessible by each CPU core for transmitting data packets to the network, the method comprising the steps of:
determining whether to perform processing of the outgoing data packet using the CPU core executing the application or using an alternative central processor unit (CPU) core;
and where it is determined to use an alternative CPU core:

selecting (340), at a session layer level of a network stack of the CPU core executing the application, an alternative CPU core for outgoing packet processing based on processor load;

establishing, at said session layer level, an application connection in said alternative CPU core;

associating the data packet with the application connection;

creating (350) an interrupt thread on the selected alternative CPU core;

handing off (350) handling of the data packet to a network stack of said alternative CPU core; and

handling (360)the data packet at levels below the session layer level in the network stack of said alternative CPU core so as to process the data packet to the network interface for transmission over the network.
A method as claimed in claim 1 or claim 2 wherein the CPU core is selected based on processor interrupt load for each CPU core.
A method as claimed in claim 3 wherein the CPU core is selected giving preference to a CPU core sharing cache memory with a CPU core performing an operation related to the received packet.
A method as claimed in claim 3 wherein the multi core processor is adapted to enable hyper-threading wherein each CPU core emulates two virtual CPUs enabling a co-thread to be executed in each virtual CPU of the CPU core and wherein a virtual CPU for establishing the application connection is selected in a CPU core selected giving preference to a non co-threaded CPU core.
A method as claimed in claim 3 wherein the processor interrupt load is monitored by an operating system scheduling function.
A method as claimed in claim 2 further comprising the step of associating affinity for the application with the selected CPU core.
A network server system (100, 200) comprising:
at least one central processor unit (CPU - 120) having a plurality of cores (130, 131); and

a network interface (150) for connecting to a network (110), and adapted to create an interrupt in a first CPU core (130) connected to the network interface (150) for processing each data packet received from the network (110),
wherein:

on receipt of a data packet from the network (110) the network server system (100, 200) is adapted to select a second CPU core (131) and establish an application connection at data link layer level of a network stack (210) of the first CPU core (130);

each data packet received from the network is associated with an application connection established in a selected CPU core;

a CPU core is selected based on processor load;

a separate interrupt thread is created on a selected CPU core associated with the application connection for processing the data packet; and

the first CPU core (130) hands off handling of a received data packet to a network stack (211) of a selected CPU core (131), further handling of the received data packet at levels above the data link layer being performed in the network stack (211) of the selected CPU core (131).
A network server system (100, 200) comprising:
at least one central processor unit (CPU -120) having a plurality of cores (130, 131); and

a network interface (150) for forming a connection to a network (110), and adapted to create an interrupt in a CPU core (130, 131) connected to the network interface (150) for processing each data packet received from the network (110), and further adapted such that each CPU core can access the network interface (150) for transmitting data packets to the network (110),
wherein:

an execution CPU core executes an application;

on transmission of a data packet to the network (110) from the application, the network server system (100, 200) is adapted to select an alternative CPU core (131) to the execution CPU core and establish an application connection at session layer level of a network stack (210) of the execution CPU core;

each data packet being sent to the network is associated with the application connection;

an alternative CPU core is selected based on processor load; and

an interrupt thread is created on an alternative CPU core associated with the application connection for processing the data packet; and

the execution CPU core hands off handling of a data packet to be transmitted to a network stack of an alternative CPU core, further handling of said data packet at levels below the session layer being performed in the network stack of the alternative CPU core so as to process the data packet to the network interface (150) for transmission over the network (110).
A network server system (100, 200) as claimed in claim 8 or claim 9 wherein the alternative CPU core is selected based on processor interrupt load for each CPU core.
A network server system (100, 200) as claimed in claim 10 wherein the alternative CPU core is selected giving preference to a CPU core sharing cache memory with a CPU core performing an operation related to the received packet.
A network server system (100, 200) as claimed in claim 10 wherein the processor interrupt load for each CPU core is monitored by a operating system scheduler function.
A network server system (100, 200) as claimed in claim 10 wherein the multi core processor is adapted to enable hyper-threading wherein each CPU core emulates two virtual CPUs enabling a co-thread to be executed in each virtual CPU of the CPU core and wherein a virtual CPU for establishing the application connection is selected in a CPU core selected giving preference to a non co-threaded CPU core.