JP2003018188A - Flow architecture for remote high-speed interface application - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem of there being the possibility that a serial queue, such as a PCI bus interface, is turned to be a bottleneck resulting in the latently annihilating benefits on performance for use of a high-speed network switching interface. SOLUTION: This system having an inter-remote bus high-speed switching interface is provided with a switching mechanism, to which a plurality of bus interfaces are connected. A programmable flow queue, having a plurality of parallel logical flow queues, is used for scheduling a packet in accordance with the protocol request of a remote bus interface.

Description

Detailed Description of the Invention

[0001]

FIELD OF THE INVENTION The present invention relates generally to methods and systems for managing communication networks,
In particular, it relates to a switching flow control mechanism in a communication network. More particularly, the present invention relates to methods and systems for replacing conventional prioritized output queuing with logical flow control mechanisms. This logical flow control mechanism is implemented inside a switch mechanism that can be used as a remote bus interconnect so that data ordering and associated flow control can be handled simultaneously.

[0002]

2. Description of the Related Art Conventionally, slow network speed and transmission control protocol / Internet protocol (TCP)
Flow control provided by higher layer protocols such as / IP) have circumvented flow control problems at the switch. With the recent development of inter-node flow control and data transmission media in communication networks, congestion in bridges or switches has become a greater problem. A flow control mechanism has been required in response to the recent increase in network speed of 1 Gbit / sec corresponding to the full-duplex transmission function. Such flow control goals include efficiency and fairness.

Server input / output (I / O) is the next generation I / O.
It is evolving in a direction defined by new initiatives such as O (NGIO), Future I / O (FIO), and System I / O (SIO). NGIO,
FIO and SIO have been integrated into the InfiniBand architecture by the InfiniBand consortium. These I / O adapters require a switch mechanism to interconnect the host channel adapter (HCA) and target channel adapter (TCA).

Like the PRIZMA switch structure made by International Business Machines (IBM),
Many conventional switching schemes use programmable packet flow priority schemes. At initialization, a value from 1 to n can be set as the number of available priorities. Therefore, such a switch is capable of supporting up to n priority levels with an n-queue prioritization scheme. A flow control mechanism can be used to control the flow of a plurality of packets having different priorities at the input / output unit of the switching mechanism. On-chip pins may be used to implement input flow control in such a switch fabric, or receive-in the header of each packet.
grant) information may be used. Flow control at the output of the switch fabric can be realized by the send enable pin.

In such a flow control system with priority, if a certain priority is disabled, all lower priorities will also be disabled.
The n queues corresponding to the n priority levels are served by the following priority rules. That is, a packet waiting in a high priority queue will be forwarded before any lower priority queue.

[0006]

The flow control thresholds and back pressure triggers in conventional switch fabrics rely on priority-based queuing to provide cumulative output queue sizes for individual queue sizes. Was used. Therefore, whenever a higher priority was disabled, all lower priority traffic was guaranteed to be disabled. Serial queues such as the PCI bus interface can become a bottleneck that can potentially diminish the performance benefits of using a high speed network switching interface. PCI is a well known 32/64 bit local bus standard designed for fast access to peripherals such as user displays, disk drives, modems in personal computers.

Appropriate buffer management must be implemented to prevent buffer overflow or underflow within the switch due to physical bus interface interrupts in the switch I / O. Serial queues such as the remote PCI bus interface become a potential bottleneck for flow control. Therefore, it will be appreciated that there is a need for a system and method that maintains parallel flow control and data ordering mechanisms such that remote bus interfaces can be remotely switched utilizing the queuing capabilities of the switches.

[0008]

A system having a high speed switching interface between remote buses is disclosed below. The system includes a switch mechanism to which multiple remote bus interfaces are connected. A programmable flow queue, including multiple parallel logical flow queues, is used to schedule packets according to the protocol requirements of the remote bus interface. That is, according to the present invention, in a device for providing a high-speed switching interface between remote buses, a switch mechanism, a remote bus, a remote bus interface for interfacing the remote bus with the switch mechanism, and a remote bus interface. And a programmable flow queue for scheduling packets according to the protocol requirements of. The programmable flow queue is embedded in the switch fabric and packets arriving at the switch fabric from the remote bus interface may be routed to the programmable flow queue before exiting the switch fabric. The remote bus interface may also be an interface that interconnects remote peripheral components.
In addition, the remote bus has a bus architecture for determining the order of data among the remote buses, and
The programmable flow queue may also include data ordering logic for scheduling packets according to the bus architecture. here,
The programmable flow queue may include multiple parallel logical flow queues. The data ordering logic may characterize multiple parallel logical flow queues according to the bus architecture. The programmable flow queue may further include logic for individually programmable adjustment of queue threshold requirements for each of the plurality of parallel logical flow queues.
The switch fabric has a source port for receiving an incoming packet from a remote bus, and the device further provides a bus architecture instruction for routing the incoming packet to a programmable flow queue according to the above bus architecture. It may also include a transaction work queue to hold. The present invention also provides a method for providing a high-speed switching interface between remote buses via a switch mechanism, wherein the switch mechanism interfaces a remote bus and schedules packets according to a protocol request of the remote bus interface. It also provides a method. The objectives, features and advantages of the present invention will be apparent from the detailed description that follows. It should be noted that, although the novel points that are considered to be the features of the present invention are listed in the scope of claims, a detailed description of the following embodiments will be given with respect to the preferred form of use, another object and advantages thereof in addition to the invention itself. It will be best understood by reference in conjunction with.

[0009]

BEST MODE FOR CARRYING OUT THE INVENTION The present invention replaces the conventional priority scheme for in-switch flows with a logical control architecture, where n conventional priority queues are replaced with n logical flow queues. . Disclosed is a configuration of a switching mechanism having a large number of flows per port to support implementations such as PCI-to-PCI switching. The present invention modifies a conventional switch mechanism such as found in the newly developed PRIZMA switch (manufactured by IBM) to enable it to efficiently support many different traffic flows.

Such a new output queue architecture allows for individual service and flow control for multiple logical flows. Such a flexible queuing architecture is required, for example, for implementations where PCI bus ordering instructions must be adhered to. Each logical flow has a queue (ie, logical port) on a physical output port, similar to a conventional priority queue. These logical queues can be individually and independently enabled and disabled for transmission. Unlike conventional priority queues, these logical flow queues are not programmed with a unique, mutually independent ordering scheme. The physical output port provides the output of the logical flow queue in a programmable determined order according to the required processing. For example, output flow from a collection of logical flow queues to physical output ports may be done in a round robin fashion, and in the case of PCI bus applications, flow scheduling was established by the rules of PCI bus command ordering. Will be programmed to fulfill the requirements.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Referring to the drawings below,
The same parts and corresponding parts are given the same reference numbers. In FIG. 1, a communication switch 100 (sometimes referred to as a switch mechanism "switch fabric")
A flow architecture according to the present invention is implemented therein. In the illustrated embodiment, the switch 100 is
Packet-based 16x1 with 16 bidirectional ports
6 switching units. 1 corresponding to 16 output ports 106 to simplify the port bidirectionality
6 input ports 102 are shown. Switch 100 is PRI
It may be a ZMA switch, with 2 ports in each direction.
It is possible to provide bandwidths of Gbps or higher. Such port speeds make switch 100 particularly well suited to provide the bandwidth required by modern high speed networking I / Os.

As further shown in FIG. 1, switch 100
Includes a shared packet memory module 104. The memory module 104 may be any of a variety of computer memory devices. A plurality of input ports 1 as described in more detail with reference to FIGS. 2 and 3.
Input packets coming in from one of the 02 are programmable. In this regard, in the illustrated embodiment, the packet size is 32 bytes and the packet includes a 3-byte header and data field. Destination port (one of the plurality of output ports 106
Are defined in the packet header. By preparing the destination port address in the bitmap format, multicasting from any output port 106 is possible. The switch 100 has programmable packet transmission (queuing) implemented within a set of logical flow queues. The set of logical flow queues includes a logical flow queue 108 associated with output port 0, a logical flow queue 110 associated with output port 1, a logical flow queue 112 associated with output port 15, and the like.

Message Structure FIG. 2 shows the internal structure of a switch packet according to the preferred embodiment of the present invention. As will be described in detail below, the switch 100 constructs a transaction packet 200 in response to a service or access request from the remote bus. As shown in FIG. 2, the transaction packet 200 includes a transaction header 208. The transaction header 208 is the switch 10
Contains routing information about the switch internal route to direct the transaction packet through 0 to the correct output (target) port. The transaction packet 200 further includes a transaction payload 206. The transaction payload 206 contains the message to be interpreted and executed by the target bus interface logic (not shown). In PCI-to-PCI switching, such a bus interface logic is
It would be a target bus PCI sequencer.

The original message in transaction packet 200 includes message header 204 and message payload 202. The message header 204 contains transaction specific information (eg command, address, byte enable) and the message payload 202 contains up to 32 bytes of data.
Messages with payload data less than 32 bytes may be tagged inside the transaction header 208. This tag is used by the target bus interface logic (sequencer) described above to generate a series of memory write commands to transfer a single or more data words.

Queue Structure A logical port adapter 300 is shown in FIG. The adapter is for queuing packets transferred from an input port to a physical output port according to the preferred embodiment of the present invention. One of the key features of the queuing system and method of the present invention is the flexible routing and output queuing, which is applicable to any particular remote bus architecture. is there. For simplicity of illustration and description, the following drawings are treated as a PCI-to-PCI switching configuration.

The logical port adapter 300 is implemented in the switch structure (sometimes referred to as a switching core) of the switch 100, which performs output port queuing. Therefore, all logical flow queues 108-112 have outputs directed "downstream" of switch 100. FIG. 3a shows the configuration of the logical flow queue implemented in the switch core. Specifically, the logical flow queue Q1,
Q2, Q3, and Q4 form a programmable flow queue block 302. An output port 310 is associated with the flow queue block 302, and a remote bus B is associated with the output port 310. Similarly, logical flow queues Q1 ', Q2', Q
3'and Q4 'comprise a programmable flow queue block 304, which is associated with an input port 308 of remote bus A. In the illustrated embodiment, buses A and B may be bidirectional, and thus may simultaneously serve as a source and a destination. Although only two buses are depicted in FIG. 3, the logical port adapter 300 is also applicable to configurations having a single or more source buses and a single or more destination buses.

Transaction packets, such as transaction packet 200, that are to be transferred to bus B are sent to output port 310 via programmable logic flow queue block 302.
The queue assigned to each type of transaction is determined according to the following criteria. First, transactions passing through the system must meet the data ordering rules used by one or both of Bus A and Bus B. Secondly, moving a large number of transactions in independent parallel paths will result in better flow and buffering efficiency. FIG. 4 illustrates a bus architecture specification rule that may be implemented within the logical adapter of FIG. 3 in accordance with the first criteria above regarding queue allocation. In this example, the packet data passing through the logical port adapter 300 satisfies the data ordering rule described in FIG. In the figure, the row-> column shows the allowable transfer. The specification rules shown in this figure will be used to describe the queuing method and command operation in the embodiments shown in FIGS. 5 and 6 below.

Command Operation FIG. 5 illustrates a switch architecture 400 according to the present invention.
Indicates. Switch architecture 400 is a bi-directional PC
It includes I-buses A and B and their corresponding PCI interfaces 404 and 418. Switch architecture 400 further includes a programmable flow queue 412 associated with the output port of bus B and a programmable flow queue 410 associated with the output port of bus A. Programmable flow queues 412 and 410 are similar to programmable flow queues 302 and 304 in that they include multiple logical flow queues. Further, as shown in FIG. 5, the switch architecture 4
00 is a transaction work queue (TWQ) 406
And TWQ416. These TWQs handle I / O packets to be transferred between PCI interfaces 418 and 404 via programmable flow queues 410 and 412. The following describes the high level configuration of the switch architecture 400 used in bidirectional command flow and data.

A Notification Memory Write In response to receiving a PCI memory write (MW) access request for a prefetchable address space within the PCI interface 404, and if memory space within the TWQ 406 is available, the PCI transaction information is within the TWQ 406. Loaded in. If there is no space available in TWQ 406, the transaction is retried until successfully placed in TWQ 406. Next, the logic inside the TWQ 406 constructs a transaction packet and sends the packet to the designated transaction queue Q1. Queue Q1 is designated as the notify memory write (PMW) queue for target bus B, as shown in FIG.
The packet is thus routed to the target bus B as described in the transaction header of the transaction packet.

PCI interface 404 or 4
Each packet entering 18 has a sequence number in its message header. The sequence number associates the PCI write transaction with the PMW packet it produces. A second sequence number is retained for each source bus. This additional sequence number is incremented each time a write transaction is performed on the source PCI bus interface. The last PMW in the sequence associated with a single PCI write operation on the source bus also has the final PWM sequence index set to one. Source bus read transaction, PM on the same bus
The read transaction, which is transmitted in the same direction as the W transaction, retrieves the sequence number from within the message header inside the TWQ 406. After arrival on the target bus B, the read transaction waits for all PMW transactions with a sequence number equal to or smaller than its own sequence number, after which the read transaction is allowed to process. Thus, in non-serial transfer, the PC
Data ordering for the I interface is maintained.

When the transfer is completed on the source bus A and all the data in the data buffer (buffer 408) related to the transaction is packetized and transmitted to the switch device, the transaction information is deleted from the TWQ 406. b Delayed Read Request and Delayed Read Completion Switch architecture 400 provides PCI I / O read request, configuration read, memory read (M
R), MRL or MRM. The transaction information encoded with the input packet for the read request is compared to the transaction information of the transaction currently being processed within TWQ 406. If matched, switch architecture 400
Inside, there will be an active delayed read request (DRR) for the transaction associated with that input packet.

In the case of DRR active (match detection), the data available flag is checked to determine if the data for the transaction is currently in the data buffer 408. If such transaction data is in the data buffer 408, it is transferred to the requesting master. If there is no such transaction data in the data buffer 408, the transaction is retried. If the transaction data encoded in the input packet (transaction request) does not match any transaction currently being processed within TWQ 406, the response of switch architecture 400 is TW.
It depends on the current availability of the queues in Q406.
Work queue not available and data buffer 4
If there is no available buffer space inside 08 (switch busy state), the input transaction request (read request) will be retried later. TWQ406
When the work queue is available internally or the buffer space is available inside the data buffer 408, the PCI transaction request is executed by the control information (command address, byte enable) related to the transaction in the TWQ 406. Once saved, the transaction is retried.

When transaction control information is added to the TWQ 406, the switch architecture 400 forms a transaction packet and transfers the packet to the delayed transaction queue (delayed read request Q2) of the destination bus B. A sequence number from the PMW sequence is assigned to this queued transaction packet. The transaction packet is then routed by the logic inside the programmable flow queue 302 to the target bus B as described in the transaction packet header. The message payload for DRR is empty. TW to copy the message header
It is held in Q406 and a memory space is allocated inside the data buffer 408. Both of these resources are utilized when the associated DRC is received from the target bus B. Additional PCI read commands may be serviced until one of the conditions listed below is detected. (1) All of the space in one or both of the TWQ 406 and the data buffer 408 has already been allocated. (2) The Q2 (DRR) queue for the target bus B is full (notified by the switch core reception permission).

When the DRR reaches the input of Q2, the sequencer logic in the programmable flow queue 302 determines the sequence number of that DRR and the sequence number carried by the PMW already arriving at the target and the "last in sequence.""PMW" flag. When all PMWs have finished as requested and all write data has been sent to the target, the sequencer logic initiates a request on target bus B to read one cache line from memory. When the data is received from the target (memory in this case), a DRC packet is constructed and transferred to Q3 (DRC / DWC queue) associated with the source bus A. When the PCI command is MRM, the sequencer logic in the programmable flow queue prefetches the data on behalf of the PCI master by initiating a request on the bus to read an additional cache line from memory. The target bus sequencer continues data prefetching as long as the master holds the PCI transfer active. When a DRC packet is received on the start bus in Q3, the packet is dropped when it reaches the head of the queue. The information in the message header is matched to the assigned TWQ 406 and the data in the message payload is already allocated in the data buffer 408.
Moved in.

Target bus B interface 418
The logic associated with can not send data to source bus A unless the data buffers in source bus A have adequate space. The prefetch logic in the target bus B interface 418 negotiates with the source bus A logic 404 for the time when additional data can be sent to the buffer in the source bus A interface 404. The priority queue Q4 is used as a highly reliable medium, and the negotiation message is transferred via Q4. FIG. 6 illustrates a preferred embodiment of the present invention.
FIG. 6 is a block diagram of programmable packet output queuing. As shown in FIG. 6, the switching interface 500 includes a programmable flow queue 504.
And the queue 504 includes a scheduler 502.
Is communicating with. The programmable flow queue has a plurality of parallel logical flow queues, as defined in FIGS. 3 and 4, for example. As shown in more detail in FIG. 6, queue threshold input 506 and data ordering input 508 are set in scheduler 502. Programmable logic in the scheduler translates the inputs from the queue threshold input 506 and the data ordering input 508 into scheduling commands for the programmable flow queue 504, so that the packets output from the switching interface 500 are protocol requirements. And according to the real-time queue status.

The preferred embodiment of the present invention includes the embodiment as a computer system programmed to carry out the various methods described herein, and as a program product. When implementing the present invention by a computer system, the instructions for implementing the disclosed method and system will be located in a storage device such as a ROM or RAM of a single or multiple computer system. Until receiving a request from the computer system, the instructions are stored in another computer storage device, for example, a disk drive (which may be a removable optical disk or floppy (R) used by being inserted into the disk drive as needed). Disk) and will be stored as a computer program product. While the present invention has been shown in a particular form and described with reference to preferred embodiments, those skilled in the art can make various changes in form and details without departing from the concept and scope of the present invention. It is easy to understand what can be done.

[Brief description of drawings]

FIG. 1 illustrates a communication switch in which a flow architecture according to the present invention is implemented.

FIG. 2 is a diagram showing a packet structure in a switch according to a preferred embodiment of the present invention.

FIG. 3 illustrates a logical port adapter for queuing packets to physical ports according to a preferred embodiment of the present invention.

4 is an illustration showing bus architecture rules that may be implemented within the logical port adapter shown in FIG.

FIG. 5 is a block diagram of a switch architecture according to a preferred embodiment of the present invention.

FIG. 6 is a block diagram illustrating programmable packet output queuing according to a preferred embodiment of the present invention.

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Metin Idomia             United States 27713, North Carolina             Saddlewood Court, Durham, County 24 (72) Inventor Andrew John Lindos, Third             United States 27516, North Carolina             State, Chapel Hill, Calderon Sutri             Heart 204 (72) Inventor Georges Roland Rodriguez             United States 27511, North Carolina             State, Cary, Evert Drive 404 F term (reference) 5B061 FF00                 5K030 GA01 HA08 HB13 HB14 KA05                       KA13 KX02 KX11 KX18 KX24                       LC01

Claims (13)

[Claims] 1. A device for providing a high-speed switching interface between remote buses, a switch mechanism, a remote bus, a remote bus interface for interfacing the remote bus to the switch mechanism, and a protocol of the remote bus interface. A programmable flow queue for scheduling packets according to requirements. 2. The programmable flow queue is embedded in the switch fabric and packets arriving at the switch fabric from the remote bus interface are sent to the programmable flow queue before leaving the switch fabric. The device of claim 1, wherein the device is routed to. 3. The remote bus interface comprises:
The apparatus of claim 1, which is an interface for interconnecting remote peripheral components. 4. The remote bus has a bus architecture for determining the order of data between the remote buses, and the programmable flow queue comprises:
The apparatus of claim 3, further comprising data ordering logic for scheduling packets according to the bus architecture. 5. The apparatus of claim 1, wherein the programmable flow queue comprises a plurality of parallel logical flow queues. 6. The apparatus of claim 5, wherein the data ordering logic characterizes the plurality of parallel logical flow queues according to the bus architecture. 7. The apparatus of claim 6, wherein the programmable flow queue further comprises logic for individually programmable adjustment of queue threshold requirements for each of a plurality of parallel logical flow queues. 8. The switch mechanism has a source port for receiving input packets from the remote bus, and the device further follows the bus architecture to route the input packets to the programmable flow queue. The apparatus of claim 4, including a transaction work queue that holds bus architecture instructions for routing. 9. A method for providing a high-speed switching interface between remote buses via a switch mechanism, the method comprising: interfacing a remote bus by the switch mechanism, wherein a packet is transmitted according to a protocol request of the remote bus interface. How to schedule. 10. The switch fabric includes a programmable flow queue, further comprising the step of routing the input packet to the programmable flow queue before the input packet exits the switch fabric.
The method according to claim 9. 11. The remote bus has a bus architecture for determining a data order across the remote bus, and the method further comprises scheduling packets according to the bus architecture. the method of. 12. The programmable flow queue is
11. The method of claim 10, comprising a plurality of concurrent logical flow queues, further comprising adjusting queue threshold requirements for each of the plurality of concurrent logical flow queues independently. 13. The switch mechanism of claim 10, wherein the switch mechanism has a source port for receiving input packets from the remote bus and routes the input packets to the programmable flow queue according to the bus architecture. The method described.