HK1250539B - Method and system for balancing storage data traffic in converged networks - Google Patents

Description

Method and system for balancing storage data traffic in a converged network

Technical Field

The present invention relates to methods and systems for enabling a computing device coupled to a network to access a storage device coupled to the network via an adapter, and to devices for implementing such methods and systems. In some embodiments, the present invention relates to balancing (e.g., in an attempt to optimize) storage data traffic in a system in which computing devices (servers) coupled to a network operate to access storage devices also coupled to the network via an adapter.

Background Art

In the past, data centers typically implemented two distinct network infrastructures: a data communications network (typically Ethernet-based) and a separate "storage" network for accessing storage devices. Typical storage networks implemented the conventional Fibre Channel protocol. The expressions "data communications network" and "data network" are used synonymously herein to refer to a network of a different class from the "storage network," in the sense that the storage network is configured and used to primarily carry "storage data" traffic (where "storage data" refers to data retrieved from or to be stored on at least one storage device), while the data network is configured and used to primarily carry other data traffic (i.e., data other than storage data).

However, implementing multiple network types (eg, separate data and storage networks) undesirably increases the capital and operating costs of operating a data center.

Recently, many data centers have begun investigating (and some have already begun using) the use of a single network to carry storage data traffic and other (non-storage data) traffic. Such a single network will be referred to herein as a "converged network." An example of a converged network is an Ethernet-based network in which all traffic is sent between servers coupled to the network and storage devices coupled to the network (via adapters). Unfortunately, the two types of network traffic to be sent over the converged network (storage data traffic and other data traffic) have different characteristics.

To carry traffic other than storage data traffic, data networks (e.g., those implementing Ethernet using the Internet Protocol) can (and therefore are typically) implemented as unmanaged or minimally managed networks. This makes it easy to add and remove computers and other hardware from the data network. For example, the DHCP protocol can typically provide a new device with all the information it needs to operate on the data network (without human intervention).

However, network loops can cause serious problems in data networks (i.e., the continuous forwarding of packets that should be discarded). For this reason, data networks often implement some kind of protocol (e.g., the Spanning Tree Protocol) to ensure that only one path is known between any two devices on the data network. Redundant data paths are rarely explicitly established on data networks. Furthermore, traffic on data networks is relatively unpredictable, and applications are often written to tolerate any available bandwidth on the data network.

In contrast, storage networks are typically managed networks. Network administrators typically manually assign which computers can communicate with which storage devices on the storage network (i.e., there is typically no self-configuration). There has been little progress in making network connections (implemented as separate storage networks from data networks) adaptable to changing conditions. Furthermore, to provide the high availability and fault tolerance levels typically required for low-level data storage, fully redundant paths typically exist between storage devices (coupled to the storage network) and computers.

Due to the differences between storage networks (and their storage data traffic) and data networks (and their non-storage data traffic), combining both storage data traffic and other traffic in a converged network can result in uneven network utilization, which can degrade the overall performance of applications in the data center. Typical embodiments of the present invention address this uneven utilization of converged networks, for example, to allow applications in the data center to approach the maximum performance available.

The following definitions apply throughout this specification, including in the claims:

"Storage device" means a device (e.g., a disk drive) configured to store and retrieve data. Typically, a storage device is accessed using a logical block address (LBA) and a number of blocks. A logical block is a fixed-size chunk of total storage capacity (e.g., 512 or 4096 bytes). A traditional rotating disk drive is an example of a storage device;

"Server" means a computing device configured to access and use storage devices across a network (converged network) for the purpose of storing and retrieving data (e.g., files and/or applications);

"Adapter" means a device configured to connect a storage device or a storage system including two or more storage devices (e.g., JBOD) to a network (e.g., a converged network). In typical embodiments of the present invention, each storage device is typically accessible to a server via two or more adapters to provide fault-tolerant access to data stored on the storage device.

"Interface" means a component of a server or adapter that connects a device (server or adapter) to a network (e.g., a converged network). Examples of interfaces are physical devices (i.e., network interface controllers (NICs)) and software-defined wrappers for multiple NICs (for link aggregation). In an exemplary embodiment of the present invention, an interface is a hardware or software element that has its own Internet Protocol (IP) address in a converged network;

"Agent" means a software or hardware component or subsystem of a server (or adapter) that is configured to run on a server (or adapter) during operation of the server (or adapter) to switch (or prepare to switch) storage data traffic on a network (e.g., a converged network). In some embodiments of the present invention, not all servers and adapters on a converged network have agents. However, coupling non-participating servers and/or adapters (servers and/or adapters without agents) to the network may limit the degree of equalization that can be achieved (according to embodiments of the present invention); and

A "data path" refers to the path for sending data between a storage device and a server via an adapter using one interface on each of the adapter and the server (i.e., a path from the storage device through the adapter interface and through the server interface to the server, or a path from the server through the server interface and the adapter interface to the storage device). In an IP network, a data path can typically be represented by a combination of the IP address of the server interface and the IP address of the adapter interface, and optionally, by the port number used at the adapter. However, in the case of link aggregation, the complete path will depend on the actual interface used for the path within the group of interfaces bound to one IP address.

When a storage system (e.g., a JBOD) comprising two or more storage devices is coupled to an adapter and both the adapter and the server are coupled to a converged network, it is envisioned that the server (in order to access the storage devices of the storage system) will typically specify (i.e., be configured to use) a particular storage device of the storage system (e.g., one disk drive of the JBOD) and a data path between the server and the storage device. According to typical embodiments of the present invention, the data path may change from time to time in order to balance the storage data traffic on the network. According to some embodiments of the present invention, the data path (between the server and the storage system) may change from time to time in order to balance the storage data traffic on the network (and, according to the present invention, the adapter's selection of a particular device of the storage system to be accessed by the server may change from time to time, but such change will not necessarily be deterministic).

Generally speaking, when storage data traffic is combined with other data traffic on a converged network, the attributes of the different types of traffic may combine to produce inefficient use of the network's overall bandwidth, thereby limiting the performance of the data communication traffic and/or the storage traffic.

For example, modern server computers typically include two or more 1 Gbps or 10 Gbps network interfaces (referred to herein as "interfaces" in the context of servers being connected to a converged network). Many such servers run software packages (e.g., the Hadoop open source software package) that allow large numbers of servers to work together to solve problems involving large amounts of data. However, such software (e.g., Hadoop) typically requires that each server have a unique name and address. Consequently, data communication traffic between servers running the software (e.g., Hadoop) will typically use only one of the two (or more) network connections available on each server.

In contrast, storage data traffic is typically configured with redundant paths between servers and disk drives to survive failures in any of these components. These redundant paths can be used to redirect storage data traffic (e.g., by spreading it across network interfaces) to avoid network interfaces being occupied by data communication traffic (non-storage traffic). However, standard mechanisms for implementing this redirection (e.g., multipath I/O or "MPIO" approaches) incur significant performance penalties for storage data traffic on converged networks. Specifically, typical storage data load-spreading mechanisms are based on sending storage commands across all available interfaces in a round-robin fashion or determining some measure of how much work is outstanding on each link (e.g., the number of outstanding commands, the total number of outstanding bytes, or some other measure) and sending commands to the "least busy" interface. These mechanisms impose significant performance penalties on storage data traffic between servers and disk drives because, to achieve maximum performance, commands executed by the disk drives must be directed to consecutive locations on the disk. If commands are not sent to access consecutive locations, a "seek" operation is required to move the disk drive's read/write head to the new location. Each such seek operation will typically reduce overall performance by about 1% or more. Conventional dispersal mechanisms (round-robin or "least busy" dispersal) increase the number of seeks required to execute a sequence of disk access commands because they frequently cause consecutive commands in the sequence to take different paths from the server to the disk drives. Because the different paths will have different processing times and latencies (due to the other operations on each path), commands issued in one sequence will typically be executed in a different order. Each reordering will cause a seek and, thereby, reduce overall data carrying capacity. It has been observed that when these conventional dispersal mechanisms are applied to Hadoop storage operations, they reduce the overall performance of storage data traffic by about 75% (i.e., the amount of storage data that can be transferred is about 25% of the amount that would be possible without the use of round-robin or least busy mechanisms).

Another conventional technique, known as "link aggregation", is sometimes applied to split traffic between a first device (typically a server) and a second device (typically another server), between a group consisting of all interfaces available for coupling the devices to a network, the first device having a plurality of interfaces available for coupling the device to the network, and the second device also having a plurality of interfaces for coupling the devices to the network. According to link aggregation, in order to achieve a kind of load balancing, a new selection of one of the interfaces of the first device and one of the interfaces of the second device is made (e.g., in a random or pseudo-random manner) before each new stream of data values (i.e., each new sequence of data values that will not be transmitted out of order) is transmitted from a selected interface of one of the devices to a selected interface of the other device over the network. This allows the data communication traffic (averaged over many streams) to use all available interfaces and maintain a rough balance between the amount of data sent on each interface (unless one interface fails).

Conventionally, it is not recommended to perform link aggregation in order to transmit storage data over a network. However, even if some form of link aggregation is used (contrary to conventional recommended practice) in an attempt to balance the storage data traffic occurring between multiple interfaces of a server and multiple interfaces of an adapter over a converged network, such use of link aggregation will prevent a significant imbalance in the storage data traffic in the converged network. The significant imbalance will be caused by design decisions necessary to maintain fault tolerance for storage traffic. That is, the need for a fully redundant path from the server (via at least one adapter) to each storage device requires that each storage device (or storage system including multiple storage devices) must be attached to the network through two completely separate network connection devices (i.e., two separate adapters), each coupled between the storage device (or storage subsystem) and the network. Otherwise, if only one adapter is present, a failure of the adapter will render the storage device (or subsystem) unavailable. Because each such adapter must be a separate device, link aggregation cannot balance the network load between two adapters that provide redundant data paths to the same storage device (or storage subsystem) and cannot prevent a significant imbalance in storage data traffic through one adapter relative to storage data traffic through another adapter that provides redundant data paths to the same storage device (or storage subsystem). Because the adapters are separate devices, one may be busier and therefore slower than one or more others that may access the same storage device. In contrast, exemplary embodiments of the present invention can mitigate storage data traffic imbalance (and prevent significant storage traffic imbalance) in a converged network even when link aggregation is being used.

Summary of the Invention

As used herein, the term "bandwidth" of a system (e.g., a network, or a device coupled to a network, or a network interface of a device that can be coupled to a network) means either the "consumed bandwidth" of the system or the "available bandwidth" of the system. The expression "consumed bandwidth" of a system is used herein to mean the data rate (bit rate) passing through the system (e.g., the rate at which data traffic occurs through the system, or an average or other statistical characterization of the rate at which data traffic has occurred through the system over a certain time interval). The expression "total available bandwidth" of a system is used herein to mean the maximum possible data rate (bit rate) of the system (i.e., the maximum rate at which data traffic will occur through the system). The expression "available bandwidth" of a system is used herein to mean the total available bandwidth of the system minus the consumed bandwidth of the system.

In some embodiments, the present invention is a method for balancing storage data traffic in a system where computing devices (referred to herein as "servers") coupled to a converged network access (via adapters) storage devices coupled to the network (e.g., in an attempt to optimize storage data traffic). A set of agents implemented on the servers ("server agents") and a set of agents implemented on the adapters ("adapter agents") are configured to detect and respond to imbalances in storage and data traffic across the network, and to redirect storage data traffic to reduce the imbalance and thereby improve overall network performance (for both data communications and storage traffic). Other embodiments include systems configured to perform this method and devices configured to implement this method or for use in such a system.

Typically, each of the agents (server agents and adapter agents) operates autonomously (except that adapter agents can, under certain circumstances, respond to requests and notifications from server agents), and there is no central computer or manager directing the agents' operations. Typically, adapter agents interact directly with server agents only when the adapter and server (where the adapter is implemented) provide a storage data path for at least one storage device; server agents never communicate directly with other server agents, and adapter agents never communicate directly with other adapter agents. However, exemplary embodiments of the present invention allow all agents to react to and influence the behavior of other agents in order to balance overall network traffic and avoid destabilizing behavior. Furthermore, if any network-coupled device fails, the surviving network-coupled devices will continue to balance network traffic (and adjust to the consequences of the failure) without any interruption.

According to typical embodiments, storage data traffic on a converged network is balanced in a fully decentralized manner, with communications being performed to achieve balancing that occurs only between the endpoints of each data path between an adapter and a server (not between servers or between adapters or from an adapter to two or more servers). A failure of any participant (e.g., a server interface, a server agent, an adapter interface, or an adapter agent) affects only the paths of which that participant is a member. Generally, there is only one-to-one communication between any server agent and an adapter agent (e.g., a server agent does not share such communication with more than one adapter agent). In contrast, conventional methods for balancing storage data traffic among multiple storage devices and multiple servers have not been decentralized in this manner.

According to a typical embodiment, the server agent and the adapter agent operate to collect information about the state of the network and to cause the server (where appropriate) to redirect all traffic destined for the storage device from one data path (between the server and the storage device) to a different data path (between the server and the storage device) selected to reduce network imbalance.

In typical embodiments of the present method, it is assumed that another entity (e.g., a management or allocation process) has informed each server (and its agents) of all data paths that can be used between the server and each storage device (e.g., disk drive) that the server can access in order to transfer data to and from the storage device. Typically, it is further assumed that each server (and its agents) has learned the preferred data path between the server and the storage device (for each storage device accessible by the server) (e.g., based on a static analysis of the network, or determined in a deterministic manner (e.g., the path to the adapter interface with the lowest IP address)).

In one class of embodiments, the present invention is a system comprising: at least one server having at least one server interface, wherein the server is configured to couple to a converged network via the server interface and the server is configured to include a server agent; at least one storage device; and at least one adapter configured to couple to the storage device and having at least one adapter interface (and optionally at least one other adapter having at least one adapter interface and configured to couple the storage device to the network), wherein the adapter is configured to couple the storage device to the network via the adapter interface and the adapter is configured to include an adapter agent.

The adapter agent is coupled and configured to:

determining whether each of the adapter interfaces is overloaded, and generating an adapter interface overload indication for each of the adapter interfaces, wherein the adapter interface overload indication for each of the adapter interfaces indicates whether the adapter interface is overloaded; and

Reporting at least one of the adapter interface overload indications to the server agent in response to a request from the server agent (e.g., causing the adapter to assert data indicating at least one of the adapter interface overload indications to at least one of the adapter interfaces in response to the request from the server agent).

The server agent is coupled and configured to:

causing the server to assert a request to the adapter agent and identifying at least one adapter interface overload indication asserted (ie, provided) by the adapter agent to the server in response to the request; and

For a path including the server interface and through which the server accesses the storage device via the adapter, it is determined whether the path is overloaded by using the adapter interface overload indication.

In some embodiments, the server agent is coupled and configured to respond to a determination that the path is overloaded, including by:

determining whether to select a new path to the storage device for subsequent use, and

After determining that the new path should be selected, causing the server to reroute the storage data traffic between the server and the storage device to the new path. Preferably, the server agent is coupled and configured to: wait for a time interval of sufficient duration after causing the server to reroute the storage data traffic between the server and the storage device to the new path so that the effect of the change to the new path can be reflected in the results of continuous monitoring of traffic on each adapter interface of the adapter agent by each adapter agent; and after the wait, begin evaluating (e.g., re-evaluating) paths to the storage device, including at least one path other than the new path. In a preferred embodiment, the time interval of the wait is determined by a random number selected as a normal variable of a selected interval (e.g., 10 seconds), subject to a predetermined minimum wait and maximum wait.

In some embodiments, the system includes a first adapter configured to couple the storage device to the network; and a second adapter configured to couple the storage device to the network (and optionally at least one additional adapter configured to couple the storage device to the network), the first adapter including at least one first adapter interface and the second adapter including at least one second adapter interface, the first adapter including a first adapter agent and the second adapter including a second adapter agent, and the server agent is coupled and configured to:

monitoring data traffic (e.g., receive traffic and transmit traffic) occurring on each of the server interfaces to determine a consumed bandwidth of each of the server interfaces, and determining an available bandwidth of each of the server interfaces based on the consumed bandwidth of each of the server interfaces; and

identifying at least one available bandwidth indication provided by the first adapter agent to the server in response to a request asserted from the server to the first adapter, wherein each of the available bandwidth indications indicates available bandwidth for one of the first adapter interfaces, and identifying at least one additional available bandwidth indication provided by the second adapter agent to the server in response to a request asserted from the server to the second adapter, wherein each of the additional available bandwidth indications indicates available bandwidth for one of the second adapter interfaces; and

An available bandwidth on a path including the server interface and one of the second adapter interfaces of the second adapter is determined as a minimum value of the available bandwidth on the server interface and the available bandwidth of the one of the second adapter interfaces.

Optionally, the adapter agent is coupled and configured to:

monitoring data traffic (e.g., receive traffic and transmit traffic) occurring on each of the adapter interfaces and generating a consumed bandwidth indication for each of the adapter interfaces, wherein the consumed bandwidth indication for each of the adapter interfaces indicates the consumed bandwidth of the adapter interface; and

generating an available bandwidth indication for each of the adapter interfaces, wherein the available bandwidth indication for each of the adapter interfaces indicates the available bandwidth of the adapter interface; and

reporting at least one of the adapter interface overload indications, and at least one of the consumed bandwidth indications and/or at least one of the available bandwidth indications to the server agent in response to a request from the server agent (e.g., causing the adapter to assert data indicating at least one of the adapter interface overload indications, and at least one of the consumed bandwidth indications and/or at least one of the available bandwidth indications to at least one of the adapter interfaces in response to the request from the server agent).

Optionally, further, the adapter agent is coupled and configured to:

estimating the adapter's ability to process the additional data (eg, the adapter's computational load capacity); and/or

A raw overload indication value is filtered to generate a filtered overload value, wherein the raw overload indication value indicates a determined overload and the filtered overload value indicates whether the determined overload is persistent, and wherein at least one of the adapter interface overload indications indicates the filtered overload value.

In some embodiments, the adapter agent is coupled and configured to generate an indication of available bandwidth for each of the adapter interfaces, wherein the indication of available bandwidth for each of the adapter interfaces indicates the available bandwidth of the adapter interface, including by aging each planned additional bandwidth usage value received from at least one of the server agents for one of the adapter interfaces, thereby generating an aged planned bandwidth usage value for the adapter interface, and maintaining, for each of the adapter interfaces, a sum of each of the aged planned bandwidth usage values for the adapter interface. In some such embodiments, the adapter agent is coupled and configured to generate the indication of available bandwidth for each of the adapter interfaces based on: the total available bandwidth of the adapter interface, at least one measurement of consumed bandwidth of the adapter interface, an indication of the adapter's ability to process additional data, and a sum of each of the aged planned bandwidth usage values for the adapter interface for the adapter interface.

In some embodiments, each server is programmed with software that implements the server agent for each server, and each adapter is programmed with software that implements the adapter agent for each adapter. In some embodiments, at least one server agent or at least one adapter agent is implemented in hardware (e.g., at least one of the servers includes a hardware subsystem that implements its server agent).

Other aspects of the present invention are an adapter (programmed or otherwise configured to implement an embodiment of the adapter agent of the present invention), a disk drive (or other storage device) integrated with such an adapter, a JBOD (or other storage device system) integrated with such an adapter, a server (programmed or otherwise configured to implement an embodiment of the server agent of the present invention), a hardware implementation of an embodiment of the server agent of the present invention, and a software implementation of an embodiment of the adapter agent of the present invention.

Other aspects of the invention are methods performed during the operation of any embodiment of the system, adapter, storage device, JBOD, server, or other device of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a block diagram of an embodiment of the system of the present invention.

FIG2 is a block diagram of another embodiment of the system of the present invention.

DETAILED DESCRIPTION

In one class of embodiments, the present invention is a system comprising at least one server coupled to a converged network via at least one server interface and at least one storage device coupled to the network via at least two adapters.

An example of such a system will be described with reference to FIG1. In the system of FIG1, servers 1 and 3 (and optionally other servers) and adapters 5, 7, 9, and 11 (and optionally other adapters) are each coupled to a converged network 20. Storage subsystem 13 is coupled to network 20 through each of adapters 5 and 7. Storage subsystem 15 is coupled to network 20 through each of adapters 9 and 11. Each of storage subsystems 13 and 15 can be a disk drive or other storage device, or a storage subsystem including multiple storage devices (e.g., JBOD).

Server 1 includes an interface 2 (the interface is configured to connect server 1 to network 20), and server 1 is configured to include an application subsystem 4 (e.g., programmed with software that implements the application subsystem). Server 1 is also configured to include a server agent subsystem 6 (e.g., programmed with software that implements the server agent subsystem). Server 3 includes an interface 8 (the interface is configured to connect server 3 to network 20), and is configured to include an application subsystem 10 (e.g., programmed with software that implements the application subsystem). Server 3 is also configured to include a server agent subsystem 12 (e.g., programmed with software that implements the server agent subsystem).

In some embodiments, each of interfaces 2 and 8 is implemented as a physical device (i.e., a network interface controller ("NIC"). In other embodiments, each of interfaces 2 and 8 is implemented as a software-defined wrapper of multiple NICs. In an exemplary embodiment of the invention, each of interfaces 2 and 8 is a hardware or software element with its own Internet Protocol (IP) address.

Adapter 5 is configured to include an adapter proxy subsystem 14 (e.g., programmed with software that implements the adapter proxy subsystem). Adapter 7 is configured to include an adapter proxy subsystem 16 (e.g., programmed with software that implements the adapter proxy subsystem). Adapter 9 is configured to include an adapter proxy subsystem 18 (e.g., programmed with software that implements the adapter proxy subsystem). Adapter 11 is configured to include an adapter proxy subsystem 22 (e.g., programmed with software that implements the adapter proxy subsystem).

In an exemplary embodiment, network 20 is an Ethernet network, and elements 1, 3, 5, 7, 9, and 11 are configured to communicate over network 20 according to the iSCSI (Internet Small Computer System Interface) networking protocol. The iSCSI protocol is a conventional Internet Protocol-based networking standard that allows data to be transmitted over a LAN, WAN, or the Internet. In this exemplary embodiment, elements 1, 3, 5, 7, 9, and 11 (as well as agents 6, 12, 14, 16, 18, and 22) use the iSCSI networking protocol in a simple manner (much simpler than in many conventional applications) in which, while communication between server 1 (or 3) and any of adapters 5, 7, 9, or 11 is allowed, only one connection path exists at a time between each server (1 or 3) and each adapter (5, 7, 9, or 11).

In an exemplary embodiment:

Adapter 5 includes an iSCSI interface for communicating with server 1 or 3 via network 20. Communication between adapter agent 14 and server agents 6 and 12 according to the present invention is implemented through this iSCSI interface. Adapter 5 is also configured to communicate with storage subsystem 13 according to the well-known Serial Attached SCSI ("SAS") protocol to enable storage data traffic between server 1 (or 3) and subsystem 13;

Adapter 7 includes an iSCSI interface for communicating with server 1 or 3 via network 20. Communication between adapter agent 16 and server agents 6 and 12 according to the present invention is implemented through this iSCSI interface. Adapter 7 is also configured to communicate with storage subsystem 13 according to the SAS protocol to implement storage data traffic between server 1 (or 3) and subsystem 13;

Adapter 9 includes an iSCSI interface for communicating with server 1 or 3 via network 20. Communication between adapter agent 18 and server agents 6 and 12 according to the present invention is implemented through this iSCSI interface. Adapter 9 is also configured to communicate with storage subsystem 15 according to the SAS protocol to implement storage data traffic between server 1 (or 3) and subsystem 15; and

Adapter 11 includes an iSCSI interface for communicating with server 1 or 3 via network 20. Communication between adapter agent 22 and server agents 6 and 12 according to the present invention is implemented through this iSCSI interface. Adapter 11 is also configured to communicate with storage subsystem 15 according to the SAS protocol to facilitate storage data traffic between server 1 (or 3) and subsystem 15.

Application subsystem 4 of server 1 is configured to initiate access to storage devices coupled to network 20 (e.g., storage devices in subsystem 13 or 15). Application subsystem 10 of server 3 is configured to initiate access to storage devices coupled to network 20 (e.g., storage devices in subsystem 13 or 15). In typical operation, an entity (e.g., a management or allocation process) has informed application subsystem 4 and agent 6 of all data paths available between server 1 and each storage device that the server can access for transferring data to or from the storage device, and application subsystem 4 and agent 6 have learned the preferred data path between server 1 and the storage devices (for each storage device accessible by server 1) (e.g., based on dynamic analysis of the network, or determined in a deterministic manner (e.g., the path to the adapter interface with the lowest IP address)). Similarly, in typical operation, an entity (e.g., a management or allocation process) has informed the application subsystem 10 and agent 12 of all data paths that can be used between the server 3 and each storage device that the server can access to transfer data to and from the storage device, and the application subsystem 10 and agent 12 have learned the preferred data path between the server 3 and the storage device (for each storage device accessible by the server 3).

In an exemplary embodiment, each of the adapter agent subsystems 14, 16, 18, and 22 (also referred to herein as adapter agents or agents) and each of the server agent subsystems 6 and 12 (also referred to herein as server agents or agents) are configured in accordance with the present invention (e.g., in a manner to be described below) to detect and respond to an imbalance in storage data traffic on the converged network 20, and to redirect the storage data traffic to reduce the imbalance and thereby improve overall network performance (for both data communications and storage traffic). For example, in an exemplary embodiment, the server agent subsystem 6 is configured in accordance with the present invention (e.g., in a manner to be described below) to detect an imbalance in storage data traffic on the network 20 and (where appropriate) respond to it by redirecting the storage data traffic from one data path between the server 1 and a particular storage device (in subsystem 13 or 15) to another data path between the server 1 and the same storage device.

Another embodiment of the system of the present invention is shown in FIG2 . In the system of FIG2 , server 21 (and optionally other servers) and adapters 25, 27, 29, and 31 (and optionally other adapters) are coupled to converged network 20 (which may be identical to network 20 in FIG1 ). Storage subsystem 23 is coupled to network 20 via each of adapters 25 and 27. Storage subsystem 33 is coupled to network 20 via each of adapters 29 and 31. Each of storage subsystems 23 and 33 is a storage subsystem that includes multiple storage devices (e.g., each is a JBOD that includes multiple disk drives).

Server 21 includes interfaces 22 and 24, each of which is a network interface controller (NIC) having its own Internet Protocol (IP) address and configured to connect server 21 to network 20. Server 21 is configured to include an application subsystem 26 (e.g., programmed with software implementing the application subsystem) and is also configured to include a server agent subsystem 28 (e.g., programmed with software implementing the server agent subsystem).

Adapter 25 includes interfaces 30 and 32, each of which is a network interface controller (NIC) having its own Internet Protocol (IP) address and configured to connect adapter 25 to network 20, and adapter 25 is configured to include an adapter proxy subsystem 38 (e.g., programmed with software implementing the adapter proxy subsystem). Adapter 25 also includes ports 34 and 36, each coupled to storage subsystem 23, and is configured to couple a storage device (within subsystem 23) to network 20 via either port 34 or 36 and either interface 30 or 32.

Adapter 27 includes interfaces 40 and 42, each of which is a network interface controller (NIC) having its own Internet Protocol (IP) address and configured to connect adapter 27 to network 20, and adapter 27 is configured to include an adapter proxy subsystem 48 (e.g., programmed with software implementing the adapter proxy subsystem). Adapter 27 also includes ports 44 and 46, each coupled to storage subsystem 23, and is configured to couple a storage device (within subsystem 23) to network 20 via either port 44 or 46 and either interface 40 or 42.

Adapter 29 includes multiple interfaces (not shown), each of which is a network interface controller (NIC) having its own Internet Protocol (IP) address and configured to connect adapter 29 to network 20, and adapter 29 is configured to include an adapter proxy subsystem 50 (e.g., programmed with software implementing the adapter proxy subsystem). Adapter 29 also includes multiple ports (not explicitly shown) each coupled to storage subsystem 33, and is configured to couple storage devices (within subsystem 33) to network 20 via any of the ports and any of the NICs of adapter 29.

Adapter 31 includes multiple interfaces (not shown), each of which is a network interface controller (NIC) having its own Internet Protocol (IP) address and configured to connect adapter 31 to network 20, and adapter 31 is configured to include an adapter proxy subsystem 52 (e.g., programmed with software implementing the adapter proxy subsystem). Adapter 31 also includes multiple ports (not explicitly shown) each coupled to storage subsystem 33, and is configured to couple storage devices (within subsystem 33) to network 20 via any of the ports and any of the NICs of adapter 31.

In an exemplary embodiment, network 20 is an Ethernet network, and elements 21, 25, 27, 29, and 31 are configured to communicate according to the iSCSI (Internet Small Computer System Interface) networking protocol over network 20. In this exemplary embodiment, elements 21, 25, 27, 29, and 31 (and agents 28, 38, 48, 50, and 52) use the iSCSI networking protocol in a simple manner (much simpler than in many conventional applications) in which, while communication between server 21 and any of adapters 25, 27, 29, or 31 is permitted, only one connection path exists between the server and each adapter (25, 27, 29, or 31) at a time.

In an exemplary embodiment:

Each of interfaces 30 and 32 of adapter 25 is an iSCSI interface for communicating with server 21 via network 20. Communication between adapter agent 38 and server agent 28 according to the present invention is implemented through this iSCSI interface. Adapter 25 is also configured to communicate with storage subsystem 23 via either port 34 or 36 in accordance with the Serial Attached SCSI ("SAS") protocol to enable storage data traffic between server 21 and subsystem 23;

Each of interfaces 40 and 42 of adapter 27 is an iSCSI interface for communicating with server 21 via network 20. Communication between adapter agent 48 and server agent 28 according to the present invention is implemented through this iSCSI interface. Adapter 27 is also configured to communicate with storage subsystem 23 via either port 44 or 46 in accordance with the Serial Attached SCSI ("SAS") protocol to enable storage data traffic between server 21 and subsystem 23;

Adapter 29 includes an iSCSI interface for communicating with server 21 via network 20. Communication between adapter agent 50 and server agent 28 according to the present invention is implemented through this iSCSI interface. Adapter 29 is also configured to communicate with storage subsystem 33 according to the SAS protocol to implement storage data traffic between server 21 and subsystem 33; and

Adapter 31 includes an iSCSI interface for communicating with server 21 via network 20. Communication between adapter agent 52 and server agent 28 according to the present invention is implemented through this iSCSI interface. Adapter 31 is also configured to communicate with storage subsystem 33 according to the SAS protocol to facilitate storage data traffic between server 21 and subsystem 33.

Application subsystem 26 of server 21 is configured to initiate access to storage devices coupled to network 20 (e.g., storage devices in subsystem 23 or 33). In typical operation, an entity (e.g., a management or allocation process) has informed application subsystem 26 and agent 28 of all data paths that can be used between server 21 and each storage device that the server can access to transfer data to or from the storage device, and application subsystem 26 and agent 28 have learned the preferred data path between server 21 and the storage devices (for each storage device accessible by server 21) (e.g., based on dynamic analysis of the network, or determined in a deterministic manner (e.g., the path to the adapter interface with the lowest IP address)).

In an exemplary embodiment, each of the adapter agent subsystems 38, 48, 50, and 52 (also referred to herein as adapter agents or agents), and the server agent subsystem 26 (also referred to herein as server agent or agent), are configured in accordance with the present invention (e.g., in a manner to be described below) to detect and respond to an imbalance in storage data traffic on the converged network 20, and to redirect the storage data traffic to reduce the imbalance and thereby improve overall network performance (for both data communications and storage traffic). For example, in an exemplary embodiment, the server agent 26 is configured in accordance with the present invention (e.g., in a manner to be described below) to detect an imbalance in storage data traffic on the network 20 and (where appropriate) respond to it by redirecting the storage data traffic from one data path between the server 21 and a particular storage device (in subsystem 23 or 33) to another data path between the server 21 and the same storage device.

There are at least four data paths (e.g., one path through each of interfaces 30, 32, 40, and 42) between each Ethernet port (NIC 22 or 24) of server 21 and each accessible storage device (typically, each a disk drive), and thus at least eight data paths between server 21 and each accessible storage device. Thus, the system of FIG. 2 provides a lot of redundancy for storage device access.

In a typical data center (e.g., a data center implementing the system of Figure 1 or the system of Figure 2), a management server (not shown in Figures 1 or 2) would be coupled to a network for use in configuring and reconfiguring the data center (e.g., including by notifying the application subsystem 26 and agent 28 of Figure 2 of all data paths that can be used between a server 21 and each storage device that the server can access via network 20 to transfer data to and from the storage device).

It is contemplated that some embodiments of the present invention's servers are programmed (e.g., the server's application subsystem is programmed) to run a data package (e.g., the Hadoop open source software package) that allows a large number of servers to work together to solve a problem (typically involving large amounts of data). It is also contemplated that multiple such servers (and multiple adapters, each configured to implement an embodiment of the present invention's adapter agent) can be coupled to a converged network in a data center (e.g., a Hadoop data center), which may be located in a single building. Each adapter will typically be coupled to a JBOD, such that the JBOD's individual disk drives are accessible to the server via the adapter over the network. The disk drives considered "local" to each server will typically be located in one JBOD (or more than one JBOD), and the one or more JBODs will typically be installed in one rack (e.g., a server can obtain three copies of the data it processes, storing one copy on two disk drives in one rack and the third copy on disk drives in another rack). In such embodiments, the servers will be coupled via the network to enable distributed processing of a set (e.g., a large set) of data in parallel (with some of the processing being performed in response to commands asserted by each server).

More generally, in an exemplary embodiment of the system, server, or adapter of the present invention, each adapter agent (e.g., agents 14, 16, 18, or 22 of FIG. 1 , or agents 38, 48, 50, or 52 of FIG. 2 ) and each server agent (e.g., agents 6 or 12 of FIG. 1 , or agent 28 of FIG. 2 ) is processing hardware configured with software (e.g., software whose source code is written in Python and/or C) to operate in accordance with an embodiment of the present invention. For example, both the server agent and the application subsystem of a server (e.g., both agent 6 and subsystem 4 of server 1 of FIG. 1 , or both agent 12 and subsystem 10 of server 3 of FIG. 1 ) can be implemented while processing hardware configured with software (e.g., a computer). Generally, no application (e.g., an application implemented by subsystem 4 of server 1 of FIG. 1 , or subsystem 10 of server 3 of FIG. 2 , or subsystem 26 of server 21 of FIG. 2 ) needs to be modified to obtain the advantages of exemplary embodiments of the present invention. Typically, each server agent and adapter agent operates in a manner that is invisible to the application, and any application that uses any of the involved server or adapter interfaces (including interfaces that perform only data communication operations) will benefit from the storage data load balancing performed in accordance with the present invention.

Next, we describe the operation of each adapter agent and each server agent during operation of an embodiment of the system of the present invention according to a class of embodiments of the present invention. In this description, "receive traffic" (or "receive data") refers to data asserted (i.e., provided) from the network to the adapter (or server), and "transmit traffic" (or "transmit data") refers to data asserted (i.e., provided) from the adapter (or server) to the network. Typically, a single adapter (or a single server) has two interfaces to the network, and it may have more than two interfaces to the network.

In some embodiments, each adapter agent (e.g., each of agents 14, 16, 18, and 22 of FIG. 1 , or each of agents 38, 48, 50, and 52 of FIG. 2 ) is configured to perform all or some of the following operations:

1. The adapter agent monitors the receive and transmit traffic (e.g., in bits per second) occurring on each interface of the adapter and generates at least one measurement of the consumed bandwidth for each of the interfaces. Typically, each monitoring sample is taken over a relatively short period of time (e.g., a few seconds), and the adapter agent determines statistical representations of the streams of receive and transmit data samples to provide short-term and long-term measurements of the consumed bandwidth (the bandwidth used on each interface) for each interface. Because modern NICs are full-duplex (typically, the adapter's NIC can transmit and receive simultaneously), separate statistics are typically maintained for the receive and transmit data on each interface. In a preferred embodiment, known methods for determining exponential moving averages of values (i.e., in this case, the exponential moving average of the receive traffic on the interface over a moving time window of fixed duration, or the exponential moving average of the transmit traffic on the interface over a moving time window of fixed duration) are used to determine the statistical representations of the receive traffic on each interface and the statistical representations of the transmit traffic on each interface, because such exponential moving averages are inexpensive to compute. An example of a method for determining such an exponential (weighted) moving average is described in U.S. Patent No. 6,438,141 (issued on August 20, 2002) with reference to FIG. 8 thereof. In a preferred embodiment, each short-term moving average approximates an arithmetic moving average over a 20-second interval (window) (or intervals substantially equal to 20 seconds), and each long-term average approximates an arithmetic moving average over a 60-second interval (window) (or intervals substantially equal to 60 seconds). Other window durations and calculation methods may be used to implement other embodiments of the present invention;

2. The adapter agent calculates (e.g., estimates) the adapter's ability to handle the additional data. In a preferred embodiment, this is the adapter's computational load capacity. Because processing the additional data will involve more computational work, if the adapter is running at its computational capacity, it may not be able to handle the additional storage data traffic, even if its interfaces are not fully utilized. In some embodiments, the adapter agent incorporates into its calculation (e.g., estimate) of the adapter's ability to handle the additional data the remaining capacity of any other resources that may be consumed by the adapter in processing the storage data traffic. Optionally, the adapter agent also determines a derating factor for the adapter, which the adapter agent multiplies (in some embodiments) by a raw estimate of the additional available bandwidth per adapter interface to determine a limited estimate of the additional available bandwidth per adapter interface (e.g., to limit the bandwidth that the adapter agent will report to be available for the interface, as described below);

3. If the server agent has indicated to the adapter agent that the server (in which the server agent is implemented) plans to use future additional bandwidth (on paths that include the adapter's interface) in the near future, the adapter agent maintains a total of the planned additional future bandwidth usage (s) that each such server agent has indicated to the adapter agent for one or more paths that include the adapter's interface. In a preferred embodiment, the adapter agent (of the adapter) will only accept notifications of planned bandwidth usage from the server when the server is accessing a storage device (e.g., a disk drive) on the path that includes the adapter. The server agent's indication of planned future bandwidth usage is not a reservation or bandwidth allocation, but rather provides notification that actual bandwidth consumption statistics determined by the adapter agent will likely change in the near future. The purpose of such indication by the server agent and the total maintained by the adapter agent is to eliminate or limit the possibility that data traffic from many storage devices will be directed to one interface of one adapter at once. The adapter agent typically decrements (i.e., "ages") each planned additional bandwidth usage notification over time and maintains an updated (aged) total of the aged planned additional usage values for each interface of the adapter. Since new traffic is actually being routed through the interface, this new actual traffic is included in the per-interface traffic measurements made by the adapter agent. In a preferred embodiment, the adapter agent ages each planned additional bandwidth usage notification by exponentially (i.e., achieving exponential decay of the indicated planned additional bandwidth usage value) decreasing the bandwidth value indicated by said notification with a half-life of 20 seconds (or a half-life substantially equal to 20 seconds). Alternatively, other mechanisms and values (e.g., an exponentially decaying half-life value) may be used to achieve the desired aging of each indicated planned additional bandwidth usage value;

4. The adapter agent determines (calculates) whether each interface of the adapter (in which the agent is implemented) is overloaded, and (in response to a request from the server agent) reports an indication of such overload (if an overload is determined to exist) to the server agent. This overload indication can be used by the server to determine whether to attempt to stop using the interface (if possible). The server will typically be configured to use the indication to determine whether a link has been almost fully utilized for a while and is still fully utilized, and if so, consider the link to be overloaded and determine whether it would be better to route some storage data traffic elsewhere. The adapter agent can filter the raw overload indication value (indicating the determined overload) to generate a filtered overload value indicating whether the determined overload persists, and then (in response to a request from the server agent) report the filtered overload value instead of the raw overload value. In a typical embodiment, the adapter agent is configured to use a selected bandwidth at which the interface is considered to be fully utilized as the overload bandwidth level. In a preferred embodiment, the overload bandwidth level is selected to be 92.5% of the full available bandwidth of the interface, and the interface (by the filtered overload value) is reported as overloaded if the overload calculation yields true at least twice in a row. In a typical embodiment, an overload calculation is considered true if any of the following conditions are true:

Both the short-term and long-term measurements of consumed transmit bandwidth (e.g., long-term and short-term transmit bandwidth averages) exceed the overload bandwidth level, or both the short-term and long-term measurements of consumed receive bandwidth (e.g., long-term and short-term receive bandwidth averages) are above the overload bandwidth level; or

The adapter's data processing capacity has been (or almost has been) reached;

5. The adapter agent calculates an estimate of the available bandwidth per adapter interface (i.e., the additional bandwidth available to accommodate data traffic from the new storage device that may be redirected to the interface by the server). This calculation does not require any knowledge of the state or capabilities of the new storage device, but rather is a determination by the adapter agent of an estimated amount of additional storage data traffic to and from the storage device that could be handled by the interface if such additional traffic were directed to the interface. This estimate of available bandwidth is typically calculated based on the interface's total available bandwidth (e.g., raw capacity in bits per second), traffic statistics for the interface (i.e., at least one measurement of the adapter interface's consumed bandwidth), the adapter's ability to handle the additional data (i.e., computational load), and the total indicated future bandwidth notification for the interface. Since storage data traffic often includes both read and write traffic, the estimated additional available traffic calculation typically assumes that the additional traffic will be either transmit or receive traffic, whichever is already busiest. This prevents any additional traffic from overloading the already heavily loaded data travel direction on the interface. In a preferred embodiment, the estimated available bandwidth is based on the average receive and transmit data for the interface plus an estimate of the normal variation in recent traffic (i.e., the standard deviation) to avoid slowing down the processing of existing work. In a preferred embodiment, the estimate of the average and expected variation in traffic is calculated via a "fast up, slow down" exponential moving average (wherein if the most recently generated statistic is greater than the previously generated statistic, a relatively greater weight is applied to the next average; if the most recently generated statistic is less than or equal to the previously generated statistic, a relatively lesser weight is applied to the next average) as described, for example, in the aforementioned U.S. Patent 6,438,141. Using simple calculations, this "fast up, slow down" exponential moving average can be approximated to the most recent average plus one standard deviation of the series. Furthermore, the estimated total raw available bandwidth can be reduced by a safety factor to account for brief bursts in traffic passing through the interface without degrading performance. In one embodiment, the estimate of the available bandwidth of the adapter interface (represented as the value "available" in the following equation) is calculated as follows (although it should be understood that additional terms and factors can be taken into account in the calculation to adjust the behavior):

Available = (Safety_Factor*(Original_Bandwidth-Worst Case))*Processing_Capacity_Derating_Factor,

Where the value "worst case" is equal to maximum(transmit_average_sum_variation, receive_average_sum_variation) + sum(aging - future - bandwidth - notification), where

"max(a,b)" means the value "a" or the value "b", whichever is greater,

transmit_mean_sum_variance is a measure of the consumed transmit bandwidth of the interface (e.g., an estimate of the average transmit data for the interface plus an estimate of the normal variation (standard deviation) of recent transmit traffic).

receive_mean_and_variance is a measure of the consumed receive bandwidth of an interface (e.g., an estimate of the average received data for the interface plus an estimate of the normal variation (standard deviation) of recent receive traffic).

"Sum (Aging-Future-Bandwidth-Notification)" is the sum of the aging planned additional bandwidth usage values of the adapter interface.

Safety_factor is the safety factor mentioned above,

Raw_Bandwidth represents the full available bandwidth of the interface, and

processing_capability_derating_factor is the derating factor of the adapter (of the above type); and/or

6. The adapter agent responds to the status request from the server agent (i.e., the status request from the server agent of the server on the same storage data path as the adapter). Typically, the status report returned to the server agent contains the current overload status and available bandwidth of the interface for each adapter interface, as described above.

In some embodiments, each server agent (e.g., each of agents 6 and 12 of FIG. 1 , or agent 28 of FIG. 2 ) is configured to perform all or some of the following operations:

1. As in the exemplary embodiment of the adapter agent, the server agent monitors the receive and transmit traffic (e.g., in bits per second) occurring on each interface of the server and generates at least one measurement of the consumed bandwidth of each of said interfaces. Typically, each monitoring sample is typically taken over a relatively short period of time (e.g., a few seconds), and the server agent determines statistical representations of the streams of receive and transmit data samples to provide short-term and long-term measurements of the consumed bandwidth of each interface (the bandwidth used on each interface). Because modern NICs are full-duplex (typically, a server's NIC can transmit and receive simultaneously), separate statistical data are typically maintained for the receive and transmit data on each interface. In a preferred embodiment, a known method for determining an exponential moving average of values (i.e., in this case, an exponential moving average of the receive traffic on the interface over a moving time window of fixed duration, or an exponential moving average of the transmit traffic on the interface over a moving time window of fixed duration) is used to determine the statistical representation of the receive traffic on each interface and the statistical representation of the transmit traffic on each interface (e.g., in the same manner as described above for the exemplary embodiment of the adapter agent of the present invention);

2. For each storage device (e.g., disk drive) that accesses a path assigned to the server (in the sense that the storage device is available for access by the server via the converged network through the path), the server agent may cause the server to issue a request to the adapter that is the other end point of the path, and the server agent receives bandwidth (consumed bandwidth and/or available bandwidth) and/or overload information of the adapter (i.e., overload and/or bandwidth reports generated by the adapter agent of the adapter in response to the request). In many cases, the same adapter is used for several storage devices and paths, so the adapter data received in response to one request may often be for multiple paths;

3. For each path through which a server (in which a server agent is implemented) accesses a storage device via an adapter, the server agent calculates whether the path is overloaded and requires load shedding, and how much available (unused) bandwidth the path has. In a typical embodiment, the server agent determines the available bandwidth on the path as the minimum of: the available bandwidth on the server interface coupled along the path or the available bandwidth on the adapter interface coupled along the path. In a typical embodiment, the server agent determines that the path is overloaded if either the server interface or the adapter interface is overloaded (typically, including by using an interface overload indication in a report asserted by the adapter agent to the server in response to a request from the server);

4. If at least one overload path is in use (used by the server to access any storage device(s)), the server agent typically performs a selection process to access each overload. In a preferred embodiment, if there are at least two overload paths in use by the server, the server agent considers them in a random order and selects only one per cycle:

If there is another path available (between the server and the adapter coupled along the overloaded path) that is not overloaded and has sufficient available bandwidth for another storage device, the server agent selects the alternative path for subsequent use. If two or more such alternative paths are available, the server agent selects the path with the highest available bandwidth;

Otherwise, if the server (and its server agent) has learned a preferred data path between the server and the storage device coupled along the overloaded path, and if the current (overloaded) path is not the path originally assigned by the server for accessing the storage device, then the preferred data path is selected for subsequent use (regardless of whether the preferred data path is overloaded). Generally, if the assignment of the current (overloaded) path has not changed (i.e., if no other path has been selected to replace the current path), then the next overloaded path is considered in the same manner as the current path;

5. If a new path assignment is made (i.e., if the server agent selects another path to replace the current path), the server agent typically performs the following actions:

Notifying the adapter agent associated with the newly selected path that the server interface (which is coupled along the newly selected path) plans to assert storage data traffic having a specific bandwidth (e.g., one disk's worth of bandwidth for future load) to a specific interface of the adapter. This immediately affects statistics and reports generated by the adapter agent for the adapter and generally prevents (indirectly) two servers from attempting to use the same excess bandwidth on an adapter interface; and

The server agent causes the server to reroute storage data traffic between the server and the associated storage device to the newly selected path; and/or

6. After causing the server to reroute storage data traffic between the server and the storage device to the newly selected path, the server agent waits for a time interval (e.g., a predetermined or randomly selected time interval) of sufficient duration so that the impact of the server agent's recent action can be reflected in the ongoing monitoring results (e.g., monitoring statistics) of traffic on each adapter interface of the adapter agent by each adapter agent. After the wait, the server agent begins evaluating (e.g., re-evaluating) paths to the storage device, including at least one path other than the new path. In a preferred embodiment, the time interval for the wait is determined by a random number selected as a normal variable of the selected interval (e.g., 10 seconds), subject to a predetermined minimum wait and maximum wait.

An exemplary method performed by an embodiment of the system of the present invention is as follows. A server agent (in this example, agent 28 of server 21 in FIG. 2 ) sends a notification of planned additional bandwidth usage to an adapter agent (in this example, adapter agent 38 of adapter 25 in FIG. 2 ) in response to determining that server 21 should access a storage device (coupled to network 20 via adapter 25) via a path through a particular interface of adapter 25 (in this example, interface 30). In response, adapter agent 38 decrements (i.e., “ages”) the planned additional bandwidth usage value indicated by the notification over time and maintains an updated (aged) sum of all aged planned additional bandwidth usage values received for interface 30 (and uses the aged sum to generate a current overload status and available bandwidth indication). Because new traffic is actually routed through interface 30, this new actual traffic is included in the per-interface traffic measurement performed by adapter agent 38 (and used to generate a current overload status and available bandwidth indication for each adapter interface). Concurrently, other adapter agents for other adapters coupled to the network independently perform their own per-adapter interface traffic measurements (and generate their own per-interface current overload status and available bandwidth indications). The server agent 28 requests (from each adapter agent) a report indicating the current overload status and available bandwidth for each interface of the adapter in which each such adapter agent is implemented and which is part of a path to the storage device used by the server, and in response, each queried adapter agent independently sends the requested report to the server agent 28. The server agent 28 uses the report and statistical representation of the traffic generated by the agent 28 itself for its own server interface to determine whether to allow the server 21 to access the storage device via the current path (as assumed by the most recently asserted notification of planned additional bandwidth usage) or whether to select another path (to replace the current path) for server 21 to access the storage device. If the server agent 28 selects a new path for server 21 to access the storage device, the server agent 28 notifies the adapter agent associated with the newly selected path that the server interface (which will be coupled to the newly selected path) plans to assert storage data traffic with a specific bandwidth to the specific interface of the adapter, and the server agent 28 causes the server 21 to reroute the storage data traffic between the server 21 and the associated storage device to the newly selected path. Thus, the system operates in a decentralized manner (independently asserting independently generated reports from independently configured adapter agents to server agents at the servers) to select the best path for access to storage devices by the servers.

In some embodiments of the present invention, a server agent (e.g., each of agents 6 and 12 of FIG. 1 , or agent 28 of FIG. 2 ) coupled to a server of a converged network is configured to detect and reroute storage traffic around network bottlenecks, in addition to those caused by adapter interface traffic or adapter capabilities. An example of such a bottleneck is a network bottleneck caused by traditional data communication traffic between the server and other server(s) that may not participate in the rebalancing mechanism.

In one preferred embodiment, a server and an adapter (e.g., elements 1, 3, 5, 7, 9, and 11 of FIG. 1 , each of which implements multiple network interfaces) are coupled to a converged network (e.g., network 20 of FIG. 1 ), which is an Ethernet network, and the server and adapter are configured to communicate over the network according to the iSCSI (Internet Small Computer System Interface) networking protocol. In such embodiments, the server agent and the adapter agent (e.g., agents 6, 12, 14, 16, 18, and 22 of FIG. 1 ) utilize the iSCSI networking protocol in a simplified manner (much simpler than in many conventional applications), wherein, while communication between a server (e.g., server 1 or 3) and any of the adapters (e.g., adapters 5, 7, 9, or 11) is permitted, only one connection path exists between each server and each adapter (to the storage device) at a time. In such embodiments, the server agent utilizes conventional multipath I/O ("MPIO") technology (or a new, simplified version of conventional MPIO technology) to achieve storage data traffic balancing according to the present invention. The expression "MPIO-type subsystem" is used herein to refer to either a processing subsystem (eg, of a server) that implements conventional MPIO or a processing subsystem that implements a simplified version of conventional MPIO.

In the described embodiment of the type, each server includes an MPIO-type subsystem (e.g., an MPIO driver in the kernel) that manages data input/output according to iSCSI via selected ones of the server interfaces. The server's server agent interacts with the MPIO-type subsystem, including by setting a storage device access "policy" that allows the server (via one of the network and the adapter) to access the storage device only through one of the server interfaces that has been selected by the server agent. This policy is similar to a conventional MPIO "failover only" policy that performs load balancing and instead uses a single valid path for network access (and the other potentially available paths are merely backup paths that are only used when the single valid path fails). However, the storage device access policy is used by the server agent of the present invention in accordance with the present invention to achieve storage data traffic balancing in a new way. When the server's server agent selects a new path (typically including the step of receiving a requested report from the adapter agent, according to any embodiment of the method of the present invention) for the server to access the storage device via the server's newly selected interface, the server agent causes the server to reroute storage data traffic (to or from the storage device) to the newly selected path by causing the MPIO-type subsystem to specify a new storage device access "policy," which allows the server to access the storage device only via the new interface among the server interfaces selected by the server agent. The server agent also causes the new storage device access path to extend to the appropriate adapter interface selected by the server agent.

Thus, in embodiments of the type described, an MPIO-type subsystem (proxyed by the server of the present invention) is used to balance storage data traffic over a converged network in accordance with the present invention.

MPIO was originally developed on isolated storage networks, and conventional MPIO load balancing will not work well on converged networks. For example, suppose an attempt is made to use MPIO in a converged network (implemented as Ethernet) to balance storage data traffic between multiple Ethernet ports of a server coupled to the network and multiple Ethernet ports of an adapter coupled to the network, where the adapter also has multiple "back-end" SAS ports coupled to a disk drive subsystem (i.e., JBOD) to be accessed by the server. In this example, all conventional load balancing "strategies" of MPIO (sending storage commands in a round-robin manner across all available Ethernet interfaces, or determining some metric of how much work is outstanding on each link and sending commands to the 'least busy' Ethernet interface) will typically increase the number of seeks required to execute a series of disk access commands because it will frequently cause consecutive commands in the sequence to take different paths from the server to the disk drives (often causing the commands to arrive at the disk drives out of order), and will therefore cause the excessive seek problem described above to be the result of changing the storage data path through the network too quickly, regardless of whether the change is desirable or not. In contrast, typical embodiments of the present invention (including those embodiments in which the server agent of the present invention uses the MPIO-type subsystem of the server to balance storage data traffic on the converged network as described above) will generally not cause excessive seek problems because they will generally only change the storage data path used to access any disk drive when necessary, and generally very infrequently (e.g., once, twice, or a few times per hour). An important advantage of typical embodiments of the present invention is that it maintains the orderly delivery of commands to disks via the converged network while adjusting cross-traffic (to perform storage data traffic balancing).

In another class of embodiments, a server implementing a server agent embodiment of the present invention also implements a user interface. During typical operation of the server, in such embodiments having a display device coupled to the server, the user interface causes the display device to display indications of the operation or status of the server agent and/or indications of reports received or determinations made by the server agent. For example, the following types of indications may be displayed: status of the server agent monitoring server interface traffic and/or bandwidth, reports received from the adapter agent (e.g., regarding adapter interface status and available bandwidth), and determinations that the current storage device access path should or should not be changed.

Advantages and features of exemplary embodiments of the present invention include the following:

1. According to an exemplary embodiment, storage data traffic is balanced on a converged network in a fully decentralized manner, with communication performed so that balancing occurs only between the endpoints of each data path (e.g., server 1 and server 5 in FIG. 1 , or server 21 and adapter 25 in FIG. 2 ) between an adapter and a server (not between servers, between adapters, or from an adapter to two or more servers). A failure of any participant (e.g., a server interface, a server agent, an adapter interface, or an adapter agent) affects only the paths of which that participant is a member. Generally, there is only one-to-one communication between any server agent and an adapter agent (e.g., a server agent does not share such communication with more than one adapter agent). In contrast, conventional methods for balancing storage data traffic across multiple storage devices and multiple servers have not been decentralized in this manner.

2. The communication required to implement storage traffic balancing occurs only between the endpoints of each data path between the adapter and the server (e.g., server 1 and adapter 5 in FIG1 , or server 21 and adapter 25 in FIG2 ). Therefore, the number of connections between a server and an adapter is limited by the number of storage devices (e.g., disks) associated with the path between the server and the adapter. Consequently, even in very large data centers with thousands of servers and adapters, the computational and network load on each server and adapter required to implement exemplary embodiments of the present invention is minimal.

3. There is no pre-reservation or locking of bandwidth for storage data traffic. Therefore, a failure of any participant (i.e., server interface, server agent, adapter interface, or adapter agent) will be immediately reflected in the overall statistics, and the resources used by the participant (before the failure) will automatically be available for use by the remaining devices. If one or more failed devices subsequently return, the performance of an exemplary embodiment of the present method will cause other servers to redirect traffic away from the path(s) used by the restored device(s) if the traffic causes an overload.

4. Even when the server sends planned additional bandwidth usage notifications to the adapter, the adapter agent (implemented in the adapter) typically decreases (i.e., "ages") the planned additional bandwidth usage value indicated by each notification over time. Aging typically decreases (to zero) the aged planned additional bandwidth usage value for each interface of the adapter relatively quickly. Consequently, planned additional bandwidth usage notifications that do not quickly result in additional observed storage data traffic are quickly ignored.

5. The data path selected by the server that causes a temporary overload is usually changed (ie, replaced by a new data path to the same storage device) in a very short time.

6. The process of announcing each server's intention to begin using a new path (i.e., each server agent sends a notification of planned additional bandwidth usage to each adapter's adapter agent, which will be affected by the actual existence of the indicated planned additional bandwidth usage) prevents many servers from making the same decision at approximately the same time. This essentially prevents any conflicts that could arise from nearly simultaneous path decisions based solely on historical data. Otherwise, all servers might see statistics indicating a slightly overloaded interface, and all servers might redirect paths to that interface, leading to a severe overload condition.

7. Using random periods (e.g., in an embodiment where the server agent waits for a randomly determined time interval after causing the server to reroute storage data traffic between the server and the storage device to a newly selected path so that the impact of the server agent's latest action can be reflected in the monitoring statistics before the server agent begins to reevaluate the path to the storage device) prevents the server from working in lock step, further avoiding making synchronization conflict decisions.

8. If the network becomes fully utilized (i.e., all interfaces are overloaded), such that there is no opportunity to redirect storage traffic, in a typical embodiment, all servers and adapters will revert to the predetermined "preferred" data paths between the servers and adapters. This means that no useless reboot attempts will be made. Furthermore, if the preferred data paths are selected in a way that statically balances all data traffic, they should constitute an optimal configuration in a fully loaded network.

9. No application (e.g., an application implemented by subsystem 4 of server 1 or subsystem 10 of server 3 in FIG. 1 , or subsystem 26 of server 21 in FIG. 2 ) needs to be changed to obtain the advantages of the exemplary embodiments of the present invention. Typically, each server agent and adapter agent operates in a manner invisible to the application, and programs and devices that use any of the involved interfaces will benefit from storage data load balancing (including those that perform only data communication operations).

Other aspects of the present invention are adapters programmed or otherwise configured to implement adapter agent embodiments of the present invention (e.g., any one of adapters 5, 7, 9, and 11 of FIG. 1 , or any one of adapters 25, 27, 29, and 31 of FIG. 2 ), disk drives (or other storage devices) integrated with such adapters (e.g., storage subsystem 15 implemented as a disk drive, integrated with adapter 9 (and adapter 11) as a single device 100, as shown in FIG. 1 ), JBODs (or other storage device systems) integrated with such adapters (e.g., storage subsystem 33 implemented as a JBOD, integrated with adapter 29 (and adapter 31) as a single device 101, as shown in FIG. 2 ), servers programmed or otherwise configured to implement server agent embodiments of the present invention (e.g., any one of servers 1 and 3 of FIG. 1 , or server 21 of FIG. 2 ), hardware implementations of server agent embodiments of the present invention (e.g., agent 6 of FIG. 1 , implemented in hardware), and hardware implementations of adapter agent embodiments of the present invention (e.g., agent 14 of FIG. 1 , implemented in hardware).

Other aspects of the present invention are methods performed in the operation of the system, adapter, storage device, JBOD, server, or other device of the present invention. One such method comprises the following steps:

asserting a request from a server to an adapter over a converged network, wherein the server is configured to include a server agent and the adapter is configured to include an adapter agent;

employing the server agent to identify at least one adapter interface overload indication asserted (i.e., provided) by the adapter agent to a server interface of the server in response to the request, wherein the adapter interface overload indication indicates whether an adapter interface of the adapter is overloaded; and

For a path including the server interface and through which the server accesses at least one storage device via the adapter, the server agent is employed to determine whether the path is overloaded by using the adapter interface overload indication.

It should be understood that although certain forms of the invention have been shown and described herein, the invention is not limited to the specific embodiments described or illustrated or the specific methods described. Claims describing methods do not imply any specific order of steps unless explicitly stated in the claim language.

Claims (49)

1. An adapter configured for use in a system comprising at least one server coupled to a converged network via at least one server interface, and at least one storage device, wherein the server includes a server agent, and the adapter comprises: At least one port, the at least one port being configured to couple the storage device to the adapter; At least one adapter interface, the at least one adapter interface being configured to couple the adapter to the network, and thereby coupling the storage device to the network via the adapter when the storage device is coupled to the at least one port; and An adapter proxy, wherein the adapter proxy is coupled and configured to: Determine whether each adapter interface is overloaded, and generate an adapter interface overload indication for each adapter interface, wherein the adapter interface overload indication for each adapter interface indicates whether the adapter interface is overloaded; and In response to a request from the server agent, the adapter asserts data indicating at least one adapter interface overload to at least one of the adapter interfaces. 2. The adapter of claim 1, wherein the adapter agent is further coupled and configured to: Monitor the data traffic occurring on each of the adapter interfaces and generate a bandwidth consumption indication for each of the adapter interfaces, wherein the bandwidth consumption indication for each of the adapter interfaces indicates the bandwidth consumed by the adapter interface; Generate an available bandwidth indication for each of the adapter interfaces, wherein the available bandwidth indication for each of the adapter interfaces indicates the available bandwidth of the adapter interface; and In response to the request from the server proxy, the adapter asserts data indicating the following to at least one of the adapter interfaces: At least one of the adapter interface overload indicators, and at least one of the bandwidth consumption indicators and/or at least one of the available bandwidth indicators. 3. The adapter of claim 2, wherein the adapter agent is further coupled and configured to filter raw overload indication values to generate filtered overload values, wherein the raw overload indication values indicate identified overloads, and the filtered overload values indicate whether the identified overloads are persistent; and in response to the request from the server agent, the adapter asserts data indicating the filtered overload values to at least one of the adapter interfaces. 4. The adapter of claim 1, wherein the adapter agent is further coupled and configured to estimate the adapter's ability to process additional data. 5. The adapter of claim 1, wherein the adapter agent is coupled and configured to generate an available bandwidth indication for each of the adapter interfaces, wherein the available bandwidth indication for each of the adapter interfaces indicates the available bandwidth of the adapter interface, including by: Each planned additional bandwidth usage value received from at least one of the server proxies for one of the adapter interfaces is aged out, thereby generating an aged planned bandwidth usage value for the adapter interface, and the sum of each of the aged planned bandwidth usage values for the adapter interface is maintained for each of the adapter interfaces. 6. The adapter of claim 5, wherein the adapter agent is coupled and configured to generate the available bandwidth indication for each of the adapter interfaces based on: the full available bandwidth of the adapter interface, at least one measurement of the bandwidth consumed by the adapter interface, an indication of the adapter's ability to process additional data, and a sum of the aging-planned bandwidth usage values for each of the adapter interfaces. 7. A system for a converged network, comprising the adapter as claimed in claim 1, and further comprising: At least one server, the at least one server having at least one server interface, wherein the server is configured to include a server proxy and is coupled to a converged network via the server interface; and At least one storage device is configured to be coupled to the adapter, such that the adapter couples the storage device to the network via the adapter interface. The server proxy is coupled and configured to: The server asserts a request to the adapter agent and identifies at least one adapter interface overload indication provided to the server by the adapter agent in response to the request; and For a path that includes the server interface and through which the server accesses the storage device via the adapter, the overload indication of the adapter interface is used to determine whether the path is overloaded. 8. The system of claim 7, wherein the server proxy is configured to respond to a determination of path overload, including by: Determine whether to select a new path to the storage device for subsequent use, and After determining that the new path should be selected, the server changes the routing of storage data traffic between the server and the storage device to the new path. 9. The system of claim 8, wherein the server agent is coupled and configured to: wait for a time interval of sufficient duration after causing the server to change the routing of storage data traffic between the server and the storage device to the new path, such that the effect of the change to the new path is reflected in the results of continuous monitoring of traffic on each adapter interface of the adapter agent by each adapter agent; and after the wait, begin evaluating the path to the storage device, including at least one path other than the new path. 10. The system of claim 9, wherein the time interval of the waiting is determined by a random number selected as a normal variable of the selected interval, subject to a predetermined minimum wait and a maximum wait. 11. The system of claim 7, wherein the adapter is a first adapter, the first adapter including at least one first adapter interface and a first adapter agent, and wherein the system comprises: A second adapter, configured to couple the storage device to the network, wherein the second adapter includes at least one second adapter interface, and the second adapter includes a second adapter agent, and the server agent is coupled and configured to: Monitoring data traffic occurring on each of the server interfaces to determine the bandwidth consumed by each of the server interfaces, and determining the available bandwidth for each of the server interfaces based on the bandwidth consumed; and At least one available bandwidth indication provided to the server by the first adapter proxy in response to a request asserted from the server to the first adapter is identified, wherein each available bandwidth indication indicates the available bandwidth of one first adapter interface; and at least one additional available bandwidth indication provided to the server by the second adapter proxy in response to a request asserted from the server to the second adapter is identified, wherein each additional available bandwidth indication indicates the available bandwidth of one second adapter interface; and The available bandwidth on the path including the server interface and one of the second adapter interfaces is determined as the minimum of the available bandwidth on the server interface and the available bandwidth of the one of the second adapter interfaces. 12. An apparatus for a converged network, configured in a system including at least one server coupled to the converged network via at least one server interface, wherein the server includes a server agent, wherein the apparatus is a storage device with an integrated adapter, and includes: Data storage subsystem; and An adapter subsystem, coupled to the data storage subsystem, wherein the adapter subsystem implements the adapter, and the adapter subsystem includes: At least one adapter interface configured to couple the adapter subsystem to the network, and thereby couple the data storage subsystem to the network via the adapter subsystem; and An adapter proxy, wherein the adapter proxy is coupled and configured to: Determine whether each of the adapter interfaces is overloaded, and generate an adapter interface overload indication for each of the adapter interfaces, wherein the adapter interface overload indication for each of the adapter interfaces indicates whether the adapter interface is overloaded; and In response to a request from the server agent, the adapter subsystem asserts data indicating at least one adapter interface overload to at least one of the adapter interfaces. 13. The device of claim 12, wherein the device is a storage device with an integrated adapter, the adapter subsystem implementing the adapter, and the data storage subsystem implementing the storage device. 14. The device of claim 13, wherein the storage device is a disk drive. 15. The device of claim 12, wherein the device is a JBOD with an integrated adapter, the adapter subsystem implementing the adapter, the data storage subsystem implementing the JBOD, and the JBOD includes a set of disk drives. 16. A server configured for use in a system including at least one storage device and at least one adapter coupled to said storage device, wherein the adapter has at least one adapter interface coupled to a converged network, the adapter coupling the storage device to the network via said adapter interface, and the adapter is configured to include an adapter agent, the server comprising: A processing subsystem, configured to include a server agent; and At least one server interface, the at least one server interface being configured to be coupled to the network, wherein the processing subsystem is coupled to the server interface and configured to access the network via the server interface when the server interface is coupled to the network, and wherein the server proxy is coupled and configured to: The processing subsystem asserts a request to the adapter agent and identifies at least one adapter interface overload indication provided by the adapter agent to the server interface in response to the request; and For a path that includes the server interface and through which the server accesses the storage device via the adapter, the overload indication of the adapter interface is used to determine whether the path is overloaded. 17. The server of claim 16, wherein the server proxy is configured to respond to a determination of path overload, including by: Determine whether to select a new path to the storage device for subsequent use by the server, and After determining that the new path should be selected, the processing subsystem changes the routing of storage data traffic between the server and the storage device to the new path. 18. The server of claim 17, wherein the server agent is coupled and configured to: wait for a time interval of sufficient duration after the processing subsystem changes the routing of storage data traffic between the server and the storage device to the new path, such that the effect of the change to the new path is reflected in the results of continuous monitoring of traffic on each adapter interface of the adapter agent by each adapter agent; and after the wait, begin evaluating paths to the storage device, including at least one path other than the new path. 19. The server of claim 18, wherein the time interval of the waiting is determined by a random number selected as a normal variable of the selected interval, subject to a predetermined minimum wait and a maximum wait. 20. The server of claim 16, wherein the system includes a first adapter configured to couple the storage device to the network and a second adapter configured to couple the storage device to the network, wherein the first adapter includes at least one first adapter interface and the second adapter includes at least one second adapter interface, the first adapter includes a first adapter proxy and the second adapter includes a second adapter proxy, wherein the server includes at least a first server interface and a second server interface, and wherein the server proxy is coupled and configured to: Monitoring data traffic occurring on each of the server interfaces determines the bandwidth consumed by each of the server interfaces, and based on the bandwidth consumed by each of the server interfaces, determines the available bandwidth for each of the server interfaces; and At least one available bandwidth indication provided by the first adapter agent to the first server interface in response to a request from the processing subsystem to the first adapter assertion is identified, wherein each available bandwidth indication indicates the available bandwidth of one first adapter interface; and at least one additional available bandwidth indication provided by the second adapter agent to the second server interface in response to a request from the processing subsystem to the second adapter assertion is identified, wherein each additional available bandwidth indication indicates the available bandwidth of one second adapter interface; and The available bandwidth on the path including the second server interface and the second adapter interface is determined as the minimum of the available bandwidth on the second server interface and the available bandwidth of the second adapter interface. 21. The server of claim 16, wherein the adapter agent is coupled and configured to generate an available bandwidth indication for each of the adapter interfaces of the adapter. Furthermore, the server proxy is coupled and configured to: Identify at least one of the available bandwidth indications provided to the server interface by the adapter agent in response to the request; and Access the path including the server interface and at least one of the adapter interfaces in a manner that uses the available bandwidth indication. 22. A method for a converged network, comprising the following steps: A request is asserted from a server to an adapter via a converged network, wherein the server is configured to include a server proxy and the adapter is configured to include an adapter proxy; The server proxy is used to identify at least one adapter interface overload indication of the server interface provided to the server by the adapter proxy in response to the request, wherein the adapter interface overload indication indicates whether the adapter's adapter interface is overloaded; and For a path that includes the server interface and through which the server accesses at least one storage device via the adapter, the server agent is used to determine whether the path is overloaded by using the adapter interface overload indication. 23. The method of claim 22, further comprising the step of: responding to the determination of path overload using the server proxy, including by: Determine whether to select a new path to the storage device for subsequent use, and After determining that the new path should be selected, the server changes the routing of storage data traffic between the server and the storage device to the new path. 24. The method of claim 23, comprising the following steps: (a) After rerouting storage data traffic between the server and the storage device to the new path, wait for a sufficient time interval such that the impact of the rerouting to the new path is reflected in the results of continuous monitoring of traffic on each adapter interface of each of the adapter agents coupled to the network; and (b) After step (a), the server agent is used to evaluate the path to the storage device, including at least one path other than the new path. 25. The method of claim 22, further comprising the step of: The server proxy is used to monitor data traffic occurring on each server interface of the server to determine the bandwidth consumed by each server interface, and based on the bandwidth consumed by each server interface, the available bandwidth for each server interface is determined; and The server proxy is used to identify at least one available bandwidth indication provided to the server by a first adapter proxy of a first adapter coupled to the network in response to a request asserted by the server to the first adapter, wherein each available bandwidth indication indicates the available bandwidth of the adapter interface of the first adapter; and to identify at least one additional available bandwidth indication provided to the server by a second adapter proxy of a second adapter coupled to the network in response to a request asserted by the server to the second adapter, wherein each additional available bandwidth indication indicates the available bandwidth of the adapter interface of the second adapter; and The server proxy is used to determine the available bandwidth on the path including one of the server interfaces and the adapter interface of the second adapter as the minimum of the available bandwidth on the server interface and the available bandwidth on the adapter interface of the second adapter. 26. The method of claim 22, further comprising the step of: The adapter agent is used to monitor data traffic occurring on each adapter interface of the adapter and to generate a bandwidth consumption indication for each adapter interface, wherein the bandwidth consumption indication for each adapter interface indicates the bandwidth consumed by the adapter interface; The adapter agent is used to generate an available bandwidth indication for each of the adapter interfaces, wherein the available bandwidth indication for each adapter interface indicates the available bandwidth of the adapter interface; and The adapter agent is used to cause the adapter to report at least one adapter interface overload indication, at least one bandwidth consumption indication, and/or at least one available bandwidth indication to the server agent. 27. An adapter configured for use in a system comprising at least one server coupled to a converged network via at least one server interface, and at least one storage device, wherein the server includes a server agent, and the adapter includes: At least one port, the at least one port being configured to couple the storage device to the adapter; At least one adapter interface, the at least one adapter interface being configured to couple the adapter to the network, and thereby coupling the storage device to the network via the adapter when the storage device is coupled to the at least one port; and An adapter proxy, wherein the adapter proxy is coupled and configured to: Monitor data traffic occurring on each of the adapter interfaces of the adapter, and generate a bandwidth consumption indication for each adapter interface, wherein the bandwidth consumption indication for each adapter interface indicates the bandwidth consumed by the adapter interface; Generate an available bandwidth indication for each of the adapter interfaces, wherein the available bandwidth indication for each adapter interface indicates the available bandwidth of the adapter interface; and In response to a request from the server agent, the adapter asserts at least one bandwidth indication data to at least one of the adapter interfaces, wherein the at least one bandwidth indication is at least one of the consumed bandwidth indications, or at least one of the available bandwidth indications, or at least one of the consumed bandwidth indications and at least one of the available bandwidth indications. 28. The adapter of claim 27, wherein the adapter agent is further coupled and configured to estimate the adapter's ability to process additional data. 29. The adapter of claim 27, wherein the adapter agent is coupled and configured to generate the available bandwidth indication for each of the adapter interfaces, including by: Each planned additional bandwidth usage value received from at least one of the server proxies for one of the adapter interfaces is aged out, thereby generating an aged planned bandwidth usage value for the adapter interface, and the sum of each of the aged planned bandwidth usage values for each of the adapter interfaces is maintained. 30. The adapter of claim 29, wherein the adapter agent is coupled and configured to generate the available bandwidth indication for each of the adapter interfaces based on: the full available bandwidth of the adapter interface, at least one measurement of the bandwidth consumed by the adapter interface, an indication of the adapter's ability to process additional data, and a sum of the aging-planned bandwidth usage values for each of the adapter interfaces. 31. A system for a converged network, comprising the adapter as claimed in claim 27, and further comprising: At least one server, the at least one server having at least one server interface, wherein the server is configured to include a server proxy and is coupled to a converged network via the server interface; and At least one storage device is configured to be coupled to the adapter, such that the adapter couples the storage device to the network via the adapter interface. The server proxy is coupled and configured to: The server asserts a request to the adapter agent and identifies at least one bandwidth indication provided to the server by the adapter agent in response to the request; and Traffic imbalances on the network can be detected by using at least one bandwidth indication provided to the server by the adapter agent in response to the request. 32. The system of claim 31, wherein the server agent is coupled and configured to: in response to detecting the imbalance, cause the server to redirect storage data traffic on the network from a data path between the server and the storage device to a new data path between the server and the storage device. 33. The system of claim 32, wherein the server agent is coupled and configured to: wait for a time interval of sufficient duration after causing the server to redirect storage data traffic between the server and the storage device from the one data path to the new data path, such that the effect of the redirection to the new data path is reflected in the results of continuous monitoring of traffic on each adapter interface of the adapter agent by each adapter agent; and after the wait, begin evaluating paths to the storage device, including at least one path other than the new data path. 34. The system of claim 33, wherein the time interval of the waiting is determined by a random number selected as a normal variable of the selected interval, subject to a predetermined minimum wait and a maximum wait. 35. The system of claim 31, wherein the adapter is a first adapter, the first adapter including at least one first adapter interface and a first adapter agent, and wherein the system comprises: A second adapter, configured to couple the storage device to the network, wherein the second adapter includes at least one second adapter interface, and the second adapter includes a second adapter agent, and the server agent is coupled and configured to: Monitoring data traffic occurring on each of the server interfaces to determine the bandwidth consumed by each of the server interfaces, and determining the available bandwidth for each of the server interfaces based on the bandwidth consumed; and At least one available bandwidth indication provided to the server by the first adapter proxy in response to a request asserted from the server to the first adapter is identified, wherein each available bandwidth indication indicates the available bandwidth of one first adapter interface; and at least one additional available bandwidth indication provided to the server by the second adapter proxy in response to a request asserted from the server to the second adapter is identified, wherein each additional available bandwidth indication indicates the available bandwidth of one second adapter interface; and The available bandwidth on the path including the server interface and one of the second adapter interfaces is determined as the minimum of the available bandwidth on the server interface and the available bandwidth of the one of the second adapter interfaces. 36. An apparatus for a converged network, configured for use in a system including at least one server coupled to the converged network via at least one server interface, wherein the server includes a server agent, wherein the apparatus is a storage device with an integrated adapter, and includes: Data storage subsystem; and An adapter subsystem, coupled to the data storage subsystem, wherein the adapter subsystem implements the adapter, and the adapter subsystem includes: At least one adapter interface configured to couple the adapter subsystem to the network, and thereby couple the data storage subsystem to the network via the adapter subsystem; and An adapter proxy, wherein the adapter proxy is coupled and configured to: Monitor data traffic occurring on each of the adapter interfaces of the adapter, and generate a bandwidth consumption indication for each adapter interface, wherein the bandwidth consumption indication for each adapter interface indicates the bandwidth consumed by the adapter interface; Generate an available bandwidth indication for each of the adapter interfaces, wherein the available bandwidth indication for each adapter interface indicates the available bandwidth of the adapter interface; and In response to a request from the server agent, the adapter asserts at least one bandwidth indication data to at least one of the adapter interfaces, wherein the at least one bandwidth indication is at least one of the consumed bandwidth indications, or at least one of the available bandwidth indications, or at least one of the consumed bandwidth indications and at least one of the available bandwidth indications. 37. The device of claim 36, wherein the device is a storage device with an integrated adapter, the adapter subsystem implementing the adapter, and the data storage subsystem implementing the storage device. 38. The device of claim 37, wherein the storage device is a disk drive. 39. The device of claim 36, wherein the device is a JBOD with an integrated adapter, the adapter subsystem implementing the adapter, the data storage subsystem implementing the JBOD, and the JBOD includes a set of disk drives. 40. A server configured for use in a system including at least one storage device and at least one adapter coupled to said storage device, wherein the adapter has at least one adapter interface coupled to a converged network, the adapter coupling the storage device to the network via said adapter interface, and the adapter is configured to include an adapter agent, the server comprising: A processing subsystem, configured to include a server agent; and At least one server interface, the at least one server interface being configured to be coupled to the network, wherein the processing subsystem is coupled to the server interface and configured to access the network via the server interface when the server interface is coupled to the network, and wherein the server proxy is coupled and configured to: The processing subsystem asserts a request to the adapter agent and identifies at least one bandwidth indication provided to the server by the adapter agent in response to the request, wherein the at least one bandwidth indication is at least one consumed bandwidth indication, or at least one available bandwidth indication, or at least one consumed bandwidth indication and at least one available bandwidth indication; and Traffic imbalances on the network can be detected by using at least one bandwidth indication provided to the server by the adapter agent in response to the request. 41. The server of claim 40, wherein the server agent is coupled and configured to: in response to detecting the imbalance, cause the server to redirect storage data traffic on the network from a data path between the server and the storage device to a new data path between the server and the storage device. 42. The server of claim 41, wherein the server agent is coupled and configured to: wait for a time interval of sufficient duration after causing the server to redirect storage data traffic between the server and the storage device from the one data path to the new data path, such that the effect of the redirection to the new data path is reflected in the results of continuous monitoring of traffic on each adapter interface of the adapter agent by each adapter agent; and after the wait, begin evaluating paths to the storage device, including at least one path other than the new data path. 43. The server of claim 42, wherein the time interval of the waiting is determined by a random number selected as a normal variable of the selected interval, subject to a predetermined minimum wait and a maximum wait. 44. The server of claim 40, wherein the system includes a first adapter configured to couple the storage device to the network and a second adapter configured to couple the storage device to the network, wherein the first adapter includes at least one first adapter interface and the second adapter includes at least one second adapter interface, the first adapter includes a first adapter proxy and the second adapter includes a second adapter proxy, wherein the server includes at least a first server interface and a second server interface, and wherein the server proxy is coupled and configured to: Monitoring data traffic occurring on each of the server interfaces determines the bandwidth consumed by each of the server interfaces, and based on the bandwidth consumed by each of the server interfaces, determines the available bandwidth for each of the server interfaces; and At least one available bandwidth indication provided by the first adapter agent to the first server interface in response to a request from the processing subsystem to the first adapter assertion is identified, wherein each available bandwidth indication indicates the available bandwidth of one first adapter interface; and at least one additional available bandwidth indication provided by the second adapter agent to the second server interface in response to a request from the processing subsystem to the second adapter assertion is identified, wherein each additional available bandwidth indication indicates the available bandwidth of one second adapter interface; and The available bandwidth on the path including the second server interface and the second adapter interface is determined as the minimum of the available bandwidth on the second server interface and the available bandwidth of the second adapter interface. 45. The server of claim 40, wherein the adapter agent is coupled and configured to generate an available bandwidth indication for each of the adapter interfaces of the adapter. Furthermore, the server proxy is coupled and configured to: Identify at least one of the available bandwidth indications provided to the server interface by the adapter agent in response to the request; and Access the path including the server interface and at least one of the adapter interfaces in a manner that uses the available bandwidth indication. 46. A method for a converged network, comprising the following steps: A request is asserted from a server to an adapter via a converged network, wherein the server is configured to include a server proxy and the adapter is configured to include an adapter proxy; The server proxy is used to identify at least one bandwidth indication provided to the server by the adapter proxy in response to the request, wherein the at least one bandwidth indication is at least one consumed bandwidth indication, or at least one available bandwidth indication, or at least one consumed bandwidth indication and at least one available bandwidth indication; and Traffic imbalances on the network can be detected by using at least one bandwidth indication provided to the server by the adapter agent in response to the request. 47. The method of claim 46, further comprising the step of: using the server proxy to redirect storage data traffic on the network from a data path between the server and the storage device to a new data path between the server and the storage device in response to detecting the imbalance. 48. The method of claim 47, further comprising the steps of: employing the server agent to wait for a sufficiently long time interval after the server redirects storage data traffic between the server and the storage device from the one data path to the new data path, such that the impact of the redirection to the new data path is reflected in the results of continuous monitoring of traffic on each adapter interface of the adapter agent by each adapter agent; and after the wait, beginning to evaluate paths to the storage device, including at least one path other than the new data path. 49. The method of claim 46, further comprising the step of: The server proxy is used to monitor data traffic occurring on each server interface of the server to determine the bandwidth consumed by each server interface, and based on the bandwidth consumed by each server interface, the available bandwidth for each server interface is determined; and The server proxy is used to identify at least one available bandwidth indication provided to the server by a first adapter proxy of a first adapter coupled to the network in response to a request asserted by the server to the first adapter, wherein each available bandwidth indication indicates the available bandwidth of the adapter interface of the first adapter; and to identify at least one additional available bandwidth indication provided to the server by a second adapter proxy of a second adapter coupled to the network in response to a request asserted by the server to the second adapter, wherein each additional available bandwidth indication indicates the available bandwidth of the adapter interface of the second adapter; and The server proxy is used to determine the available bandwidth on the path including one of the server interfaces and the adapter interface of the second adapter as the minimum of the available bandwidth on the server interface and the available bandwidth on the adapter interface of the second adapter.