JP2002509311A - Scalable single-system image operating software for multi-processing computer systems - Google Patents

Abstract

Abstract: A scalable single system image ("S31") operating system architecture for a multi-processing computer system having separate service (16) and computation (18) processors, which is unique to these processors. However, both have shared, common access to all of the computer system memory. Compute processors do not have input / output ("I / O") devices mapped directly to them, while service processors have full I / O capability. The single operating system software image provides a single system application programming interface to application programs across all of the processors, and the communication mechanism between the compute processor and the service processor uses multiple requests to the service processor and each request. And a fast and asynchronous interrupt response. The compute scheduler provides an interface to a service processor running on each compute processor and running operating software.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of multi-processing computer systems. More specifically, the present invention relates to scalable single system image ("S31") operating software for a multi-processing computer system.

[0002] The connection of two or more homogeneous processors to a single monolithic central memory is called multiprocessing. Until recently, due to hardware and software limitations, the total number of physical processors that could efficiently access a single memory was minimal. As the number of processors in a computer system increases, these limitations have reduced the ability to maintain computational efficiency and have reduced the overall scalability of the system. With the advent of faster and cheaper microprocessors, large processor count systems are becoming a reality for hardware.

[0003] Advances in hardware have allowed interconnected networks to multiplex hundreds of processors to a single memory, with minimal loss of efficiency. Due to performance issues with software locking primitives in large configurations, system software designers still do not have the operating software scalability needed to efficiently accommodate large numbers of processors .

[0004]

SUMMARY OF THE INVENTION

In response to these computer system architecture efficiency issues, SRC Computers, Inc., of Colorado Springs, Colorado, has provided systems with large shared memory, Utilizing commercial processors to obtain high-bandwidth input / output ("I / O") functionality has developed an affordable, high-performance computer that replaces traditional high-performance supercomputers. This is achieved by balancing processor speed, memory size, and I / O bandwidth to achieve a high degree of efficiency between the system hardware and software, thus achieving parallelism. The degree of the bigger.

[0005] Disclosed herein is a computer system that utilizes a scalable single system image ("S31") operating software architecture. This architecture enables efficient scalability of operating software from a few processors to hundreds of processors in a multiprocessor environment that efficiently provides a single system image. The S31 architecture of the present invention substantially eliminates the need for massively parallel ("MPP") architectures. This is simply because the distributed memory and message passing synchronization primitives are no longer needed. The flat memory model, which is simple and easy to program, replaces message passing. Common application programming interface / application binary interface ("API /
ABI ") is given to all applications. Eliminating the message passing paradigm greatly improves performance both computationally and in terms of input / output ("I / O").

[0006] Disclosed herein is a scalable single system image ("S31") operating system architecture for a multi-processing computer system having a separate service processor and a compute processor, with a processor between the processors. Have unique differences, but both have shared common access to all computer system memory. Compute processors have no input / output ("I / O") devices mapped directly on them,
The service processor can control the added device. The single operating system software image provides a single system application programming interface to the application program across all processors, and the communication mechanism between the compute processor and the service processor uses multiple requests for the service processor and for each request Enables fast asynchronous interrupt response. The compute scheduler runs on each compute processor and provides an interface to the service processor where the operating system software runs.

[0007] The S31 architecture also improves cache performance by reducing cache conflicts. This conflict is reduced because the operating software must be able to process application requests during the process.
Because it no longer forces application data from the cache. As the processor speed in terms of memory latency increases, as the latest generation of multiprocessors shows, this performance improvement becomes more important.

[0008] Specifically disclosed herein is a multiprocessor computer system that includes operating software. The computer system includes a first plurality of service processors functioning in conjunction with operating software, the service processor handling all input / output functions in the computer system. The second plurality of computation processors function in conjunction with the computation scheduler. The operating software and the computation scheduler provide a communication medium between the service processor and the computation processor.

[0009] Further disclosed herein is a multiprocessor computer system that includes operating software. The computer system includes a first plurality of service processors and a second plurality of computing processors. Each of these services works in conjunction with operating software to handle all input / output functions in the computer system. Each of these calculations works in conjunction with a calculation scheduler, and the operating software and calculation scheduler provide a communication medium between the service processor and the calculation processor.

By referring to the following description of the preferred embodiment in connection with the accompanying drawings,
The above and other features and objects of the present invention and the manner of achieving them will become more apparent and the invention itself is best understood.

[0011]

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to FIGS. 1A and 1B, a multi-processing computer system 10 according to the present invention is shown. Exemplary computer system 10, in pertinent part, including interconnected segments 12 _{0-12 15} any number of principles of the invention, equally applicable to multiple processors of any scalable system It is possible. Various segments 1 as described in detail below
2 _{0-12 15} is coupled via a number of trunk lines 14 ₀ to 14 _15.

[0012] Each of the segments 12, 16 ₃ from the service processor 16 ₀ (service processor 16 ₀ further functioning as the master boot device) and a number of functionally distinct processing elements in the form of a calculation processor 18 ₀ 18 ₁₅ including. The service processor 16 is coupled to a number of peripheral component interconnect ("PCI") interface modules 20, and in the illustrated embodiment, each service processor is coupled to two such modules 20 and the service processor 16 Allows all I / O functions of segment 12 to be performed.

Computer system 10 further includes a serial-PCI interface 22 that couples system console 24 to at least one segment 12 of computer system 10. The system console 24 operates to allow a user of the computer system 10 to download boot information to the computer system 10, configure devices, monitor status, and perform diagnostic functions. No matter how many segments 12 are configured in computer system 10, only one system console 24 is required.

A boot device 26 (eg, Iomega Corporation, Roy, Utah)
, Roy UT), available from, JAZ (R) removable disk computer mass storage device) is also coupled to the master boot service processor 16 ₀ through one of the PCI module 20. PCI module is coupled from the service processor 16 ₁ to 16 ₃ 20, for example, be utilized to couple the segment 12 to any other peripheral devices such as ₅ from ₀ disk array 28 28, one of any these The above may be replaced with, for example, an Ethernet connection.

Computer system 10 includes sophisticated hardware and building blocks, which are commercially available and are somewhat improved to accommodate the specifics of high performance computing (“HPC”). On the hardware side, the basic unit of computer system 10 is segment 12. Each segment 12 includes a compute and service processor 18, 16 elements, memory, power supplies and a crossbar switch assembly. Computer system 10 is "scalable" in that end users can configure a system consisting of one to sixteen interconnected segments 12. Each segment 12 includes a total of 20 processors, namely, 16 computation processors 18 and 4 service processors 16. In the preferred embodiment, compute processor 18 has four processors (eg, Intel Corporat, Santa Clara, Calif.).
ion, Santa Clara, CA), available from Deschutes ^TM microprocessor) and eight interface chips (i.e. two calculation processor 18 per)
And may be on individual assemblies including Each calculation processor 18 has 300
Having an internal processor clock rate of greater than 100 MHz and a system clock rate of greater than 100 MHz, the interface chip provides a connection between the computing processor 18 and a memory switch that connects to the memory, as described and described in detail below. .

The service processor 16 may be included on a sub-processor assembly,
It is responsible for all inputs and outputs at computer system 10. Each of the service processor assemblies is a processor (same type as the computation processor 18)
Includes two interface chips, two 1 Mbyte I / O buffers and two bidirectional PCI buses. Each PCI bus has a single connector. All I / O ports have equal priority DMA capabilities to the processor. P
The CI module 20 serves two purposes, depending on which service processor 16 uses them. PCI connector master boot service processor 16 on ₀ is used to connect to the boot device 26 and the system console 24. PCI module 20 from the regular service processor 16 ₁ 16 on ₃ is used for all other peripheral devices. Supported P
Some of the CI-based interconnects include Small Computer System Interface ("SCSI"), Fiber Distributed Data Interface ("FDDI")
, High Performance Parallel Interface ("HIPPI") and others. Each PC
The I-bus has a corresponding commercial host adapter.

Separating the service functions from the computation functions allows simultaneous execution of numerical processing and operating system duty and services of external peripherals.

Referring still to FIG. 2, the computer system 10 of FIGS. 1A and 1B
The interconnect strategy has 16 trunk lines 14₀From 14_Fifteen16 segments 12 interconnected by₀From 12_FifteenThis is shown in more detail in an implementation that employs As shown, a number of memory banks 50₀From 50_FifteenAre the calculation processors 18₀From 18_Fifteen(Since there are 16 memory banks 50 per segment 12, there are 16 segment banks.
In the computer system 10 with the memory 12, a total of 256 memory banks 50 are provided.
Will form part of the computer system 10
14₀From 14_FifteenAnd the same number of memory switches 52₀From 52_FifteenThrough each
Are combined. Memory bank 50₀From 50_FifteenThe memory utilized for synchronous random access memory ("SSRAM") or other suitable high speed
It may be a memory device. As shown, segment 12₀From 12_FifteenEach, for example, the trunk line 14₀From 14_FifteenAnd the same number of processor switches 54₀From 54_Fifteen20 processors (4 service processors 16) coupled via a corresponding one of₀From 16_ThreeAnd 16 calculation processors 18 ₀ From 18_Fifteen)including.

Each segment 12 is connected to all other segments 1 via a crossbar switch.
Interconnect with 2. Computer system 10 crossbar switch technology allows segments 12 to cross segment boundaries and even within individual segments 12.
It allows to have a uniform memory access time. It also allows computer system 10 to employ a single memory access protocol for all memories in the system. Crossbar switches can utilize high speed field programmable gate arrays ("FPGAs") to provide an interconnect path between memory and processor, regardless of where the processor and memory are physically located. . This crossbar switch interconnects every segment 12 and allows processors and memories located in different segments 12 to communicate with uniform latency. In the preferred embodiment, each crossbar switch has a latency of one clock per layer, which includes the reconfiguration time. 16 segments 1 utilizing 320 processors 16, 18
In a computer system 10 with two, only two crossbar layers are required.

As mentioned above, the computer system 10 preferably includes the memory bank 5
0 may use SSRAM. This provides a component cycle time of 6 nanoseconds. Each memory bank 50 has 64 to 256
Supports M bytes of memory. Each compute processor 18 supports one memory bank 50, each memory bank 50 being 256 bits wide, plus 32 parity bits, for a total of 288 bits. In addition, if the size of the memory bank 50 is designed to match the size of the cache line, it is possible to make one bank access to all the cache lines. Completing the parity check on the address and data packets may provide read and write memory error correction.

The parity check for an address packet can be the same for both read and write functions. The new and old parity bits are compared to determine whether the memory read or write should continue or abort. If a memory "write" occurs, a parity check may be made for each data packet arriving at the memory. Each of these data packets has an 8-bit parity code appended to it. When a data packet arrives in memory, a new 8-bit parity code is generated for the data packet and the old and new parity codes are compared. This comparison yields one of two types of codes: a single bit error ("SBE") or a double or multiple bit error ("DBE"). Single bit errors may be corrected on the data packet before it is put into memory. For double or multiple bit errors, the data packet is not written to memory and is reported to the processor, which retries the data packet reference. When a memory "read" occurs, each of the data packets read from the memory generates an 8-bit parity code. This parity code is sent to the processor together with the data. The processor performs single error correction and double error detection ("SECEDED") on each data packet.

Still referring to FIG. 3, 16 segments 12 containing 64 service processors 16 and 256 compute processors 18, for a total of 320 processors
A simplified diagram of a computer system 10 comprising a is shown. Service processor 16 and compute processor 18 interface to computer program application software through a scalable single system image ("S31") layer 60, as described in more detail below. The service processor 16 handles all I / O operations as described above, and also handles the execution of the computer system 10 operating system.

Still referring to FIG. 4, the S31 layer 60 is shown in more detail, which is associated with the computer program code application 62 software and service and computing processors 16,18. The S31 layer 60 is on top of the application 62 and includes a common API / ABI layer 64 and further a calculation scheduler 66 layer and an operating software 68 layer. Calculating the scheduler 66 is the interface to the calculation processor 18 ₀ to 18 _15, while the operating software 68 interface from the service processor 16 ₀ to 16 _3. The operating software 68 provides an interrupt response signal 70 to the calculation scheduler to control the operation of the calculation processor 18 as shown. A plurality of memory communication buffer 72 receives the 18 ₁₅ or data from the various computing processors 18 ₀ provides data from the service processor 16 ₀ to 16 _3.

[0024] As shown described above, a preferred realization flounder scalable single system interconnect architecture of the present invention, a uniform memory access from a plurality of memory banks 50 ₀ over common shared memory containing 50 _N Provided on a multiprocessor computer system 10. As described above, the processor subsystem, two groups, i.e., those having an I / O connection, and the service processor 16 ₀ 16 _N, having no I / O connections, i.e. from the calculation processor 18 ₀ 18 _N.

S 31 utilizes a software environment consisting of the service and computing processors 16, 18. A single copy of operating system software 68
It exists across all processors 16 in the service partition. A separate computation scheduler 66 resides within each computation processor 18. This software model works with powerful "single system images" to ensure a global resource sharing paradigm. To use this architecture efficiently,
A highly scalable thread of execution must be present in both the operating system software 68 and the user application 62 software design model in conjunction with the advanced software "multithreading."

Due to the powerful shared memory hardware architecture of computer system 10, this choice of operating software 68 features eliminates the need for a message passing paradigm for communication between processors or between hardware boundaries. However, some simple communication is required between the compute processor 18 and the service processor 16. No physical hardware boundaries are apparent to the application 62 program. The physical processors 16, 18 all provide the same application programming interface (API) to the end user.

To complete the scalability model, the user application 62
Multiple threads of execution in application user space can be started and terminated, further eliminating the overhead of operating system software 68 and increasing scalability as the number of physical processors increases. .

Under the S31 architecture, a user application 62 makes a request to the system within a normal system software mechanism. Application 62
Does not know if the application 62 is running on the service processor 16 or on the computation processor 18. When the request is executed by the service processor 16, the request is entered directly into the operating system software 68 via the normal operating system bus for processing.

If the request is to be executed on compute processor 18, the request
Processed by The requesting thread is placed on the pending queue of the service processor 16 and the compute processor 18 issues a request to the service processor 16 to check the queue. Operating software 68 running in service processor 16 examines the request queue and processes the request as if it had occurred on the service processor.

The requesting processor 18 is suspended until an interrupt response or is placed in a general scheduling table maintained by operating software 68 to dispatch further work. The service processor 16 queues the application thread, which is
8 and responds to the original request by restoring the original application context, which returns the application 62 to execution.

Very small computation scheduler 6 to reference operating software 68
It is noted that no major changes in operating software 68 are required, other than the addition of 6 and smaller additional components. Any physical processors 16 in the service partition can execute simultaneously in operating system 68. Critical data areas can be locked using standard locking primitives currently found in the underlying hardware.

The basic component of the software environment is operating system software 68. In a preferred embodiment, the computer system 10 is a Sun Microsystems, Inc., Palo Alto, California.
c. Palo Alto, CA) available from SunSoft, Solaris 2.6 Operating Sy.
stem), modified to achieve better performance across multiple compute and service processors 18, 16 by restricting the operating system to run only in service processor 16. May be. As described above, this technique further includes a calculation scheduler 6
This is accomplished by utilizing OS 6 to communicate operating system requests and scheduling information between service processor 16 and compute processor 18.

In other words, a single copy of operating system software 68
Running in all service processors 16, while a separate computation scheduler 6
6 is in each calculation processor 18. This software model, coupled with powerful scalable single system images, provides for global resource sharing. The computer system 10 of the present invention, in conjunction with advanced software "multithreading", provides a highly scalable thread of execution in both the operating system software 68 and the user application software 62.

Although the principles of the present invention have been described above in connection with a particular computer architecture, any number of services and / or computing processors may be utilized,
It is clear that the above description is to be regarded as illustrative and not restrictive of the scope of the invention. In particular, it is recognized that the teachings of the above disclosure will suggest other modifications to those skilled in the art. Such variations would include other features that are known per se and may be used instead of or in addition to the features described herein. Although the claims are expressly stated in this application for a particular combination of features, the scope disclosed herein also includes any novel features or modifications that would be apparent to those skilled in the art.
Includes any novel combination of features, either explicitly or implicitly disclosed, or any generalization or variation thereof, which relates to the same invention as claimed herein in any claim. Whether or not to do so, whether or not it alleviates all or any of the same technical problems addressed by the present invention. Applicant reserves the right to expressly claim new claims for such features or / and combinations of such features during the prosecution of this application or of any further application derived therefrom.

[Brief description of the drawings]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A is an implementation of the invention that includes one to sixteen segments connected to each other by an equal number of trunk lines, each segment including a number of compute and service processors and also a memory and crossbar switch assembly. 1 is a functional block system overview diagram illustrating a computer system according to an example.

FIG. 1B is an implementation of the present invention that includes one to sixteen segments connected to each other by an equal number of trunk lines, each segment including a number of compute and service processors and also a memory and crossbar switch assembly. 1 is a functional block system overview diagram illustrating a computer system according to an example.

FIG. 2 is a simplified functional block diagram of an interconnect strategy for the computer system of FIGS. 1A and 1B.

FIG. 3 includes a 256 computation processor and 64 service processors that interface to a computer program application software via a scalable single system image (“S31”) in accordance with the present invention.
FIG. 3 is a simplified functional block diagram of the computer system of FIGS. 1A, 1B and 2 illustrating a segment system.

FIG. 4 illustrates the system of FIG. 3 illustrating an interface to computer program application software via a common API / ABI to a compute processor via a compute scheduler and a service processor via operating software to provide an interrupt response to the compute scheduler. FIG. 4 is a more detailed block diagram of a single-segment computer system corresponding to part of FIG.

[Procedure amendment]

[Submission date] July 24, 2000 (2000.7.24)

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Claims

[Correction method] Change

[Correction contents]

[Claims]

──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Bertoni, Jonathan El, United States, 80918 Colorado, Colorado Springs, Paragon Drive, 2630, Apartment E F-term (reference) 5B045 AA01 BB12 BB28 BB29 DD12 EE02 GG07 GG08 GG12 5B098 AA10 GA01 GA02 GC16 [Continuation of summary]

Claims (19)

[Claims] 1. A multi-processor computer system including operating software, the system including a first plurality of service processors operative in conjunction with the operating software, wherein the service processor is configured to provide all input / outputs in the computer system. Processing an output function, further comprising a second plurality of computation processors, wherein the computation processors function in conjunction with a computation scheduler, wherein the operating software and the computation scheduler are located between the service processor and the computation processors. Give communication medium,
Multiprocessor computer system. 2. The multiprocessor computer system according to claim 1, wherein said communication medium operates to enable a plurality of input / output requests to said service processor. 3. The multiprocessor computer system according to claim 2, wherein said communication medium is operative to enable an asynchronous response to said input / output request. 4. The multiprocessor computer system of claim 1, wherein said service processor and said computing processor have shared access to a plurality of associated memory banks. 5. The system of claim 1, further comprising: a common application programming interface operatively coupled to the computation scheduler and the operating software;
The multiprocessor computer system of claim 1, wherein the computation scheduler and the operating software include a single system image application programming interface. 6. The multiprocessor computer system of claim 5, wherein said single system image application programming interface is scalable across said first plurality of service processors and said second plurality of computing processors. . 7. The multiprocessor of claim 1, further comprising a system console coupled to at least one of said first plurality of service processors to enable a user of said computer system to interact therewith. Computer system. 8. The at least one of the first plurality of service processors.
A boot device coupled to the computer and booting the computer system.
A multiprocessor computer system according to claim 7. 9. The multiprocessor computer system according to claim 1, further comprising at least one computer mass storage device coupled to at least one of said first plurality of service processors. 10. The first plurality of service processors and the second plurality of computation processors include a first computer system segment, wherein the computer system functions in conjunction with the operating software. Further comprising at least one additional computer system segment comprising: a plurality of service processors; and a fourth plurality of compute processors operative in conjunction with the compute scheduler.
The multiprocessor computer system according to claim 1. 11. A multiprocessor computer system including operating software, the system including a first plurality of service processors and a second plurality of computing processors, each of the service processors associated with the operating software. Functioning to handle all input / output functions in the computer system, each of the computing processors functioning in conjunction with a computing scheduler, wherein the operating software and the computing scheduler operate between the service processor and the computing processor. Multiprocessor computer system that provides a communication medium to 12. The multiprocessor computer system according to claim 11, wherein said communication medium operates to enable a plurality of input / output requests to said service processor. 13. The multiprocessor computer system according to claim 12, wherein said communication medium operates to enable asynchronous response to said input / output request. 14. The multiprocessor computer system according to claim 11, wherein said services and computations have shared access to a plurality of associated memory banks. 15. The system further comprising a common application programming interface operatively coupled to the computation scheduler and the operating software, wherein the application programming interface, the computation scheduler and the operating software include a single system image application programming interface. The multiprocessor computer system of claim 11, wherein: 16. The multiprocessor computer system of claim 15, wherein said single system image application programming interface is scalable over said N computational segments. 17. A system console coupled to at least one of said first plurality of service processors of one of said N computational segments to allow a user of said computer system to interact therewith. The multiprocessor computer system of claim 11, comprising: 18. The multi-processor of claim 17, further comprising a boot device coupled to the at least one of the first plurality of service processors in one of the N computational segments to boot the computer system. Processor computer system. 19. The multi-processor of claim 11, further comprising at least one computer mass storage device coupled to at least one of the first plurality of service processors in one of the N computational segments. Processor computer system.