WO2024213056A1 - Method for controlling high-performance computing cluster, and electronic device and storage medium - Google Patents

Abstract

Disclosed in the present application are a method for controlling a high-performance computing cluster, and an electronic device and a storage medium. The method comprises: upon receiving a job migration request, a scheduling node acquiring a job state of a target job, wherein the job migration request is used for migrating the target job; when the job state is a running state, the scheduling node determining a first computing node from a high-performance computing cluster, and determining a target process on the first computing node, wherein the target job is run on the first computing node, and the target process is a process, on the first computing node, which corresponds to the target job; and the scheduling node sending migration information to the first computing node, wherein the migration information is used for controlling the first computing node to dump a target process mirror image into a preset storage device. The present application solves the technical problem in the related art of the relatively low reliability of a high-performance computing cluster.

Description

Control method, electronic device and storage medium of high performance computing cluster

This application claims priority to a Chinese patent application filed with the Chinese Patent Office on April 13, 2023, with application number 202310441527.7 and invention name “Control method, electronic device and storage medium for high-performance computing cluster”, the entire contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of control of computing clusters, and in particular to a control method, electronic device and storage medium for a high-performance computing cluster.

Background Art

Traditional high-performance computing clusters (HPC clusters) are mainly composed of login nodes, scheduling nodes, and computing nodes. After the scheduler on the scheduling node of the high-performance computing cluster receives a job submission request, it can select a suitable computing node from the HPC cluster to run the job. Once the job starts running on the computing node, it is difficult to perceive the computing node where the job is located. If a computing node fails, the entire job calculation will fail.

To address the above-mentioned problems, no effective solution has been proposed yet.

Summary of the invention

The embodiments of the present application provide a control method, an electronic device, and a storage medium for a high-performance computing cluster, so as to at least solve the technical problem of low reliability of the high-performance computing cluster in the related art.

According to one aspect of an embodiment of the present application, a control method for a high-performance computing cluster is provided, including: when a scheduling node receives a job migration request, obtaining the job status of a target job, wherein the job migration request is used to migrate the target job; when the job status is running, the scheduling node determines a first computing node from the high-performance computing cluster, and determines a target process on the first computing node, wherein the target job is running on the first computing node, and the target process is a process on the first computing node corresponding to the target job; the scheduling node sends migration information to the first computing node, wherein the migration information is used to control the first computing node to dump the target process image to a preset storage device.

According to one aspect of an embodiment of the present application, a control method for a high-performance computing cluster is also provided, including: a first computing node receives migration information sent by a scheduling node, wherein a target job is running on the first computing node, the migration information is sent by the scheduling node when receiving a job migration request and determining that the job status of the target job is running, and the job migration request is used to migrate the target job; based on the migration information, the first computing node dumps a target process image corresponding to the target job to a preset storage device

According to another aspect of an embodiment of the present application, a high-performance computing cluster is also provided, including: a computing node for running jobs; a preset storage device for storing process images; a scheduling node connected to the computing node, and used to obtain the job status of the target job when a job migration request is received, and when the job status is running, determine the first computing node and determine the target process on the first computing node, wherein the job migration request is used to migrate the target job, wherein the target job is running on the first computing node, and the target process is the process on the first computing node corresponding to the target job; the first computing node is connected to the preset storage device, and is used to store the target process image to the preset storage device; Set up a storage device.

According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes any one of the methods in the above embodiments when running.

According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the storage medium is located is controlled to execute any one of the methods in the above embodiments.

In the embodiment of the present application, first, the scheduling node obtains the job status of the target job when receiving the job migration request, wherein the job migration request is used to migrate the target job; when the job status is running, the scheduling node determines the first computing node from the high-performance computing cluster, and determines the target process on the first computing node, wherein the target job is running on the first computing node, and the target process is the process corresponding to the target job on the first computing node; the scheduling node sends migration information to the first computing node, wherein the migration information is used to control the first computing node to dump the target process image to the preset storage device, so as to achieve the purpose of improving the reliability of the high-performance computing cluster. It is easy to notice that, when the job status is running, the computing nodes in the high-performance cluster can be checked, and the first computing node running the target job can be determined, and the target process image on the first computing node can be dumped to the preset storage device, so as to achieve the dumping of the target process corresponding to the target job without the user's perception, and the target job on the potential fault node can be easily migrated to reduce the risk of job failure, thereby improving the reliability of the target job operation and improving the reliability of the high-performance computing cluster. Thus, the technical problem of low reliability of the high-performance computing cluster in the related technology is solved.

It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

FIG1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a control method for a high-performance computing cluster according to an embodiment of the present application;

FIG2 is a block diagram of a computing environment according to an embodiment of the present application;

FIG3 is a structural block diagram of a service grid according to an embodiment of the present application;

FIG4 is a flow chart of a control method for a high performance computing cluster according to Embodiment 1 of the present application;

FIG5 is a schematic diagram of a traditional high performance computing cluster according to an embodiment of the present application;

FIG6 is a schematic diagram of a control structure of a high performance computing cluster according to an embodiment of the present application;

7 is a flow chart of a high performance computing cluster performing a dump operation process to persistent storage according to an embodiment of the present application;

8 is a flowchart of restoring a high performance computing cluster job from a dumped process image according to an embodiment of the present application;

FIG9 is a schematic diagram of a control structure of another high performance computing cluster according to an embodiment of the present application;

FIG10 is a schematic diagram of a control structure of another high performance computing cluster according to an embodiment of the present application;

FIG11 is a flow chart of a control method for a high performance computing cluster according to Embodiment 2 of the present application;

FIG12 is a schematic diagram of a high performance computing cluster according to Embodiment 2 of the present application;

FIG13 is a schematic diagram of a control device for a high performance computing cluster according to Embodiment 4 of the present application;

FIG14 is a schematic diagram of a control device for a high performance computing cluster according to Embodiment 5 of the present application;

FIG. 15 is a structural block diagram of a computer terminal according to an embodiment of the present application.

DETAILED DESCRIPTION

In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

Job migration: a job can be migrated from one node computer to another node computer with a lighter workload or suitable for processing the job;

High Performance Computing (HPC) clusters: These clusters can handle complex computing problems simultaneously by connecting multiple machines.

HPC job scheduling: job scheduling performed by a scheduler on a high-performance computing cluster;

Mirror dump process: It is the information storage process that produces a mirror view of the same data on any two or more disks.

Example 1

According to an embodiment of the present application, a control method for a high-performance computing cluster is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

The method embodiment provided in Embodiment 1 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. FIG1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a control method for a high-performance computing cluster according to an embodiment of the present application. As shown in FIG1 , a computer terminal 10 (or mobile device) may include one or more (shown in the figure as 102a, 102b, ..., 102n) processors 102, a processor for storing data A memory 104, and a transmission module 106 for communication functions, wherein the processor 102 may include but is not limited to a processing device such as a microcontroller unit (MCU) or a programmable logic device (FPGA). In addition, it may also include: a display, an input/output interface (I/O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and/or a camera. It will be appreciated by those skilled in the art that the structure shown in FIG. 1 is only illustrative and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may also include more or fewer components than those shown in FIG. 1, or have a configuration different from that shown in FIG. 1.

It should be noted that the one or more processors 102 and/or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

The memory 104 can be used to store software programs and modules of application software, such as the program instructions/data storage device corresponding to the control method of the high-performance computing cluster in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the control method of the high-performance computing cluster described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

The display may be, for example, a touch screen liquid crystal display (LCD), which enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

The hardware structure block diagram shown in FIG1 can be used not only as an exemplary block diagram of the above-mentioned computer terminal 10 (or mobile device), but also as an exemplary block diagram of the above-mentioned server. In an optional embodiment, FIG2 shows an embodiment of using the computer terminal 10 (or mobile device) shown in FIG1 as a computing node in a computing environment 201 in a block diagram. FIG2 is a structural block diagram of a computing environment according to an embodiment of the present application. As shown in FIG2, the computing environment 201 includes multiple (210-1, 210-2, ..., are used in the figure to illustrate) computing nodes (such as servers) running on a distributed network. The computing nodes all contain local processing and memory resources, and the terminal user 202 can remotely run applications or store data in the computing environment 201. The application can be provided as multiple services 220-1, 220-2, 220-3 and 220-4 in the computing environment 201, representing services "A", "D", "E" and "H" respectively.

The end user 202 can provide and access services through a web browser or other software application on the client. In some embodiments, the end user 202's provision and/or request can be provided to the ingress gateway 230. The ingress gateway 230 may include a corresponding agent to handle the provision of services (one or more services provided in the computing environment 201). Should and/or request.

Services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and/or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine to execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container can be started to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.

In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, as shown in FIG2 , a service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). A Pod can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers in a Pod process requests related to one or more corresponding functions of the service, and the proxy 245 typically controls network functions related to the service, such as routing, load balancing, etc. Other services can also be equipped with similar Pods.

During operation, executing a user request from the end user 202 may require invoking one or more services in the computing environment 201, and executing one or more functions of one service may require invoking one or more functions of another service. As shown in FIG2 , service “A” 220-1 receives a user request from the end user 202 from the ingress gateway 230, service “A” 220-1 may call service “D” 220-2, and service “D” 220-2 may request service “E” 220-3 to execute one or more functions.

The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.

In another optional embodiment, FIG3 shows a block diagram of an embodiment of using the computer terminal 10 (or mobile device) shown in FIG1 as a service grid. FIG3 is a structural block diagram of a service grid according to an embodiment of the present application. As shown in FIG3, the service grid 300 is mainly used to facilitate secure and reliable communication between multiple microservices. Microservices refer to decomposing an application into multiple smaller services or instances and distributing them on different clusters/machines to run.

As shown in Fig. 3, microservices may include application service instance A and application service instance B, which form a functional application layer of the service grid 300. In one embodiment, application service instance A runs in the form of container/process 308 on machine/workload container group 314 (Pod), and application service instance B runs in the form of container/process 310 on machine/workload container group 316 (Pod).

In one implementation, application service instance A may be a product query service, and application service instance B may be a product ordering service.

As shown in FIG3 , application service instance A and grid proxy (sidecar) 303 coexist in machine workload container group 614, and application service instance B and grid proxy 305 coexist in machine workload container 314. Grid proxy 303 and grid proxy 305 form the data plane layer (dataplane) of service grid 300. Grid proxy 303 and grid proxy 305 run in the form of container/process 304 and container/process 306, respectively, and can receive request 312 for commodity query service, and grid proxy 303 and application service instance A can communicate bidirectionally, and grid proxy 305 and application service instance B can communicate bidirectionally. In addition, grid proxy 303 and grid proxy 305 can also communicate bidirectionally.

In one embodiment, the traffic of application service instance A is routed to the appropriate destination through grid proxy 303. The network traffic of application service instance B is routed to the appropriate destination through the grid proxy 305. It should be noted that the network traffic mentioned here includes but is not limited to Hyper Text Transfer Protocol (HTTP), Representational State Transfer (REST), high-performance, general open source framework (google Remote Procedure Call, gRPC), open source in-memory data structure storage system (Redis), etc.

In one embodiment, the function of extending the data plane layer can be achieved by writing a custom filter for the proxy (Envoy) in the service mesh 300. The service mesh proxy configuration can be to enable the service mesh to correctly proxy service traffic and achieve service intercommunication and service governance. Mesh proxy 303 and mesh proxy 305 can be configured to perform at least one of the following functions: service discovery, health checking, routing, load balancing, authentication and authorization, and observability.

As shown in FIG3 , the service grid 300 also includes a control plane layer. The control plane layer may be a group of services running in a dedicated namespace, and these services are hosted by a hosted control plane component 301 in a machine/workload container group (machine/Pod) 302. As shown in FIG3 , the hosted control plane component 301 communicates bidirectionally with the grid agent 303 and the grid agent 305. The hosted control plane component 301 is configured to perform some control management functions. For example, the hosted control plane component 301 receives telemetry data transmitted by the grid agent 303 and the grid agent 305, and can further aggregate the telemetry data. For these services, the hosted control plane component 301 can also provide a user-oriented application programming interface (API) to more easily manipulate network behavior and provide configuration data to the grid agent 303 and the grid agent 305.

In the above operating environment, the present application provides a control method for a high-performance computing cluster as shown in FIG4. FIG4 is a flow chart of a control method for a high-performance computing cluster according to Embodiment 1 of the present application. As shown in FIG4, the method includes:

Step S402: upon receiving the job migration request, the scheduling node obtains the job status of the target job.

The job migration request is used to migrate the target job.

The above-mentioned target job (Job) can be a task to be processed. In the e-commerce field, the target job can be an order, product information, etc. In the medical field, the job can be patient information, medical images, etc. There is no limitation on the target job here, and it can be set according to actual conditions. It can be a task to be processed in any field.

The target job may be a job on a node with relatively low utilization, or may be a job on a potential faulty computing node. The risk of job failure may be reduced by migrating the job on the potential faulty computing node.

The above-mentioned job migration request may be generated according to the demand for monitoring the target job, or may be generated periodically, which is not limited here, and the job migration request of the target job may be generated according to the actual situation.

The above-mentioned scheduling node may be a scheduler.

In an optional embodiment, the cluster administrator may generate a job migration request by triggering a checkpoint operation on the target job. After receiving the target job migration request, the scheduler may check the job status of the target job to migrate the target job according to the job status.

In another optional embodiment, when a job migration request is received, the job status of the target job can be checked to determine whether the job status of the target job is running. If the job status is that the target job is running, the target process of the target job can be dumped to facilitate monitoring of the target job.

Step S404: When the job status is running, the scheduling node determines a first computing node from the high-performance computing cluster, and determines a target process on the first computing node.

The target job is running on the first computing node, and the target process is a process on the first computing node corresponding to the target job.

The above-mentioned high-performance computing cluster can be used to represent a computing cluster that processes complex computing problems simultaneously by connecting multiple machines.

The first computing node mentioned above may be one or more, which is not limited here.

In an optional embodiment, when the job status is running, the scheduler can search the scheduling database for a list of computing nodes corresponding to the target job and a list of process identifiers (IDs) on the computing nodes, that is, determine the first computing node from the high-performance cluster and determine the target process on the first computing node.

Step S406: The scheduling node sends migration information to the first computing node.

The migration information is used to control the first computing node.

The above-mentioned preset storage device may be any pre-set storage device. For example, the preset storage device may be a device including a persistent storage path, but is not limited thereto. This is only described as an example.

In an optional embodiment, a checkpoint-restart service in a high-performance cluster may be called to perform a checkpoint dump on a target process of a target job, and the target process may be dumped to a preset storage device.

Furthermore, after the target process image is dumped to the preset storage device, the checkpoint-restart service can return the result to the scheduler. The scheduler comprehensively calculates the results of each node in the node list. When each node on the node list returns success, the checkpoint operation of the target job is considered to be successfully completed. At this time, the scheduling node can update the job status and move the target job to the waiting queue, and set the conditions for being scheduled again.

If the checkpoint-restart service returns a list of computing nodes for the job and some nodes return success or some nodes return failure, the scheduling node communicates with the successful nodes to resume operation and can return the operation failure and related information to the administrator.

Through the above-mentioned method of the present application, the utilization rate of the HPC cluster can be improved and the cluster throughput can be increased. The target job in the high-performance cluster can be migrated, so as to free up more available nodes for the subsequent jobs, and the jobs that could not be put into operation immediately on the scheduling queue can be put into operation as soon as possible, thereby increasing the number of jobs that can be executed by the cluster per unit time, thereby improving the cluster throughput and increasing efficiency.

Through the above-mentioned method of the present application, the failure probability of HPC jobs can be reduced. By monitoring and analyzing the computer cluster, the running HPC jobs can be migrated to other nodes in the cluster in advance for nodes with potential failure risks, thereby reducing job failures caused by unexpected downtime or other failures.

Through the above-mentioned method of the present application, it is also possible to provide convenience for cluster resource maintenance. When cluster software updates or other operation and maintenance actions need to be implemented, the system administrator can migrate the jobs on the nodes to be operated, and then implement resource maintenance operations without waiting for the HPC job to be completed.

Through the above-mentioned method of the present application, the power consumption of the HPC cluster can also be reduced, which is helpful for carbon neutrality. The jobs on the nodes with relatively low utilization on the HPC cluster can be migrated. For example, the jobs can be concentrated on certain cabinets in the computer room, and then the unloaded nodes or cabinets and other equipment can be powered off to save power or put into hibernation, which helps to reduce power consumption.

Through the above steps, the scheduling node first obtains the job status of the target job when receiving the job migration request, wherein the job migration request is used to migrate the target job; when the job status is running, the scheduling node The scheduling node determines the first computing node from the high-performance computing cluster and determines the target process on the first computing node, wherein the target job is running on the first computing node, and the target process is the process corresponding to the target job on the first computing node; the scheduling node sends migration information to the first computing node, wherein the migration information is used to control the first computing node to dump the target process image to the preset storage device, so as to achieve the purpose of improving the reliability of the high-performance computing cluster. It is easy to notice that when the job status is running, the computing nodes in the high-performance cluster can be checked, and the first computing node running the target job can be determined, and the target process image on the first computing node can be dumped to the preset storage device, so as to achieve the dumping of the target process corresponding to the target job without the user's perception, and the target job on the potential fault node can be easily migrated to reduce the risk of job failure, thereby improving the reliability of the target job operation and achieving the purpose of improving the reliability of the high-performance computing cluster. Thus, the technical problem of low reliability of the high-performance computing cluster in the related technology is solved.

Currently, there are mature virtual machine hot migration methods in cloud computing technology. If an HPC cluster is built using virtual machines, the virtual machines are generally used as job clusters to act as job computing nodes, but this does not include actions in the HPC job scripts where users create or use virtual machines. That is, virtual machines are generally not considered in HPC job systems. In HPC scenarios, bare metal machines are generally used to improve performance. Virtual machine migration solutions implemented using programs are not suitable for use in HPC scenarios.

FIG5 is a schematic diagram of a traditional high-performance computing cluster according to an embodiment of the present application. As shown in FIG5 , after a cluster user logs in to the login node, he can submit an HPC job to the job scheduling node of the cluster. Usually, an HPC job consists of a group of multi-machine parallel computing processes, which are specifically controlled by the script and submission command for submitting the job. After the scheduler on the job scheduling node receives the job submission request, it will select a suitable computing node from the computer cluster to run the job, wherein multiple computing nodes in the computer cluster can share storage. After the job starts running on the computing node, the HPC scheduler cannot perceive the computing node where the mobile job is located except for conventional process control and monitoring. If a computing node fails, the entire job will fail to calculate.

FIG6 is a schematic diagram of a control structure of a high performance computing cluster according to an embodiment of the present application. As shown in FIG6 ,

The first step is that when the computing cluster is started, the computing nodes contained in the computing cluster will start the checkpoint-restart service, which implements the checkpoint-restart action for the process corresponding to the HPC job and provides an access interface or application programming interface (Application Programming Interface, referred to as API) to the scheduling node;

In the second step, the user logs in to the login node and submits the job to the HPC cluster. Cluster users submit jobs according to the normal process and can ignore the details of the job submission process.

In the third and fourth steps, the user submits the job to the scheduler in the scheduling queue and waits for the HPC cluster scheduler to select a computing node to execute the job. In order to facilitate the job scheduling node to support job migration, the relevant algorithms, queues, and databases on the scheduling node need to be modified as follows:

For traditional HPC job states, there are R (running) and Q (queuing), etc. The addition of C (CheckPointed) indicates that the corresponding job is in the checkpoint, and the process corresponding to the job has been dumped to persistent storage. Among them, the job in the checkpoint state does not occupy the CPU to run, and the job state is transferred to persistent storage.

Generally, the dump process time is related to the memory size occupied by the job process and the storage write speed. For jobs that occupy a large amount of memory, it takes a long time. You can add a checkpoint node C _t (Checkpointing) as needed to indicate the state in the dump. When the cluster administrator performs a checkpoint operation on the job, the job returns from the running queue to the waiting queue and sets the conditions for resuming the job process on the compute node. Jobs in the C state are not restored until the cluster resources meet the conditions or When the cluster administrator initiates scheduling, the process corresponding to the job is restored on the computing node. Checkpoint operations can be performed through commands, interfaces, button controls, etc.

The scheduling algorithm in Figure 6 is used to increase the communication and call process of the checkpoint-restart service with the computing node, schedule the checkpointed job back to the waiting queue and set it to a new state. It also sets the conditions for rescheduling back to the computing node, and resumes the job execution when the conditions are met, and updates the job and the computing node list and the corresponding process ID.

In FIG6 , the job in the scheduling queue is dumped and then returned to the scheduling queue in a new state.

The scheduling database in FIG6 can add job states C, Ct, and the job corresponding storage path.

In the fifth step, the job enters the running state and the computing node starts running the job. After the job starts running, the computing node list of the job in the computer cluster and the process ID of the job on the computing node list can be determined. The scheduler updates the job status and stores the job-related information in the corresponding scheduling database.

In the sixth and seventh steps, the cluster administrator triggers the job to perform a checkpoint operation. After receiving the request, the scheduler checks the status of the job. If the job is running, the scheduler searches the scheduling database for the list of computing nodes corresponding to the job and the list of process IDs on the computing nodes. The scheduler communicates with the nodes on the computing node list and calls the program checkpoint node (Job Progress Check Point) of the checkpoint-restart service. The checkpoint-restart service starts to checkpoint the process corresponding to the above job and dumps the process image to the specified persistent storage path. After the dump is completed, the checkpoint-restart service returns the result to the scheduler, and the scheduler integrates the results of each node in the computing node list.

When all nodes in the computing node list return success, the checkpoint operation of the entire job is considered to be successfully completed. At this time, the scheduling node updates the job status to C and moves the job from the running queue to the waiting queue, and sets the conditions for being scheduled again; when some nodes in the job computing node list return success or all nodes return failure, the scheduling node communicates with the successful nodes to resume operation and returns the operation failure and related information to the management.

In steps 8 and 9, for jobs that have completed the checkpoint operation, the HPC cluster administrator can schedule the job to execute on a new computing node by presetting conditions or directly controlling the scheduler. When the scheduler receives a request from the cluster administrator or meets the above presetting conditions, the scheduler and the cluster administrator can specify a new computing node list or a scheduling algorithm to determine the computing node communication in order to call the program recovery node (Job Progress Restart) of the checkpoint restart service on the computing node. The checkpoint restart service can restore the execution of the job on the computing node from the job process image on the corresponding storage path. The recovery process may also encounter errors, such as the process ID is already occupied, in which case the computing node returns failure to the scheduling node.

After the scheduling node receives the node checkpoint restart service return node on each new computing node list, the job resumes when the node in the computing node list returns successfully. The scheduler can then update the job status and the job database's new node information. If there is a computing node that fails to resume the job process, the scheduler stops the completed recovery process on other computing nodes and returns the corresponding information to the cluster administrator.

FIG. 7 is a flow chart of a high performance computing cluster performing a dump operation process to persistent storage according to an embodiment of the present application. As shown in FIG. 7 , the method includes:

Step S701, the scheduler receives a request to start a checkpoint;

Step S702, checking the operation status;

Step S703, determine whether the job is running, if yes (Y), execute step S704, if no (N), execute step S717;

Step S704, querying the job node list from the scheduling database;

Step S705, establishing a checkpoint-restart service communication call with each computing node;

Step S706, receiving the operation result returned by the computing node;

Step S707, whether each computing node returns that the operation is successful, if so, execute step S708, if not, execute step S710;

Step S708, update the job status, adjust the job queue, and set the re-running conditions;

Step S709, returning to indicate successful completion of the operation;

Step S710, resume the operation process of some successful nodes to continue running;

Step S711, return failure and end the operation;

Step S712, the checkpoint-restart service receives the request;

The received request may be a JobProgressCheckPoint request.

Step S713, obtaining the process identifier and dump path;

Step S714, determine whether the job is running, if so, execute step S715, if not, execute step S706;

Step S715, performing a checkpoint dump operation on the process;

Step S716, returning the operation result to the scheduling node;

Step S717, end the operation and return.

FIG8 is a flow chart of restoring a high performance computing cluster job from a dumped process image according to an embodiment of the present application. As shown in FIG8 , the method includes:

Step S801, the cluster administrator controls the job recovery or checkpoint preset conditions to be met;

Step S802, the scheduler starts to execute the job process recovery;

Step S803, the scheduler runs the scheduling algorithm to obtain a list of new computing nodes where the job is restored;

Step S804, establishing checkpoints of different computing nodes - restarting service communication calls;

Step S805, receiving the operation result returned by the computing node;

Step S806, determining whether the return operation of each computing node is successful, if so, executing step S807, if not, executing step S809;

Step S807, updating the job status and adjusting the job queue;

Step S808, returning to indicate successful completion of the operation;

Step S809, stop restoring the partially successful node operation process and continue to run;

Step S810, return failure and end the operation;

Step S811, the checkpoint-restart service receives the request;

Step S812, obtaining the job process image from the dump path;

Step S813, checkpoint-restart service to resume image execution;

Step S814, returning the operation result to the scheduling node.

Figure 9 is a schematic diagram of the control structure of another high-performance computing cluster according to an embodiment of the present application. Compared with the method shown in Figure 6, the present application supplements three steps. First, during the HPC job submission phase, after the scheduler selects a candidate computing node from the computing cluster, before executing the job, the container for the job is first started in the computing node, and then the job process is executed in the container; secondly, when a scenario requiring HPC job migration is encountered during the cluster job execution phase.

There are two ways to checkpoint and restart the job process to dump the job. First, since the container is also a process for the operating system, you can directly checkpoint the container process. Second, checkpoint the job in the container. The process performs a checkpoint operation. Compared with the former, the number of job processes in the container may be more than one; finally, for the cluster job recovery stage, if the container is packed and dumped when the process is dumped, the process corresponding to the container is restored. When performing the checkpoint operation, the dump container job process can be executed in different ways. One way is as shown in Figure 9. The container can be started first and then the HPC job process recovery operation can be run in the container; Figure 10 is a control structure diagram of another high-performance computing cluster according to an embodiment of the present application. As shown in Figure 10, another way is to restore the job process to a node without a container environment. In addition, for the job process mentioned in Figure 6, the container environment can also be started first, and then the execution can be restored in the container environment.

The control method of the high-performance computing cluster proposed in this application can facilitate cluster users or administrators to further control the migration of cluster jobs in the nodes in the computing cluster. The risk of job failure can be reduced by migrating jobs on potential faulty nodes. The cluster utilization rate can also be improved through appropriate migration. At the same time, the power consumption of the entire cluster can be reduced by cooperating with measures such as cabinet nodes.

In the above embodiment of the present application, the scheduling node includes a scheduler, a running queue and a queue to be scheduled, and the scheduling node sends migration information to the first computing node. The method also includes: the scheduler receives the dump result fed back by the first computing node, wherein the dump result is used to indicate whether the target process image is successfully dumped; when the dump result is that the target process dump is successful, the scheduler updates the job status of the target job to the migration status, and moves the target job from the running queue to the queue to be scheduled, wherein the running queue is used to store jobs whose job status is running, and the queue to be scheduled is used to store jobs whose job status is not running; the scheduler outputs a first prompt message, wherein the first prompt message is used to prompt that the migration of the target job is successful.

The above-mentioned scheduler is used to move the job according to the dump result fed back by the computing node.

The above-mentioned job status is used to indicate that the target job is being executed in the first computing node.

In an optional embodiment, the scheduler can determine whether the target process image is dumped successfully based on the dump result fed back by the first computing node. When the dump result is that the target process image is dumped successfully, the scheduler can update the job status of the target job to the migration status, thereby terminating the operation of the target job. The scheduler can execute the operation of moving the target job from the running queue to the to-be-scheduled queue. When the scheduler outputs the first prompt information, it means that the target job has been migrated successfully.

In another optional embodiment, after the scheduler updates the job status of the target job to the migration status, the scheduler may store the relevant information of the updated status in the scheduling database corresponding to the target job.

In the above embodiments of the present application, the migration status includes: migration completed and migration in progress. The scheduler updates the job status of the target job to the migration status, including: the scheduler obtains the amount of memory occupied by the target process and the storage speed corresponding to the preset storage device; the scheduler updates the job status to migration completed or migration in progress based on the amount of memory and the storage speed.

In an optional embodiment, the dump process time is related to the memory size occupied by the job process and the storage write speed. Jobs that occupy a large amount of memory require a longer time. Therefore, the scheduler can obtain the amount of memory occupied by the target process and the storage speed corresponding to the preset storage device, and determine the time required to complete the migration based on the amount of memory and the stored data, so as to update the job status to migration completed or migrating according to the migration time.

In the above embodiments of the present application, the scheduler updates the job status to migration completed or migrating based on the memory amount and the storage speed, including: when the memory amount is greater than the preset memory amount and the storage speed is less than the preset speed, the scheduler updates the job status to migrating; when the memory amount is less than or equal to the preset memory amount, or the storage speed is greater than or equal to the preset speed, the scheduler updates the job status to migration completed.

The above-mentioned preset memory amount may be a memory amount pre-set according to the memory of a preset storage device, and the preset memory amount may also be set according to actual conditions.

The above-mentioned preset speed can be a speed preset according to the storage speed of a preset storage device, and the preset speed can also be set according to actual conditions.

In an optional embodiment, when the amount of memory is greater than the preset memory amount and the storage speed is less than the preset speed, it means that the target process of the target job has not yet been migrated, so the memory amount is large and the speed is small, and the job status can be updated to migrating at this time; when the amount of memory is less than or equal to the preset memory amount and the storage speed is greater than or equal to the preset speed, it means that the target process of the target job has been migrated, so the memory amount of the preset storage device is small and the speed can be large, and the job status can be updated to migration completed at this time; when the amount of memory is less than or equal to the preset storage amount and the storage speed is less than the preset speed, it means that the target process of the target job has been migrated, so the memory amount of the preset storage device is small and the speed can be small, and the job status can be updated to migration completed.

In another optional embodiment, the job status during the migration process may be marked with different identifiers, thereby facilitating the scheduler to update the job status.

In the above embodiment of the present application, when there are multiple first computing nodes, the method also includes: when the dump result of at least one first computing node is a failure to dump the target process, the scheduler determines a second computing node among the multiple first computing nodes, wherein the dump result of the second computing node is a successful dump of the target process; the scheduler sends a resume operation request to the second computing node, wherein the resume operation request is used to request the second computing node to continue running the target process; the scheduler outputs a second prompt message, wherein the second prompt message is used to prompt that the target job migration has failed.

The prompting method of the second prompting information is not limited to text, voice, image, etc., and the prompting method of the second prompting information can be determined according to actual needs.

In an optional embodiment, when the dump result of at least one first computing node is a target process dump failure, the scheduler can determine a second computing node from multiple first computing nodes to send a resume request to the second computing node, thereby avoiding the second computing node from being affected by the migration failure, so that the second computing node can continue to run the target process.

In the above embodiment of the present application, after the scheduling node sends the migration information to the first computing node, the method also includes: when the target job meets the recovery conditions or receives a scheduling request, the scheduling node determines a third computing node from the high-performance computing cluster, wherein the scheduling request is used to schedule the target job to the third computing node; the scheduling node sends recovery information to the third computing node, wherein the recovery information is used to control the third computing node to read the target process from a preset storage device and run the target process.

The above-mentioned recovery condition may be a pre-set condition, wherein the recovery condition may be set in a scheduling algorithm, but is not limited thereto.

You can increase the communication and calling process of the checkpoint-restart service of the computing node, schedule the checkpointed jobs back to the waiting queue and set them to a new state, set the recovery conditions for rescheduling them back to the computing node, and resume job execution when the conditions are met.

The above scheduling request can be generated by the cluster administrator to trigger the job to perform checkpoint operation.

The third computing node mentioned above may be a computing node in a new computing node list specified by a cluster administrator, or may be a computing node in a new computing node list calculated by a scheduling algorithm.

In an optional embodiment, after receiving the scheduling request, the scheduler can determine the The third computing node is determined, and the target process can be read from the preset storage device by calling the first preset interface, so as to restore the operation of the target process on the third computing node.

In the above embodiment of the present application, the scheduling node determines the third computing node from the high-performance computing cluster, including: when the target job meets the recovery condition, the scheduling node determines the third computing node from the high-performance computing cluster based on the scheduling algorithm, wherein the scheduling algorithm is used to allocate the target resources of the third computing node to the target job so that the target job is executed by the target resources; when receiving a scheduling request, the scheduling node determines the computing node corresponding to the scheduling request from the high-performance computing cluster to obtain the third computing node.

The third computing node may be re-determined through the existing scheduling algorithm of the cluster, and resources may be allocated to the third computing node, so that the target job may be executed by the target resources.

The above scheduling algorithm can be used to determine the communication and calling process of the checkpoint-restart service. It can schedule the jobs that have completed the checkpoint back to the waiting queue and set them to a new state, and set the conditions for rescheduling them back to the computing node. When the conditions are met, the job execution is resumed. The scheduling algorithm can also update the job and the computing node list and the corresponding process ID.

In an optional embodiment, the computing node in the new computing node list may be calculated according to a scheduling algorithm, so as to determine that the computing node is the third computing node.

In the above embodiment of the present application, the scheduling node includes a scheduler, a running queue and a queue to be scheduled. After the scheduling node sends recovery information to the third computing node, the method also includes: the scheduler receives the recovery result fed back by the third computing node, wherein the recovery result is used to characterize whether the target process has recovered successfully; when the recovery result is that the target process has recovered successfully, the scheduler updates the job status of the target job to running, and moves the target job from the queue to be scheduled to the running queue; the scheduler outputs a third prompt message, wherein the third prompt message is used to prompt that the target job has recovered successfully.

In an optional embodiment, the scheduler receives the recovery result fed back by the third computing node. If the recovery result is that the target process has been successfully recovered, the scheduler can update the job status of the target job to running, so as to facilitate determining the job status of the target job. At this time, the target job can be moved from the to-be-scheduled queue to the running queue, and the scheduler can output a third prompt message to facilitate the user to understand that the target job has been successfully recovered.

In the above embodiment of the present application, when there are multiple third computing nodes, the method also includes: when the recovery result of at least one third computing node is that the target process recovery fails, the scheduler determines a fourth computing node among the multiple third computing nodes, wherein the recovery result of the fourth computing node is that the target process recovery is successful; the scheduler sends a stop operation request to the fourth computing node, wherein the stop operation request is used to request the fourth computing node to stop running the target process; the scheduler outputs a fourth prompt message, wherein the fourth prompt message is used to prompt that the target job recovery fails.

The target process in the fourth computing node has been successfully restored, that is, the action of restoring the target process has been completed in the fourth computing node.

In an optional embodiment, if the recovery result of multiple third computing nodes is that the target process recovery failed, it is necessary to determine a fourth computing result that the target process in the multiple third computing nodes has been successfully recovered, and it is necessary to stop the target process of the fourth computing node so that the recovery status of the target processes on the third computing nodes is unified. At this time, the fourth prompt information can be output through the scheduler so as to prompt the user that the target job recovery has failed through the fourth prompt information.

By integrating the job process checkpoint-restart technology into the HPC job system through the method proposed in this application, the scheduling algorithm and strategy of the job system can be improved, so that the jobs submitted by users to the HPC cluster can achieve the user-imperceptible migration function, thereby benefiting the HPC cluster system in operation and maintenance, fault handling, cluster power consumption, etc.

Example 2

According to an embodiment of the present application, another control method embodiment of a high-performance computing cluster is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

FIG. 11 is a flow chart of a method for controlling a high performance computing cluster according to Embodiment 2 of the present application. As shown in FIG. 11 , the method may include the following steps:

Step S1102: The first computing node receives migration information sent by the scheduling node.

The target job is running on the first computing node, and the migration information is sent by the scheduling node when it receives a job migration request and determines that the job status of the target job is running. The job migration request is used to migrate the target job.

Step S1104: the first computing node dumps the target process image corresponding to the target job to a preset storage device based on the migration information.

In the above embodiment of the present application, the migration information includes at least: process information of the target process and a storage path of a preset storage device. The first computing node dumps the target process image corresponding to the target job to the preset storage device based on the migration information, including: the first computing node determines whether there is a target process on the first computing node based on the process information; when the target process exists on the first computing node, the first computing node dumps the target process image to the preset storage device based on the storage path.

In an optional embodiment, the first computing node obtains the migration information sent by the scheduling node through a first preset interface.

The above-mentioned first preset interface can be Job Progress CheckPoint, wherein the input information of the first preset interface can be a job process ID set and a job process mirror storage path, the return information of the first preset interface can be success or failure, and the action of the first preset interface can be to transfer the checkpoint of the corresponding process to the specified storage.

Because there may be multiple processes corresponding to the same job on a computing node, it is necessary to specify a set of job process IDs. The process dump image can be placed in any valid storage. Generally, in order to facilitate migration actions, it can be placed in the shared storage mounted on the cluster computing node. This is only an example and is not limited.

The process information of the target process mentioned above may be the name, data, etc. of the target process.

The storage path of the above-mentioned preset storage device may be a path name, path information, etc. of the storage path, wherein the storage path of the preset storage device may be a persistent storage path.

In an optional embodiment, the first computing node can obtain the migration information sent by the scheduling node through the first preset interface, and can determine the process information of the target process and the storage path of the preset storage device through the migration information, and can determine whether there is a target process of the target job on the first computing node based on the process information. If there is a target process, it means that the process of the target job has not ended, and the target process image can be dumped to the preset storage device; if there is no target process, it means that the process of the target job has ended, and there is no need to dump the target process image to the preset storage device.

In the above embodiment of the present application, after the first computing node dumps the target process image corresponding to the target job to the preset storage device based on the migration information, the method also includes: the third computing node receives the recovery information sent by the scheduling node, wherein the recovery information is sent by the scheduling node when the target job meets the recovery conditions or receives the scheduling request; the third computing node reads the target process from the preset storage device; the third computing node runs the target process.

In an optional embodiment, the third computing node obtains the recovery information sent by the scheduling node through the second preset interface.

The above-mentioned second preset interface can be Job Progress Restart, wherein the input information of the second preset interface can be the job image path, the return information of the second preset interface can be success or failure, and the action of the second preset interface can be restoring the dumped job image to the computing node.

Since the job process ID information is saved in the dump image, you only need to specify the job image path in the input information.

In an optional embodiment, the third computing node can obtain the recovery information sent by the scheduling node through the second preset interface, and the third computing node can actively read the target process from the preset storage device according to the storage path to facilitate restoring the target process of the target job to the third computing node.

In the above embodiment of the present application, running the target process on the third computing node includes: the third computing node running the target process includes: the third computing node reading the process information of the target process from a preset storage device; the third computing node determines whether there is a process corresponding to the process information; if it is determined that there is a process corresponding to the process information, the third computing node stops running the target process; if it is determined that there is no process corresponding to the process information, the third computing node runs the target process.

During the recovery of the target process, errors may occur, such as the process ID being occupied, in which case the compute node returns a failure to the scheduling node. After the scheduling node receives the node checkpoint restart service return on the new compute node list, the job resumes when the nodes in each compute node list return successfully, and the scheduler updates the job status and the new node information of the job in the database; if there is a compute node that fails to return the job recovery process, the scheduler stops the recovery process that has been completed on other compute nodes and returns the corresponding information to the cluster administrator.

In an optional embodiment, the third computing node can read the process information of the target process from a preset storage device, and can determine whether the target process is occupied through the process information of the target process. If it is determined that there is a process corresponding to the process information, it means that the target process is occupied, and the third computing node stops running the target process. If it is determined that there is no process corresponding to the process information, it means that the target process is not occupied, and the third computing node runs the target process.

It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, can be in the form of a software product. It is embodied that the computer software product is stored in a storage medium (such as ROM/RAM, disk, CD), including a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of various embodiments of the present application.

Example 3

According to an embodiment of the present application, a high-performance computing cluster for implementing the control method of the above-mentioned high-performance computing cluster is also provided. Figure 12 is a schematic diagram of a high-performance computing cluster according to Example 2 of the present application. As shown in Figure 12, the high-performance computing cluster 1200 includes: a computing node 1202, a preset storage device 1204, and a scheduling node 1206.

Among them, the computing node 1202 is used to run jobs; the preset storage device 1204 is used to store processes; the scheduling node 1206 is connected to the computing node, and is used to obtain the job status of the target job when a job migration request is received, and when the job status is running, determine the first computing node and determine the target process on the first computing node, wherein the job migration request is used to migrate the target job, wherein the target job is running on the first computing node, and the target process is the process on the first computing node corresponding to the target job; the first computing node is connected to the preset storage device, and is used to store the target process image to the preset storage device.

The first computing node mentioned above may be a computing node where the target job is located among the computing nodes.

The above-mentioned preset storage device may be used to store one or more processes, wherein the target process may be a process corresponding to the target job stored in the preset storage device.

Through the above embodiment, the computing node is used to run the job; the preset storage device is used to store the process image; the scheduling node is connected to the computing node, and is used to obtain the job status of the target job when receiving the job migration request, and when the job status is running, determine the first computing node, and determine the target process on the first computing node, wherein the job migration request is used to migrate the target job, wherein the target job is running on the first computing node, and the target process is the process corresponding to the target job on the first computing node; the first computing node is connected to the preset storage device, and is used to store the target process image to the preset storage device, so as to achieve the purpose of improving the reliability of the high-performance computing cluster. It is easy to notice that when the job status is running, the computing nodes in the high-performance cluster can be checked, and the first computing node running the target job can be determined, and the target process image on the first computing node can be dumped to the preset storage device, so as to achieve the dumping of the target process corresponding to the target job without the user's perception, and the target job on the potential fault node can be easily migrated to reduce the risk of job failure, thereby improving the reliability of the target job operation and improving the reliability of the high-performance computing cluster. Thus, the technical problem of low reliability of the high-performance computing cluster in the related technology is solved.

It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

Example 4

According to an embodiment of the present application, a control device for a high-performance computing cluster for implementing the above-mentioned control method of the high-performance computing cluster is also provided. Figure 13 is a schematic diagram of a control device for a high-performance computing cluster according to Example 4 of the present application. As shown in Figure 13, the device 1300 includes: an acquisition module 1302, a determination module 1304, and a sending module 1306.

The acquisition module is used to obtain the job status of the target job when the scheduling node receives the job migration request, wherein the job migration request is used to migrate the target job; the determination module is used to determine the first computing node from the high-performance computing cluster through the scheduling node when the job status is running, and determine the target process on the first computing node, wherein the target job is running on the first computing node, and the target process is the process on the first computing node that is connected to the target job. The process corresponding to the job; the sending module is used to send migration information to the first computing node through the scheduling node, wherein the migration information is used to control the first computing node to dump the target process image to a preset storage device.

It should be noted that the acquisition module 1302, the determination module 1304, and the sending module 1306 correspond to steps S402 to S406 in Example 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

In the above embodiments of the present application, the scheduling node includes a scheduler, a running queue and a queue to be scheduled, and the device also includes: a receiving module, an updating module, and an output module.

The receiving module is used to receive the dump result fed back by the first computing node through the scheduler, wherein the dump result is used to indicate whether the target process has been dumped successfully; the updating module is used to update the job status of the target job to the migration status when the dump result is that the target process has been dumped successfully, and move the target job from the running queue to the to-be-scheduled queue, wherein the running queue is used to store jobs whose job status is running, and the to-be-scheduled queue is used to store jobs whose job status is not running; the output module is used for the scheduler to output the first prompt information, wherein the first prompt information is used to prompt that the migration of the target job is successful.

In the above embodiments of the present application, the migration status includes: migration completed and migration in progress. The update module is also used to obtain the amount of memory occupied by the target process and the storage speed corresponding to the preset storage device through the scheduler. The update module is also used to update the job status to migration completed or migration in progress based on the memory amount and storage speed through the scheduler.

In the above embodiments of the present application, the update module is also used to update the job status to migrating through the scheduler when the memory amount is greater than the preset memory amount and the storage speed is less than the preset speed; the update module is also used to update the job status to migration completed through the scheduler when the memory amount is less than or equal to the preset memory amount and the storage speed is greater than or equal to the preset speed.

In the above embodiments of the present application, the device also includes: a determination module.

Among them, the determination module is also used to, when the dump result of at least one first computing node is a failure to dump the target process, the scheduler determines a second computing node among multiple first computing nodes, wherein the dump result of the second computing node is a successful dump of the target process, and the scheduler sends a resume operation request to the second computing node, wherein the resume operation request is used to request the second computing node to continue running the target process, and the scheduler outputs a second prompt message, wherein the second prompt message is used to prompt that the target job migration has failed.

In the above embodiment of the present application, the determination module is also used for the scheduling node to determine a third computing node from the high-performance computing cluster when the target job meets the recovery conditions or receives a scheduling request, wherein the scheduling request is used to schedule the target job to the third computing node, and the scheduling node sends recovery information to the third computing node, wherein the recovery information is used to control the third computing node to read the target process from a preset storage device and run the target process.

In the above embodiment of the present application, the determination module is also used to determine a third computing node from the high-performance computing cluster based on a scheduling algorithm through a scheduling node when the target job meets the recovery conditions, and to obtain the third computing node by determining the computing node corresponding to the scheduling request from the high-performance computing cluster through the scheduling node when a scheduling request is received.

In the above embodiment of the present application, the scheduling node includes a scheduler, a running queue and a queue to be scheduled, and the receiving module is also used The scheduler receives the recovery result fed back by the third computing node, wherein the recovery result is used to indicate whether the target process has recovered successfully; the update module is used to update the job status of the target job to running when the recovery result is that the target process has recovered successfully, and move the target job from the to-be-scheduled queue to the running queue; the output module is also used for the scheduler to output a third prompt message, wherein the third prompt message is used to prompt that the target job has recovered successfully.

In the above embodiments of the present application, the device further includes: a sending module.

Among them, the determination module is also used for the scheduler to determine a fourth computing node among multiple third computing nodes when the recovery result of at least one third computing node is a failure to recover the target process, wherein the recovery result of the fourth computing node is a successful recovery of the target process; the sending module is used for the scheduler to send a stop operation request to the fourth computing node, wherein the stop operation request is used to request the fourth computing node to stop running the target process; the output module is also used for the scheduler to output a fourth prompt message, wherein the fourth prompt message is used to prompt that the recovery of the target job has failed.

It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

Example 5

According to an embodiment of the present application, a control device for a high-performance computing cluster for implementing the above-mentioned control method of the high-performance computing cluster is also provided. Figure 14 is a schematic diagram of a control device for a high-performance computing cluster according to Example 5 of the present application. As shown in Figure 14, the device 1400 includes: a receiving module 1402 and a dump module 1404.

Among them, the receiving module is used to receive the migration information sent by the scheduling node through the first computing node, wherein the target job is running on the first computing node, and the migration information is sent by the scheduling node when it receives the job migration request and determines that the job status of the target job is running, and the job migration request is used to migrate the target job; the dump module is used for the first computing node to dump the target process image corresponding to the target job to a preset storage device based on the migration information.

In the above embodiments of the present application, the migration information includes at least: process information of the target process and a storage path of a preset storage device. The dump module is also used to determine whether the target process exists on the first computing node based on the process information through the first computing node. If the target process exists on the first computing node, the target process image is dumped to the preset storage device through the first computing node based on the storage path.

In the above embodiments of the present application, the device includes: a reading module and an operating module.

Among them, the receiving module is used for the third computing node to receive the recovery information sent by the scheduling node, wherein the recovery information is sent by the scheduling node when the target job meets the recovery conditions or receives a scheduling request; the reading module is used for the third computing node to read the target process from the preset storage device; and the running module is used for the third computing node to run the target process.

In the above embodiment of the present application, the running module is also used for the third computing node to read the process information of the target process from a preset storage device, and the third computing node determines whether there is a process corresponding to the process information. If it is determined that there is a process corresponding to the process information, the third computing node stops running the target process. If it is determined that there is no process corresponding to the process information, the third computing node runs the target process.

It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

Example 6

The embodiment of the present application can provide a computer terminal, which can be any one of the computer terminal groups. Optionally, in this embodiment, the computer terminal may be replaced by a terminal device such as a mobile terminal.

Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the control method of the high-performance computing cluster: when the scheduling node receives a job migration request, it obtains the job status of the target job, wherein the job migration request is used to migrate the target job; when the job status is running, the scheduling node determines the first computing node from the high-performance computing cluster, and determines the target process on the first computing node, wherein the target job is running on the first computing node, and the target process is the process on the first computing node corresponding to the target job; the scheduling node sends migration information to the first computing node, wherein the migration information is used to control the first computing node to dump the target process image to a preset storage device.

Optionally, Figure 15 is a block diagram of a computer terminal according to an embodiment of the present application. As shown in Figure 15, the computer terminal A may include: one or more (only one is shown in the figure) processors 102, a memory 104, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

Among them, the memory can be used to store software programs and modules, such as the program instructions/modules corresponding to the control method and device of the high-performance computing cluster in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the control method of the high-performance computing cluster mentioned above. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The processor can call the information and application programs stored in the memory through the transmission device to execute the following steps: when the scheduling node receives a job migration request, it obtains the job status of the target job, wherein the job migration request is used to migrate the target job; when the job status is running, the scheduling node determines the first computing node from the high-performance computing cluster, and determines the target process on the first computing node, wherein the target job is running on the first computing node, and the target process is the process on the first computing node corresponding to the target job; the scheduling node sends migration information to the first computing node, wherein the migration information is used to control the first computing node to dump the target process image to a preset storage device.

Optionally, the processor may also execute program code of the following steps: the scheduler receives a dump result fed back by the first computing node, wherein the dump result is used to indicate whether the target process is successfully dumped; when the dump result is that the target process is successfully dumped, the scheduler updates the job status of the target job to a migration status, and moves the target job from the running queue to the to-be-scheduled queue, wherein the running queue is used to store jobs whose job status is running, and the to-be-scheduled queue is used to store jobs whose job status is not running; the scheduler outputs a first prompt message, wherein the first prompt message is used to prompt that the migration of the target job is successful.

Optionally, the processor may also execute program code of the following steps: the scheduler obtains the amount of memory occupied by the target process and the storage speed corresponding to the preset storage device; the scheduler updates the job status to migration completed or in progress based on the memory amount and storage speed.

Optionally, the processor may further execute the following program code: when the memory amount is greater than the preset memory amount and the storage speed is less than the preset speed, the scheduler updates the job status to migrating; when the memory amount is less than or equal to When the preset memory amount is reached and the storage speed is greater than or equal to the preset speed, the scheduler updates the job status to migration completed.

Optionally, the processor may also execute program code of the following steps: when the target job meets the recovery conditions or receives a scheduling request, the scheduling node determines a third computing node from the high-performance computing cluster, wherein the scheduling request is used to schedule the target job to the third computing node; the scheduling node sends recovery information to the third computing node, wherein the recovery information is used to control the third computing node to read the target process from a preset storage device and run the target process.

Optionally, the processor may also execute the program code of the following steps: when the target job meets the recovery conditions, the scheduling node determines a third computing node from the high-performance computing cluster based on the scheduling algorithm; when a scheduling request is received, the scheduling node determines the computing node corresponding to the scheduling request from the high-performance computing cluster to obtain the third computing node.

Optionally, the processor may also execute the program code of the following steps: when the target job meets the recovery conditions, determining a third computing node from the high-performance computing cluster based on the scheduling algorithm; when a scheduling request is received, determining the computing node corresponding to the scheduling request from the high-performance computing cluster to obtain the third computing node.

Optionally, the processor may also execute program code of the following steps: the third computing node obtains recovery information sent by the scheduling node through the second preset interface, wherein the recovery information includes at least: a storage path of a preset storage device; and the third computing node reads the target process from the preset storage device based on the storage path.

Optionally, the processor may also execute program code of the following steps: the third computing node reads process information of the target process from a preset storage device; the third computing node determines whether there is a process corresponding to the process information; if it is determined that there is a process corresponding to the process information, the third computing node stops running the target process; if it is determined that there is no process corresponding to the process information, the third computing node runs the target process.

Optionally, the processor may also execute program code of the following steps: the scheduler receives a recovery result fed back by a third computing node, wherein the recovery result is used to indicate whether the target process has recovered successfully; when the recovery result is that the target process has recovered successfully, the scheduler updates the job status of the target job to running, and moves the target job from the to-be-scheduled queue to the running queue; the scheduler outputs a third prompt message, wherein the third prompt message is used to prompt that the target job has recovered successfully.

Optionally, the processor may also execute program code of the following steps: when the recovery result of at least one third computing node is that the target process recovery fails, the scheduler determines a fourth computing node among multiple third computing nodes, wherein the recovery result of the fourth computing node is that the target process recovery is successful; the scheduler sends a stop operation request to the fourth computing node, wherein the stop operation request is used to request the fourth computing node to stop running the target process; the scheduler outputs a fourth prompt message, wherein the fourth prompt message is used to prompt that the target job recovery fails.

The processor can call the information and application programs stored in the memory through the transmission device to execute the following steps: the first computing node receives the migration information sent by the scheduling node, wherein a target job is running on the first computing node, and the migration information is sent by the scheduling node after receiving a job migration request and determining that the job status of the target job is running, and the job migration request is used to migrate the target job; based on the migration information, the first computing node dumps the target process image corresponding to the target job to a preset storage device.

Optionally, the processor may also execute program code of the following steps: the first computing node determines whether a target process exists on the first computing node based on process information; when the target process exists on the first computing node, the first computing node dumps the target process image to a preset storage device based on a storage path.

Optionally, the processor may also execute program code of the following steps: the third computing node receives recovery information sent by the scheduling node, wherein the recovery information is sent by the scheduling node when the target job meets the recovery conditions or receives a scheduling request; the third computing node reads the target process from a preset storage device; the third computing node runs the target process.

Optionally, the processor may also execute program code of the following steps: the third computing node reads process information of the target process from a preset storage device; the third computing node determines whether there is a process corresponding to the process information; if it is determined that there is a process corresponding to the process information, the third computing node stops running the target process; if it is determined that there is no process corresponding to the process information, the third computing node runs the target process.

According to the embodiment of the present application, first, when the scheduling node receives a job migration request, it obtains the job status of the target job, wherein the job migration request is used to migrate the target job; when the job status is in operation, the scheduling node determines the first computing node from the high-performance computing cluster, and determines the target process on the first computing node, wherein the target job is running on the first computing node, and the target process is the process corresponding to the target job on the first computing node; the scheduling node sends migration information to the first computing node, wherein the migration information is used to control the first computing node to dump the target process image to a preset storage device, so as to achieve the purpose of improving the reliability of the high-performance computing cluster. It is easy to notice that when the job status is in operation, the computing nodes in the high-performance cluster can be checked, and the first computing node running the target job can be determined, and the target process image on the first computing node can be dumped to the preset storage device, so as to achieve the dumping of the target process corresponding to the target job without the user's perception, so as to facilitate the migration of the target job on the potential fault node to reduce the risk of job failure, thereby improving the reliability of the target job operation and achieving the purpose of improving the reliability of the high-performance computing cluster. Thus, the technical problem of low reliability of the high-performance computing cluster in the related technology is solved.

It can be understood by those skilled in the art that the structure shown in FIG. 15 is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. FIG. 15 does not limit the structure of the above-mentioned electronic device. For example, the computer terminal A may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG. 15, or have a configuration different from that shown in FIG. 15.

A person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

Example 7

The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the control method of the high performance computing cluster provided in the above embodiment 1.

Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: when the scheduling node receives a job migration request, it obtains the job status of the target job, wherein the job migration request is used to migrate the target job; when the job status is running, the scheduling node determines a first computing node from the high-performance computing cluster and determines a target process on the first computing node, wherein the target job is running on the first computing node. job, the target process is the process on the first computing node corresponding to the target job; the scheduling node sends migration information to the first computing node, wherein the migration information is used to control the first computing node to dump the target process image to a preset storage device.

The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims (16)

A control method for a high performance computing cluster, characterized by comprising: The scheduling node obtains the job status of the target job when receiving the job migration request, wherein the job migration request is used to migrate the target job; When the job status is running, the scheduling node determines a first computing node from the high-performance computing cluster, and determines a target process on the first computing node, wherein the target job is running on the first computing node, and the target process is a process on the first computing node corresponding to the target job; The scheduling node sends migration information to the first computing node, wherein the migration information is used to control the first computing node to dump the target process image to a preset storage device. The method according to claim 1, characterized in that the scheduling node includes a scheduler, a running queue and a queue to be scheduled, and after the scheduling node sends the migration information to the first computing node, the method further includes: The scheduler receives the dump result fed back by the first computing node, wherein the dump result is used to indicate whether the target process image is successfully dumped; When the dump result is that the target process dump is successful, the scheduler updates the job status of the target job to a migration status, and moves the target job from the running queue to the queue to be scheduled, wherein the running queue is used to store jobs whose job status is running, and the queue to be scheduled is used to store jobs whose job status is not running; The scheduler outputs first prompt information, wherein the first prompt information is used to prompt that the target job is successfully migrated. The method according to claim 2, wherein the migration status includes: migration completed and migration in progress, and the scheduler updates the job status of the target job to the migration status, including: The scheduler obtains the amount of memory occupied by the target process and the storage speed corresponding to the preset storage device; The scheduler updates the job status to the migration completed or the migration in progress based on the memory amount and the storage speed. The method according to claim 3, characterized in that the scheduler determines, based on the memory amount and the storage speed, that the job status is updated to the migration completed or the migration in progress, comprising: When the memory amount is greater than a preset memory amount and the storage speed is less than a preset speed, the scheduler updates the job status to being migrated; When the memory amount is less than or equal to the preset memory amount, or the storage speed is greater than or equal to the preset speed, the scheduler updates the job status to migration completion. The method according to claim 2, characterized in that, when there are multiple first computing nodes, the method further comprises: In a case where the dump result of at least one of the first computing nodes is that the target process image dump fails, the scheduler determines a second computing node among the plurality of the first computing nodes, wherein the dump result of the second computing node is that the target process image dump succeeds; The scheduler sends a resume operation request to the second computing node, wherein the resume operation request is used to request Requesting the second computing node to continue running the target process; The scheduler outputs second prompt information, wherein the second prompt information is used to prompt that the migration of the target job fails. The method according to claim 1, characterized in that after the scheduling node sends the migration information to the first computing node, the method further comprises: When the target job meets the recovery condition or receives a scheduling request, the scheduling node determines a third computing node from the high-performance computing cluster, wherein the scheduling request is used to schedule the target job to the third computing node; The scheduling node sends recovery information to the third computing node, wherein the recovery information is used to control the third computing node to read the target process from the preset storage device and run the target process. The method according to claim 6, wherein the scheduling node determines the third computing node from the high-performance computing cluster, comprising: In a case where the target job meets the recovery condition, the scheduling node determines the third computing node from the high-performance computing cluster based on a scheduling algorithm, wherein the scheduling algorithm is used to allocate a target resource of the third computing node to the target job so that the target job is executed by the target resource; When receiving the scheduling request, the scheduling node determines the computing node corresponding to the scheduling request from the high-performance computing cluster to obtain the third computing node. The method according to claim 6, characterized in that the scheduling node includes a scheduler, a running queue and a queue to be scheduled, and after the scheduling node sends the recovery information to the third computing node, the method further includes: The scheduler receives the recovery result fed back by the third computing node, wherein the recovery result is used to indicate whether the target process is successfully recovered; When the recovery result is that the target process is successfully recovered, the scheduler updates the job status of the target job to running, and moves the target job from the to-be-scheduled queue to the running queue; The scheduler outputs third prompt information, wherein the third prompt information is used to prompt that the target job is successfully restored. The method according to claim 8, characterized in that, when there are multiple third computing nodes, the method further comprises: In a case where the recovery result of at least one of the third computing nodes is that the target process fails to recover, the scheduler determines a fourth computing node among the plurality of the third computing nodes, wherein the recovery result of the fourth computing node is that the target process succeeds to recover; The scheduler sends a stop operation request to the fourth computing node, wherein the stop operation request is used to request the fourth computing node to stop running the target process; The scheduler outputs fourth prompt information, wherein the fourth prompt information is used to prompt that the target job recovery fails. A control method for a high performance computing cluster, characterized by comprising: The first computing node receives migration information sent by the scheduling node, wherein a target job is running on the first computing node, the migration information is sent by the scheduling node when receiving a job migration request and determining that the job status of the target job is running, and the job migration request is used to migrate the target job; The first computing node dumps the target process image corresponding to the target job to a preset storage device based on the migration information. The method according to claim 10, characterized in that the migration information at least includes: process information of the target process and a storage path of the preset storage device, and the first computing node dumps the target process image corresponding to the target job to the preset storage device based on the migration information, comprising: The first computing node determines, based on the process information, whether the target process exists on the first computing node; In a case where the target process exists on the first computing node, the first computing node dumps the target process image to a preset storage device based on the storage path. The method according to claim 10, characterized in that after the first computing node dumps the target process image corresponding to the target job to a preset storage device based on the migration information, the method further comprises: The third computing node receives the recovery information sent by the scheduling node, wherein the recovery information is sent by the scheduling node when the target job meets the recovery condition or receives the scheduling request; The third computing node reads the target process from the preset storage device; The third computing node runs the target process. The method according to claim 12, wherein the third computing node runs the target process, comprising: The third computing node reads the process information of the target process from the preset storage device; The third computing node determines whether there is a process corresponding to the process information; In the case of determining that there is a process corresponding to the process information, the third computing node stops running the target process; In a case where it is determined that there is no process corresponding to the process information, the third computing node runs the target process. A high performance computing cluster, comprising: Compute nodes, used to run jobs; Preset storage device for storing process images; a scheduling node connected to the computing node, and configured to obtain a job status of a target job upon receiving a job migration request, and to determine a first computing node and a target process on the first computing node when the job status is running, wherein the job migration request is used to migrate the target job, wherein the target job is running on the first computing node, and the target process is a process on the first computing node corresponding to the target job; The first computing node is connected to the preset storage device and is used to store the target process image in the preset storage device. An electronic device, comprising: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 13 when running. A computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored An execution program, wherein when the executable program is running, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 13.