CN116661957A - Cluster resource scheduling method, device, computer equipment and storage medium - Google Patents

Abstract

The application discloses a cluster resource scheduling method, device, computer equipment, and storage medium. The cluster resource scheduling method includes: obtaining historical task execution information of the cluster, which includes multiple queues for executing tasks; The information determines the initial resource demand information of each queue in the cluster, and the initial resource demand information includes the initial resource value of each queue at each moment in the preset first time period; obtains the real-time resource demand information of each queue, according to the real-time resource demand information and The initial resource demand information determines the resource correction value of each queue at the next moment; according to the initial resource demand information and the resource correction value, resource allocation is performed on each queue. In the embodiment of the present application, cluster resources can be dynamically allocated to each queue, which improves the utilization rate of cluster resources, improves task execution efficiency with as few resources as possible, and reduces task execution time.

Description

Cluster resource scheduling method, device, computer equipment and storage medium

technical field

The present application relates to the technical field of resource scheduling, and in particular to a cluster resource scheduling method, device, computer equipment, and storage medium.

Background technique

With the rapid development of science and technology, more and more data are generated, and various big data products based on frameworks such as Hadoop clusters are more and more favored by users. How to schedule available cluster resources to save costs and improve computing efficiency has become an important research direction for major data platforms. In the prior art, when resources are allocated to each queue in the cluster, idle resources are allocated to each queue according to a resource policy preset by the user. This fixed allocation of cluster resources will lead to uneven resource allocation.

Contents of the invention

The embodiment of the present application provides a cluster resource scheduling method, device, computer equipment, and storage medium, which dynamically allocate cluster resources to each queue according to the historical task execution information of the cluster and the real-time resource demand information of the queue, thereby improving the utilization rate of the cluster resources. It can improve the efficiency of task execution with as few resources as possible and reduce the time of task execution.

On the one hand, the present application provides a cluster resource scheduling method, and the cluster resource scheduling method includes:

Obtain historical task execution information of the cluster, where the cluster includes multiple queues for executing tasks;

determining initial resource requirement information of each queue in the cluster according to the historical task execution information, where the initial resource requirement information includes the initial resource value of each queue at each moment within a preset first time period;

Acquiring the real-time resource requirement information of each queue, and determining the resource correction value of each queue at the next moment according to the real-time resource requirement information and the initial resource requirement information;

Resource allocation is performed on each of the queues according to the initial resource requirement information and the resource correction value.

In some implementations of the present application, the determining the initial resource requirement information of each queue in the cluster according to the historical task execution information includes:

According to the historical task execution information, determine the historical resource demand value of each queue in the cluster at each moment within the preset first time period;

Determine initial resource requirement information of each queue in the cluster according to the historical resource requirement value.

In some implementations of the present application, the determining the historical resource demand value of each queue in the cluster at each moment within the preset first time period according to the historical task execution information includes:

According to the historical task execution information, determine the task time sequence of each queue in the cluster and the average resource demand value of the tasks in each queue within the execution time;

According to the task time series and the average resource demand value, determine the historical resource demand value of each queue in the cluster at each moment within a preset first time period.

In some implementations of the present application, the determining the initial resource requirement information of each queue in the cluster according to the historical resource requirement value includes:

determining the initial resource value of each queue at each moment within the first time period according to the historical resource demand value;

The initial resource requirement information of each queue in the cluster is determined according to the initial resource value of each queue at each moment in the first time period.

In some implementations of the present application, the determining the resource correction value of each queue at the next moment according to the real-time resource requirement information and the initial resource requirement information includes:

According to the real-time resource demand information and the initial resource demand information, respectively determine the real-time resource demand value and the initial resource value of each queue at each moment within a preset second time period, the second time period being included in within said first period of time;

According to the real-time resource demand value and the initial resource value of each queue at each moment within the second time period, determine the resource correction value of each queue at a next moment.

In some implementations of the present application, the determining the resource correction value of each queue at the next moment according to the real-time resource demand value and initial resource value of each queue at each moment within the second time period includes: :

determining the resource difference of each queue at each moment in the second time period according to the real-time resource demand value and the initial resource value of the queues at each moment in the second time period;

According to the resource difference value, the resource correction value of each queue at the next moment is determined.

In some implementations of the present application, the resource allocation for each queue according to the initial resource requirement information and the resource correction value includes:

determining the target resource demand value of each queue at the next moment according to the initial resource demand information and the resource correction value;

Resource allocation is performed on each of the queues according to the target resource demand value.

In some implementations of the present application, the determining the target resource demand value of each queue at the next moment according to the initial resource demand information and the resource correction value includes:

determining the initial resource value of each queue at the next moment according to the initial resource requirement information;

Determine the target resource demand value of each queue at the next moment according to the initial resource value of each queue at the next moment and the resource correction value.

In another aspect, the present application provides a cluster resource scheduling device, and the cluster resource scheduling device includes:

An information acquisition unit, configured to acquire historical task execution information of the cluster, where the cluster includes multiple queues for executing tasks;

An information determining unit, configured to determine initial resource requirement information of each queue in the cluster according to the historical task execution information, where the initial resource requirement information includes the initial resource requirement information of each queue at each moment within a preset first time period. resource value;

A correction value determining unit, configured to acquire real-time resource demand information of each queue, and determine a resource correction value of each queue at a next moment according to the real-time resource demand information and the initial resource demand information;

A resource scheduling unit, configured to allocate resources to each of the queues according to the initial resource requirement information and the resource correction value.

In some embodiments of the present application, the information determination unit is specifically used for:

According to the historical task execution information, determine the historical resource demand value of each queue in the cluster at each moment within the preset first time period;

Determine initial resource requirement information of each queue in the cluster according to the historical resource requirement value.

In some embodiments of the present application, the information determination unit is specifically further configured to:

According to the historical task execution information, determine the task time sequence of each queue in the cluster and the average resource demand value of the tasks in each queue within the execution time;

According to the task time series and the average resource demand value, determine the historical resource demand value of each queue in the cluster at each moment within a preset first time period.

In some embodiments of the present application, the information determination unit is specifically further configured to:

determining the initial resource value of each queue at each moment within the first time period according to the historical resource demand value;

The initial resource requirement information of each queue in the cluster is determined according to the initial resource value of each queue at each moment in the first time period.

In some embodiments of the present application, the correction value determination unit is specifically used for:

According to the real-time resource demand information and the initial resource demand information, respectively determine the real-time resource demand value and the initial resource value of each queue at each moment within a preset second time period, the second time period being included in within said first period of time;

According to the real-time resource demand value and the initial resource value of each queue at each moment within the second time period, determine the resource correction value of each queue at a next moment.

In some embodiments of the present application, the correction value determining unit is specifically further used for:

determining the resource difference of each queue at each moment in the second time period according to the real-time resource demand value and the initial resource value of the queues at each moment in the second time period;

According to the resource difference value, the resource correction value of each queue at the next moment is determined.

In some implementations of the present application, the resource scheduling unit is specifically configured to:

determining the target resource demand value of each queue at the next moment according to the initial resource demand information and the resource correction value;

Resource allocation is performed on each of the queues according to the target resource demand value.

In some embodiments of the present application, the resource scheduling unit is further configured to:

determining the initial resource value of each queue at the next moment according to the initial resource requirement information;

Determine the target resource demand value of each queue at the next moment according to the initial resource value of each queue at the next moment and the resource correction value.

On the other hand, the present application also provides a kind of computer equipment, and described computer equipment comprises:

one or more processors;

storage; and

One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the cluster resource scheduling method described in any one of the first aspects.

In the fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, and the computer program is loaded by a processor to execute the cluster resource scheduling method described in any one of the first aspect. step.

This application determines the initial resource demand information of each queue according to the historical task execution information of the cluster, determines the resource correction value of each queue at the next moment according to the initial resource demand information of each queue and the real-time resource demand information of each queue, and determines the resource correction value of each queue according to the initial resource demand Information and resource correction values allocate resources to each queue, which can improve the efficiency of task execution with as few resources as possible, improve the utilization rate of cluster resources, and reduce the time for task execution.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is a schematic diagram of a scenario of a cluster resource scheduling system provided by an embodiment of the present application;

FIG. 2 is a schematic flow chart of an embodiment of a cluster resource scheduling method provided in the embodiment of the present application;

Fig. 3 is a schematic flow chart of a specific embodiment of the cluster resource scheduling method provided in the embodiment of the present application;

FIG. 4 is a schematic structural diagram of an embodiment of a cluster resource scheduling device provided in an embodiment of the present application;

Fig. 5 is a schematic structural diagram of an embodiment of a computer device provided in an embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts belong to the scope of protection of this application.

In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", " The orientation or positional relationship indicated by "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation shown in the drawings Or positional relationship is only for the convenience of describing the present application and simplifying the description, but does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present application. In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of said features. In the description of the present application, "plurality" means two or more, unless otherwise specifically defined.

In this application, the word "exemplary" is used to mean "serving as an example, illustration or illustration". Any embodiment described in this application as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is given to enable any person skilled in the art to make and use the application. In the following description, details are set forth for purposes of explanation. It should be understood that one of ordinary skill in the art would recognize that the present application may be practiced without these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of the present application with unnecessary detail. Thus, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed in this application.

It should be noted that since the methods in the embodiments of the present application are executed in computer equipment, the processing objects of each computer equipment exist in the form of data or information, such as time, which is essentially time information. It can be understood that in the subsequent embodiments If the size, quantity, location, etc. are mentioned, there are corresponding data for processing by computer equipment, and the details will not be repeated here.

Embodiments of the present application provide a cluster resource scheduling method, device, computer equipment, and storage medium, which will be described in detail below.

Please refer to FIG. 1. FIG. 1 is a schematic diagram of a scenario of a cluster resource scheduling system provided by an embodiment of the present application. The cluster resource scheduling system may include a computer device 100, and a cluster resource scheduling device is integrated in the computer device 100, as shown in FIG. 1 computer equipment.

In the embodiment of the present application, the computer device 100 is mainly used to obtain the historical task execution information of the cluster, which includes multiple queues for executing tasks; determine the initial resources of each queue in the cluster according to the historical task execution information demand information, the initial resource demand information includes the initial resource value of each queue at each moment within the preset first time period; obtain the real-time resource demand information of each queue, and according to the real-time resource demand information and the The initial resource demand information determines the resource correction value of each queue at the next moment; according to the initial resource demand information and the resource correction value, resource allocation is performed on each queue, and the computer device 100 executes according to the historical tasks of the cluster. Information and the real-time resource demand information of each queue in the cluster dynamically allocate cluster resources for each queue, which improves the utilization rate of cluster resources, improves the efficiency of task execution with as few resources as possible, and reduces the time for task execution.

In the embodiment of the present application, the computer device 100 can be an independent server, or a server network or server cluster composed of servers, for example, the computer device 100 described in the embodiment of the present application, which includes but not limited to computers, network A host, a single web server, a collection of multiple web servers, or a cloud server composed of multiple servers. Wherein, the cloud server is composed of a large number of computers or network servers based on cloud computing (Cloud Computing).

It can be understood that the computer device 100 used in the embodiment of the present application may be a device including both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing bidirectional communication on a bidirectional communication link. Such devices may include cellular or other communication devices having a single line display or a multi-line display or a cellular or other communication device without a multi-line display. Specifically, the computer device 100 may be a desktop terminal or a mobile terminal, and the computer device 100 may also be specifically one of a mobile phone, a tablet computer, a notebook computer, and the like.

Those skilled in the art can understand that the application environment shown in Figure 1 is only an application scenario related to the solution of this application, and does not constitute a limitation on the application scenario of the solution of this application. More or fewer computer devices are shown, for example, only one computer device is shown in FIG. 1 , it can be understood that the cluster resource scheduling system may also include one or more other services, which are not specifically limited here.

In addition, as shown in Figure 1, the cluster resource scheduling system may also include a memory 200 for storing data, such as historical task execution information of the cluster, such as the task ID executed in the YARN cluster, the task start time, and the task end time , the CPU required for task execution, etc., such as the initial resource requirement information of each queue in the cluster, specifically, such as the initial resource requirement information of queue A, the initial resource requirement information of queue B, and the initial resource requirement information of queue C in the cluster.

It should be noted that the schematic diagram of the scenario of the cluster resource scheduling system shown in Figure 1 is just an example. The cluster resource scheduling system and the scenario described in the embodiment of this application are for more clearly illustrating the technical solution of the embodiment of the application, and are not intended to Constituting a limitation on the technical solutions provided by the embodiments of the present application, those skilled in the art know that with the evolution of the cluster resource scheduling system and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems Be applicable.

First, an embodiment of the present application provides a cluster resource scheduling method. The execution subject of the cluster resource scheduling method is a cluster resource scheduling device. The cluster resource scheduling device is applied to computer equipment. The cluster resource scheduling method includes: obtaining the history of the cluster task execution information, the cluster includes a plurality of queues for executing tasks; determine the initial resource requirement information of each queue in the cluster according to the historical task execution information, and the initial resource requirement information includes that each queue is in Presetting the initial resource value at each moment in the first time period; obtaining the real-time resource demand information of the queues, and determining the resources of the queues at the next time according to the real-time resource demand information and the initial resource demand information Correction value: according to the initial resource requirement information and the resource correction value, perform resource allocation to each of the queues.

As shown in Figure 2, it is a schematic flow chart of an embodiment of the cluster resource scheduling method in the embodiment of the present application. The cluster resource scheduling method may include the following steps 301-304, specifically as follows:

301. Acquire historical task execution information of a cluster, where the cluster includes multiple queues for executing tasks.

The historical task execution information of the cluster refers to the task information executed on the cluster in the past period of time. Usually, the cluster includes multiple queues for executing tasks, such as queue 1, queue 2, queue 3, etc., each queue Contains a certain amount of cluster resources, and tasks entering the cluster will be assigned to a queue for processing. Historical task execution information includes task ID, task start time, task end time, CPU required to execute the task (unit: cu/min), task level, and the queue to which the task belongs. The historical task execution information of the cluster is in the historical log of the cluster. When cluster resource scheduling is required, you can request the cluster background to obtain the cluster historical log, and then obtain the historical task execution information of the cluster from the cluster historical log. For example, when the cluster is a YARN cluster , you can request the YARN cluster background to obtain historical task execution information in the cluster history log.

302. Determine initial resource requirement information of each queue in the cluster according to the historical task execution information, where the initial resource requirement information includes the initial resource value of each queue at each moment within a preset first time period.

The initial resource demand information includes the initial resource value of each queue in the cluster at each moment in the preset first time period. The resource forecast value required at any time, the preset first time period can be set according to user needs, it can be from 0:00 to 24:00 in a day, or any other time period in a day, such as 8:00 a.m. to 8:00 p.m. hour. In this embodiment, after acquiring the historical task execution information of the cluster, the initial resource requirement information of each queue in the cluster is determined according to the historical task execution information, so that the cluster resources can be dynamically scheduled according to the initial resource requirement information in subsequent steps.

In a specific implementation manner, as shown in FIG. 3, determining the initial resource requirement information of each queue in the cluster according to the historical task execution information in step 302 may include the following steps 401-402, specifically as follows:

401. According to the historical task execution information, determine the historical resource demand value of each queue in the cluster at each moment within a preset first time period;

402. Determine initial resource requirement information of each queue in the cluster according to the historical resource requirement value.

The historical resource demand value of each queue at each moment is the sum of the resource requirements of the running tasks of each queue at each moment. For example, queue 1 runs task A and task B at time t, and task A needs The resource is 5cu/min, and the required resource per minute of task B is 6cu/min, then the historical resource demand value of queue 1 at time t is 11cu/min. The historical task execution information mentioned in the preceding steps includes the task ID, task start time, task end time, CPU required to execute the task (unit: cu/min), task level, and the queue to which the task belongs. information, the tasks executed by each queue at each moment in the preset first time period and the resource requirements of the tasks executed at each moment can be determined according to the historical task execution information, and then it can be determined that each queue is executed at the preset first time The historical resource demand value at each moment in the period, and then determine the initial resource demand information of each queue in the cluster according to the historical resource demand value of each queue at each moment in the preset first time period.

In a specific implementation manner, in step 401, according to the historical task execution information, determining the historical resource demand value of each queue in the cluster at each moment within the preset first time period may include the following steps 501- 502, as follows:

501. According to the historical task execution information, determine the task time sequence of each queue in the cluster and the average resource demand value of the tasks in each queue within the execution time;

502. Determine, according to the task time series and the average resource demand value, the historical resource demand value of each queue in the cluster at each moment within a preset first time period.

The task time sequence of each queue refers to the task sequence executed by each queue in each time period. According to the task time sequence of each queue, the tasks executed by each queue at each moment can be determined. For example, as shown in Table 1, queue 1 is at 0 - Execute task A within 5 minutes, execute task B within 1-5 minutes, and execute task C within 5-15 minutes. According to the task time sequence of queue 1, it can be determined that queue 1 executes task A within 0-1 minutes, and within 1-5 minutes Execute task A and task B, and perform task C within 5-15 minutes. The average resource demand of a task within the execution time refers to the average amount of resources required by the task per minute during the execution time, and its value is divided by the amount of resources required by the task divided by the running time of the task. The running time of A is 5 minutes, and the amount of resources required by task A is 30 (cu/min), then the average resource demand value of task A during the execution time is 30/5=6 (cu/min).

Table 1 Task data table

Historical task execution information includes task ID, task start time, task end time, CPU required to execute the task (unit: cu/min), task level, and the queue to which the task belongs. To determine the historical resource demand value at any time, you can first determine the task time sequence of each queue in the cluster based on the task ID, task start time, task end time, task level, and task queue in the historical task execution information, and determine the task time sequence according to the historical task execution The task start running time, the task ending running time and the CPU required to execute the task in the information determine the average resource demand value of the task in each queue within the execution time, and then traverse the task time series of each queue, and according to the task in each queue The average resource demand value of the task within the execution time determines the historical resource demand value of each queue in the cluster at each moment within the preset first time period. For example, in Table 1, task A is executed in 0-1 minutes, task A and task B are executed in 1-5 minutes, and task C is executed in 5-15 minutes. The average resource demand value of task A is 6 (cu/min), The average resource demand value of task B is 5 (cu/min), and the average resource demand value of task C is 5 (cu/min), then the historical resource demand value of 0-1 minute is 6 (cu/min), 1 - The historical resource demand value for 5 minutes is 11 (cu/min), and the historical resource demand value for 5-15 minutes is 5 (cu/min).

In a specific implementation manner, as shown in FIG. 3, determining the initial resource requirement information of each queue in the cluster according to the historical resource requirement value in step 402 may include the following steps 503-504, specifically as follows:

503. Determine the initial resource value of each queue at each moment in the first time period according to the historical resource demand value;

504. Determine initial resource requirement information of each queue in the cluster according to the initial resource value of each queue at each moment in the first time period.

In this embodiment, resource allocation may be performed to each queue in real time, or resource allocation may be performed to each queue at intervals of a preset time. When allocating resources to each queue in real time, the initial resource value of each queue at each moment in the first time period can be the historical resource demand value of each queue at each moment in the first time period, then it is determined that each queue is at After the historical resource demand value at each moment in a time period, the historical resource demand value of each queue at each moment in the first time period can be directly used as the initial resource value of each queue at each moment in the first time period, And according to the initial resource value of each queue at each moment in the first time period, the initial resource requirement information of each queue in the cluster is determined.

When resources are allocated to each queue at intervals of preset time intervals, in order to meet the task resource requirements within the preset time intervals, in this embodiment, the preset first time period is divided according to preset intervals, and the first time period is divided into There are several time windows, and the maximum value of the historical resource demand value in the time window is used as the initial resource value at each moment in the time window. For example, the first time period is from 0:00 to 24:00, and the first time period is divided according to an interval of 15 minutes. Assuming that the tasks executed in the time window of 0-15 minutes are as shown in Table 1, then 0-15 The initial resource value at each moment in a minute is y ₀ =Max(6+5,5)=11.

When the initial resource value at each moment is the maximum value among the historical resource demand values within the time window to which this moment belongs, the second can be determined according to the historical resource demand values of each queue at each moment within the preset first time period. The maximum value of the historical resource demand value in the time window to which each moment belongs in a time period, and then determine each queue according to the maximum value in the historical resource demand value in the time window to which each moment belongs to in the first time period The initial resource value at each moment in the first time period, and then determine the initial resource demand information of each queue in the cluster according to the initial resource value of each queue at each moment in the first time period.

303. Acquire real-time resource requirement information of each queue, and determine a resource correction value of each queue at a next moment according to the real-time resource requirement information and the initial resource requirement information.

The real-time resource demand information is the real-time resource value required by each queue, the initial resource demand information is the predicted resource demand value of each queue determined by the historical task execution information, and the resource correction value is determined according to the real-time resource demand information and the initial resource demand information. The deviation value used to represent the deviation between the resource demand forecast value and the real-time resource demand value of each queue. In this embodiment, after obtaining the real-time resource demand information of each queue, the resource correction value of each queue at the next moment is determined according to the real-time resource demand information and the initial resource demand information, so as to allocate resources to each queue according to the resource correction value in subsequent steps.

In a specific implementation manner, continuing to refer to FIG. 3 , determining the resource correction value of each queue at the next moment according to the real-time resource requirement information and the initial resource requirement information in step 303 may include the following steps 403~404, the details are as follows:

403. According to the real-time resource requirement information and the initial resource requirement information, respectively determine the real-time resource requirement value and the initial resource value of each queue at each moment within a preset second time period, the second time period included within said first time period;

404. Determine a resource correction value of each queue at a next moment according to the real-time resource demand value and the initial resource value of each queue at each moment within the second time period.

The preset second time period is included in the preset first time period, that is to say, the second time period is a part of the first time period, for example, the first time period is from 0:00 to 24:00, and the second time period From 1 o'clock to 2 o'clock, in order to more accurately determine the resource correction value of each queue at the next moment, the preset second time period is the time period including the current moment, for example, the next moment is t+1 time, the second time The segment is from time t-n to time t. In this embodiment, when determining the resource correction value of each queue at the next moment, the real-time resource demand value at each moment within the preset second time period can be determined according to the real-time resource demand information, and the real-time resource demand value of each queue can be determined according to the initial resource demand information. Preset the initial resource value at each moment in the second time period, and then determine the resource correction value of each queue at the next moment according to the real-time resource demand value and initial resource value of each queue at each moment in the preset second time period .

In a specific implementation manner, in step 404, according to the real-time resource demand value and the initial resource value of each queue at each moment within the second time period, determine the resource correction value of each queue at the next moment , may include the following steps 505-506, specifically as follows:

505. Determine the resource difference value of each queue at each moment in the second time period according to the real-time resource demand value and the initial resource value of each queue at each moment in the second time period;

506. Determine a resource correction value of each queue at a next moment according to the resource difference.

The resource difference of each queue at each moment in the second time period is the difference between the real-time resource demand value and the initial resource value of each queue at each moment in the second time period. When calculating the resource difference at each moment in the period, the difference operation may be performed on the real-time resource demand value and the initial resource demand value of each queue at each moment in the second time period. For example, when the second time period is from time T _s to time T _e , the resource difference R _i =rt _i -y _i of each queue at each moment in the second time period, where R _i is the resource at time i Difference, i∈T _s →T _e , rt _i is the real-time resource demand value at time i, and y _i is the initial resource value at time i.

After obtaining the resource differences of each queue at each moment in the second time period, the resource differences of each queue at each moment in the second time period are respectively summed and then averaged to obtain the resource difference of each queue at the next moment Resource modifier. For example, when the second time period is from time T _s to time T _e , the resource correction value of each queue at the next time Among them, D is the resource correction value at the next moment, T _s is the start time of the second time period, T _e is the end time of the second time period, i∈T _s →T _e , rt _i is the real-time resource at time i demand value, y _i is the initial resource value at time i, and (rt _i -y _i ) is the resource difference at time i.

304. Perform resource allocation to the queues according to the initial resource requirement information and the resource correction value.

The initial resource demand information includes the initial resource value of each queue at each moment in the preset first time period, and the initial resource value is the resource demand forecast value determined according to the historical task execution information, and there is usually a gap between the real-time resource demand value and the real-time resource demand value. There is a certain deviation. In this embodiment, the resource correction value of each queue at the next moment is obtained, and the initial resource value in the initial resource demand information is corrected according to the resource correction value, and then resource allocation for each queue is performed according to the corrected initial resource value. Allocation makes the cluster resources allocated to each queue more even and reasonable, improves the efficiency of task execution with as few resources as possible, improves the utilization of cluster resources, and reduces the time for task execution.

In a specific implementation manner, continuing to refer to FIG. 3 , in step 304, according to the initial resource requirement information and the resource correction value, performing resource allocation to each of the queues may include the following steps 405-406, details as follows:

405. Determine the target resource demand value of each queue at the next moment according to the initial resource demand information and the resource correction value;

406. Perform resource allocation on each of the queues according to the target resource demand value.

The target resource demand value is the resource value required by each queue at the next moment determined according to the initial resource demand information and the resource correction value. In this embodiment, after determining the resource correction value of each queue at the next moment, according to the initial resource demand information and The resource correction value determines the target resource demand value of each queue at the next moment, and then allocates resources for each queue according to the target resource demand value of each queue at the next moment and the priority of each queue, so that cluster resources can be dynamically allocated to each queue , improve the utilization rate of cluster resources, improve the efficiency of task execution with as few resources as possible, and reduce the time of task execution. For example, if the target resource demand value of queue 1 is determined to be y _next1 , the target resource demand value of queue 2 is y _next2 , and the target resource demand value of queue 3 is y _next3 , then the priority of queue 1, queue 2, and queue 3 can be determined And the determined target resource demand value of each queue allocates cluster resources to each queue. Before allocating resources to each queue according to the target resource demand value, this embodiment also compares the target resource demand value of each queue at the next moment with the real-time resource demand value of each queue at the next moment. When the target resource demand value does not meet the real-time resource demand value of each queue at the next moment, continue to obtain the real-time resource demand information of each queue, and determine the real-time resource demand information of each queue according to the real-time resource demand information and the initial resource demand information. According to the resource correction value of the initial resource demand information and the resource correction value, the step of determining the target resource demand value of each queue at the next moment until the target resource demand value of each queue at the next moment meets the requirements of each queue at the next moment Real-time resource demand value at a moment.

In a specific implementation manner, in step 405, according to the initial resource requirement information and the resource correction value, determining the target resource requirement value of each queue at the next moment may include the following steps 507-508, specifically as follows :

507. Determine the initial resource value of each queue at the next moment according to the initial resource requirement information;

508. Determine the target resource demand value of each queue at a next time according to the initial resource value of each queue at the next time and the resource correction value.

The initial resource value of each queue at the next moment is the predicted resource demand value of each queue at the next moment determined according to the historical task execution information, and the initial resource demand information includes the initial resource value of each queue at the next moment. In this embodiment, when determining the target resource demand value of each queue at the next moment, first determine the initial resource value of each queue at the next moment according to the initial resource demand information, and then calculate the initial resource value of each queue at the next moment according to the resource correction value Correction is performed to obtain the target resource demand value of each queue at the next moment. For example, the resource demand value of each queue at the next moment is y _t , then the target resource demand value of each queue at the next moment y _next =y _t +D, where y _next is the target resource demand value at the next moment, and D is The resource modifier for the next moment.

In order to better implement the cluster resource scheduling method in the embodiment of the present application, on the basis of the cluster resource scheduling method, the embodiment of the present application also provides a cluster resource scheduling device, as shown in FIG. 4 , the cluster resource scheduling device 600 include:

An information acquisition unit 601, configured to acquire historical task execution information of a cluster, where the cluster includes multiple queues for executing tasks;

An information determining unit 602, configured to determine initial resource requirement information of each queue in the cluster according to the historical task execution information, where the initial resource requirement information includes the initial resource value;

A correction value determining unit 603, configured to acquire real-time resource demand information of each queue, and determine a resource correction value of each queue at a next moment according to the real-time resource demand information and the initial resource demand information;

The resource scheduling unit 604 is configured to allocate resources to the queues according to the initial resource requirement information and the resource correction value.

In the embodiment of the present application, the initial resource demand information of each queue is determined according to the historical task execution information of the cluster, and the resource correction value of each queue at the next moment is determined according to the initial resource demand information of each queue and the real-time resource demand information of each queue, and Allocating resources to each queue according to the initial resource demand information and the resource correction value can improve the efficiency of task execution with as few resources as possible, improve the utilization rate of cluster resources, and reduce the time of task execution.

In some embodiments of the present application, the information determining unit 602 is specifically configured to:

According to the historical task execution information, determine the historical resource demand value of each queue in the cluster at each moment within the preset first time period;

Determine initial resource requirement information of each queue in the cluster according to the historical resource requirement value.

In some embodiments of the present application, the information determining unit 602 is further configured to:

According to the historical task execution information, determine the task time sequence of each queue in the cluster and the average resource demand value of the tasks in each queue within the execution time;

According to the task time series and the average resource demand value, determine the historical resource demand value of each queue in the cluster at each moment within a preset first time period.

In some embodiments of the present application, the information determining unit 602 is further configured to:

determining the initial resource value of each queue at each moment within the first time period according to the historical resource demand value;

The initial resource requirement information of each queue in the cluster is determined according to the initial resource value of each queue at each moment in the first time period.

In some embodiments of the present application, the correction value determination unit 603 is specifically configured to:

According to the real-time resource demand information and the initial resource demand information, respectively determine the real-time resource demand value and the initial resource value of each queue at each moment within a preset second time period, the second time period being included in within said first period of time;

According to the real-time resource demand value and the initial resource value of each queue at each moment within the second time period, determine the resource correction value of each queue at a next moment.

In some embodiments of the present application, the correction value determination unit 603 is specifically further configured to:

determining the resource difference of each queue at each moment in the second time period according to the real-time resource demand value and the initial resource value of the queues at each moment in the second time period;

According to the resource difference value, the resource correction value of each queue at the next moment is determined.

In some embodiments of the present application, the resource scheduling unit 604 is specifically configured to:

determining the target resource demand value of each queue at the next moment according to the initial resource demand information and the resource correction value;

Resource allocation is performed on each of the queues according to the target resource demand value.

In some embodiments of the present application, the resource scheduling unit 604 is specifically further configured to:

determining the initial resource value of each queue at the next moment according to the initial resource requirement information;

Determine the target resource demand value of each queue at the next moment according to the initial resource value of each queue at the next moment and the resource correction value.

The embodiment of the present application also provides a computer device, which integrates any cluster resource scheduling device provided in the embodiment of the present application, and the computer device includes:

one or more processors;

storage; and

One or more application programs, wherein the one or more application programs are stored in the memory and are configured to be executed by the processor as described in any one of the cluster resource scheduling method embodiments above. Steps in a resource scheduling method.

The embodiment of the present application further provides a computer device, which integrates any cluster resource scheduling device provided in the embodiment of the present application. As shown in Figure 5, it shows a schematic structural diagram of the computer equipment involved in the embodiment of the present application, specifically:

The computer device may include a processor 701 of one or more processing cores, a memory 702 of one or more computer-readable storage media, a power supply 703, an input unit 704 and other components. Those skilled in the art can understand that the structure of the computer device shown in FIG. 5 does not constitute a limitation on the computer device, and may include more or less components than shown in the figure, or combine some components, or arrange different components. in:

The processor 701 is the control center of the computer equipment. It uses various interfaces and lines to connect various parts of the entire computer equipment. By running or executing software programs and/or modules stored in the memory 702, and calling the Data, perform various functions of computer equipment and process data, so as to monitor the computer equipment as a whole. Optionally, the processor 701 may include one or more processing cores; preferably, the processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces, and application programs, etc. , the modem processor mainly handles wireless communications. It can be understood that the foregoing modem processor may not be integrated into the processor 701 .

The memory 702 can be used to store software programs and modules, and the processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702 . The memory 702 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playback function, an image playback function, etc.); Data created by the use of computer equipment, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 702 may further include a memory controller to provide the processor 701 with access to the memory 702 .

The computer device also includes a power supply 703 for supplying power to various components. Preferably, the power supply 703 can be logically connected to the processor 701 through a power management system, so that functions such as charging, discharging, and power consumption management can be realized through the power management system. The power supply 703 may also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

The computer device can also include an input unit 704, which can be used to receive input numbers or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

Although not shown, the computer device may also include a display unit, etc., which will not be repeated here. Specifically, in this embodiment, the processor 701 in the computer device loads the executable file corresponding to the process of one or more application programs into the memory 702 according to the following instructions, and the processor 701 runs the executable file stored in the The application program in memory 702, thus realizes various functions, as follows:

Obtain historical task execution information of the cluster, where the cluster includes multiple queues for executing tasks;

determining initial resource requirement information of each queue in the cluster according to the historical task execution information, where the initial resource requirement information includes the initial resource value of each queue at each moment within a preset first time period;

Acquiring the real-time resource requirement information of each queue, and determining the resource correction value of each queue at the next moment according to the real-time resource requirement information and the initial resource requirement information;

Resource allocation is performed on each of the queues according to the initial resource requirement information and the resource correction value.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by instructions controlling related hardware, and the instructions can be stored in a computer-readable storage medium, and is loaded and executed by the processor.

To this end, an embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, etc. . A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any cluster resource scheduling method provided in the embodiments of the present application. For example, the computer program being loaded by the processor may perform the following steps:

Obtain historical task execution information of the cluster, where the cluster includes multiple queues for executing tasks;

determining initial resource requirement information of each queue in the cluster according to the historical task execution information, where the initial resource requirement information includes the initial resource value of each queue at each moment within a preset first time period;

Acquiring the real-time resource requirement information of each queue, and determining the resource correction value of each queue at the next moment according to the real-time resource requirement information and the initial resource requirement information;

Resource allocation is performed on each of the queues according to the initial resource requirement information and the resource correction value.

In the above-mentioned embodiments, the descriptions of each embodiment have their own emphases. For the part that is not described in detail in a certain embodiment, refer to the detailed description of other embodiments above, and will not be repeated here.

During specific implementation, each of the above units or structures can be implemented as an independent entity, or can be combined arbitrarily as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, here No longer.

For the specific implementation of the above operations, reference may be made to the foregoing embodiments, and details are not repeated here.

The above is a detailed introduction of a cluster resource scheduling method, device, computer equipment and storage medium provided by the embodiments of the present application. In this paper, specific examples are used to illustrate the principles and implementation methods of the present application. The description of the above embodiments It is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation and application scope. In summary, this The content of the description should not be understood as limiting the application.

Claims (11)

1. A cluster resource scheduling method, characterized in that, the cluster resource scheduling method comprises: Obtain historical task execution information of the cluster, where the cluster includes multiple queues for executing tasks; determining initial resource requirement information of each queue in the cluster according to the historical task execution information, where the initial resource requirement information includes the initial resource value of each queue at each moment within a preset first time period; Acquiring the real-time resource requirement information of each queue, and determining the resource correction value of each queue at the next moment according to the real-time resource requirement information and the initial resource requirement information; Resource allocation is performed on each of the queues according to the initial resource requirement information and the resource correction value. 2. The cluster resource scheduling method according to claim 1, wherein the determining the initial resource requirement information of each queue in the cluster according to the historical task execution information comprises: According to the historical task execution information, determine the historical resource demand value of each queue in the cluster at each moment within the preset first time period; Determine initial resource requirement information of each queue in the cluster according to the historical resource requirement value. 3. The cluster resource scheduling method according to claim 2, wherein, according to the historical task execution information, the historical resource requirements of each queue in the cluster are determined at each moment within the preset first time period values, including: According to the historical task execution information, determine the task time sequence of each queue in the cluster and the average resource demand value of the tasks in each queue within the execution time; According to the task time series and the average resource demand value, determine the historical resource demand value of each queue in the cluster at each moment within a preset first time period. 4. The cluster resource scheduling method according to claim 2, wherein the determining the initial resource demand information of each queue in the cluster according to the historical resource demand value comprises: determining the initial resource value of each queue at each moment within the first time period according to the historical resource demand value; The initial resource requirement information of each queue in the cluster is determined according to the initial resource value of each queue at each moment in the first time period. 5. The cluster resource scheduling method according to claim 1, wherein the determining the resource correction value of each queue at the next moment according to the real-time resource requirement information and the initial resource requirement information comprises: According to the real-time resource demand information and the initial resource demand information, respectively determine the real-time resource demand value and the initial resource value of each queue at each moment within a preset second time period, the second time period being included in within said first period of time; According to the real-time resource demand value and the initial resource value of each queue at each moment within the second time period, determine the resource correction value of each queue at a next moment. 6. The cluster resource scheduling method according to claim 5, characterized in that, according to the real-time resource demand value and initial resource value of each queue at each moment in the second time period, the The resource correction value of the queue at the next moment, including: determining the resource difference of each queue at each moment in the second time period according to the real-time resource demand value and the initial resource value of the queues at each moment in the second time period; According to the resource difference value, the resource correction value of each queue at the next moment is determined. 7. The cluster resource scheduling method according to claim 1, wherein the resource allocation for each queue according to the initial resource requirement information and the resource correction value comprises: determining the target resource demand value of each queue at the next moment according to the initial resource demand information and the resource correction value; Resource allocation is performed on each of the queues according to the target resource demand value. 8. The cluster resource scheduling method according to claim 7, wherein the determining the target resource demand value of each queue at the next moment according to the initial resource demand information and the resource correction value comprises: determining the initial resource value of each queue at the next moment according to the initial resource requirement information; Determine the target resource demand value of each queue at the next moment according to the initial resource value of each queue at the next moment and the resource correction value. 9. A cluster resource scheduling device, characterized in that the cluster resource scheduling device comprises: An information acquisition unit, configured to acquire historical task execution information of the cluster, where the cluster includes multiple queues for executing tasks; An information determining unit, configured to determine initial resource requirement information of each queue in the cluster according to the historical task execution information, where the initial resource requirement information includes the initial resource requirement information of each queue at each moment within a preset first time period. resource value; A correction value determining unit, configured to acquire real-time resource demand information of each queue, and determine a resource correction value of each queue at a next moment according to the real-time resource demand information and the initial resource demand information; A resource scheduling unit, configured to allocate resources to each of the queues according to the initial resource requirement information and the resource correction value. 10. A computer device, characterized in that the computer device comprises: one or more processors; storage; and One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the cluster resource scheduling according to any one of claims 1 to 8 method. 11. A computer-readable storage medium, wherein a computer program is stored thereon, and the computer program is loaded by a processor to execute the method in the cluster resource scheduling method according to any one of claims 1 to 8. step.