WO2025227845A1 - Training task processing method and related device - Google Patents

Abstract

Provided in the embodiments of the present application are a training task processing method and a related device. The method may be applied to a model training stage, and comprises: receiving request information for requesting the execution of a training task for a first model; if the priority of the training task for the first model is higher than the priority of a second training task, executing the training task for the first model, and suspending the execution of the second training task; and if the priority of the training task for the first model is lower than or equal to the priority of the second training task, sending response information, wherein the response information is used for indicating a delay or refusal to execute the training task for the first model, or the response information comprises information of the second training task, and the second training task is a training task currently executed by means of a training device. The priorities of training tasks are used to decide to suspend, delay or refuse the execution of some training tasks, thereby helping to prevent a training device from being overloaded and helping to avoid interruptions to processes for executing the training tasks or data loss.

Description

A method for processing training tasks and related equipment

This application claims priority to Chinese Patent Application No. 202410547036.5, filed on April 30, 2024, entitled "A Method for Processing Training Tasks and Related Equipment", the entire contents of which are incorporated herein by reference.

Technical Field

This application relates to the field of artificial intelligence, and in particular to a method for processing training tasks and related equipment.

Background Technology

With the continuous development of intelligence, artificial intelligence (AI) technology is being applied in more and more fields, that is, machine learning (ML) models are being applied in more and more fields. Before a machine learning model is applied, it needs to be trained.

The requesting device of the machine learning model can send a request message to the training device, which requests the training device to perform the task of training the machine learning model. The training device can perform the training task according to the received request message.

With the widespread application of AI, a single training device can receive requests from multiple requesting devices. In other words, the training task currently being executed by the same training device may include multiple training tasks. However, since the computer resources in the training device are limited, if the training device executes multiple training tasks simultaneously, it may overload the training device, resulting in the interruption of the process used to execute the training tasks or the loss of data.

Summary of the Invention

This application provides a method for processing training tasks and related equipment. The training equipment can use the priority of training tasks to decide to pause, delay or refuse to execute certain training tasks, which helps to avoid overloading of the training equipment, thereby helping to avoid interruption of the process used to execute training tasks or to avoid data loss, and improving the stability of the training equipment in the process of executing training tasks.

The embodiments of this application provide the following technical solutions:

In a first aspect, embodiments of this application provide a method for processing training tasks. This method can be applied to the training phase of a model. The method includes: after a training device receives request information from a first device (hereinafter referred to as "first request information" for easy distinction), the first request information is used to request the training device to execute a training task of a first model (hereinafter referred to as the first training task, i.e., the training task of the first model and the first training task can be relatedly replaced); if the priority of the training task of the first model is higher than the priority of the second training task, the training device executes the training task of the first model and suspends the execution of the second training task; wherein, the second training task is one or more training tasks currently being executed by the training device, or, the second training task is the training task with the lowest priority among the training tasks currently being executed by the training device.

For example, if the second training task is one of multiple training tasks currently being executed by the training device, then "the priority of the first training task is higher than the priority of the second training task" can be understood as the priority of the first training task being higher than the priority of each training task included in the second training task. The training device starting to execute the first training task means that the first training task has been added to the training tasks currently being executed by the training device; "the training device pausing to execute the second training task" can be referred to as the training device suspending the second training task, so that the second training task is no longer included in the training tasks currently being executed by the training device.

And/or,

After receiving the first request information from the first device, if the priority of the training task of the first model is lower than or equal to the priority of the second training task, the training device sends a response information to the first device. The response information is used to notify the first device to delay or refuse to execute the training task of the first model, or the response information includes information about the second training task; wherein, the second training task is one or more training tasks currently being executed by the training device, or the second training task is the training task with the lowest priority among the training tasks currently being executed by the training device.

In this implementation, after receiving the first request information, the training device can also determine whether to execute the training task of the first model based on the priority of the training task of the first model. If the priority of the training task of the first model is higher than the priority of the second training task, the training task of the first model can be executed and the execution of the second training task can be suspended. If the priority of the training task of the first model is lower than or equal to the priority of the second training task, the execution of the training task of the first model can be delayed or rejected. The second training task is the training task currently being executed by the training device. That is, the training device can use the priority of the training task to decide to pause, delay or reject the execution of certain training tasks, which helps to avoid the overload of the training device, thereby helping to avoid the interruption of the process used to execute the training task or the loss of data, and improving the stability of the training device in the process of executing the training task.

In one possible implementation, the second training task may include one or more training tasks with the lowest priority among all currently executed training tasks, and the sum of the idle resources of the training device and the occupied resources of the second training task is greater than or equal to the required resources of the training task of the first model.

In one possible implementation, the first request information includes the priority of the training task of the first model, which can also be understood as the first request information including the priority of the first training task; for example, the first device determines the priority of the first training task based on a first factor, which may include the inference type of the first model; optionally, the first factor may also include: the resource requirements of the first training task and the immediacy requirements of the first model.

In this implementation, if the first request information includes the priority of the first training task, the training device can directly obtain the priority of the first training task from the first request information. This allows for a faster comparison between the priority of the first training task and the priorities of other training tasks. Since the decision on whether to execute the first training task can only be made after determining the comparison result between the priority of the first training task and the priority of the second training task, this also facilitates a faster determination of whether to execute the first training task. Furthermore, obtaining the priority of the first training task directly from the first request information reduces the resources consumed in the process of "the training device determining the priority of the first training task," allowing the training device to allocate more resources to executing the training task and obtain a model that has completed the training task as quickly as possible.

In one possible implementation, the information of the second training task includes the priority of each training task included in the second training task; optionally, the information of the second training task may also include identification information of each training task in the second training task; optionally, it may also include the inference type corresponding to each training task in the second training task. The second request information corresponding to the second training task may include the priority of the second training task, and optionally, the training device may also adjust the priority of the second training task.

In this implementation, if the priority of the first training task is lower than or equal to the priority of the second training task, the training device will delay or refuse to execute the first training task. The response information sent by the training device to the first device includes the priority of the second training task. Thus, the first device can not only know that the training device has decided to delay or refuse to execute the first training task, but also know that the training device is executing the second training task with a higher priority, so it decides to delay or refuse to execute the first training task. In other words, the first device can know the current status of the training device, which makes it easier for the first device to determine a more suitable processing method by combining the current status of the training device and the needs of the first model. It also helps to make the use of resources in the training device better meet the needs of the current scenario.

In one possible implementation, after the training device suspends the execution of the second training task, the method further includes: the training device notifying the second device corresponding to the second training task to suspend the execution of the second training task; or, the training device notifying the second device corresponding to the second training task of the training device's idle resources. For example, the idle resources of the training device may include idle storage resources in the training device; optionally, it may also include idle processor resources in the training device. In this implementation, after suspending the execution of the second training task, the training device will promptly notify the second device corresponding to the second training task that the training device has suspended the execution of the second training task, thereby enabling the second device to quickly know that the training device has suspended the execution of the second training task, making the execution process of the second training task more controllable.

In one possible implementation, the training device notifies the second device corresponding to the second training task to suspend the execution of the second training task, including: the training device sending a reason value to the second device, the reason value indicating the reason for suspending the execution of the second training task. In this implementation, after determining that the second training task has been suspended, the training device sends a reason value to the second device, which indicates the reason for suspending the execution of the second training task. Thus, the second device can not only promptly know that the second training task has been suspended, but also know the reason for the suspension. This facilitates the second device in promptly determining the handling method for the second training task after confirming that it has been suspended, and also allows the second device to determine a more suitable handling method based on the reason for suspending the second training task.

In one possible implementation, the reason is that the second training task is the lowest priority training task on the training device. In this implementation, the second training task that the training device pauses execution is the lowest priority training task on the training device. That is, the resources in the training device can be allocated to higher priority training tasks as much as possible, so that the resources in the training device can be used more efficiently.

In one possible implementation, the cause value includes the priority of each of at least one third training task currently being performed by the training device, wherein the priority of each third training task is higher than the priority of the second training task.

For example, "at least one third training task" can include all training tasks currently being executed by the training device. It should be noted that since the training device may pause executing old training tasks or begin executing new training tasks, the training tasks included in "the training tasks currently being executed by the training device" in this application can vary. If, when the training device sends a cause value to the second device, it has paused executing the second training task and begun executing the first training task, then the training tasks currently being executed by the training device can include the first training task; that is, at least one third training task can include the first training task. Alternatively, "at least one third training task" can include only the first training task. Or, "at least one third training task" can include a preset number of training tasks selected from all training tasks currently being executed by the training device, etc.

Optionally, the above cause value may also include identification information for each third training task; alternatively, the above cause value may also include the inference type for each third training task.

In this implementation, the reason value sent by the training device to the second device includes the priority of each of the at least one third training task currently being executed by the training device. The priority of each training task is higher than that of the second training task. After receiving the reason value, the second device can not only know that the execution of the second training task is suspended due to the low priority, but also know the priority of the third training task that is currently occupying the resources of the training device. That is, the second device can know a more detailed resource usage of the training device. This makes it easier for the second device to determine a more suitable processing method by combining the resource usage of the training device with the needs of the second model. It also helps to make the resource usage of the training device better meet the needs of the current scenario.

In one possible implementation, the method further includes: the training device receiving a processing method for the second training task from the second device, wherein the processing method for the second training task is: termination, waiting, or adjustment of resource usage. In this implementation, after learning that the execution of the second training task is paused, the second device can send a processing method for the second training task to the training device. Since the second device has a clearer understanding of its needs for the second model, it can determine whether to terminate, wait, or adjust resource usage based on its needs for the second model. This helps improve the adaptability of the processing method for the second training task to specific application scenarios.

In one possible implementation, the method further includes: when the processing mode of the second training task is termination, the training device deletes the second request information corresponding to the second training task; for example, the training device may delete the second request information corresponding to the second training task. Alternatively, when the processing mode of the second training task is waiting, the training device waits until the idle resources are greater than or equal to the resources required by the second training task before continuing to execute the second training task. Alternatively, when the processing mode of the second training task is adjusting resource usage, the training device reduces the number of parameters of the model corresponding to the second training task, or reduces the accuracy when executing the second training task, or reduces the batch training scale when executing the second training task.

A training task involves multiple training processes on a model. Batch training refers to dividing the multiple training processes of a training task into multiple batches. In the training process of a single batch, n training samples are obtained from the training dataset, where n is an integer greater than or equal to 1. The batch size represents the number of training samples used in the training process of a single batch, which is also the number of n. For example, the "batch size" affects the memory resources required by the model in the training process of a single batch.

For example, "reducing the number of parameters in the model corresponding to the second training task" can be achieved by pruning the second model. The specific pruning algorithm can be flexibly determined based on the actual application scenario, thereby reducing not only the storage resources used but also the processor resources used. "Reducing the precision when performing the second training task" can be achieved by replacing the high-precision data format with a low-precision data format when performing the second training task, such as reducing from FP32 to FP16, or from FP64 to mixed precision, etc. This example is only for the purpose of understanding the solution, thereby reducing not only the storage resources used but also the processor resources used. "Reducing the batch training size when performing the second training task" can be achieved by reducing the number of training samples used in a single batch of training, thereby reducing the storage resources required for a single batch of training.

In this implementation, different processing methods are provided for the second training task fed back by the second device, and specific processing schemes of the training device are provided. Thus, no matter what processing method the second device feeds back, the training device has a corresponding processing scheme, which helps to improve the smoothness and stability of the application during execution. Furthermore, when the processing method of the second training task is termination, the second request information is deleted, thereby releasing all resources related to the second request information in the training device in a timely manner, which helps to avoid wasting resources in the training device.

In one possible implementation, the method is applied to the following scenarios: the resource requirements of the first training task exceed the idle resources of the training device; or, the number of training tasks currently being executed by the training device exceeds a preset threshold. Optionally, the resource requirements of the first training task may include the storage resources required by the first training task, and optionally, may also include the processor resources required by the first training task. The storage resources include video memory resources and/or system memory resources.

In this implementation, the training task processing method provided in this application is started when the required resources of the first training task exceed the idle resources of the training device, or when the number of training tasks currently being executed by the training device exceeds a preset threshold. That is, in the aforementioned scenarios, the training device will pause, delay, or refuse to execute certain training tasks, providing two application scenarios for this application and improving the implementation flexibility of this solution. In addition, in the aforementioned two application scenarios, the load limit of the training device is almost reached, which is conducive to avoiding overload of the training device while making the maximum use of the resources in the training device.

In one possible implementation, adjusting resource usage includes: reducing the number of parameters in the model corresponding to the second training task, reducing the accuracy when performing the second training task, or reducing the batch training scale when performing the second training task. In this implementation, when the processing method for the second training task is to adjust resource usage, the second device can further indicate how to adjust the resources used when performing the second training task. This helps to make the final second model more compatible with the application scenario of the second model, that is, it helps to obtain a more satisfactory second model under the premise of limited resources.

In one possible implementation, each training task currently performed by the training device is a task of training a model in the communication domain. This implementation provides a specific application domain for the method of this application, increasing the degree of integration between this application and a specific application domain.

Secondly, this application provides a method for processing training tasks, which can be applied to the training phase of a model. The method includes: a first device sending request information to a training device, the request information being used to request the execution of a training task of a first model; the first device receiving response information from the training device, the response information being used to notify the first device to delay or refuse to execute the training task of the first model, or the response information including information of a second training task; wherein the priority of the training task of the first model is lower than or equal to the priority of the second training task, the second training task being one or more training tasks currently being executed by the training device, or the second training task being the lowest priority training task among the training tasks currently being executed by the training device.

In one possible implementation, the first device sends the processing method of the training task of the first model to the training device. The processing method of the training task of the first model is: termination, waiting, or adjusting the occupied resources.

In one possible implementation, adjusting resource usage includes: reducing the number of parameters of the model corresponding to the training task of the first model, reducing the accuracy when performing the first training task, or reducing the batch training size when performing the first training task.

In one possible implementation, the determining factors for the processing method of the first model's training task include: the immediacy requirement of the first model, the accuracy requirement of the first model, and/or, the degree of reduction in the accuracy of the first model caused by adjusting resource usage. The "immediacy requirement of the first model" can be understood as the time requirement for the first model after completing the first training task, or as how quickly the first model needs to be deployed. The shorter the deployment time, the higher the immediacy requirement of the first model. The "degree of reduction in the accuracy of the first model caused by adjusting resource usage" can be determined by at least one of the following parameters: the first accuracy range of the first model obtained after adjusting the resources used during the execution of the first training task, and/or, the second accuracy range, where the second accuracy range represents the decrease in accuracy of the final obtained first model due to the adjustment of resource usage by a second accuracy range.

In this implementation, the processing method for the second training task is determined from two dimensions: time requirements and accuracy requirements. This approach is beneficial for obtaining a better processing method under the premise of limited resources.

In the second aspect of this application, the first device may also perform the steps performed by the first device in the first aspect and various implementations of the first aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the second aspect can all be found in the first aspect, and will not be repeated here.

Thirdly, this application provides a method for processing training tasks, which can be applied to the training phase of a model. The method includes: a second device sending a request message to a training device, the request message being used to request the training device to execute a training task of a second model (hereinafter referred to as the second training task); the second device receiving a notification message from the training device, the notification message indicating to suspend the execution of the training task of the second model, or the notification message indicating the idle resources of the training device; wherein, the priority of the second training task is lower than the priority of the first training task, the first training task being a training task added by the training device.

In one possible implementation, the notification information includes a reason value, which indicates the reason for pausing the training task of the second model.

In one possible implementation, the method further includes: the first device sending the processing method of the training task of the second model to the training device, wherein the processing method of the training task of the second model is: termination, waiting, or adjusting the occupied resources.

In one possible implementation, the determining factors for how the training task of the second model is handled include: the immediacy requirements of the second model, the accuracy requirements of the second model, and/or, the degree of reduction in the accuracy of the second model caused by adjusting resource usage.

In the third aspect of this application, the second device may also perform the steps performed by the second device in the first aspect and various implementations of the first aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the third aspect can all be found in the first aspect, and will not be repeated here.

Fourthly, embodiments of this application provide a training task processing apparatus, which can be applied to the training phase of a model and can be used in a training device. The training task processing apparatus includes: a receiving module, configured to receive request information from a first device, the request information being used to request the execution of a training task of a first model; an execution module, configured to execute the training task of the first model and suspend the execution of the second training task if the priority of the training task of the first model is higher than the priority of the second training task; and/or, a sending module, configured to send response information to the first device if the priority of the training task of the first model is lower than or equal to the priority of the second training task, the response information being used to notify the first device to delay or refuse the execution of the training task of the first model, or the response information including information of the second training task; wherein the second training task is one or more training tasks currently being executed by the training device, or the second training task is the training task with the lowest priority among the training tasks currently being executed by the training device.

In one possible implementation, the requested information includes the priority of the training task for the first model.

In one possible implementation, the information for the second training task includes the priority of the second training task.

In one possible implementation, the training task processing device further includes: a notification module for notifying the second device corresponding to the second training task to suspend the execution of the second training task; or, a notification module for notifying the second device corresponding to the second training task of the idle resources of the training device.

In one possible implementation, the notification module is specifically used to send a reason value to the second device, the reason value indicating the reason for suspending the execution of the second training task.

In one possible implementation, the reason is that the second training task is the lowest priority training task on the training device.

In one possible implementation, the cause value includes the priority of each of at least one third training task currently being performed by the training device, wherein the priority of each third training task is higher than the priority of the second training task.

In one possible implementation, the receiving module is further configured to receive the processing method of the second training task from the second device, wherein the processing method of the second training task is: termination, waiting, or adjustment of occupied resources.

In one possible implementation, the training task processing apparatus further includes: a deletion module, used to delete the request information corresponding to the second training task when the processing mode of the second training task is termination; or, an execution module, used to wait until the idle resources are greater than or equal to the resources required by the second training task when the processing mode of the second training task is waiting, and then continue to execute the second training task; or, an adjustment module, used to reduce the number of parameters of the model corresponding to the second training task, or reduce the accuracy when executing the second training task, or reduce the batch training scale when executing the second training task when the processing mode of the second training task is adjusting the occupied resources.

In one possible implementation, the method is applied to the following scenarios: the resource requirements of the first training task are greater than the available resources of the training device; or, the number of training tasks currently being executed by the training device is greater than a preset threshold.

In one possible implementation, adjusting resource usage includes: reducing the number of parameters in the model corresponding to the second training task, reducing the accuracy when performing the second training task, or reducing the batch training size when performing the second training task.

In one possible implementation, each training task currently performed by the training device is a task of training a model in the field of communications.

In the fourth aspect of this application, the training task processing device can also execute the steps performed by the training device in the first aspect and various implementations of the first aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the fourth aspect can all be found in the first aspect, and will not be repeated here.

Fifthly, embodiments of this application provide a training task processing apparatus, which can be applied to the training phase of a model. This apparatus can be used in a first device. The training task processing apparatus includes: a sending module, configured to send request information to the training device, the request information being used to request the execution of a training task for a first model; and a receiving module, configured to receive response information from the training device, the response information being used to notify the first device to delay or refuse to execute the training task for the first model, or the response information including information about a second training task; wherein the priority of the first training task is lower than or equal to the priority of the second training task, the second training task is one or more training tasks currently being executed by the training device, or the second training task is the lowest priority training task among the training tasks currently being executed by the training device.

In one possible implementation, the sending module is further configured to send the processing method of the first training task to the training device, wherein the processing method of the first training task is: termination, waiting, or adjustment of occupied resources.

In one possible implementation, adjusting resource usage includes: reducing the number of parameters in the model corresponding to the first training task, reducing the accuracy when performing the first training task, or reducing the batch training size when performing the first training task.

In one possible implementation, the determining factors for how the first training task is handled include: the immediacy requirements of the first model, the accuracy requirements of the first model, and/or, the degree of reduction in the accuracy of the first model caused by adjusting resource usage.

In the fifth aspect of this application, the training task processing device can also execute the steps executed by the first device in the second aspect and various implementations of the second aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the fifth aspect can all be found in the second aspect, and will not be repeated here.

Sixthly, embodiments of this application provide a training task processing apparatus, which can be applied to the training phase of a model and can be used in a second device. The training task processing apparatus includes: a sending module, used to send request information to the training device, the request information being used to request the training device to execute a training task of a second model; and a receiving module, used to receive notification information from the training device, the notification information indicating to suspend the execution of the training task of the second model, or the notification information indicating the idle resources of the training device; wherein, the priority of the second training task is lower than the priority of the first training task, and the first training task is a training task added by the training device.

In one possible implementation, the notification information includes a reason value, which indicates the reason for pausing the training task of the second model.

In one possible implementation, the sending module is further configured to send the processing method of the training task of the second model to the training device, wherein the processing method of the training task of the second model is: termination, waiting, or adjustment of resource usage.

In the sixth aspect of this application, the training task processing device can also execute the steps performed by the second device in the third aspect and various implementations of the third aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the sixth aspect can all be found in the second aspect, and will not be repeated here.

In a seventh aspect, embodiments of this application provide an apparatus including a processor and a memory, the processor being coupled to the memory, the memory being used to store a program; and the processor being used to execute the program in the memory, causing the apparatus to perform the methods described in the first, second, or third aspects above.

Eighthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first, second, or third aspects above.

Ninthly, embodiments of this application provide a computer program product, which includes a program that, when run on a computer, causes the computer to perform the methods described in the first, second, or third aspects above.

In a tenth aspect, this application provides a chip system including a processor for supporting the implementation of the functions involved in the foregoing aspects, such as transmitting or processing data and/or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for a terminal device or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

Attached Figure Description

Figure 1 is a schematic diagram of a structure;

Figure 2 is a schematic diagram of the training and application phases of the model;

Figure 3a is a schematic diagram of an architecture for a training task processing system;

Figure 3b is a schematic diagram of another architecture for the training task processing system;

Figure 3c is a schematic diagram of another architecture for the training task processing system;

Figure 4 is a schematic diagram of a training task processing method provided in an embodiment of this application;

Figure 5 is another schematic diagram of the training task processing method provided in the embodiments of this application;

Figure 6 is another schematic diagram of the training task processing method provided in the embodiments of this application;

Figure 7 is a schematic diagram of a training task processing device provided in an embodiment of this application;

Figure 8 is a schematic diagram of another structure of the training task processing device provided in the embodiment of this application;

Figure 9 is a schematic diagram of another structure of the training task processing device provided in the embodiment of this application;

Figure 10 is a schematic diagram of the structure of a device provided in an embodiment of this application.

Detailed Implementation

The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

In the embodiments of this application, "send" and "receive" refer to the direction of signal transmission. For example, "send information to device XX" can be understood as the destination of the information being device XX, which may include direct transmission via the air interface or indirect transmission by other units or modules via the air interface. "Receive information from device YY" can be understood as the source of the information being device YY, which may include direct reception from device YY via the air interface or indirect reception from device YY via other units or modules via the air interface. "Send" can also be understood as the "output" of the chip interface, and "receive" can also be understood as the "input" of the chip interface. In other words, sending and receiving can occur between devices or within devices, for example, through buses, traces, or interfaces between components, modules, chips, software modules, or hardware modules within a device. It is understood that information may undergo necessary processing, such as encoding and modulation, between the source and destination of information transmission, but the destination can understand the valid information from the source. Similar expressions in this application can be understood in a similar way and will not be elaborated further.

In the embodiments of this application, "instruction" can include direct and indirect instructions, as well as explicit and implicit instructions. The information indicated by a certain piece of information (hereinafter referred to as instruction information) is called the information to be instructed. In specific implementation, there are many ways to indicate the information to be instructed, such as, but not limited to, directly indicating the information to be instructed, such as the information to be instructed itself or its index. It can also indirectly indicate the information to be instructed by indicating other information, where there is an association between the other information and the information to be instructed; or it can indicate only a part of the information to be instructed, while the other parts are known or pre-agreed upon. For example, the instruction can be implemented by using a pre-agreed (e.g., protocol predefined) arrangement of various information, thereby reducing the instruction overhead to a certain extent. This application does not limit the specific method of instruction. It is understood that for the sender of the instruction information, the instruction information can be used to indicate the information to be instructed; for the receiver of the instruction information, the instruction information can be used to determine the information to be instructed.

First, the overall workflow of an artificial intelligence system is described, as shown in Figure 1, which is a structural diagram. The framework of the aforementioned artificial intelligence theme is then elaborated from two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

(1) Infrastructure

The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips, which can specifically employ hardware acceleration chips such as central processing units (CPUs), neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

(2) Data

The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

(3) Data processing

Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

(4) General ability

After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

(5) Smart Products and Industry Applications

Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: communications, intelligent driving, intelligent terminals, intelligent transportation, smart homes, intelligent healthcare, intelligent security, intelligent manufacturing, and smart cities.

The method provided in this application can be applied to various application fields of artificial intelligence technology, specifically to the training stage of models in various application fields. Optionally, the method provided in this application can be applied to application scenarios where multiple training tasks are performed by the same training device, with each training task being a task of training a model. The "model" in this application can also be called a "machine learning model." For example, the machine learning model in this application can specifically be represented as a neural network, or as a non-neural network model, etc., which can be determined based on the actual application scenario.

For example, in the field of communications, models can be deployed in core network equipment, network equipment, and/or terminal equipment. The functions implemented by the models may include, but are not limited to: predicting the movement trajectory of terminal equipment, compressing the codebook in the Channel State Index Reference Signal (CSI-RS), decompressing the codebook in the CSI-RS, beamforming, load balancing of network equipment, or other functions. The specific devices in which models are deployed to implement which functions can be determined based on the actual application scenario.

For example, core network equipment refers to cloud servers that carry various network functions. For instance, the functions implemented through core network equipment include, but are not limited to: network data analytics function (NWDAF), access and mobility management function (AMF), session management function (SMF), authentication server function (AUSF), network exposure function (NEF), network repository function (NRF), network slice selection function (NSSF), unified data management (UDM), user plane function (UPF), etc., which will not be listed exhaustively here.

Network equipment can refer to devices that provide wireless access services in a wireless network. For example, a network device can be a device that connects terminal devices to a wireless network, and can also be called a base station; the aforementioned base station can be various forms of macro base stations, micro base stations, relay stations, or access points, etc. In wireless communication systems employing different wireless access technologies, the names of network devices with base station functions may differ. For example, a base station can be called an evolved Node B (eNB), a Node B (NB), the next-generation Node B (gNB) in a 5th generation (5G) communication system, a home base station (e.g., home evolved Node B, or home Node B, HNB), a base band unit (BBU), a wireless fidelity (Wi-Fi) access point (AP), a transmission reception point (TRP), or a radio network controller (RNC), etc. In another possible scenario, the terminal device can achieve wireless access through the cooperation of multiple network nodes, with each network node performing a portion of the base station's functions. For example, network nodes can be central units (CU), distributed units (DU), CU-control plane (CP), CU-user plane (UP), or radio units (RU), etc. CU and DU can be set up separately or included in the same network element, such as a baseband unit (BBU). RU can be included in radio frequency equipment or radio frequency units, such as remote radio units (RRU), active antenna units (AAU), or remote radio heads (RRH). In different systems, CU (or CU-CP and CU-UP), DU, or RU may have different names, but their meanings will be understood by those skilled in the art. For example, in an ORAN system, a CU can also be called an open CU (O-CU), a DU can also be called an open DU (O-DU), a CU-CP can also be called an open CU-CP (O-CU-CP), a CU-UP can also be called an open CU-UP (O-CU-UP), and an RU can also be called an open RU (O-RU). Any of the CU (or CU-CP, CU-UP), DU, and RU units can be implemented through software modules, hardware modules, or a combination of software and hardware modules. This application does not limit the specific device form of the network equipment.

A terminal device refers to a wireless terminal device capable of receiving scheduling and instruction information sent by network devices. Terminal devices can be handheld devices, vehicle-mounted devices, wearable devices, computing devices, or other processing devices with wireless communication capabilities, etc., without exhaustive list. Terminal devices can communicate with one or more core network devices or the Internet via a wireless access network (RAN). For example, terminal devices can be portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile devices that exchange voice and/or data with the RAN. For example, a terminal device can be a user agent, a cellular phone, a smartphone, a personal digital assistant (PDA), a tablet PC, a modem, a handset, a laptop computer, a personal communication service (PCS) phone, a remote station, an access point (AP), a remote terminal, an access terminal, a customer premises equipment (CPE), a terminal, a user equipment (UE), or a mobile terminal (MT), etc.

For example, terminal devices can also be wearable devices, such as glasses, gloves, watches, clothing, and shoes. Furthermore, terminal devices can also be drones, robots, terminal devices in device-to-device (D2D) communication, vehicle-to-everything (V2X) communication, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, terminal devices in self-driving, remote medical care, smart grids, smart cities, and smart homes.

In addition, the terminal device can also be a terminal device in a communication system after 5G (such as a sixth-generation (6G) communication system) or a terminal device in a future evolved public land mobile network (PLMN), etc. The embodiments of this application do not limit the device form of the terminal device.

For example, in the field of intelligent driving, functions implemented using models include: predicting the trajectory of obstacles around the vehicle, determining the vehicle's lateral and longitudinal directions, predicting the location areas the vehicle may reach in the future, performing trajectory planning, and other functions. Similarly, in the field of smart terminals, functions implemented using models include: image style transfer, image inpainting, predicting the category of objects in an image, speech recognition, text translation, and other functions. These are just a few examples of the various functions implemented using models in different application areas.

To facilitate understanding of this solution, please refer to Figure 2 first. Figure 2 is a schematic diagram of the training and application phases of the model. As shown in Figure 2, the entire system may include a request device 200, a training device 210, a database 220, an execution device 230, and a data storage system 240. The execution device 230 includes a computing module 231.

The requesting device 200 and the training device 210 can be communicatively connected. The requesting device 200 is a device that sends request information to the training device 210, which is used to request the execution of a training task for the model. For example, in the field of communication, the requesting device 200 can be a terminal device, a network device, or a core network device. Optionally, the requesting device 200 can be a terminal device responsible for the operation, management, and maintenance of the network. Technicians can directly interact with the aforementioned terminal device to achieve network management and maintenance.

Database 220 stores a training dataset. During the training phase of model 201, training device 210 acquires model 201 before the training operation is performed, and uses the training dataset to iteratively train model 201 until the preset convergence condition is met, thus obtaining model 201 after the training operation is performed. "Model 201 after the training operation is performed" can also be called model 201 after training. For example, "iteratively training model 201" can be understood as updating the weight parameters in model 201 multiple times.

The trained model 201 can be deployed to the computing module 231 of the execution device 230. The execution device 230 can access data, code, etc., from the data storage system 240, and can also store data, instructions, etc., in the data storage system 240. The data storage system 240 can be located within the execution device 230, or it can be an external storage device relative to the execution device 230. During the application phase of the model 201, the execution device 230 can input the data to be processed into the model 201 to obtain the prediction information generated by the model 201, thereby realizing the function of the model 201.

Please refer to Figure 2. The requesting device 200 and the training device 210 can be independent devices, and the training device 210 and the execution device 230 can be independent devices. For a more intuitive understanding of this scheme, please refer to Figure 3a. Figure 3a is a schematic diagram of an architecture of a training task processing system. In Figure 3a, the training device is taken as a core network device in the communication field, and the execution device is taken as a base station in the communication field. The requesting device can be a terminal device that communicates with the core network device. As shown in Figure 3a, the requesting device can send request information to the core network device. This request information is used to request the execution of the training task. When the core network device completes the training task based on the request information, it can obtain a model that has undergone training operations. The model that has undergone training operations is deployed to multiple base stations. It should be understood that the example in Figure 3a is only for the convenience of understanding this scheme and is not intended to limit this scheme.

It is worth noting that Figure 3a is merely a schematic diagram of one architecture for the training task processing system, and the positional relationships between the devices, components, modules, etc., shown in the figure do not constitute any limitation. For example, the training device 210 and the execution device 230 can also be integrated into the same device. For a more intuitive understanding of this solution, please refer to Figure 3b, which is a schematic diagram of another architecture for the training task processing system. In Figure 3b, the training device and the execution device are both the same core network device as an example. The requesting device can be a terminal device that communicates with the core network device. As shown in Figure 3b, the requesting device can send request information to the core network device. When the core network device completes the training task based on the request information, it can obtain the model that has undergone training operations. The model that has undergone training operations is deployed to the core network device. It should be understood that the example in Figure 3b is only for the convenience of understanding this solution and is not intended to limit this solution.

Please refer to Figure 3c. Figure 3c is a schematic diagram of another architecture of the training task processing system. In Figure 3b, the training device and the execution device are both the same base station as an example. The requesting device can be a terminal device that communicates with the core network device. The requesting device can send request information to the core network device, and the core network device will then forward the request information to each base station. After receiving the request information, if the base station completes the training task based on the request information, it can obtain the model that has been trained. The model that has been trained is deployed to the base station. It should be understood that the example in Figure 3c is only for the convenience of understanding this scheme and is not intended to limit this scheme.

It should be noted that in other embodiments, the requesting device can also be integrated with the training device in the same device. For example, in some application scenarios in the field of communication, when the core network device determines to perform a training operation on a certain model, it can also generate request information and then perform the training task based on the request information. That is, the requesting device and the training device can be integrated in the same core network device. For another example, in some application scenarios in the field of communication, when the base station determines to perform a training operation on a certain model, it can also generate request information and then perform the training task based on the request information. That is, the requesting device and the training device can be integrated in the same base station, etc. The specific product forms of the "requesting device", "training device" and "execution device" can be determined in combination with the actual application scenario.

Furthermore, Figures 3a, 3b, and 3c above are only examples of applications in the field of communication. The method provided by this application can also be applied to other fields. For example, in the field of intelligent driving, the requesting device and the training device are integrated into the same device, both of which are cloud servers, and the executing device is a vehicle, etc. The situations in each application field will not be listed here one by one.

Since a single training device can receive request information from multiple requesting devices, the training task currently being executed by the same training device may include multiple training tasks. However, the computer resources within the same training device are limited. Executing multiple training tasks may overload the training device, leading to process interruption or data loss. To address the aforementioned issues, this application discloses that after receiving a request information for executing a training task of a first model (hereinafter referred to as "first request information" for ease of distinction), the training device can determine whether to execute the first training task based on its priority. The first training task includes the training task of the first model. If the priority of the first training task is higher than the priority of the second training task, the first training task can be executed, and the execution of the second training task can be suspended. If the priority of the first training task is lower than or equal to the priority of the second training task, the execution of the first training task can be delayed or rejected. The second training task is the training task currently being executed by the training device. In other words, the training device can use the priority of training tasks to decide whether to pause, delay, or reject the execution of certain training tasks, which helps to avoid overloading the training device, thereby preventing process interruption or data loss and improving the stability of the training device during the execution of training tasks.

Based on the above description, the detailed implementation process of the training task processing method provided in this application will be described below. Specifically, please refer to Figure 4, which is a schematic diagram of a training task processing method provided in an embodiment of this application. As shown in Figure 4, the training task processing method provided in this application may include:

401. The training device receives a first request message from the first device, the first request message being used to request the execution of the training task of the first model.

The first request information can be a single message or information within a message, without restriction.

It should be noted that the training task of the first model and the first training model in this application can be interchanged without restriction.

Specifically, when the first device determines to train the first model, it can send the first request information to the training device. Correspondingly, the training device receives the first request information and, based on it, determines that it requests to execute the training task of the first model. Therefore, it can create a first training task, which includes the training task of the first model. It should be noted that the creation of the first training task by the training device does not mean that the training device immediately begins executing the first training task. The training device can determine whether to begin executing the first training task in subsequent steps; the specific determination process will be described in subsequent steps.

For example, the training device creating a first training task may include: the training device generating identification information for the first training task. The training device can then place the first training task into a waiting queue; for example, the training device can place the identification information of the first training task into the waiting queue, and optionally, the training device can place the identification of the first training task and first request information into the waiting queue.

The first device can be a requesting device. As described in Figures 2 to 3c above, the first device (i.e., the requesting device) can specifically be a terminal device, a network device, a core network device, or other types of devices, depending on the actual application scenario. The training device can be a network device, a core network device, a cloud server, or other types of devices.

The first training task is the training task of the first model, which can also be called the first machine learning model. The first model can be a neural network or a non-neural network model.

For example, the first request information may include information indicating the inference type of the first model; the training device can determine what kind of model the first training task is based on the inference type of the first model.

It should be noted that the inference type of the first model can also be referred to as the type of task performed by the first model, or other names, etc., which will not be exhaustively listed here. The inference type of the first model can indicate the function implemented by the first model. For example, if the inference type of the first model is to predict the movement trajectory of the terminal device, then the function implemented by the first model is to predict the movement trajectory of the terminal device; for another example, if the inference type of the first model is to compress the codebook in CSI-RS, then the function implemented by the first model is to compress the codebook in CSI-RS; for yet another example, if the inference type of the first model is image classification, then the function implemented by the first model is to predict the category of objects in the image, and so on. It should be understood that the examples here are only for the convenience of understanding the relationship between "the inference type of the first model" and "the function implemented by the first model".

For example, the first request information indicates the inference type of the first model by including at least one of the following: a description of the inference type of the first model, an identifier of the inference type of the first model, a first model that has not yet been trained, an identifier of the first model, or other information.

If the first request information includes a first model that has not yet been trained, the training device, upon acquiring the untrained first model, can determine what kind of information the first model outputs, that is, determine the inference type of the first model. For example, if the first model outputs the movement trajectory of the terminal device over a future period, the inference type of the first model is predicting the movement trajectory of the terminal device; as another example, if the first model outputs compressed information of the codebook in CSI-RS, the inference type of the first model is compressing the codebook in CSI-RS; as yet another example, if the first model outputs the category of an object in an image, the inference type of the first model is image classification, etc. These examples are only provided to facilitate understanding of how to determine the inference type of the first model.

If the first request information includes the identification information of the first model, assuming that the training device stores the identification information of multiple models and the one-to-one correspondence between multiple models, then after obtaining the identification information of the first model, the training device can determine the first model corresponding to the identification information of the first model according to the correspondence. After obtaining the first model, it can determine what kind of information the first model outputs, that is, it can determine the inference type of the first model.

Optionally, the first request information may further include: the accuracy requirement of the first training task, and/or, the batch training size requirement of the first training task. The "accuracy requirement of the first training task" can be used to indicate the precision to be used when performing the first training task. For example, the precision used when performing a training task can be single-precision floating-point (FP32), half-precision floating-point (FP16), double-precision floating-point (FP64), mixed precision, or other precision, etc., which are not limited here.

A training task can include multiple training processes for a model. Batch training refers to dividing the multiple training processes of a training task into multiple batches. In the training process of a single batch, n training samples are obtained from the training dataset, where n is an integer greater than or equal to 1. The batch size represents the number of training samples used in the training process of a single batch, which is also the number of n. For example, the batch size affects the GPU memory resources required by the model in the training process of a single batch.

The training device can determine the required precision when performing the first training task based on the precision requirements of the first training task, and/or, the training device can determine the number of training samples used in the training process of a single batch based on the batch training scale requirements of the first training task.

Optionally, the first request information may also include the priority of the training task of the first model, which can also be understood as the first request information including the priority of the first training task; for example, the priority of the first training task may be expressed as a number, such as 1, 2, 3, 4, 5 or other numbers, with the larger the number, the higher the priority; or, the priority of the first training task may be expressed as text, such as first level, second level, third level, etc.

Optionally, the first device determines the priority of the first training task before sending the first request information to the training device. Specifically, the determination of this priority can refer to the following two cases.

Scenario 1: The first device determines the priority of the first training task based on the first factor.

The first factor may include at least one of the following: the inference type of the first model, the resource requirements of the first training task, or the immediacy requirements of the first model.

The impact of the inference type of the first model on the priority of the first training task can be described in the following description. For example, the less resources the first training task requires, the higher its priority can be; the more resources the first training task requires, the lower its priority can be. The higher the immediacy requirement of the first model, the higher its priority can be; the lower the immediacy requirement of the first model, the lower its priority can be.

Optionally, the resource requirements for the first training task may include storage resources needed to execute the first training task, which may include video memory and/or system memory resources, without limitation. Alternatively, the resource requirements for the first training task may also include processor resources or other resources needed to execute the first training task.

For example, the immediacy requirement of the first model can be understood as the time requirement for the first model after completing the first training task, or as the time within which the first model needs to be deployed. The shorter the deployment time, the higher the immediacy requirement of the first model. For instance, if Model 1 is used to compress or decompress the codebook in CSI-RS, then Model 1 has a high immediacy requirement; as another example, if Model 2 is used to implement load balancing of base stations, since Model 2 is updated periodically, then Model 2 has a lower immediacy requirement, etc. The examples here are only for the convenience of understanding this scheme.

In one alternative implementation, the first factor includes the inference type of the first model; assuming the first device stores a correspondence between inference types and priorities, the first device can determine the priority corresponding to the inference type of the first model, i.e., the priority of the first training task, based on the correspondence.

In another alternative implementation, it is assumed that the first factor includes not only the inference type of the first model, but also: the resource requirements of the first training task, and/or the immediacy requirements of the first model. The first device acquires a first score corresponding to the inference type of the first model, and acquires a second score corresponding to the resource requirements of the first training task, and/or acquires a third score corresponding to the immediacy requirements of the first model. The first device can perform a weighted summation of all the acquired scores to obtain the total score of the first training task, thereby determining the priority of the first training task. All acquired scores may include the first score, as well as the second and/or third scores. The higher the total score of the first training task, the higher its priority; the lower the total score of the first training task, the lower its priority.

For example, assuming the first device stores a correspondence between inference types and scores, the first device can determine the first score corresponding to the inference type of the first model based on the correspondence.

The more resources required for the first training task, the lower the second score; conversely, the fewer resources required for the first training task, the higher the second score. In one optional implementation, the first device can store a correspondence between required resources and scores, and then the first device can determine the second score corresponding to the required resources of the first training task based on this correspondence. In another optional implementation, the first device inputs the required resources of the first training task into a first preset algorithm to obtain the second score corresponding to the required resources of the first training task, whereby the first preset algorithm indicates the mapping relationship between required resources and scores.

The higher the immediacy requirement of the first model, the higher the third score of the first training task; the lower the immediacy requirement of the first model, the lower the third score of the first training task. In one optional implementation, the first device can store the correspondence between immediacy requirements and scores, and then the first device can determine the third score corresponding to the immediacy requirement of the first model based on the correspondence. In another optional implementation, the first device inputs the immediacy requirement of the first model into a second preset algorithm to obtain the third score corresponding to the immediacy requirement of the first model. The second preset algorithm indicates the mapping relationship between immediacy requirements and scores.

It should be noted that the first factor may include more or fewer elements. The examples above are only for the convenience of understanding this solution and are not intended to limit this solution.

Scenario 2: The first device determines the priority of the first training task based on the first operation.

Specifically, the first device can receive a first operation input by the user, and then determine the priority of the first training task based on the first operation.

The first operation can be a selection operation for the priority of the first training task, or it can be the priority of the first training task input through a text box, or it can be the priority of the first training task input by voice, etc. The specific operation can be determined according to the actual product form.

It should be noted that the first request information may also include other information, such as the training data set of the first model, or the performance requirements of the first model, etc., which will not be listed here.

For example, after receiving the first request information, the training device can create a first training task based on the first request information, and then determine whether to execute the first training task. Optionally, when the training device is in a first scenario, the training device can determine whether to execute the first training task based on the priority of the first training task; when the training device is not in a first scenario, the training device can start executing the first training task; for example, the first scenario includes: the resource requirement of the first training task is greater than the idle resources of the training device, and/or, the number of training tasks currently being executed by the training device is greater than a preset threshold, which will be described below.

Scenario 1: The first scenario involves the resource requirements of the first training task exceeding the available resources of the training equipment.

Specifically, the training device can determine whether the resource requirement of the first training task is greater than the available resources of the training device. If the resource requirement of the first training task is greater than the available resources of the training device, the training device can determine the second training task from all currently executing training tasks, and then determine whether the priority of the first training task is higher than the priority of the second training task. If the priority of the first training task is higher than the priority of the second training task, then proceed to step 402; if the priority of the first training task is lower than or equal to the priority of the second training task, then proceed to step 403. If the resource requirement of the first training task is less than or equal to the available resources of the training device, then the first training task can be started.

For example, the training device can determine the resource requirements of the first training task in the following three ways.

In one alternative implementation, the first request information may further include information indicating the required resources for the first training task, so that the training device can determine the required resources for the first training task based on the first request information.

In another alternative implementation, the training device can determine the resource requirements of the first training task based on the inference type of the first model, the accuracy requirements of the first training task, and the batch training scale requirements of the first training task. Specifically, the inference type of the first model indicates which type of first model to train; the more parameters the first model has, the more resources the first training task requires; the higher the accuracy requirements of the first training task, the more resources it requires; and the larger the batch training scale requirements of the first training task, the more GPU memory resources are required during the training of a single batch.

In another alternative implementation, the training device can store a correspondence between inference types and required resources, and then the training device can determine the required resources of the first training task corresponding to the inference type of the first model based on the correspondence.

Optionally, the idle resources of the training device may include idle storage resources in the training device; optionally, it may also include idle processor resources in the training device, or it may include other types of idle resources, etc., which can be determined according to the actual application scenario. For example, the training device's determination of whether the resources required by the first training task are greater than the idle resources of the training device may include: the training device determining whether the storage resources required by the first training task are greater than the idle storage resources in the training device, for example, the aforementioned storage resources may include video memory resources and/or system memory resources. Optionally, it may also include: the training device determining whether the processor resources required by the first training task are greater than the idle processor resources in the training device, etc., which can be set according to the actual application scenario.

In one scenario, the second training task is the lowest priority training task among all training tasks currently being executed by the training device. Therefore, the higher priority of the first training task than the second training task can also be understood as the higher priority of the first training task than the lowest priority training task among all currently executed training tasks.

In another scenario, the second training task comprises S training tasks currently being executed by the training device, where S is an integer greater than or equal to 1. That is, the second training task includes one or more training tasks currently being executed by the training device. In this case, the higher priority of the first training task than the second training task indicates that the first training task has a higher priority than any of the S training tasks. Conversely, the lower or equal priority of the first training task than the second training task indicates that the first training task has a lower or equal priority than any of the S training tasks.

Regarding the specific implementation method for determining the second training task by the training device, in one implementation, after determining that the resource requirement of the first training task is greater than the idle resources of the training device, the training device can determine a first difference between the resource requirement of the first training task and the idle resources of the training device. Based on the first difference, the training device determines S training tasks from all currently executed training tasks, which are the second training tasks. The S training tasks can be the S lowest priority training tasks among all currently executed training tasks, and the sum of the resources occupied by the S training tasks is greater than or equal to the first difference, that is, the sum of the idle resources of the training device and the resources occupied by all the training tasks among the S training tasks is greater than or equal to the resource requirement of the first training task.

Optionally, the resources occupied by the training task may include the storage resources occupied by the training task, and optionally, the processor resources occupied by the training task, or other types of resources, which can be determined according to the actual application scenario.

In another implementation, S can be a preset value, and the training device can determine the S lowest priority training tasks from all currently executed training tasks. Alternatively, S can be a preset value, and the training device can determine the S resource-intensive training tasks from all currently executed training tasks. Another implementation can also involve the training device randomly determining S training tasks from all currently executed training tasks, etc. The specific implementation method for the training device to determine the second training task can be determined based on the actual application scenario.

Scenario 2: The first scenario includes the number of training tasks currently being executed by the training device being greater than a preset threshold.

Specifically, the training device can determine whether the number of training tasks currently being executed by the training device is greater than a preset threshold. If the number of training tasks currently being executed by the training device is greater than the preset threshold, the training device determines whether the priority of the first training task is higher than the priority of the second training task. If the priority of the first training task is higher than the priority of the second training task, then proceed to step 402; if the priority of the first training task is lower than or equal to the priority of the second training task, then proceed to step 403. If the number of training tasks currently being executed by the training device is less than or equal to the preset threshold, then the first training task can be started. The preset threshold can be an integer greater than or equal to 1, for example, the preset threshold can be 4, 5, 6 or other values, which can be determined according to the actual application scenario.

Scenario 3: The first scenario includes situations where the resource requirements of the first training task exceed the available resources of the training device, and the number of training tasks currently being executed by the training device exceeds a preset threshold.

Specifically, the training device can determine whether the resource requirement of the first training task is greater than the idle resources of the training device, and whether the number of training tasks currently being executed by the training device is greater than a preset threshold. If the resource requirement of the first training task is greater than the idle resources of the training device, or the number of training tasks currently being executed by the training device is greater than the preset threshold, the training device will determine whether the priority of the first training task is higher than the priority of the second training task. If the resource requirement of the first training task is less than or equal to the idle resources of the training device, and the number of training tasks currently being executed by the training device is less than or equal to the preset threshold, then the first training task can be started.

In this embodiment, the training task processing method provided in this application is started when the required resources of the first training task are greater than the idle resources of the training device, or when the number of training tasks currently being executed by the training device is greater than a preset threshold. That is, in the aforementioned scenarios, the training device will pause, delay, or refuse to execute certain training tasks, providing two application scenarios for this application and improving the implementation flexibility of this solution. In addition, in the aforementioned two application scenarios, the load limit of the training device is almost reached, which is conducive to avoiding overload of the training device while making the maximum use of the resources in the training device.

Regarding the specific implementation methods for determining the priority of the first training task by the training device, in one implementation, the training device can obtain the priority of the first training task from the first request information. In another implementation, the training device can determine the priority of the first training task based on a first factor. The specific implementation methods of the aforementioned steps can be found in the above description of the first device determining the priority of the first training task based on the first factor, and will not be repeated here.

Optionally, the training device can also adjust the priority of the first training task. The priority in the first request information and the priority of the first training task determined by the training device based on the first factor can both be understood as the initial priority of the first training task. The training device can increase the initial priority of the first training task according to the first duration to obtain the current priority of the first training task. The first duration is the duration between the time when the training device receives the first request information and the current time. For example, the longer the first duration, the higher the current priority of the first training task, and the shorter the first duration, the lower the current priority of the first training task.

The specific implementation method for determining the priority of each training task included in the second training task by the training device is similar to the specific implementation method for determining the priority of the first training task by the training device, and will not be repeated here.

In this embodiment, if the first request information includes the priority of the first training task, the training device can directly obtain the priority of the first training task from the first request information. This allows for a faster comparison between the priority of the first training task and the priorities of other training tasks. Since the decision on whether to execute the first training task can only be made after determining the comparison result between the priority of the first training task and the priority of the second training task, this also facilitates a faster determination of whether to execute the first training task. Furthermore, obtaining the priority of the first training task directly from the first request information reduces the resources consumed by the training device in determining the priority of the first training task. This allows the training device to allocate more resources to executing the training task, which is beneficial for obtaining a model that has completed the training task as quickly as possible.

Optionally, each training task currently performed by the training device is a task of training a model in the communications domain. Examples of models in the communications domain can be found in the above description and will not be repeated here. This provides a specific application domain for the method in this application, improving the degree of integration between this application and specific application domains.

402. If the priority of the first training task is higher than that of the second training task, the training device executes the training task of the first model and suspends the execution of the second training task.

The second training task can be one or more training tasks currently being executed by the training device, or the second training task can be the training task with the lowest priority among the training tasks currently being executed by the training device.

In this embodiment of the application, the execution of the first training task by the training device means that the first training task has been added to the training tasks that the training device is currently executing; the suspension of the execution of the second training task by the training device can be referred to as the suspension of the second training task by the training device, so that the second training task is no longer included in the training tasks currently being executed by the training device.

For example, the training device may also maintain a waiting queue. When the training device pauses the execution of the second training task, the second request information corresponding to the second training task can be put into the waiting queue.

Optionally, the training device can also store information about the first training task. This information may include the identification information of the first training task, and optionally, it may also include the priority of the first training task. If the training device can adjust the priority of the first training task, then the information about the first training task includes its current priority. The information about the first training task may also include other types of information, such as the inference type of the first model corresponding to the first training task. The specific information that the first training task's information may include can be determined based on the actual application scenario.

403. If the priority of the first training task is lower than or equal to the priority of the second training task, the training device sends a response message to the first device.

The response information can be used to notify the first device to delay or refuse to execute the training task of the first model, or the response information can include information about the second training task.

Wherein, the second training task is one or more training tasks currently being executed by the training device, or the second training task is the training task with the lowest priority among the training tasks currently being executed by the training device.

In this embodiment, if the priority of the first training task is lower than or equal to the priority of the second training task, it will not affect the training device's execution of the second training task, and the training device will send response information to the first device, meaning the training device will not directly execute the first training task. Correspondingly, the first device can receive response information from the training device, which indicates that the training device has not directly executed the first training task. For example, the response information may include an indication value corresponding to delay or rejection. For instance, the indication value corresponding to delay or rejection could be 1111, representing delay or rejection of executing the first training task; or, the response information may include delay or rejection, thereby informing the first device to delay or reject the execution of the first training task, etc. The specific details can be determined based on the actual application scenario.

For example, the information of the second training task may include the information of each of the S training tasks, and the information of each of the S training tasks may include the identification information of each of the S training tasks; optionally, it may also include the priority of each of the S training tasks, that is, the information of the second training task may include the priority of the second training task; optionally, it may also include the inference type corresponding to each of the S training tasks, etc., all of which can be determined in combination with the actual application scenario.

Optionally, the response information may include information about all training tasks currently being executed by the training device that have a higher priority than the first training task, wherein all training tasks currently being executed by the training device that have a higher priority than the first training task include the second training task.

In this embodiment, the training device can use the priority of training tasks to decide to pause, delay, or refuse to execute certain training tasks, which helps to avoid overloading of the training device, thereby helping to avoid interruption of the process used to execute training tasks or to avoid data loss, and improving the stability of the training device in the process of executing training tasks.

Optionally, if the priority of the first training task is lower than or equal to the priority of the second training task, the training device will delay or refuse to execute the first training task. The response information sent by the training device to the first device includes the priority of the second training task. Thus, the first device can not only know that the training device has decided to delay or refuse to execute the first training task, but also know that the training device is executing the second training task with a higher priority. In other words, the first device can know the current status of the training device, which makes it easier for the first device to determine a more suitable processing method by combining the current status of the training device and the needs of the first model. This also helps to make the use of resources in the training device better meet the needs of the current scenario.

Optionally, based on the embodiment corresponding to Figure 4 above, please refer to Figure 5. Figure 5 is another schematic diagram of the training task processing method provided by the embodiments of this application. As shown in Figure 5, the training task processing method provided by this application may include:

501. The second device sends a second request message to the training device.

In this embodiment of the application, the second device and the first device are different requesting devices. When the second device determines to train the second model, it can send a second request message to the training device.

The second request information is used to request the execution of the training task for the second model. The training task for the second model can be called the second training task, and the two can be interchanged.

Correspondingly, the training device can receive the second request information from the second device, and then determine the second training task based on the second request information; the meaning of the terms in step 501 and the specific implementation of the steps can be referred to the description of step 401 in the embodiment corresponding to Figure 4 above, the difference being that "first device" is replaced with "second device", "first model" is replaced with "second model", and "first request information" is replaced with "second request information", which will not be repeated here.

Referring to the description in the embodiment corresponding to Figure 4 above, the second training task may include one or more training tasks. In one case, the second training task includes one training task, in which case the second request information may include one request message, and the second device may include one device. In another case, the second training task includes at least two training tasks, in which case the second request information may also include at least two request messages corresponding one-to-one with the at least two training tasks. Each of the at least two request messages is used to request the execution of one of the training tasks in the second training task, and the second device may include all devices that sent the aforementioned at least two request messages.

After receiving the second request information, the training device can determine whether to start executing the second training task, and then start executing the second training task. For example, after receiving each request information included in the second request information, the training device can determine whether to start executing the training task corresponding to each request information, and then start executing the training task corresponding to each request information. The process by which the training device determines whether to start executing the training task can be referred to the description of the training device determining whether to start executing the first training task in the embodiment corresponding to Figure 4 above, and will not be repeated here.

502. The training device receives the first request information from the first device.

The first request information can be used to request the execution of the training task of the first model.

503. If the priority of the training task of the first model is higher than the priority of the training task of the second model, the training device shall execute the training task of the first model and suspend the execution of the second training task.

The training task of the first model can be referred to as the first training task.

In this embodiment, the meanings of the terms in steps 502 and 503 and the specific implementation of the steps can be found in the description of steps 401 and 402 in the embodiment corresponding to Figure 4 above, and will not be repeated here.

504. The training equipment sends a notification message to the second equipment corresponding to the second training task.

The notification information can be used to notify the second device to suspend the training task of the second model, or it can be used to notify the second device of the idle resources of the training device.

In this embodiment, step 504 is an optional step. After pausing the execution of the second training task, the training device can also send a notification message to the second device corresponding to the second training task, and the second device can receive the notification message from the training device.

In one scenario, the notification information is used to instruct the second device to suspend the execution of the second training task, that is, in step 504, the training device notifies the second device corresponding to the second training task to suspend the execution of the second training task.

Optionally, the training device notifying the second device to suspend the execution of the second training task may include: the training device sending a reason value to the second device, that is, the notification information includes a reason value, which indicates the reason for suspending the execution of the second training task. Correspondingly, the second device may receive the reason value from the training device.

For example, since the second device includes one or more devices, the sending of a cause value by the training device to the second device can be understood as the training device sending the cause value to each device included in the second device. The receiving of a cause value from the training device by the second device can be understood as each device included in the second device receiving a cause value from the training device.

Optionally, this reason includes the second training task being the lowest priority training task on the training device. In this embodiment, the second training task that the training device suspends execution from is the lowest priority training task on the training device, meaning that the resources in the training device can be allocated to higher priority training tasks as much as possible, so that the resources in the training device can be used more efficiently.

In one scenario, the aforementioned cause value includes the priority of each of at least one third training task currently being performed by the training device, where the priority of each third training task is higher than the priority of the second training task.

For example, at least one third training task may include all training tasks currently being executed by the training device. It should be noted that since the training device may pause the execution of old training tasks or begin executing new training tasks, the training tasks currently being executed by the training device in this application can vary. When the training device sends a cause value to the second device, the training device has paused the execution of the second training task and begun executing the first training task; therefore, the training tasks currently being executed by the training device may include the first training task, that is, at least one third training task may include the first training task. Alternatively, at least one third training task may include only the first training task. Or, at least one third training task may include a preset number of training tasks selected from all training tasks currently being executed by the training device, etc., which can be determined based on the actual application scenario.

Optionally, the above-mentioned cause value may also include the identification information of each third training task; alternatively, the above-mentioned cause value may also include the inference type of each third training task, etc., which can be determined in combination with the actual application scenario.

In this embodiment, the reason value sent by the training device to the second device includes the priority of each of the at least one third training tasks currently being executed by the training device. The priority of each training task is higher than that of the second training task. After receiving the reason value, the second device can not only know that the execution of the second training task is suspended due to the low priority, but also know the priority of the third training task that is occupying the resources of the training device. That is, the second device can know a more detailed resource usage of the training device, which makes it easier for the second device to determine a more suitable processing method by combining the resource usage of the training device and the needs of the second model. It also helps to make the resource usage of the training device better meet the needs of the current scenario.

In another scenario, the aforementioned reason value can also be represented by a letter, such as "LL", which means that the execution of the second training task is paused because the second training task is the lowest priority training task on the training device.

In another scenario, the aforementioned reason value can also be expressed as a number. For example, the reason value can be expressed as "000000", which means that the second training task is paused because it is the lowest priority training task on the training device. Alternatively, the aforementioned reason value can also be expressed in other forms, which can be set according to the actual application scenario.

In this embodiment, after determining that the second training task has been paused, the training device sends a reason value to the second device. This reason value indicates the reason for pausing the second training task. This allows the second device to promptly know that the second training task has been paused and, more importantly, the reason for the pause. This facilitates the second device in determining the appropriate handling method for the second training task after confirming its pause, and also allows it to determine a more suitable handling method based on the reason for the pause.

Specifically, the notification information may include an indication value corresponding to the pause. For example, the indication value corresponding to the pause may be 2222, representing the pause of the second training task. Alternatively, the notification information may include "pause execution," thereby informing the second device to pause the execution of the second training task. The specific details can be determined based on the actual application scenario.

In this embodiment, after the training device suspends the execution of the second training task, it will promptly notify the second device corresponding to the second training task that the training device has suspended the execution of the second training task. This allows the second device to quickly know that the training device has suspended the execution of the second training task, making the execution process of the second training task more controllable.

In another scenario, the notification information is used to inform the second device of the idle resources of the training device. That is, in step 504, the training device notifies the second device of the idle resources of the training device. Correspondingly, the second device can determine that the training device has paused the execution of the second training task, and the second device can know how many idle resources are in the training device. The concept of the idle resources of the training device can be referred to the description in Figure 4 above, and will not be repeated here.

For example, the notification information may include the number of idle storage resources in the training device; optionally, the notification information may also include the number of idle processor resources in the training device, or the notification information may also include the number of other types of idle resources in the training device, which can be determined in combination with the actual application scenario.

In this embodiment, after the training device suspends the execution of the second training task, it can notify the second device of the idle resources of the training device. Thus, the second device can not only know that the training device has suspended the execution of the second training task, but also determine how to handle the situation of the training device suspending the execution of the second training task based on the idle resources of the training device, which is conducive to obtaining a processing method that is more suitable for the current state of the training device.

505. The method for the second device to send a second training task to the training device.

The second training task can be handled by terminating, waiting, or adjusting the resources used.

In this embodiment, step 505 is an optional step. After determining that the training device has paused the execution of the second training task, the second device can determine the processing method for the second training task and then send the processing method for the second training task to the training device.

For example, the processing method of the second device sending the second training task to the training device can be understood as the processing method of each device included in the second device sending any one of the training tasks in the second training task to the training device. The processing method of the second training task can be understood as the processing method of any one of the training tasks included in the second training task.

For example, when the processing method of the second training task is termination, the second device can send an indication value corresponding to termination to the training device. For example, the indication value corresponding to termination can be 000000, DDDDD, or other types of indication values. Alternatively, the second device can send feedback information to the training device to indicate "cancel the execution of this training task". The specific implementation method can be determined according to the actual application scenario.

When the processing mode of the second training task is waiting, the second device can send an indication value corresponding to waiting to the training device; or, the second device can send feedback information to the training device to indicate "waiting", etc., without limitation.

When the processing method for the second training task involves adjusting resource usage, in one scenario, this resource adjustment can further include: reducing the number of parameters in the model corresponding to the second training task, reducing the accuracy when executing the second training task, or reducing the batch training scale when executing the second training task. For example, reducing the number of parameters in the model corresponding to the second training task can be achieved by pruning the second model. The specific pruning algorithm can be flexibly determined based on the actual application scenario, thereby reducing not only storage resources but also processor resources. Reducing the accuracy when executing the second training task can be achieved by replacing the high-precision data format with a low-precision data format, such as reducing from FP32 to FP16, or from FP64 to mixed precision. This example is only for ease of understanding the solution, thus reducing not only storage resources but also processor resources. Reducing the batch training scale when executing the second training task can be achieved by reducing the number of training samples used in a single batch of training, thereby reducing the storage resources required for a single batch of training.

For example, the second device can send first feedback information to the training device. The first feedback information instructs the training device to process the second training task in a first manner. The first manner is to reduce the number of parameters of the model corresponding to the second training task, reduce the accuracy when performing the second training task, or reduce the batch training scale when performing the second training task. That is, the second device not only instructs the processing method of the second training task to adjust the occupied resources, but also instructs what method to use to adjust the occupied resources when performing the second training task.

In this embodiment, after learning that the second training task is suspended, the second device can send the processing method of the second training task to the training device. Since the second device has a clearer understanding of the requirements of the second model, it can determine whether to terminate, wait, or adjust the resource usage of the second training task based on the requirements of the second model. This is beneficial to improving the adaptability between the processing method of the second training task and the specific application scenario.

When the processing method of the second training task is to adjust the occupied resources, the second device can further indicate how to adjust the resources occupied when executing the second training task. This is beneficial to make the final second model more compatible with the application scenario of the second model, that is, to obtain a more satisfactory second model under the premise of limited resources.

In another scenario, the second device can send an indication value to the training device corresponding to adjusting the occupied resources; or, the second device can send feedback information to the training device to indicate "adjusting the occupied resources," so that the training device can know that the processing method of the second training task is to adjust the occupied resources, and then the training device can determine how to adjust the occupied resources when performing the second training task.

Optionally, the factors determining the processing method of the second training task include: the immediacy requirements of the second model, the accuracy requirements of the second model, and/or, the degree of reduction in the accuracy of the second model caused by adjusting the resources used.

Optionally, for a second model with high immediacy and accuracy requirements, the training device can terminate the process, allowing it to promptly request other devices to execute the second training task. For a second model with low immediacy requirements, the training device can wait. For a second model with high immediacy and low accuracy requirements, the training device can adjust resource usage.

The immediacy requirement of the second model can be referred to the description of the "immediacy requirement of the first model" in the corresponding embodiment of Figure 4 above, except that "first model" is replaced with "second model", which will not be repeated here. The "accuracy requirement of the second model" can be understood as the accuracy requirement of the second model after performing the second training task, or it can also be understood as the performance requirement of the second model after performing the second training task, etc. "Model accuracy" or "model performance" can be understood as the accuracy of the prediction information generated by the model.

The degree of reduction in the accuracy of the second model caused by adjusting the occupied resources can be determined by at least one of the following parameters: the first accuracy range of the second model obtained after adjusting the resources occupied when performing the second training task, and/or the second accuracy range, where the second accuracy range represents the decrease in the accuracy of the final second model caused by adjusting the occupied resources, or may include other parameters, etc., which are not exhaustively listed in this application embodiment.

For example, the immediacy requirement of the second model is that the second model, after completing the second training task, needs to be online within a second duration. The accuracy requirement of the second model is that the accuracy of the second model after completing the second training task needs to be within a first preset accuracy range. The second device can determine whether the second duration is greater than the preset duration. If the second duration is greater than the preset duration, the second device can determine that the processing method for the second training task is to wait. If the second duration is less than the preset duration, in one implementation, the second device can determine whether the first accuracy range is within the first preset accuracy range. If the first accuracy range is within the first preset accuracy range, the second device can determine that the processing method for the second training task is to adjust the occupied resources. If the first accuracy range is not within the first preset accuracy range, that is, if there is an accuracy outside the first preset accuracy range within the first accuracy range, the second device can determine that the processing method for the second training task is to terminate, and then the second device can request other devices to execute the second training task.

In another implementation, the second device can determine whether the second precision range is within the second preset precision range. If the second precision range is within the second preset precision range, the second device can determine that the processing method for the second training task is to adjust the occupied resources. If the second precision range is not within the second preset precision range, that is, if there is a precision outside the second preset precision range, the second device can determine that the processing method for the second training task is to terminate, and then the second device can request other devices to execute the second training task.

In this embodiment, the second device can determine the processing method of the second training task based on the immediacy requirements of the second model, the accuracy requirements of the second model, and/or the degree of reduction in the accuracy of the second model caused by the use of resources. That is, the processing method of the second training task is determined from the two dimensions of time requirements and accuracy requirements, which is beneficial to obtain a better processing method under the premise of limited resources.

506. The training equipment processes data based on the processing method of the second training task.

In this embodiment, step 506 is an optional step. Step 506 may include: when the processing mode of the second training task is termination, the training device deletes the second request information corresponding to the second training task; for example, the training device may delete the second request information corresponding to the second training task from the waiting queue. Alternatively,

When the processing mode of the second training task is waiting, the training device waits until the idle resources are greater than or equal to the resources required by the second training task before continuing to execute the second training task. Optionally, referring to the description in the embodiment corresponding to Figure 4 above, after the training device pauses the execution of the second training task, the second request information corresponding to the second training task can be placed in a waiting queue. When the head of the waiting queue is the second request information, and the idle resources of the training device are greater than or equal to the resources required by the second training task, the second training task can continue to be executed. Alternatively,

Step 506 may include: when the processing method of the second training task is to adjust the occupied resources, regardless of whether the second device or the training device determines the method to adjust the occupied resources when executing the second training task, the training device can know the method to adjust the occupied resources when executing the second training task. In this way, the training device can reduce the number of parameters of the model corresponding to the second training task, or reduce the accuracy when executing the second training task, or reduce the batch training scale when executing the second training task. The specific implementation methods of the above three methods can be referred to the description in step 505 above, which will not be repeated here.

In this embodiment, different processing methods for the second training task fed back by the second device are provided, and specific processing schemes for the training device are provided. Thus, no matter what processing method the second device feeds back, the training device has a corresponding processing scheme, which helps to improve the smoothness and stability of the execution process of this application. Furthermore, when the processing method of the second training task is termination, the second request information is deleted, thereby releasing all resources related to the second request information in the training device in a timely manner, which helps to avoid wasting resources in the training device.

Optionally, after adjusting the resources occupied by the second training task, the training device can also adjust the priority of the second training task, for example, by increasing the priority of the second training task.

Optionally, based on the embodiments corresponding to Figures 4 and 5 above, please refer to Figure 6. Figure 6 is another schematic diagram of the training task processing method provided by the embodiments of this application. As shown in Figure 6, the training task processing method provided by this application may include:

601. The second device sends a second request message to the training device.

In this embodiment, the meaning of the terms in step 601 and the specific implementation of the steps can be found in the description of step 501 in the embodiment corresponding to Figure 5 above, and will not be repeated here.

602. The training device receives a first request message from the first device, the first request message being used to request the execution of the training task of the first model.

603. If the priority of the training task of the first model is lower than or equal to the priority of the second training task, the training device sends a response message to the first device.

The training task of the first model can be replaced by a first training task. The response information can be used to notify the first device to delay or refuse to execute the training task of the first model, or the response information may include information about the second training task.

In this embodiment, the meanings of the terms in steps 602 and 603 and the specific implementation methods of the steps can be found in the description of steps 401 and 403 in the embodiment corresponding to Figure 4 above, and will not be repeated here.

604. The method for the first device to send the training task of the first model to the training device.

The training task of the first model can be handled by terminating, waiting, or adjusting the resources used.

605. The training equipment processes data based on the processing method of the first training task.

In this embodiment, the meanings of the terms in steps 604 and 605 and the specific implementation of the steps can be found in the descriptions of steps 505 and 506 in the embodiment corresponding to Figure 5 above. The difference is that "second device" is replaced with "first device", "second training task" is replaced with "first training task", "second model" is replaced with "first model", and "second request information" is replaced with "first request information". These will not be elaborated on here.

Based on the embodiments corresponding to Figures 4 to 6, in order to better implement the above-described solutions of the embodiments of this application, related devices for implementing the above-described solutions are also provided below. Specifically, referring to Figure 7, Figure 7 is a schematic diagram of a training task processing device provided in an embodiment of this application. The training task processing device 700 can be applied in a training device. The training task processing device 700 includes: a receiving module 701, used to receive request information from a first device, the request information being used to request the execution of a training task of a first model; an execution module 702, used to execute the training task of the first model and suspend the execution of the second training task if the priority of the training task of the first model is higher than the priority of the second training task; and/or, a sending module 703, used to send response information to the first device if the priority of the training task of the first model is lower than or equal to the priority of the second training task, the response information being used to notify the first device to delay or refuse the execution of the training task of the first model, or the response information including information about the second training task; wherein, the second training task is one or more training tasks currently being executed by the training device, or, the second training task is the training task with the lowest priority among the training tasks currently being executed by the training device.

Optionally, the requested information may include the priority of the training task for the first model.

Optionally, the information for the second training task includes the priority of the second training task.

Optionally, the training task processing device 700 further includes: a notification module 704, used to notify the second device corresponding to the second training task to suspend the execution of the second training task; or, the notification module 704, used to notify the second device corresponding to the second training task of the idle resources of the training device.

Optionally, the notification module 704 is specifically used to send a reason value to the second device, the reason value being used to indicate the reason for suspending the execution of the second training task.

Optionally, the reason may include the second training task being the lowest priority training task on the training device.

Optionally, the reason value includes the priority of each of the at least one third training task currently being performed by the training device, wherein the priority of each third training task is higher than the priority of the second training task.

Optionally, the receiving module 701 is also used to receive the processing mode of the second training task from the second device, wherein the processing mode of the second training task is: termination, waiting, or adjusting the occupied resources.

Optionally, the training task processing device 700 further includes: a deletion module 705, used to delete the request information corresponding to the second training task when the processing mode of the second training task is termination; or, an execution module 702, used to wait until the idle resources are greater than or equal to the resources required by the second training task when the processing mode of the second training task is waiting, and then continue to execute the second training task; or, an adjustment module 706, used to reduce the number of parameters of the model corresponding to the second training task, or reduce the accuracy when executing the second training task, or reduce the batch training scale when executing the second training task when the processing mode of the second training task is adjusting the occupied resources.

Optionally, the method can be applied to the following scenarios: the resource requirements of the first training task are greater than the available resources of the training device; or, the number of training tasks currently being executed by the training device is greater than a preset threshold.

Optionally, adjusting the resources used may include: reducing the number of parameters of the model corresponding to the second training task, reducing the accuracy when performing the second training task, or reducing the batch training size when performing the second training task.

Optionally, each training task currently being performed by the training device is a task of training a model in the field of communications.

It should be noted that the information interaction and execution process between the modules/units in the training task processing device 700 are based on the same concept as the various method embodiments corresponding to Figures 4 to 6 in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

Please refer to Figure 8. Figure 8 is a schematic diagram of another structure of the training task processing device provided in this application embodiment. The training task processing device 800 can be applied in a first device. The training task processing device 800 includes: a sending module 801, used to send request information to the training device, the request information being used to request the execution of a training task of a first model; and a receiving module 802, used to receive response information from the training device, the response information being used to notify the first device to delay or refuse to execute the training task of the first model, or the response information including information of a second training task; wherein, the priority of the training task of the first model is lower than or equal to the priority of the second training task, the second training task is one or more training tasks currently being executed by the training device, or the second training task is the training task with the lowest priority among the training tasks currently being executed by the training device.

Optionally, the sending module 801 is also used to send the processing method of the training task of the first model to the training device. The processing method of the training task of the first model is: termination, waiting, or adjusting the occupied resources.

Optionally, adjusting the resources used may include: reducing the number of parameters of the model corresponding to the training task of the first model, reducing the accuracy when performing the training task of the first model, or reducing the batch training scale when performing the training task of the first model.

Optionally, the determining factors for how the training task of the first model is handled include: the immediacy requirements of the first model, the accuracy requirements of the first model, and/or, the degree of reduction in the accuracy of the first model caused by adjusting the resources used.

It should be noted that the information interaction and execution process between the modules/units in the training task processing device 800 are based on the same concept as the various method embodiments corresponding to Figures 4 to 6 in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

Based on the embodiments corresponding to Figures 4 to 6, in order to better implement the above-described solutions of this application, related devices for implementing the above-described solutions are also provided below. Specifically, referring to Figure 9, Figure 9 is another structural schematic diagram of the training task processing device provided in this application embodiment. The training task processing device 900 can be applied to a second device. The training task processing device 900 includes: a sending module 901, used to send request information to the training device, the request information being used to request the training device to execute a training task of a second model; a receiving module 902, used to receive notification information from the training device, the notification information indicating to suspend the execution of the training task of the second model, or the notification information indicating idle resources of the training device; the second training task includes the training task of the second model; wherein, the priority of the second training task is lower than the priority of the first training task, the first training task being a newly added training task by the training device.

Optionally, the notification information includes a reason value, which indicates the reason for pausing the training task of the second model.

Optionally, the sending module 901 is also used to send the processing method of the training task of the second model to the training device. The processing method of the training task of the second model is: termination, waiting, or adjusting the occupied resources.

It should be noted that the information interaction and execution process between the modules/units in the training task processing device 900 are based on the same concept as the various method embodiments corresponding to Figures 4 to 6 in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

Please refer to Figure 10, which is a schematic diagram of the structure of a device provided in an embodiment of this application. The device 1000 can specifically be the training device, the first device, or the second device in the above embodiments. Referring to Figure 10, the device 1000 includes at least one processor 1001 and at least one memory 1002. The processor 1001 and the memory 1002 are connected, for example, through a bus.

The memory 1002 is primarily used to store software programs. The memory 1002 can exist independently and be connected to the processor 1001. Optionally, the memory 1002 can be integrated with the processor 1001, for example, integrated within a single chip. The memory 1002 can store program code that executes the technical solutions of the embodiments of this application, and its execution is controlled by the processor 1001. The various types of computer program code being executed can also be considered as drivers for the processor 1001.

The processor 1001 mainly executes the software program stored in the memory 1002 to implement the functions corresponding to the training device, the first device, or the second device in any of the embodiments shown in Figures 4 to 6.

Figure 10 shows only one memory and one processor. In actual devices, there can be multiple processors and multiple memories. Memory can also be called storage medium or storage device, etc. Memory can be a storage element on the same chip as the processor, i.e., an on-chip storage element, or it can be a separate storage element; this application does not limit this.

This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the steps executed by the training device in the methods described in the embodiments shown in Figures 4 to 6, or causes the computer to perform the steps executed by the first device in the methods described in the embodiments shown in Figures 5 to 6, or causes the computer to perform the steps executed by the second device in the methods described in the embodiments shown in Figures 5 to 6.

This application also provides a computer program product, which includes a program that, when run on a computer, causes the computer to perform the steps performed by the training device in the methods described in the embodiments shown in Figures 4 to 6, or causes the computer to perform the steps performed by the first device in the methods described in the embodiments shown in Figures 5 to 6, or causes the computer to perform the steps performed by the second device in the methods described in the embodiments shown in Figures 5 to 6.

This application also provides a circuit system including a processing circuit configured to perform the method described in the embodiments shown in Figures 4 to 6 above.

This application also provides a training processing system, which includes a training device, a first device, and a second device. The training device is used to execute the steps performed by the training device in the methods described in the embodiments shown in Figures 4 to 6. The first device is used to execute the steps performed by the first device in the methods described in the embodiments shown in Figures 5 to 6. The second device is used to execute the steps performed by the second device in the methods described in the embodiments shown in Figures 5 to 6.

The training device, first device, second device, or training task processing apparatus provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input/output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip to execute the methods described in the embodiments shown in Figures 1 to 6. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of a program in the first aspect of the method.

It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CLUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims (25)

A method for processing training tasks, characterized in that the method includes: The training device receives a request from the first device, the request being used to request the execution of a training task for the first model; If the priority of the training task of the first model is higher than the priority of the training task of the second model, then the training device executes the first training task and pauses the execution of the second training task, wherein the first training task includes the training task of the first model; and/or, If the priority of the training task of the first model is lower than or equal to the priority of the second training task, the training device sends a response message to the first device. The response message is used to notify the first device to delay or refuse to execute the training task of the first model, or the response message includes information about the second training task. Wherein, the second training task is one or more training tasks currently being executed by the training device, or the second training task is the training task with the lowest priority among the training tasks currently being executed by the training device. According to the method of claim 1, the request information includes the priority of the training task of the first model. The method according to claim 1 or 2, wherein the information of the second training task includes the priority of the second training task. The method according to claim 1 or 2, characterized in that, after the training device suspends the execution of the second training task, the method further includes: The training device notifies the second device corresponding to the second training task to suspend the execution of the second training task; or... The training device notifies the second device corresponding to the second training task of the training device's available resources. According to the method of claim 4, the training device notifies the second device corresponding to the second training task to suspend the execution of the second training task, comprising: The training device sends a reason value to the second device, the reason value being used to indicate the reason for pausing the execution of the second training task. The method according to claim 5 is characterized in that the reason includes the second training task being the lowest priority training task on the training device. The method according to claim 6, wherein the cause value includes the priority of each of the at least one third training tasks currently being executed by the training device, and the priority of each of the third training tasks is higher than the priority of the second training task. The method according to any one of claims 4 to 7, characterized in that the method further comprises: The training device receives the processing method of the second training task from the second device. The processing method of the second training task is: terminate, wait, or adjust the occupied resources. The method according to claim 8, characterized in that the method further comprises: When the processing mode of the second training task is termination, the training device deletes the request information corresponding to the second training task; or, When the processing mode of the second training task is waiting, the training device waits until the idle resources are greater than or equal to the resources required by the second training task before continuing to execute the second training task; or, When the processing method of the second training task is to adjust the occupied resources, the training device reduces the number of parameters of the model corresponding to the second training task, or reduces the accuracy when executing the second training task, or reduces the batch training scale when executing the second training task. The method according to any one of claims 1 to 9, characterized in that the method is applied to the following scenarios: The training task of the first model requires more resources than the idle resources of the training device; or, The number of training tasks currently being executed by the training device is greater than a preset threshold. According to the method of claim 8, the adjustment of resource usage includes: reducing the number of parameters of the model corresponding to the second training task, reducing the accuracy when performing the second training task, or reducing the batch training scale when performing the second training task. The method according to any one of claims 1 to 11, wherein each training task currently performed by the training device is a task of training a model in the field of communication. A method for processing training tasks, characterized in that the method includes: The first device sends a request message to the training device, the request message being used to request the execution of the training task of the first model; The first device receives response information from the training device, the response information being used to notify the first device to delay or refuse to execute the training task of the first model, or the response information includes information about the second training task; Wherein, the priority of the training task of the first model is lower than or equal to the priority of the second training task, the second training task is one or more training tasks currently being executed by the training device, or the second training task is the training task with the lowest priority among the training tasks currently being executed by the training device. The method according to claim 13, characterized in that the method further comprises: The first device sends the processing method of the training task of the first model to the training device. The processing method of the training task of the first model is: terminate, wait, or adjust the occupied resources. According to the method of claim 14, the method of adjusting the occupied resources includes: reducing the number of parameters of the model corresponding to the training task of the first model, reducing the accuracy when performing the first training task, or reducing the batch training scale when performing the training task of the first model. The method according to claim 14 or 15 is characterized in that the factors determining the processing method of the training task of the first model include: the immediacy requirement of the first model, the accuracy requirement of the first model, and/or the degree of reduction in the accuracy of the first model caused by the adjustment of resource occupation. A method for processing training tasks, characterized in that the method includes: The second device sends a request message to the training device, the request message being used to request the training device to perform the training task of the second model; The second device receives a notification from the training device, the notification indicating to suspend the training task of the second model, or the notification indicating the idle resources of the training device; In this context, the training task of the second model has a lower priority than the training task of the first model, which is a newly added training task by the training device. The method according to claim 17, wherein the notification information includes a reason value, the reason value being used to indicate the reason for suspending the training task of the second model. The method according to claim 17 or 18, characterized in that the method further comprises: The first device sends the processing method of the training task of the second model to the training device. The processing method of the training task of the second model is: terminate, wait, or adjust the occupied resources. A processing apparatus for training tasks, characterized in that the apparatus is applied in a training device, the apparatus comprising: A receiving module is configured to receive request information from a first device, the request information being used to request the execution of a training task for a first model; The execution module is configured to execute the training task of the first model and pause the execution of the second training task if the priority of the training task of the first model is higher than the priority of the second training task; and/or, The sending module is configured to send response information to the first device if the priority of the training task of the first model is lower than or equal to the priority of the second training task. The response information is used to notify the first device to delay or refuse to execute the training task of the first model, or the response information includes information about the second training task. Wherein, the second training task is one or more training tasks currently being executed by the training device, or the second training task is the training task with the lowest priority among the training tasks currently being executed by the training device. A processing apparatus for training tasks, characterized in that the apparatus utilizes a first device, the apparatus comprising: The sending module is used to send request information to the training device, the request information being used to request the execution of the training task of the first model; A receiving module is configured to receive response information from the training device, the response information being used to notify the first device to delay or refuse to execute the training task of the first model, or the response information includes information about a second training task; Wherein, the priority of the training task of the first model is lower than or equal to the priority of the second training task, the second training task is one or more training tasks currently being executed by the training device, or the second training task is the training task with the lowest priority among the training tasks currently being executed by the training device. A processing apparatus for a training task, characterized in that the apparatus is applied to a second device, the apparatus comprising: The sending module is used to send request information to the training device, the request information being used to request the training device to perform the training task of the second model; A receiving module is configured to receive notification information from the training device, wherein the notification information indicates that the training task of the second model should be suspended, or the notification information indicates that the training device has idle resources. In this context, the training task of the second model has a lower priority than the training task of the first model, which is a newly added training task by the training device. An apparatus characterized in that it includes a processor coupled to a memory storing program instructions which, when executed by the processor, implement the method of any one of claims 1 to 19. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a program that, when the program is run on a computer, causes the computer to perform the method as described in any one of claims 1 to 19. A computer program product, characterized in that the computer program product includes a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 19.