WO2023035899A1 - Model deployment method, platform and computer-readable medium - Google Patents

Abstract

The embodiments of the present invention relate to an edge computing technology, and in particular to a model deployment method, a platform, and a computer-readable medium. The method comprises: a Kubernetes platform (100) starting a first service (21) (S301); the first service (21) downloading a model (30) to a first Pod (51) (S302); the Kubernetes platform (100) starting a second service (22) (S303); the second service (22) notifying the Kubernetes platform (100) to start at least one second Pod (52) (S304); the Kubernetes platform (100) starting the at least one second Pod (52) (S305); and each second Pod (52) downloading the model (30) from the first Pod (51) (S306). In this way, the flow used by each Pod to download a model can be reduced.

Description

Method, platform and computer readable medium for deploying models technical field

Embodiments of the present invention relate to edge computing technology, and in particular to a method, platform and computer-readable medium for deploying a model.

Background technique

Frameworks such as TensorFlow service and KFServing can be used for serverless AI deployment models (online services). Most of them are based on the Kubernetes cluster platform to achieve load balancing, and can perform automatic capacity adjustment based on K8s.

When storing models in these frameworks, in the Kubernetes platform 100 shown in Figure 1, multiple Pods 50 can be started in one node 40, and each Pod 50 performing inference needs to store a model 30 in the disk and memory respectively. copy. Assuming that the size of the model 30 is 109MB, each Pod 50 will use 109MB of disk storage space to download the model 30 from the cloud or other places, and use 109M of memory storage space to load the model 30 from the disk storage space for efficient inference process , which wastes both space and traffic.

Contents of the invention

Embodiments of the present invention provide a method, device, and computer-readable medium for deploying a model, so as to solve the problem of wasting traffic when the model is downloaded in the platform shown in FIG. 1 .

In a first aspect, a method for deploying a model on a Kubernetes platform is provided. In the method, the Kubernetes platform starts a first service; the first service downloads a model to the first Pod; the Kubernetes platform starts a second service; The second service notifies the Kubernetes platform to start at least one second Pod; the Kubernetes platform starts the at least one second Pod; and each second Pod downloads the model from the first Pod.

In a second aspect, a Kubernetes platform is provided, including: a deployment module configured to start a first service; the first service configured to download a model to the first Pod; the deployment module configured to Start a second service; the second service is configured to notify the Kubernetes platform to start at least one second Pod; the deployment module is also configured to start the at least one second Pod; each second Pod is configured to configured to download the model from the first Pod.

In a third aspect, a Kubernetes platform is provided, including: at least one memory configured to store computer-readable codes; at least one processor configured to invoke the computer-readable codes to execute each of the methods provided in the first aspect step.

In a fourth aspect, a computer-readable medium, where computer-readable instructions are stored on the computer-readable medium, and when the computer-readable instructions are executed by a processor, the processor executes the method provided in the first aspect each step.

Among them, by deploying the first service and the first Pod on the Kubernetes platform, the model is first downloaded to the newly deployed first Pod, and the second Pod of each deployed model downloads the model from the first Pod, which greatly reduces the The traffic used by each Pod to download models from the cloud or other external spaces.

For any of the above aspects, optionally, when the second service notifies the Kubernetes platform to start at least one second Pod, it can determine the number of the second Pod that needs to be started according to the quantity of the model deployed according to the received request. quantity; and notify the Kubernetes platform to start the at least one second Pod according to the determined quantity. In this way, the automatic capacity adjustment function of the Kubernetes platform is realized.

For any of the above aspects, optionally, when each second Pod downloads the model from the first Pod, it can directly download the model from the first Pod to the memory of the corresponding second Pod, so that , there is no need to download the model to the disk space first, and then download the model from the disk space to the memory, which further saves the storage space of the Pod.

For any aspect above, optionally, each second Pod is configured to access the IP address of the first service. In this way, it is realized that the second Pod no longer downloads the model from the cloud or external resources when it is started, but directly downloads the model from the first Pod through the first service.

Description of drawings

FIG. 1 is a schematic diagram of a method for deploying a model on a Kubernetes platform in the prior art.

FIG. 2 is a schematic structural diagram of a Kubernetes platform provided by an embodiment of the present invention.

Fig. 3 is a flowchart of a method for deploying a model provided by an embodiment of the present invention.

List of reference signs:

100: Kubernetes Platform

30: Model 40: Nodes in the Kubernetes Platform 100

50: Pods in each node 40 in the Kubernetes platform 100

10: Deployment Modules in the Kubernetes Platform 100

21: First service 22: Second service

51: First Pod 52: Second Pod

41, 42, 43: Nodes in the Kubernetes platform 100

300: The method for deploying a model on the Kubernetes platform 100 provided by the embodiment of the present invention

S301～S306: steps in the method 300

Detailed ways

The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and realize the subject matter described herein, and is not intended to limit the protection scope, applicability or examples set forth in the claims. Changes may be made in the function and arrangement of the elements discussed without departing from the scope of the embodiments of the invention. Various examples may omit, substitute, or add various procedures or components as needed. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with respect to some examples may also be combined in other examples.

As used herein, the term "comprising" and its variants represent open terms meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment." The term "another embodiment" means "at least one other embodiment." The terms "first", "second", etc. may refer to different or the same object. The following may include other definitions, either express or implied. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

Below, the embodiment of the present invention will be described in detail with reference to FIG. 2 and FIG. 3 .

FIG. 2 is a schematic structural diagram of a Kubernetes platform 100 provided by an embodiment of the present invention. As shown in Figure 2, the Kubernetes platform 100 includes:

- a deployment module 10 (deployment), configured to start the first service 21 .

In the Kubernetes platform, the deployment module (deployment) can maintain multiple .yaml files, and each .yaml file corresponds to a service (service), which defines the Pod managed by the service to be started and the container (docker) contained in the Pod Information. After the Kubernetes platform starts, read the .yaml file and start the corresponding service.

- A first service 21 configured to download a model 30 to a first Pod 51 .

Here, the first service 21 is newly defined in the embodiment of the present invention, and its operation is to download the model 30 from the cloud or other external resources, such as an AI model. Usually, AI models can be trained on servers in the cloud or other servers with sufficient processing resources, and the trained models can be sent to the edge side for model inference and other processing, that is, deployed on the edge side. Here, through the newly added first service 21, the model 30 can be downloaded to the newly defined Pod, that is, the first Pod51, and other Pods can download the model 30 from the first Pod51. Traffic used when downloading to each Pod.

- The deployment module 10 is also configured to start the second service 22 . Wherein, the principle of starting the second service 22 is the same as the principle of starting the second service 22, and will not be repeated here.

- A second service 22 configured to notify the deployment module 10 to start at least one second Pod 52 .

- a deployment module 10 further configured to start at least one second Pod 52 . Here, each second Pod 52 performs various model deployment operations such as model inference, and is configured to download the model 30 from the first Pod 51 . Wherein, each second Pod52 can be configured as an IP address for accessing the first service, so that the second Pod52 can download the model 30 from the first Pod51.

Here, optionally, when the second service 22 notifies the deployment module 10 to start the second Pod 52, it can determine the number of the second Pod 52 to be started according to the number of the received request deployment model 30; and notify the deployment module 10 to follow the determined The number starts at least one second Pod 52 . In this way, the automatic capacity adjustment function of the Kubernetes platform 100 is realized.

Optionally, each second Pod52 downloads the model 30 from the first Pod51 to the memory of the corresponding second Pod52. Compared with each Pod shown in FIG. 1 downloading two copies of the model 30 to the disk and memory respectively, this optional implementation can further save the storage space of the Pod.

In the Kubernetes platform 100 shown in Figure 2, the first service 21, the second service 22 and the first Pod 51 can be located in the same node of the Kubernetes platform 100, such as: the first node 41, the second node 42 or the third node 43 middle. And each second Pod 52 can be located in the same node or different nodes of the Kubernetes platform 100 .

The Kubernetes platform 100 shown in FIG. 2 can be deployed in one computer, or in a computer cluster composed of multiple computers, so as to execute the method 300 for deploying the model in the embodiment of the present invention. In terms of hardware structure, it may include at least one memory, which includes a computer-readable medium, such as a random access memory (RAM); it may also include at least one processor coupled with the at least one memory. Computer-executable instructions are stored in at least one memory and, when executed by at least one processor, may cause the at least one processor to perform various operations performed by the Kubernetes platform 100 .

Wherein, at least one processor may include a microprocessor, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a state machine, and the like. Examples of computer readable media include, but are not limited to, floppy disks, CD-ROMs, magnetic disks, memory chips, ROM, RAM, ASICs, configured processors, all-optical media, magnetic tape or other magnetic media, or from which a computer processor can Any other medium that reads instructions. In addition, various other forms of computer-readable media can transmit or carry instructions to the computer, including routers, private or public networks, or other wired and wireless transmission devices or channels. Instructions may include code in any computer programming language, including C, C++, C++, Visual Basic, java, and JavaScript.

Fig. 3 is a flowchart of a method for deploying a model provided by an embodiment of the present invention. The method 300 can be executed by the aforementioned Kubernetes platform 100, and can include the following steps:

-S301: the Kubernetes platform 100 starts the first service 21;

-S302: the first service 21 downloads the model 30 to the first Pod51;

-S303: the Kubernetes platform 100 starts the second service 22;

- S304: the second service 22 notifies the Kubernetes platform 100 to start at least one second Pod 52;

-S305: The Kubernetes platform 100 starts at least one second Pod52;

-S306: Each second Pod52 downloads the model 30 from the first Pod51.

Optionally, in step S304, the second service 22 may determine the number of second Pods 52 to be started according to the number of received request deployment models 30, and notify the Kubernetes platform 100 to start at least one second Pod 52 according to the determined number. In step S306, each second Pod52 downloads the model 30 from the first Pod51 to the memory of the corresponding second Pod52.

For other optional implementation manners of the method, reference may be made to the foregoing description of the Kubernetes platform 100 , which will not be repeated here.

In addition, the embodiments of the present invention also provide a computer-readable medium, on which computer-readable instructions are stored. When executed by a processor, the computer-readable instructions cause the processor to execute the aforementioned deployment model. method. Examples of computer readable media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tape, non- Volatile memory card and ROM. Alternatively, the computer readable instructions may be downloaded from a server computer or cloud by a communication network.

It should be noted that not all the steps and modules in the above processes and system structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as required. The system structures described in the above embodiments may be physical structures or logical structures, that is, some modules may be realized by the same physical entity, or some modules may be realized by multiple physical entities, or may be realized by multiple Certain components in individual devices are implemented together.

Claims (12)

A method (300) for deploying a model on a Kubernetes platform, characterized in that it includes: -Kubernetes platform (100) starts (S301) first service (21); - said first service (21) downloads (S302) a model (30) to said first Pod (51); - the Kubernetes platform (100) starts (S303) the second service (22); - said second service (22) notifies (S304) said Kubernetes platform (100) to start at least one second Pod (52); - the Kubernetes platform (100) starts (S305) the at least one second Pod (52); - Each second Pod (52) downloads (S306) said model (30) from said first Pod (51). The method according to claim 1, wherein the second service (22) notifies (S304) that the Kubernetes platform (100) starts at least one second Pod (52), including: - the second service (22) determines the number of the second Pods (52) that need to be started according to the number of deployed models (30) received; - The second service (22) notifies the Kubernetes platform (100) to start the at least one second Pod (52) according to the determined number. The method according to claim 1, wherein each second Pod (52) downloads (S306) the model (30) from the first Pod (51), comprising: - Each second Pod (52) downloads said model (30) from said first Pod (51) into the memory of the corresponding second Pod (52). The method of claim 1, wherein each second Pod (52) is configured to access the IP address of the first service. The method according to any one of claims 1 to 4, characterized in that, - said first service (21), said second service (22) and said first Pod (51) are located in a first node (41) of said Kubernetes platform (100); - said at least one second Pod (52) is located in the same node or a different node of said Kubernetes platform (100). A kind of Kubernetes platform (100), is characterized in that, comprises: - a deployment module (10) configured to start the first service (21); - said first service (21), configured to download a model (30) to said first Pod (51); - said deployment module (10), further configured to start a second service (22); - said second service (22), configured to notify said deployment module (10) to start at least one second Pod (52); - said deployment module (10), further configured to start said at least one second Pod (52); - Each second Pod (52), configured to download said model (30) from said first Pod (51). The Kubernetes platform according to claim 6, wherein said second service (22) is specifically configured as: - determining the number of said second Pods (52) that need to be started according to the number of deployed models (30) received; - Informing said deployment module (10) to start said at least one second Pod (52) by the determined number. The Kubernetes platform according to claim 6, wherein each second Pod (52) is configured as: - downloading said model (30) from said first Pod (51) into the memory of a corresponding second Pod (52). The Kubernetes platform according to claim 6, wherein each of the second Pods (52) is configured to access the IP address of the first service. The Kubernetes platform according to any one of claims 6 to 9, wherein, - said first service (21), said second service (22) and said first Pod (51) are located in a first node (41) of said Kubernetes platform (100); - said at least one second Pod (52) is located in the same node or a different node of said Kubernetes platform (100). A kind of Kubernetes platform (100), is characterized in that, comprises: at least one memory configured to store computer readable code; At least one processor configured to call the computer-readable code to execute the method according to any one of claims 1-5. A computer-readable medium, characterized in that, computer-readable instructions are stored on the computer-readable medium, and when the computer-readable instructions are executed by a processor, the processor can perform the operation according to any one of claims 1-5. one of the methods described.

Images