WO2025189874A1 - Representation sequence compression method and apparatus, and related device - Google Patents

Abstract

A representation sequence compression method and apparatus, and a related device, relating to the technical field of artificial intelligence. The method comprises: acquiring a representation sequence, evaluating the importance of different representations (tokens) in the representation sequence in a first network layer set in an AI model, and dividing the representations in the representation sequence into a first representation set and a second representation set on the basis of the importance; and determining at least one virtual representation on the basis of representations in the first representation set, and using a compressed representation sequence consisting of the virtual representation and representations in the second representation set as an input sequence of a second network layer set in the AI model, wherein the number of determined virtual representations is less than the number of the representations in the first representation set. In this way, by trimming some of the representations in the representation sequence, the amount of resources needing to be occupied by forward calculation of the AI model can be reduced, and information of the trimmed representations is recycled using the virtual representation and participates in the subsequent reasoning process, so that it can be ensured that the reasoning precision of the AI model reaches a high level.

Description

Method, device and related equipment for expressing sequence compression

This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on March 14, 2024, with application number 202410295573.5 and application name “Method, device and related equipment for expressing sequence compression”, the entire contents of which are incorporated by reference into this application.

Technical Field

The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, and related equipment for representing sequence compression.

Background Art

With the development of artificial intelligence (AI) technology, AI models such as generative pre-trained transformer (GPT) and vision transformer (ViT) have been widely used in various fields such as natural language processing and vision processing.

In actual application scenarios, the length of the representation (token) sequence used as input to the AI model is on an increasing trend. For example, the number of token sequences input to the AI model may reach 2,000, etc., which will result in excessive hardware resources required for the AI model to be trained or reasoned based on the token sequence. For this reason, the representation sequence used as input to the AI model can usually be compressed to reduce the AI model's demand for hardware resources during training or reasoning. Specifically, in the forward calculation stage of the model training or reasoning process, the importance of each token can be identified in the shallow layer of the AI model (such as the first 5 network layers, etc.), and the input token sequence can be pruned according to the importance of each token, so that only multiple tokens with higher importance are retained and passed to the subsequent network layers for further calculation. In this way, the AI model can perform a forward calculation process based on a smaller number of tokens, thereby reducing the demand for hardware resources.

However, in actual application scenarios, this method of compressing and representing sequences can easily lead to low reasoning accuracy of AI models.

Summary of the Invention

This application provides a method for compressing representation sequences, aiming to reduce the number of resources required by AI models during inference (or training) while improving the inference accuracy of AI models. Furthermore, this application also provides a representation sequence compression apparatus, a computing device, a computer-readable storage medium, and a computer program product.

In the first aspect, the present application provides a method for compressing a representation sequence, which can be performed by a corresponding representation sequence compression device. Specifically, the representation sequence compression device obtains a first representation sequence, which is a sequence obtained based on the input data of an AI (artificial intelligence) model. For example, when the input data is a paragraph of text, the representation (token) in the first representation sequence can be a symbol or character (or word) in the paragraph of text. The AI model includes a first network layer set and a second network layer set, and the first network layer set and the second network layer set are cascaded front and back, such as the output data of the first network layer set is the input data of the second network layer set. Then, the representation sequence compression device evaluates the importance of different representations in the first representation sequence in the first network layer set, and the importance can be measured, for example, by an importance score. Then, the representation sequence compression device divides the representations in the first representation sequence into a first representation set and a second representation set according to the importance of each representation in the first representation sequence. Generally, the importance of the representations in the second representation set is higher than the importance of the representations in the first representation set. Finally, the representation sequence compression device determines at least one first virtual representation based on the representations in the first representation set, and uses the compressed representation sequence composed of the at least one first virtual representation and the representations in the second representation set as the input sequence for the second network layer set. The number of the determined at least one virtual representation is less than the number of representations in the first representation set. In this case, the representations in the first representation set are the pruned representations from the first representation sequence.

In this way, virtual representation is used to recover information of representations that have been pruned in the first network layer set because of their low importance. This allows the information recovered by the virtual representation to ensure the reasoning accuracy of the AI model even if the representations with higher importance to the second network layer set are mistakenly pruned in the first network layer set. This can avoid completely pruning the representation and causing the reasoning accuracy of the AI model to decrease. At the same time, after pruned part of the representations in the first representation sequence, the AI model performs forward calculations based on a smaller number of representations, and the number of resources required can be effectively reduced. Moreover, during the forward calculation process, the virtual representation and the retained representation (that is, the representation in the second representation set) can be independent of each other, which can avoid mutual interference between the virtual representation and the retained representation and affect the reasoning accuracy of the AI model.

In one possible embodiment, when the representation sequence compression device determines at least one first virtual representation based on the representation in the first representation set, it can specifically determine the value of at least one first virtual representation based on the value of the representation in the first representation set. At this time, the number of first virtual representations can be a pre-configured fixed number. In this way, using the first virtual representation to recycle the information of the representation in the pruned first representation set (that is, the value of the representation) can avoid completely pruning the representation, which leads to a decrease in the reasoning accuracy of the AI model.

In one possible embodiment, when the representation sequence compression device determines at least one first virtual representation based on the representations in the first representation set, it can specifically first determine the number of at least one first virtual representation based on the number of representations in the first representation set, and then determine the value of the at least one first virtual representation based on the value of the representation in the first representation set. At this time, the number of first virtual representations can also be determined based on the number of pruned representations, such as the larger the number of pruned representations, the larger the number of determined first virtual representations, etc. In this way, the representation sequence compression device can dynamically configure the number of virtual representations based on the actual number of pruned representations, thereby improving the flexibility of using virtual representations to recover information of pruned representations.

In one possible embodiment, the first network layer set in the AI model includes a target parameter, which is used to calculate the importance of the representation in the first representation sequence in the first network layer set. Then, the representation sequence compression device can also calculate a loss value according to a loss function during the training of the AI model. The loss function includes a variance regularization term corresponding to the target parameter, and updates the target parameter in the first network layer set according to the loss value. In this way, the representation sequence compression device can improve the discrimination of the importance of different representations calculated by the target parameter by updating the value of the target parameter, thereby helping to improve the accuracy of the measurement of the importance of the representation.

In one possible implementation, the representations in the first representation sequence include words (including characters or phrases) or image blocks. For example, a sequence of characters or phrases in a text input by a user can be used as the first representation sequence; or a sequence of image blocks in an image provided by a user can be used as the first representation sequence.

In one possible implementation, the representations in the first representation sequence include words. In this case, at least one first virtual representation used to recycle the information of the pruned representation is located at the beginning of the compressed representation sequence. In this way, the position of the virtual representation in the compressed representation sequence is set based on the importance of the position of the word in the text to the text, which can help improve the accuracy of the AI model's reasoning based on the compressed representation sequence.

In one possible implementation, the representations in the first representation sequence include image blocks. In this case, at least one first virtual representation used to recover information from the cropped representation is located in the middle of the compressed representation sequence. In this way, positioning the virtual representations in the compressed representation sequence based on the importance of the image block's position in the image can help improve the accuracy of the AI model's reasoning based on the compressed representation sequence.

In one possible embodiment, the AI model also includes a third network layer set, wherein the second network layer set and the third network layer set are cascaded front and back, that is, the output data of the second network layer set can be used as the input data of the third network layer set. Then, the representation sequence compression device can also evaluate the importance of different representations in the second representation set in the second network layer set, and divide the representations in the second representation set into a first subset and a second subset according to the importance of different representations in the second representation set in the second network layer set, wherein the importance of the representations in the second subset is higher than the importance of the representations in the first subset; then, the representation sequence compression device can also determine at least one second virtual representation based on the representations in the first subset, the number of the at least one virtual representation is less than the number of representations in the first subset, and use the compressed representation sequence composed of the at least one first virtual representation, the at least one second virtual representation and the representations in the second subset as the input sequence of the third network layer set. In this way, the representation sequence compression device can further reduce the number of resources required in the forward calculation stage by cutting the representation sequence multiple times in different network layer sets. Moreover, the representation sequence compression device can utilize different virtual representations to recover the representation information that has been pruned in different sets of network layers, and participate in the calculation of subsequent network layers based on the virtual representation, which can improve the reasoning accuracy of the AI model.

In one possible embodiment, the AI model also includes a third network layer set, wherein the second network layer set and the third network layer set are cascaded. The representation sequence compression device can then evaluate the importance of different representations in the second representation set in the second network layer set, and based on the importance of different representations in the second representation set in the second network layer set, divide the representations in the second representation set into a first subset and a second subset, wherein the importance of the representations in the second subset is higher than the importance of the representations in the first subset. The representation sequence compression device can then determine the value of at least one first virtual representation in the second network layer set based on the representations in the first subset, and use the compressed representation sequence composed of the at least one first virtual representation and the representations in the second subset as the input sequence for the third network layer set. In this way, the representation sequence compression device can further reduce the number of resources required in the forward computation phase by pruning the representation sequence multiple times in different network layer sets. Furthermore, the representation sequence compression device can use the same virtual representation to recycle information about representations pruned in different network layer sets, which can further reduce the number of resources consumed in the forward computation process.

In one possible embodiment, the method flow of the representation sequence compression device for compressing the first representation sequence can be applied to the deployment stage of the AI model. Then, the representation sequence compression device can also update the model calculation graph corresponding to the AI model according to the compressed representation sequence, so that the representation sequence compression device can reduce the amount of resources required for the forward calculation process by compressing the representation sequence during the process of performing forward calculation according to the updated model calculation graph, and at the same time use virtual representation to recycle the information of the cropped representation to ensure the reasoning accuracy of the AI model.

In one possible implementation, the method flow for compressing the first representation sequence by the representation sequence compression device can be applied during the development phase of the AI model. For example, an SDK (software development kit) for implementing the compressed representation sequence can be added to the model file corresponding to the AI model during the development phase. In this way, when the representation sequence compression device runs the developed AI model, it can compress the representation sequence to reduce the amount of resources consumed in the forward calculation process, and at the same time, use virtual representation to recycle the cropped representation information to ensure the reasoning accuracy of the AI model.

In a second aspect, the present application provides a representation sequence compression device, which includes an acquisition module for acquiring a first representation sequence, where the first representation sequence is a sequence obtained based on input data of an artificial intelligence AI model, wherein the AI model includes a first network layer set and a second network layer set, and the first network layer set and the second network layer set are cascaded front and back; an evaluation module for evaluating the importance of different representations in the first representation sequence in the first network layer set; a division module for dividing the representations in the first representation sequence into a first representation set and a second representation set according to the importance; a determination module for determining at least one first virtual representation based on the representations in the first representation set, where the number of at least one virtual representation is less than the number of representations in the first representation set, and using a compressed representation sequence composed of at least one first virtual representation and the representations in the second representation set as the input sequence of the second network layer set.

In a possible implementation, the determination module is configured to determine a value of at least one first virtual representation according to values of representations in the first representation set.

In a possible implementation, the determination module is configured to: determine the number of at least one first virtual representation based on the number of representations in the first representation set; and determine the value of at least one first virtual representation based on the value of the representation in the first representation set.

In one possible embodiment, the first network layer set includes a target parameter, which is used to calculate the importance of the representation in the first representation sequence in the first network layer set; the representation sequence compression device also includes a training module, which is used to: in the process of training the AI model, calculate the loss value according to the loss function, and the loss function includes a variance regularization term corresponding to the target parameter; update the target parameter according to the loss value.

In a possible implementation, the representations in the first representation sequence include words or image blocks.

In one possible implementation, the representation includes words, and the at least one first virtual representation is located at a starting position in the compressed representation sequence.

In one possible implementation, the representation comprises image blocks, and the at least one first virtual representation is located at an intermediate position in a sequence of compressed representations.

In one possible embodiment, the AI model also includes a third network layer set, and the second network layer set and the third network layer set are cascaded front and back; the evaluation module is also used to evaluate the importance of different representations in the second representation set in the second network layer set; the division module is also used to divide the representations in the second representation set into a first subset and a second subset according to the importance of different representations in the second representation set in the second network layer set; the determination module is also used to determine at least one second virtual representation based on the representations in the first subset, the number of at least one virtual representation is less than the number of representations in the first subset, and the compressed representation sequence composed of at least one first virtual representation, at least one second virtual representation and the representation in the second subset is used as the input sequence of the third network layer set.

In one possible embodiment, the AI model also includes a third network layer set, and the second network layer set and the third network layer set are cascaded front and back; the evaluation module is also used to evaluate the importance of different representations in the second representation set in the second network layer set; the division module is also used to divide the representations in the second representation set into a first subset and a second subset according to the importance of different representations in the second representation set in the second network layer set; the determination module is also used to determine the value of at least one first virtual representation in the second network layer set based on the representation in the first subset, and use the compressed representation sequence composed of at least one first virtual representation and the representation in the second subset as the input sequence of the third network layer set.

In one possible implementation, the representation sequence compression device is applied to the deployment stage of the AI model, and the representation sequence compression device also includes an update module for updating the model calculation graph corresponding to the AI model according to the compressed representation sequence.

In a possible implementation, the representation sequence compression device is applied to the development stage of the AI model.

The representation sequence compression device provided in the second aspect corresponds to the representation sequence compression method provided in the first aspect. Therefore, the technical effects of the second aspect and any implementation method in the second aspect can be found in the relevant description of the technical effects of the first aspect and the corresponding implementation method in the first aspect, and will not be repeated here.

In a third aspect, the present application provides a computing device comprising a processor and a memory; wherein the memory is used to store instructions, and the processor executes the instructions stored in the memory to perform the operating steps of the representation sequence compression method described in the first aspect and any one of the implementations of the first aspect.

In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computing device, the computing device executes the operating steps of the representation sequence compression method described in the first aspect or any implementation of the first aspect.

In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the operational steps of the representation sequence compression method described in the first aspect or any one of the implementations of the first aspect.

Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a schematic diagram of the reasoning process of the AI model;

FIG2 is a schematic diagram of the reasoning process of an exemplary AI model provided in this application;

FIG3 is a flow chart of a method for compressing a sequence provided by the present application;

FIG4 is a schematic diagram of inserting m virtual tokens into token sequence 1 provided by the present application;

FIG5 is a schematic diagram of the trimming token and recycling token information provided by this application;

Figure 6 is a schematic diagram of reducing the amount of data calculation after token clipping;

FIG7 is a flow chart of another method for sequence compression provided by the present application;

FIG8 is a schematic structural diagram of a sequence compression device provided by the present application;

FIG9 is a schematic diagram of the hardware structure of a computing device provided in this application.

DETAILED DESCRIPTION

To make the above-mentioned objects, features, and advantages of the present application more clearly understood, various non-limiting embodiments of the embodiments of the present application will be exemplified below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained based on the above content are within the scope of protection of this application.

Referring to FIG1 , a schematic diagram of the reasoning process of an AI model is shown, where the AI model can be run through one or more computing nodes.

Exemplarily, the AI model may be, for example, a generative pre-trained transformer model (GPT), a bidirectional encoder representations from transformer (BERT) model, a vision transformer (ViT) model, a contrastive language-image pre-training (CLIP) model, etc., or may be other types of AI models, without limitation.

The computing node that supports the operation of the AI model can be an accelerator card, which can be, for example, a deep-learning processing unit (DPU), a data processing unit (DPU), a graphics processing unit (GPU), a neural network processing unit (NPU), or other types of accelerator cards. Alternatively, the computing node can be a general-purpose processor, such as a central processing unit (CPU). Alternatively, the computing node can also include a computing device of a CPU and an accelerator card. This application does not limit the specific implementation of the computing node.

For ease of description, the following uses an example of running an AI model on a computing node.

As shown in Figure 1, the input data of the AI model can be an image, and the AI model can perform corresponding reasoning based on the image. For example, the AI model can generate text information based on the input image data, and the text information can be text information that can describe the content of the image.

When using an AI model to perform inference based on image data, the computing node can first decompose the image into multiple image blocks (each image block can be of the same size) and then expand the multiple image blocks in a specified order, such as from the upper left corner to the lower right corner of the image, to obtain a sequence of image blocks. The computing node can then use each image block as a token to obtain a corresponding token sequence, as shown in Figure 1.

Then, the computing node can use the AI model to perform the corresponding reasoning process based on the token sequence. In this process, the longer the input token sequence is, the more computing resources and storage resources are required for the AI representation sequence compression. Therefore, the computing node can prune the input tokens. Specifically, as shown in Figure 1, the computing node can calculate the importance of each token in the input token sequence at the shallow layer of the AI model, and perform mask calculation based on the importance score of each token, so as to use the mask result to identify the less important tokens in the token sequence, so that the computing node can eliminate the less important tokens in the token sequence according to the mask result, and use the remaining multiple tokens after elimination to continue to participate in the calculation of subsequent network layers in the AI model. Among them, the AI model includes multiple network layers, and, in the order of forward calculation, the network layers with the highest execution order in the multiple network layers can be described as shallow layers (such as the first 5 network layers, etc.), and the multiple network layers with the lowest execution order can be described as deep layers (such as the last 10 network layers, etc.). This allows subsequent network layers to perform calculations based on a smaller number of tokens, reducing the overall resources consumed by the AI model for inference. While reducing the number of tokens in the token sequence for inference will reduce the AI model's inference accuracy, the remaining tokens are more important, which helps maintain a relatively high level of AI model inference accuracy.

Generally speaking, the more tokens a computing node eliminates in the shallow layers of an AI model, the fewer resources the AI model uses to perform reasoning based on the input token sequence. However, the greater the reduction in the AI model's reasoning accuracy, as shown in Figure 1. Conversely, the fewer tokens a computing node eliminates in the shallow layers of an AI model, the more resources the AI model uses to perform reasoning based on the input token sequence, and accordingly, the higher the AI model's reasoning accuracy, as shown in Figure 1. Furthermore, tokens that are less important in the shallow layers of an AI model may be more important in the deeper layers of the AI model. Therefore, eliminating tokens in the shallow layers that are more important to the deeper layers can also have a significant impact on the AI model's reasoning accuracy.

Based on this, the present application provides a token sequence compression method, which aims to reduce the resource requirements of the AI model during the reasoning (or training) process while improving the reasoning accuracy of the AI model.

In specific implementation, as shown in FIG2, in the forward calculation stage of the AI model, the computing node can calculate the importance of each token in the input token sequence at the shallow layer of the AI model. Specifically, it can be to evaluate the importance of each token in the shallow layer (the importance can be measured by the importance score), and divide the multiple tokens in the token sequence into two token sets according to the importance of each token, namely token set 1 and token set 2, each token set includes at least one token. Among them, the importance of the tokens in token set 2 in the shallow layer of the AI model is higher than the importance of the tokens in token set 1 in the shallow layer of the AI model. For example, the computing node can calculate the importance of each token and perform mask calculation based on the importance score of each token, so as to use the mask result to identify tokens with higher importance and tokens with lower importance, thereby realizing the division of the token sequence into two token sets. The computing node will then determine at least one virtual token based on the tokens in token set 1, so that it can use this virtual token to recover the token information in token set 1. The number of these virtual tokens is less than the number of tokens in token set 1. Therefore, the computing node will use the compressed token sequence composed of the at least one virtual token and the tokens in token set 2 as the input sequence for the deeper layer of the AI model, that is, pass the virtual token and the retained tokens in token set 2 to the deeper layer of the AI model for calculation. At this time, the tokens in token set 1 are the pruned tokens in the token sequence, and after the computing node prunes the less important tokens at the shallow layer, it will use the virtual tokens to recover the information of the pruned tokens and participate in the deeper calculation.

In this way, virtual tokens are used to recycle information about tokens that were pruned at shallow layers due to their low importance. This allows the AI model's reasoning accuracy to be maintained even if tokens of high importance to deeper layers are mistakenly pruned at shallow layers. This prevents the AI model's reasoning accuracy from being reduced due to the complete pruned token, as shown in Figure 2. At the same time, after pruning some tokens from the token sequence, the AI model performs forward calculations based on a smaller number of tokens, effectively reducing the number of resources required. In actual test scenarios, using virtual tokens to recycle information about pruned tokens can improve the AI model's reasoning accuracy to 94.65%, while also reducing the amount of memory resources required by 63%.

Moreover, during the forward calculation process, the virtual token and the retained token can be independent of each other, which can avoid mutual interference between the virtual token and the retained token and affect the reasoning accuracy of the AI model.

In actual application scenarios, computing nodes can use configured plug-ins to trim the token sequence input to the AI model; alternatively, computing nodes can trim the token sequence input to the AI model based on the code logic of the software development kit (SDK) built into the model file corresponding to the AI model.

In the first implementation, during the deployment phase of the AI model, a target plug-in can be configured in the compute node, which can be used to trim token sequences. Then, when training or inference is performed based on the AI model, the compute node can activate and run the target plug-in and use it to update the model computation graph (forward computation phase) corresponding to the AI model. For example, the compute node can use the target plug-in to call the dynamic graph engine in the compute node to analyze and obtain the model computation graph corresponding to the AI model. The model computation graph can be used to indicate the computational logic executed by the AI model during the forward computation phase. The compute node can then use the target plug-in and dynamic graph instrumentation technology (such as Torch.FX tools) to insert computational logic for trimming token sequences into the model computation graph. This computational logic includes calculating token importance scores, generating masks to indicate trimmed and retained tokens, and configuring and inserting virtual tokens. In this way, the compute node can execute the forward computation process based on the updated model computation graph, achieving corresponding trimming of the input token sequence in the AI model.

In the second implementation, during the AI model development phase, such as when developing an AI model based on a framework like Pytorch or Mindspoe, users can call the SDK interface and add the SDK for implementing token sequence trimming to the model file corresponding to the AI model. In this way, during training and inference based on the AI model, the computing node can execute the model file corresponding to the AI model, thereby trimming the input token sequence accordingly by running the SDK in the model file.

For ease of understanding, an embodiment of the representation sequence compression method provided by the present application is described below with reference to the accompanying drawings.

See Figure 3, which is a flow chart of a method for compressing a representation sequence provided by an embodiment of the present application. As shown in Figure 3, the method for compressing a representation sequence may specifically include:

S301: The computing node obtains token sequence 1, which is a sequence obtained based on the input data of the AI model.

Typically, token sequence 1 may include multiple tokens.

For example, each token can be a word. When the AI model performs forward computation based on a word sequence, the computing node can use each word in the word sequence as a token, thereby obtaining token sequence 1. For example, a user can provide the computing node with a text message: "The weather is great today, what activities are suitable for going out with friends?" The computing node can then use each word and each punctuation mark in the text message as a token. Alternatively, the computing node can segment the text message and use each segmented word as a token, etc., without limitation.

Alternatively, each token can be an image block. When the AI model performs forward computation based on one or more images, the computing node can decompose the image into multiple image blocks in a specified order, and use each image block as a token. Based on the order of the decomposed images, the computing node can obtain a corresponding token sequence 1. For example, a user can provide a computing node with a size of 2560×1440 pixels. The computing node can then decompose the image into 16 image blocks of 160×90 pixels in order from the upper left corner to the lower right corner of the image, obtaining a sequence of 16 tokens.

In one possible implementation, the computing node can provide a client, which can be, for example, an application running on a user's device or a web browser. The user can provide text or images to the client, which then forwards the text or images to the computing node. The computing node can then use the received text or images as input data for the AI model and generate a corresponding token sequence 1 based on the input data in the manner described above.

After obtaining the token sequence 1, the computing node can perform forward computing based on the token sequence 1. The forward computing process performed by the computing node based on the input token sequence 1 can be a forward computing process in an inference scenario or a forward computing process in a training scenario, and this is not limited.

S302: The computing node evaluates the importance of different tokens in token sequence 1 in the first network layer set, wherein the AI model includes the first network layer set and the second network layer set, and the first network layer set and the second network layer set are cascaded front and back.

In this embodiment, the AI model may include multiple network layers, and the multiple network layers may be divided into a first network layer set and a second network layer set according to their depth in the AI model. The first network layer set and the second network layer set each include at least one network layer. In actual application, the network layers in the first network layer set may also be referred to as shallow layers in the AI model, and the network layers in the second network layer set may also be referred to as deep layers in the AI model. Moreover, the first network layer set and the second network layer set are cascaded front and back, that is, in the forward calculation stage, the input data of the second network layer set can be obtained based on the output data of the first network layer set.

After obtaining token sequence 1, the computing node can perform forward calculations based on token sequence 1 at each network layer of the AI model. Furthermore, when token sequence 1 is passed to the first network layer set of the AI model, the computing node can calculate the importance score of each token in token sequence 1 within the first network layer set. This importance score is used to measure the importance of the token within the first network layer set. Generally, a larger importance score indicates a higher importance.

As an implementation example, the computing node can first calculate the norm of the feature vector of the token at different spatial positions in the token sequence 1 in the first network layer set, and calculate the importance score 1 of the token based on the norm of the feature vector. Then, the computing node calculates the attention score between the token and the token at the starting position in the token sequence 1, and calculates the importance score 2 of the token based on the attention score. Among them, the implementation method of calculating the importance score 1 and the importance score 2 has relevant applications in actual scenarios and will not be elaborated here. Finally, the computing node calculates the final importance score of the token in the first network layer set by performing a weighted sum of the importance score 1 and the importance score 2 of the token.

In other embodiments, the computing node may also calculate the importance score of each token in the first network layer set based on a learned token pruning (LTP) algorithm, or may calculate the importance score of each token in the first network layer set based on the auto-scaling vision transformer (AS-ViT) framework. Furthermore, the importance of a token in the first network layer set may also be measured by other methods, which are not limited to this.

S303: The computing node divides the tokens in the token sequence 1 into a token set 1 and a token set 2 according to the importance of each token.

In this embodiment, the importance of the tokens in token set 2 in the first network layer set is higher than the importance of the tokens in token set 1 in the first network layer set of the AI model.

As an implementation example, when the importance is measured by an importance score, the computing node can compare the importance score of each token in the first network layer set with threshold 1, and classify tokens with importance scores less than threshold 1 into token set 1, and tokens with importance scores greater than or equal to threshold 1 into token set 2. Thus, the computing node can divide the multiple tokens included in token sequence 1 into token set 1 and token set 2, and the importance of the tokens in token set 2 is higher than the importance of the tokens in token set 1. Threshold 1 can be a value pre-set by a technician, or can be determined by the AI model through self-learning during the training process.

Among them, the tokens in the token set 1 with lower importance are the tokens that need to be pruned in the AI model. Accordingly, in the forward calculation process, the computing node can retain the tokens in the token set 2 to participate in the calculation of the subsequent network layer. In actual application scenarios, when the token is specifically a word, the first token in the token sequence 1 is the first word in the word sequence, and the first word is usually more important for the natural language processing process of the word sequence. Accordingly, the first token in the token sequence 1 is also more important in the forward calculation process of the AI model, so the computing node can usually retain the token in the token set 2. When the token is specifically an image block, the importance of the token in the middle of the token sequence 1 is usually higher. Therefore, the computing node can usually retain the token in the middle in the token set 2.

In actual applications, the computing node may also divide the multiple tokens in the token sequence 1 in other ways, and there is no limitation on this.

S304: The computing node determines at least one virtual token based on the tokens in token set 1, wherein the number of the at least one virtual token is less than the number of tokens in token set 1.

In this embodiment, if tokens with lower importance in the first network layer are directly pruned, the AI model's reasoning accuracy may be affected because these tokens are more important in the subsequent second network layer. Therefore, the computing node can use virtual tokens to recycle the pruned token information, so that the recycled token information can be used to compensate for the AI model's reasoning accuracy.

In a first possible implementation, before partitioning token sequence 1, the computing node may pre-configure a target number of virtual tokens for token sequence 1 (e.g., 4 virtual tokens) and insert this target number of virtual tokens into token sequence 1 obtained by the computing node. The inserted virtual tokens are used to reclaim token information from token set 1 that was pruned from the first network layer set. The target number of virtual tokens may be pre-set by a technician, for example, through a limited number of experiments.

When the token is a word, the computing node can insert a virtual token into the starting position of token sequence 1. For example, as shown in Figure 4, token sequence 1 can include a token marked with "cls" (also known as a classification character), which is used to mark the beginning of token sequence 1 (it can already be used to indicate the category of the text). At the same time, token sequence 1 can also include tokens corresponding to n+1 words (n is a positive integer), which are identified by _x0 to _xn in Figure 4. In token sequence 1, the next token of the token marked with "cls" (i.e., _x0 ) is the token corresponding to the first word in the word sequence. Therefore, before dividing token sequence 1, the computing node can first insert m virtual tokens (m is a positive integer) in token sequence 1 before the token marked with "cls" to obtain a new token sequence. In Figure 4, _v1 to _xm are used to identify the inserted virtual tokens. At this point, as shown in Figure 4, in the newly generated token sequence, the first token is a dummy token, the m+1th token is the token corresponding to the "cls" tag, and the m+2th token is the token corresponding to the first word in the word sequence. That is, the absolute position of each token in token sequence 1 in the newly generated token sequence is pushed back by m tokens. The computing node can then partition the new token sequence, including the dummy token, to obtain a relatively important token set 2. Typically, when the computing node partitions to obtain token set 2, the token labeled "cls" will also be retained in this token set 2.

When the token is an image block, the computing node can insert the virtual token into the middle position of the token sequence 1. For example, assuming that the token sequence 1 includes N tokens (N is a positive integer greater than 1), the computing node can insert the virtual token from the first Starting from the token, m virtual tokens are inserted in sequence.

In actual applications, the computing node can insert the virtual token at any other location within token sequence 1, such as at the very end of token sequence 1, and there is no limitation on this. Furthermore, after inserting the virtual token, the computing node can initialize the value of the virtual token, such as by setting all values of the virtual token to 1.

Accordingly, when the computing node divides multiple tokens in the token sequence 1, it may specifically divide the token sequence obtained by inserting the virtual token. At this time, the computing node may divide all the virtual tokens in the token sequence into the token set 2 by default (the virtual token may not participate in the calculation of the importance score).

After obtaining token set 2, the computing node can use the inserted virtual token (i.e., the virtual token in token set 2) to recover the token information in the pruned token set 1. In specific implementation, the computing node can use a mapping algorithm to calculate the value of each virtual token based on the values of the tokens in token set 1. For example, as shown in Figure 5, assuming that the pruned tokens are _x1 to _x1 , the value of each virtual token can be calculated based on the values of the pruned tokens. Exemplarily, the mapping algorithm used to calculate the value of the virtual token can be, for example, a facility location problem (FLP) algorithm. Specifically, the virtual token value is used as the address, the construction cost of the virtual token, and the difference between the virtual token and the pruned token values as the objective (i.e., minimizing the recovery cost objective). The virtual token value is quickly solved by establishing a dual program that relaxes the original problem, thereby completing the mapping from the pruned token to the virtual token. In actual applications, the computing node may also adopt other mapping algorithms to calculate the value of the virtual token according to the value in the token set 1, and there is no limitation on this.

In the first implementation described above, the computing node first configures a fixed number of virtual tokens and then divides the sequence containing the virtual tokens. In a second possible implementation, the computing node may also first divide token sequence 1 and then insert virtual tokens into the token sequence corresponding to the more important token set 2.

In specific implementation, after obtaining token set 1, the computing node can first determine the number 2 of virtual tokens based on the number 1 of tokens in token set 1, and the number 2 is less than the number 1. For example, when the number of tokens in token set 1 that need to be pruned is 20, the number of virtual tokens can be 4. Then, the computing node can configure a virtual token based on the number 2, and insert the virtual token into the token sequence corresponding to token set 2, such as inserting the virtual token at the starting position or the middle position of the token sequence. The ratio between the number of pruned tokens and the number of virtual tokens can be configured in advance by technical personnel. The computing node can determine the number 2 of virtual tokens based on any method, and there is no limitation on this.

Next, the computing node can determine the value of the inserted virtual token based on the token value in the pruned token set 1, such as by using the FLP algorithm to determine the value of the virtual token, so as to use the virtual token to recover the information of the pruned token. In this way, the computing node can dynamically configure the number of virtual tokens based on the actual number of pruned tokens, thereby improving the flexibility of using virtual tokens to recover the pruned token information. For example, when the number of pruned tokens is large, the computing node will insert more virtual tokens in the token sequence corresponding to token set 2, so as to use more virtual tokens for information recovery.

It is understandable that the above-mentioned implementation methods of configuring virtual tokens and determining virtual token values are only some exemplary explanations. In other embodiments, the computing node may also use other methods to configure the number and value of virtual tokens, and this is not limited.

S305: The computing node uses a compressed token sequence composed of at least one virtual token and tokens in token set 2 as an input sequence of the second network layer set.

In this embodiment, after the computing node prunes the less important tokens in the first network layer set, it can obtain a new sequence based on the inserted virtual tokens and the retained more important tokens. The number of tokens included in this new sequence is less than the number of tokens included in token sequence 1, thus compressing token sequence 1. For ease of distinction, the newly obtained sequence is referred to as a compressed token sequence below, so that the computing node can use this compressed token sequence as the input sequence of the second network layer set, so as to pass this compressed token sequence to the second network layer set in the AI model for continued calculation.

In a specific implementation, when the token is a word, the computing node can generate a token sequence 2 (that is, the compressed token sequence mentioned above) based on the inserted virtual token and the token in the token set 2, and the virtual token is located at the starting position of the token sequence 2, so that the computing node can pass the token sequence 2 to the second network layer set for calculation. In the second network layer set, the computing node can perform corresponding calculations based on the value of the virtual token and the value of the (non-virtual) token, so that the AI model continues to complete the forward calculation process. In this way, the amount of data calculated by the computing node in the second network layer set can be effectively reduced, so that the required resource consumption can be calculated. For example, as shown in Figure 6, before the token sequence 1 input to the AI model is trimmed, the activation value generated by the computing node in the second network layer set using the activation function (such as the Sigmoid function, etc.) based on the token sequence 1 is a value of N×d dimensions (token sequence 1 includes N tokens). After trimming token sequence 1 and obtaining token sequence 2, the computing node in the second network layer set uses the activation function to calculate based on token sequence 2 to generate an activation value of k×d dimension (token sequence 1 includes k tokens, k is a positive integer less than N). In this way, the computing node can reduce the data calculation of (N-k)×d dimension, that is, it can reduce the resource consumption generated by the calculation of this part of the data volume.

When the token is an image block, the computing node can generate a token sequence 3 (i.e., the compressed token sequence described above) based on the inserted virtual token and the tokens in token set 2. Furthermore, the virtual token is located in the middle of token sequence 3, so that the computing node can pass token sequence 3 to the second network layer set for calculation. Similarly, by reducing the number of tokens (including virtual tokens and non-virtual tokens) passed to the second network layer set, the resource consumption required by the computing node for calculation in the second network layer set can be reduced.

It should be noted that even if one or more tokens in token sequence 1 that are more important to the second network layer set are mistakenly clipped because they are less important in the first network layer set, since the information of the one or more tokens is recycled into the value of the virtual token, the computing node can also ensure the accuracy of the AI model's reasoning based on the tokens in token set 2 by performing calculations based on the virtual tokens in the second network layer set, and can approximately achieve the accuracy of the AI model's reasoning based on the original token sequence 1 in the second network layer set. At the same time, the total number of tokens involved in the calculation in the second network layer set (including virtual tokens and non-virtual tokens) can effectively reduce the resource consumption generated during the compression process of the AI representation sequence.

Furthermore, during the training of the AI model, the computing node may perform a forward calculation process according to the method flow described in steps S301 to S305 above, and update the parameter values in each network layer of the AI model based on the difference between the result calculated in the forward calculation phase and the actual result. The updated parameter values in the first network layer set include the value of the target parameter, which is a parameter for calculating the importance of each token in token sequence 1 in the first network layer set.

In this embodiment, when the computing node updates the value of the target parameter, it can improve the differentiation of the importance of different tokens calculated by the target parameter.

In specific implementation, the computing node may include a variance regularization term corresponding to the target parameter in the loss function used to train the AI model. The loss function is used to calculate the loss value, which can be used to measure the difference between the inference result calculated by the AI model in the forward calculation phase and the actual result corresponding to token sequence 1. Normally, when the loss value is small enough (such as less than a preset value), the AI model training ends. The variance regularization term is used to enhance the discrimination of the importance of the target parameter calculated for different tokens.

In this way, in the forward calculation stage, after the computing node calculates the importance score of each token using the target parameters, it can calculate the variance of the importance scores corresponding to the first network layer set based on the importance scores of each token, and bring the variance into the loss function to calculate the loss value, and update the value of the target parameter in the first network layer set according to the loss value.

In the embodiment shown in FIG2 above, the example of a computing node performing a single pruning of a token sequence during the forward computation phase is used for illustration. In other embodiments, the computing node may perform multiple pruning of the token sequence at different network layer sets to further reduce the number of resources required during the forward computation phase. This is exemplified below with reference to FIG7 .

7 , which shows a flowchart of another method for sequence compression. As shown in FIG7 , the method may specifically include:

S701: The computing node obtains token sequence 1, which is a sequence obtained based on the input data of the AI model.

S702: The computing node evaluates the importance of different tokens in token sequence 1 in the first network layer set, wherein the AI model includes the first network layer set and the second network layer set, and the first network layer set and the second network layer set are cascaded front and back.

S703: The computing node divides the tokens in the token sequence 1 into a token set 1 and a token set 2 according to the importance of each token.

Among them, the importance of the tokens in token set 2 in the first network layer set is higher than the importance of the tokens in token set 1 in the first network layer set of the AI model.

S704: The computing node determines m virtual tokens according to the tokens in token set 1, wherein the number of the m virtual tokens is less than the number of tokens in token set 1.

S705: The computing node uses the compressed token sequence composed of the m virtual tokens and the tokens in the token set 2 as the input sequence of the second network layer set.

The specific implementation process of steps S701 to S705 can be found in the relevant description of steps S301 to S305 above, and will not be repeated here.

S706: The computing node evaluates the importance of different tokens in token set 2 in the second network layer set, wherein the AI model also includes a third network layer set, and the second network layer set and the third network layer set are cascaded front and back.

S707: The computing node divides the tokens in token set 2 into subset 1 and subset 2 according to the importance of each token.

Among them, the importance of the tokens in subset 2 in the second network layer set is higher than the importance of the tokens in subset 1 in the second network layer set.

As an implementation example, when importance is measured using an importance score, the computing node may calculate the importance score of each token in token set 2 within the second network layer set. This importance score indicates the importance of the token within the second network layer set, with a higher importance score indicating a higher importance. The computing node may then compare the importance score of each token within the second network layer set with threshold 2. For tokens with an importance score less than threshold 2, the computing node may assign the token to subset 1; for tokens with an importance score greater than or equal to threshold 2, the computing node may assign the token to subset 2. In this embodiment, the manner in which the computing node divides the multiple tokens in token set 2 can be described in the description of the division of token sequence 1 in the embodiment of FIG. 3 above and is not further elaborated here. Threshold 2 can be a value pre-set by a technician or a value learned by the AI model during training, and this is not limited to this. Furthermore, the thresholds used to measure token importance in different network layer sets can be the same or different, such as threshold 1 being greater than threshold 2.

In this embodiment, the tokens in the subset 1 with lower importance are the tokens that need to be pruned in the second network layer, while the tokens in the subset 2 with higher importance will be passed to the subsequent network layers in the AI model for calculation.

S708: The computing node determines p virtual tokens based on the tokens in subset 1, where the number of the p virtual tokens is less than the number of tokens in subset 1.

Among them, the value of p can be a preset value. For example, before performing reasoning based on the input token sequence 1, the computing node can pre-configure m virtual tokens (such as 4 virtual tokens) for the first network layer set, and pre-configure p virtual tokens (such as 2 virtual tokens) for the second network layer set. Then, the computing node can insert the configured (m+p) virtual tokens into the token sequence 1. The specific implementation method can refer to the description of the relevant parts about inserting virtual tokens in the embodiment shown in Figure 3 above, which will not be repeated here. Among them, the configured (m+p) virtual tokens in the first network layer set and the second network layer set may not participate in the importance calculation.

Alternatively, the value of p can be dynamically determined by the computing node. For example, after the computing node divides the multiple tokens in token set 2 and obtains subsets 1 and 2, it can determine the value of p based on the number of tokens included in subset 1. For example, when the number of tokens included in subset 1 is 10, the value of p determined by the computing node can be 2. The computing node can then insert the p virtual tokens into the token sequence corresponding to the multiple tokens in subset 1. The positions of the p virtual tokens inserted by the computing node in the token sequence are not limited.

For the configured p virtual tokens, the computing node can determine the values of the p virtual tokens based on the values of the tokens in subset 1. For example, the computing node can map the token values in subset 1 to the values of the p virtual tokens using the FLP algorithm or other algorithm. The specific implementation method can be found in the description of step S304 in the embodiment of FIG. 3 above, and is not further described here.

S709: The computing node uses the compressed token sequence composed of the m virtual tokens, the p virtual tokens and the tokens in the subset 2 as the input sequence of the third network layer set.

Specifically, the computing node can concatenate (m+p) virtual tokens and the tokens in subset 2 into a new compressed token sequence, that is, further compress the token sequence, and use the new compressed token sequence as the input sequence of the third network layer set, so as to pass the compressed token sequence to the third network layer set for further calculation.

It is worth noting that in this embodiment, the computing node may prune the token sequence multiple times at different network layers, and each time the token sequence is pruned, a new virtual token is used to retrieve the pruned token information. In other embodiments, the computing node may use the same virtual token to retrieve the information of different tokens pruned each time the token sequence is pruned multiple times.

Taking the example of token sequence pruning in the first and second network layer sets, after the computing node divides token sequence 1 into token set 1 and token set 2, it can use m virtual tokens to recover token information from the pruned token set 1 and pass the m virtual tokens and tokens from token set 2 to the second network layer set for calculation. Then, in the second network layer set, the computing node divides the multiple tokens from token set 2 into subset 1 and subset 2, and still uses the m virtual tokens to recover token information from the pruned subset 1. For example, the computing node can first use the FLP algorithm to map the values of the tokens in the pruned subset 1 to the values of the m virtual tokens. Then, the computing node can sum the values of the m virtual tokens obtained by mapping with the current values of the m virtual tokens to calculate the new values of the m virtual tokens. Finally, the computing node passes the m virtual tokens and tokens from subset 2 to the third network layer set in the AI model for calculation. In this way, in the forward calculation phase, during the process of multiple token pruning, the computing node uses m virtual tokens to recycle the information of the pruned tokens each time, thereby further reducing the number of tokens participating in the calculation in the network layer set and reducing the number of resources required in the forward calculation phase.

Furthermore, during AI model training, the computing nodes can update the parameter values used to calculate token importance scores across multiple network layers. In this case, the computing nodes can add a variance regularization term to the AI model's corresponding loss function for these parameters to enhance the discriminability of the importance scores of different tokens calculated using these parameters.

In a specific implementation, taking the updating of the parameter values used to calculate importance scores in the first and second network layer sets as an example, in the forward computation phase, after calculating the importance score of each token using the parameters in the first network layer set, the computing node can calculate the variance 1 of the importance score corresponding to the first network layer set based on the importance scores of each token. Then, using the parameters in the second network layer set, the computing node calculates the importance score of each token (excluding virtual tokens) passed to the second network layer set, and calculates the variance 2 of the importance score corresponding to the second network layer set based on the importance scores of each token passed to the second network layer set. Next, the computing node can calculate the target variance based on variance 1 and variance 2. For example, the computing node can perform a weighted sum of variance 1 and variance 2, and the resulting sum is the target variance; the weight of variance 1 is greater than the weight of variance 2. Finally, the computing node can substitute the target variance into the loss function to calculate the loss value, and based on this loss value, update the values of the parameters used to calculate importance scores in the first and second network layer sets.

It is worth noting that other reasonable step combinations that can be thought of by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be familiar with that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by this application.

The above describes the representation sequence compression method provided in the embodiment of the present application in conjunction with Figures 1 to 7. Next, the structure of the representation sequence compression device and the computing device provided in the embodiment of the present application will be described in conjunction with the accompanying drawings.

8 , which shows a schematic diagram of the structure of a processor, the representation sequence compression device 800 includes:

An acquisition module 801 is configured to acquire a first token sequence, where the first token sequence is a sequence obtained based on input data of an artificial intelligence (AI) model, wherein the AI model includes a first network layer set and a second network layer set, and the first network layer set and the second network layer set are cascaded.

An evaluation module 802 is configured to evaluate the importance of different tokens in the first token sequence in the first network layer set;

A division module 803 is configured to divide the tokens in the first token sequence into a first token set and a second token set according to their importance;

Determination module 804 is used to determine at least one first virtual token based on the tokens in the first token set, where the number of the at least one virtual token is less than the number of tokens in the first token set, and use a compressed token sequence composed of the at least one first virtual token and the tokens in the second token set as an input sequence of the second network layer set.

In a possible implementation, the determination module 804 is configured to determine a value of at least one first virtual token according to values of tokens in the first token set.

In a possible implementation, the determining module 804 is configured to:

Determining the number of at least one first virtual token based on the number of tokens in the first token set;

The value of at least one first virtual token is determined according to the value of the token in the first token set.

In one possible implementation, the first network layer set includes a target parameter, and the target parameter is used to calculate the importance of a token in the first token sequence in the first network layer set;

The representation sequence compression apparatus 800 further includes a training module 805, configured to:

During the training of the AI model, the loss value is calculated according to the loss function, which includes the variance regularization term corresponding to the target parameter;

According to the loss value, update the target parameters.

In a possible implementation, the tokens in the first token sequence include words or image blocks.

In a possible implementation, the tokens in the first token sequence include words, and at least one first virtual token is located at a starting position in the compressed token sequence.

In a possible implementation, the tokens in the first token sequence include image blocks, and at least one first virtual token is located in the middle of the compressed token sequence.

In one possible implementation, the AI model further includes a third network layer set, and the second network layer set and the third network layer set are cascaded front and back;

The evaluation module 802 is further configured to evaluate the importance of different tokens in the second token set in the second network layer set;

The division module 803 is further configured to divide the tokens in the second token set into a first subset and a second subset according to the importance of different tokens in the second token set in the second network layer set;

The determination module 804 is also used to determine at least one second virtual token based on the tokens in the first subset, where the number of at least one virtual token is less than the number of tokens in the first subset, and use a compressed token sequence composed of at least one first virtual token, at least one second virtual token and the tokens in the second subset as an input sequence of the third network layer set.

In one possible implementation, the AI model further includes a third network layer set, and the second network layer set and the third network layer set are cascaded front and back;

The evaluation module 802 is further configured to evaluate the importance of different tokens in the second token set in the second network layer set;

The division module 803 is further configured to divide the tokens in the second token set into a first subset and a second subset according to the importance of different tokens in the second token set in the second network layer set;

The determination module 804 is also used to determine the value of at least one first virtual token in the second network layer set based on the token in the first subset, and use the compressed token sequence composed of at least one first virtual token and the token in the second subset as the input sequence of the third network layer set.

In one possible implementation, the representation sequence compression device 800 is applied to the deployment stage of the AI model, and the representation sequence compression device 800 further includes an update module 806 for updating the model calculation graph corresponding to the AI model according to the compressed token sequence.

In a possible implementation, the representation sequence compression apparatus 800 is applied to the development phase of an AI model.

Since the representation sequence compression device 800 shown in Figure 8 corresponds to the computing node in the embodiment shown in Figure 3 or Figure 7 above, the specific implementation method of the representation sequence compression device 800 shown in Figure 8 and the technical effects thereof can be found in the relevant descriptions in the embodiment shown in Figure 3 or Figure 7 above, and will not be repeated here.

FIG9 is a schematic diagram of the hardware structure of a computing device 900 provided in the present application. The computing device 900 can, for example, implement the computing nodes in the embodiments shown in FIG3 or FIG7 .

As shown in Figure 9, the computing device 900 includes a processor 901, a memory 902, and a communication interface 903. The processor 901, the memory 902, and the communication interface 903 communicate via a bus 904, and may also communicate via other means such as wireless transmission. The memory 902 is used to store instructions, and the processor 901 is used to execute the instructions stored in the memory 902. Furthermore, the computing device 900 may also include a memory unit 905, which may be connected to the processor 901, the storage medium 902, and the communication interface 903 via a bus 904. The memory 902 stores program code, and the processor 901 may call the program code stored in the memory 902 to perform the following operations:

Obtaining a first representation sequence, where the first representation sequence is a sequence obtained based on input data of an artificial intelligence (AI) model, wherein the AI model includes a first network layer set and a second network layer set, and the first network layer set and the second network layer set are cascaded in a front-to-back manner;

evaluating the importance of different representations in the first representation sequence in the first network layer set;

dividing the representations in the first representation sequence into a first representation set and a second representation set according to the importance;

determining at least one first virtual representation from the representations in the first representation set, the number of the at least one virtual representation being less than the number of representations in the first representation set;

A compressed representation sequence formed according to the at least one first virtual representation and representations in the second representation set is used as an input sequence of the second network layer set.

It should be understood that in this embodiment, the processor 901 may be a CPU, or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

The memory 902 may include a read-only memory and a random access memory, and provides instructions and data to the processor 901. The memory 902 may also include a nonvolatile random access memory.

The memory 902 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. The nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

The communication interface 903 is used to communicate with other devices connected to the computing device 900. In addition to the data bus, the bus 904 may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus 904 in the figure.

It should be understood that the computing device 900 according to the embodiment of the present application may correspond to the representation sequence compression device 800 in the embodiment of the present application, and may correspond to executing the method executed by the computing node in the method shown in Figure 3 or Figure 7 in the embodiment of the present application. The above-mentioned and other operations and/or functions implemented by the computing device 900 are respectively for implementing the process of the corresponding method in Figure 3 or Figure 7. For the sake of brevity, they will not be repeated here.

The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned representation sequence compression method.

The present application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the computer program product generates, in whole or in part, the process or function described in the present application.

The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

The computer program product may be a software installation package, which may be downloaded and executed on a computing device when any of the aforementioned methods for representing sequence compression is required.

The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

The terms used in the above embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and claims of this application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless the context clearly indicates otherwise. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; the character "/" generally indicates that the objects associated with each other are in an "or" relationship. In the embodiments of the present application. "Simultaneously" means within the same time period, including situations at the same time. The terms "first", "second", etc. in the specification, claims and drawings of this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, and this is merely a way of distinguishing objects with the same properties when describing them in the embodiments of the present application.

References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims (25)

A method for compressing a representation sequence, characterized in that the method comprises: Obtaining a first representation sequence, where the first representation sequence is a sequence obtained based on input data of an artificial intelligence (AI) model, wherein the AI model includes a first network layer set and a second network layer set, and the first network layer set and the second network layer set are cascaded in a front-to-back manner; evaluating the importance of different representations in the first representation sequence in the first network layer set; dividing the representations in the first representation sequence into a first representation set and a second representation set according to the importance; determining at least one first virtual representation from the representations in the first representation set, the number of the at least one virtual representation being less than the number of representations in the first representation set; A compressed representation sequence formed according to the at least one first virtual representation and representations in the second representation set is used as an input sequence of the second network layer set. The method according to claim 1, wherein determining at least one first virtual representation based on representations in the first representation set comprises: A value of the at least one first virtual representation is determined based on the values of representations in the first set of representations. The method according to claim 1, wherein determining at least one first virtual representation based on representations in the first representation set comprises: determining a number of the at least one first virtual representation based on the number of representations in the first representation set; A value of the at least one first virtual representation is determined based on the values of representations in the first set of representations. The method according to any one of claims 1 to 3, characterized in that the first network layer set includes a target parameter, and the target parameter is used to calculate the importance of the representations in the first representation sequence in the first network layer set; The method further comprises: During the training of the AI model, a loss value is calculated according to a loss function, where the loss function includes a variance regularization term corresponding to the target parameter; The target parameter is updated according to the loss value. The method according to any one of claims 1 to 4, characterized in that the representations in the first representation sequence include words or image blocks. The method according to claim 5, characterized in that the representation includes words and the at least one first virtual representation is located at a starting position in the compressed representation sequence. The method according to claim 5, characterized in that the representation includes image blocks and the at least one first virtual representation is located at an intermediate position in the sequence of compressed representations. The method according to any one of claims 1 to 7, characterized in that the AI model further includes a third network layer set, the second network layer set and the third network layer set are cascaded front and back, and the method further includes: evaluating the importance of different representations in the second representation set in the second network layer set; Dividing the representations in the second representation set into a first subset and a second subset according to importance of different representations in the second network layer set; determining at least one second virtual representation from the representations in the first subset, the number of the at least one virtual representation being less than the number of representations in the first subset; A compressed representation sequence formed according to the at least one first virtual representation, the at least one second virtual representation and the representations in the second subset is used as an input sequence of the third network layer set. The method according to any one of claims 1 to 7, characterized in that the AI model further includes a third network layer set, the second network layer set and the third network layer set are cascaded front and back, and the method further includes: evaluating the importance of different representations in the second representation set in the second network layer set; Dividing the representations in the second representation set into a first subset and a second subset according to importance of different representations in the second network layer set; determining a value of the at least one first virtual representation in the second set of network layers based on the representations in the first subset; A compressed representation sequence formed according to the at least one first virtual representation and the representations in the second subset is used as an input sequence of the third network layer set. The method according to any one of claims 1 to 9 is characterized in that the method is applied to the deployment stage of the AI model, and the method further includes: updating the model computation graph corresponding to the AI model according to the compressed representation sequence. The method according to any one of claims 1 to 9, characterized in that the method is applied to the development stage of the AI model. A representation sequence compression device, characterized in that the device comprises: an acquisition module, configured to acquire a first representation sequence, where the first representation sequence is a sequence obtained based on input data of an artificial intelligence (AI) model, wherein the AI model includes a first set of network layers and a second set of network layers, wherein the first set of network layers and the second set of network layers are cascaded in series; an evaluation module, configured to evaluate the importance of different representations in the first representation sequence in the first network layer set; a division module, configured to divide the representations in the first representation sequence into a first representation set and a second representation set according to the importance; A determination module is used to determine at least one first virtual representation based on the representations in the first representation set, where the number of the at least one virtual representation is less than the number of representations in the first representation set, and to use a compressed representation sequence composed of the at least one first virtual representation and the representations in the second representation set as an input sequence of the second network layer set. The device according to claim 12 is characterized in that the determination module is used to determine the value of the at least one first virtual representation based on the values of the representations in the first representation set. The device according to claim 12, wherein the determining module is configured to: determining a number of the at least one first virtual representation based on the number of representations in the first representation set; A value of the at least one first virtual representation is determined based on the values of representations in the first set of representations. The apparatus according to any one of claims 12 to 14, wherein the first network layer set includes a target parameter, and the target parameter is used to calculate the importance of the representations in the first representation sequence in the first network layer set; The apparatus further comprises a training module for: During the training of the AI model, a loss value is calculated according to a loss function, where the loss function includes a variance regularization term corresponding to the target parameter; The target parameter is updated according to the loss value. The device according to any one of claims 12 to 15, characterized in that the representations in the first representation sequence include words or image blocks. The apparatus of claim 16, wherein the representations include words and the at least one first virtual representation is located at a starting position in the compressed representation sequence. The apparatus of claim 16, wherein the representations comprise image blocks, and wherein the at least one first virtual representation is located at an intermediate position in the sequence of compressed representations. The device according to any one of claims 12 to 18, characterized in that the AI model further includes a third network layer set, and the second network layer set and the third network layer set are cascaded front and back; The evaluation module is further configured to evaluate the importance of different representations in the second representation set in the second network layer set; The division module is further configured to divide the representations in the second representation set into a first subset and a second subset according to the importance of different representations in the second network layer set; The determination module is further used to determine at least one second virtual representation based on the representations in the first subset, where the number of the at least one virtual representation is less than the number of representations in the first subset, and to use a compressed representation sequence composed of the at least one first virtual representation, the at least one second virtual representation and the representations in the second subset as an input sequence of the third network layer set. The device according to any one of claims 12 to 18, characterized in that the AI model further includes a third network layer set, and the second network layer set and the third network layer set are cascaded front and back; The evaluation module is further configured to evaluate the importance of different representations in the second representation set in the second network layer set; The division module is further configured to divide the representations in the second representation set into a first subset and a second subset according to the importance of different representations in the second network layer set; The determination module is also used to determine the value of the at least one first virtual representation in the second network layer set based on the representation in the first subset, and use the compressed representation sequence composed of the at least one first virtual representation and the representation in the second subset as the input sequence of the third network layer set. The device according to any one of claims 12 to 20 is characterized in that the device is applied to the deployment stage of the AI model, and the device further includes an update module for updating the model calculation graph corresponding to the AI model according to the compressed representation sequence. The device according to any one of claims 12 to 20 is characterized in that the device is applied to the development stage of the AI model. A computing device, characterized in that the computing device includes a processor and a memory; The memory is used to store instructions, and the processor executes the instructions stored in the memory to enable the computing device to perform the method according to any one of claims 1 to 11. A computer-readable storage medium, characterized by comprising instructions, which, when executed on a computing device, cause the computing device to execute the method according to any one of claims 1 to 11. A computer program product comprising instructions, characterized in that when the computer program product is run on at least one computing device, the computer program product causes the at least one computing device to perform the method according to any one of claims 1 to 11.