WO2025031095A1 - Method for updating parameters of pre-trained model and data processing method for pre-trained model - Google Patents

Abstract

The present disclosure relates to the fields of large model technology and data processing. Disclosed are a method for updating parameters of a pre-trained model and a data processing method for a pre-trained model. The method for updating the parameters of the pre-trained model comprises: acquiring feature data output by a backbone network of the pre-trained model, wherein the backbone network comes from an initial backbone network; calling an initial bypass network to convert the feature data, wherein the initial bypass network is constructed on the basis of at least one tuning module, and the tuning module is extracted from the initial backbone network; updating parameters of the initial bypass network on the basis of the converted feature data to obtain a target bypass network, wherein in the process of updating the parameters of the initial bypass network, data streams of the initial bypass network are independent of data streams of the backbone network, and the parameters of the initial bypass network are used for representing the effect of the tuning module on the feature data.

Description

Updating parameters in pre-trained models and data processing methods for pre-trained models

Cross-references

This disclosure claims the priority of the Chinese patent application filed with the China Patent Office on August 9, 2023, with application number 202311003358.5 and invention name “Updating parameters in a pre-trained model and data processing method for a pre-trained model”, the entire contents of which are incorporated by reference in this disclosure.

Technical Field

The present disclosure relates to the field of large models and data processing, and in particular, to a method for updating parameters in a pre-trained model and processing data of the pre-trained model.

Background Art

At present, for efficient parameter migration of large-scale pre-trained models, different parts of the original training architecture of the large-scale pre-trained models are usually lightweight adjusted according to the different tasks performed by the pre-trained models. However, when training the pre-trained models, it is still necessary to perform redundant calculations on different tasks through the initial backbone network, which makes the model training process consume a lot of resources and leads to low efficiency in obtaining feedback results in generative dialogue products. As a result, there are technical problems such as high resource consumption and low computational efficiency in the model training process.

To address the above-mentioned problems, no effective solution has been proposed yet.

Summary of the invention

The disclosed embodiments provide a method for updating parameters in a pre-trained model and processing data of the pre-trained model, so as to at least solve the technical problems of high resource consumption and computational efficiency in the training process of the model.

According to one aspect of an embodiment of the present disclosure, a method for updating parameters in a pre-trained model is provided. The method may include: obtaining feature data output by a backbone network of the pre-trained model, wherein the backbone network comes from an initial backbone network; calling an initial bypass network to convert the feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; updating the parameters of the initial bypass network based on the converted feature data to obtain a target bypass network, wherein, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the feature data.

According to another aspect of the embodiment of the present disclosure, a data processing method for a pre-trained model is also provided. The method may include: responding to an inquiry message received in a generative interactive interface, wherein the inquiry message includes at least: keywords of text generation information; calling the backbone network of the pre-trained model to at least analyze the text generation information and output text feature data, wherein the backbone network comes from the initial backbone network; calling the target bypass network to convert the text feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted text feature data, generating at least one reply result that matches the inquiry instruction; displaying the reply result The generated interactive interface.

According to another aspect of the embodiment of the present disclosure, another data processing method for a pre-trained model is also provided. The method may include: obtaining feature data obtained by processing conditional data by a backbone network in a pre-trained model, wherein the backbone network comes from an initial backbone network, and the conditional data is used to determine the generation conditions of target data; calling a target bypass network to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, generating target data corresponding to the conditional data, wherein the type of the target data includes at least one of the following: text information, image information, video information, and voice information.

According to another aspect of the embodiment of the present disclosure, another data processing method for a pre-trained model is also provided. The method may include: inputting multimodal information in a dialogue interface, wherein the type of the multimodal information includes at least one of the following: text information containing character information, video frame information containing frame image information, and audio information; calling the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, wherein the backbone network comes from the initial backbone network; calling the target bypass network to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, generating reply information corresponding to the multimodal information, wherein the type of the reply information includes at least one of the following: text information, image information, video information, and voice information.

According to another aspect of the embodiment of the present disclosure, a data processing system of a pre-trained model is also provided. The system may include: a client, configured to display a dialogue interface and capture multimodal information input in the dialogue interface, wherein the type of multimodal information includes at least one of the following: inquiry information containing character information, video frame information containing frame image information, and audio information; a server, configured to call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, and call the target bypass network to convert the feature data, and based on the converted feature data, generate reply information corresponding to the multimodal information, wherein the backbone network comes from the initial backbone network, the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network, and the type of reply information includes at least one of the following: text information, image information, video information, and voice information.

According to another aspect of the embodiment of the present disclosure, another data processing method of a pre-trained model is also provided. The method may include: inputting a speech to be converted on a virtual reality VR device or an augmented reality AR device; extracting feature data from the speech to be converted using a backbone model in the pre-trained model, wherein the backbone network comes from an initial backbone network; calling a target bypass network in the pre-trained model to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, determining the image information corresponding to the speech to be converted; using the image information to activate the VR device or AR device, and displaying the image information in the VR device or AR device.

According to another aspect of an embodiment of the present disclosure, an electronic device is also provided, which may include a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the above-mentioned computer-executable instructions are executed by the processor, the above-mentioned method of any one of the above-mentioned items is implemented.

According to another aspect of an embodiment of the present disclosure, a processor is further provided, the processor being used to run a program, wherein any one of the above methods is executed when the program is running.

According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute any one of the above methods.

According to another aspect of an embodiment of the present disclosure, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the methods described above are implemented.

In an embodiment of the present disclosure, feature data output by a backbone network of a pre-trained model is obtained, wherein the backbone network comes from an initial backbone network; an initial bypass network is called to convert the feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; parameters of the initial bypass network are updated based on the converted feature data to obtain a target bypass network, wherein, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the feature data. That is, in this embodiment, a tuning module extracted from the backbone network is used to construct an initial bypass network. At the same time, during the adjustment of the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is cut off to obtain a target bypass network (for example, a new tuning module) independent of the backbone network. As a result, during the training of the initial bypass network, there is no need to further calculate the parameter gradient of the backbone network, thereby saving memory and improving the training speed, thereby achieving the technical effect of reducing resource consumption during the training process of the model, and solving the technical problems of high resource consumption and computing efficiency during the training process of the model.

It is easily noted that the above general description and the following detailed description are merely for exemplifying and explaining the present disclosure, and do not constitute a limitation of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

FIG1 is a schematic diagram of an application scenario of a method for updating parameters in a pre-training model according to an embodiment of the present disclosure;

FIG2 is a flow chart of a method for updating parameters in a pre-training model according to an embodiment of the present disclosure;

FIG3 is a flow chart of a data processing method for a pre-training model according to an embodiment of the present disclosure;

FIG4 is a flow chart of another method for processing data of a pre-training model according to an embodiment of the present disclosure;

FIG5 is a flow chart of another method for processing data of a pre-training model according to an embodiment of the present disclosure;

FIG6 is a schematic diagram of a data processing result of a pre-training model according to an embodiment of the present disclosure;

FIG7 is a flow chart of another method for processing data of a pre-training model according to an embodiment of the present disclosure;

FIG8( a ) is a schematic diagram of a data processing system for a pre-training model according to an embodiment of the present disclosure;

FIG8( b ) is a schematic diagram of a tuning module according to an embodiment of the present disclosure;

FIG8( c ) is a schematic diagram of another tuning module according to an embodiment of the present disclosure;

FIG9 is a schematic diagram of a pre-training model according to an embodiment of the present disclosure;

FIG10 is a schematic diagram of another pre-training model according to an embodiment of the present disclosure;

FIG11 is a schematic diagram of a device for updating parameters in a pre-training model according to an embodiment of the present disclosure;

FIG12 is a schematic diagram of a data processing device for a pre-training model according to an embodiment of the present disclosure;

FIG13 is a schematic diagram of another data processing device for a pre-training model according to an embodiment of the present disclosure;

FIG14 is a schematic diagram of another data processing device for a pre-training model according to an embodiment of the present disclosure;

FIG15 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure;

FIG16 is a block diagram of an electronic device according to a method for updating parameters in a pre-training model according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

In order to enable those skilled in the art to better understand the scheme of the present disclosure, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present disclosure.

It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

The technical solution provided by the present disclosure is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than ten trillion model parameters. The large model can also be called a cornerstone model/foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision and other fields, and can be specifically applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiments of the present disclosure, data processing through pre-trained models in scenarios such as machine intelligence technology, basic visual intelligence, and video artificial intelligence (AI) are used as examples for explanation.

First, some nouns or terms that appear in the process of describing the embodiments of the present disclosure are subject to the following explanations:

Pre-trained models can be produced after large-scale training and tuning on representative data sets. They can be used as initialization models for training other downstream tasks, and can be used to speed up training or achieve better results.

Foundation Model: A model trained in different fields using large amounts of data, powerful computing power, and a well-designed structure, which can be used to process downstream tasks.

Transfer Learning can improve the learning of new tasks by transferring knowledge from related tasks that have already been learned;

Parameter-efficient transfer learning (PETL) refers to a tuning training method that modifies a small number of parameters or adds a small number of additional parameters based on a pre-trained model.

Memory-efficient Transfer Learning (METL) is a method for fine-tuning training methods based on pre-trained models using relatively small amounts of memory.

The deep learning model (Transformer) can be a network architecture consisting of several layers of encoders and decoders;

Vision Transformer can be a network architecture, which can be a migration application of Transformer architecture in the visual field;

Submodule (TransformerBlock), which can be a submodule of Transformer, can be composed of multi-head attention and feedforward network;

Multi-head Attention (MHA) can be a submodule in a Transformer, which can be a module that calculates the related vectors of query, key, and value;

Feed-forward Network (FFN) can be a submodule in a Transformer and can be composed of multiple fully connected layers and activation functions;

Multi-Layer Perceptron (MLP) can be a network module. It can be composed of one or more hidden layers and activation functions;

Adapter, which can be a tuning training method, can act on a feedforward network and can contain a small module consisting of two fully connected layers and an activation function;

Prompt tuning method (Prompt) can be a tuning training method, which can refer to tuning training with the help of a learnable parameter spliced with the input;

Prefix tuning can be a tuning method that uses two learnable parameters concatenated with the key and value in the multi-head attention layer to perform tuning training;

Discriminative tasks can refer to tasks that discriminate input data, such as discriminating Y based on X, for example, image classification tasks;

Generative tasks can be tasks that generate operations on input data, such as generating Y based on X, for example, image generation tasks;

Convolutional Neural Network (U-Net) can be a network architecture that can be composed of a downsampling encoder, an upsampling decoder, and skip connections;

The deep learning text-to-image generation model (StableDiffusion) can be a network structure for generation tasks, which can be composed of an autoencoder, a text encoder, and a U-Net structure.

According to an embodiment of the present disclosure, a method for updating parameters in a pre-trained model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

Considering the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the method for updating parameters in the above-mentioned pre-trained model provided in the embodiment of the present disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. Figure 1 is a schematic diagram of an application scenario of a method for updating parameters in a pre-trained model according to an embodiment of the present disclosure. In the application scenario shown in Figure 1, the large model is deployed in a server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface, for example, by inputting classification conditions in the graphical user interface, so as to realize the call of the large model, and then realize the method provided in the embodiment of the present disclosure.

In the disclosed embodiment, the system composed of the client device and the server can execute the following steps: the client device obtains the image to be classified. The server executes step S101 to obtain the feature data obtained by processing the image to be classified by the backbone network in the pre-trained model; step S102, calling the target bypass network to convert the feature data, wherein the target bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the backbone network; step S103, based on the converted feature data, determining the category to which the image to be classified belongs. The server 10 can transmit the category to which the image to be classified belongs to the client and display it in the client's graphical user interface. Need It is noted that, when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiments of the present disclosure can be performed in the client device.

In the above operating environment, the present disclosure provides a method for updating parameters in a pre-trained model as shown in Figure 2 in the case of a discriminant task. Figure 2 is a flow chart of a method for updating parameters in a pre-trained model according to an embodiment of the present disclosure. As shown in Figure 2, the method may include the following steps.

Step S202, obtaining feature data output by a backbone network of the pre-trained model, wherein the backbone network comes from an initial backbone network.

In the technical solution provided in the above step S202 of the present disclosure, the backbone network can be obtained from the initial backbone network. Feature data output by the backbone network can be obtained. Among them, the pre-trained model can also be called a pre-trained network, which can be a model produced after large-scale training and tuning on a representative data set, which can be used as an initialization model when training other downstream tasks, and can be used to speed up the training speed or obtain better results. Feature data can also be called output features, which can be data obtained by the backbone network processing voice, image and other information, and can be represented by x ₀ and x _1. This is only for example, and no specific restrictions are made on the representation of feature data. The backbone network can be used for tasks such as image classification, target detection and semantic segmentation, and can include feedforward network modules, multi-head attention modules, etc., and can be a visual task processing network (Vision Transformer, referred to as ViT), a convolutional neural network (Convolutional Neural Network, referred to as CNN), etc. This is only for example, and no specific restrictions are made on the content contained in the backbone network and the type of the backbone network. The feature data may be an abstract feature representation, which may be a vector, a pixel-level label, a bounding box, a category probability, etc. This is only an example and does not impose any specific restrictions on the type of feature data.

Optionally, the backbone network can output feature data during the forward propagation process.

For example, taking the image classification task as an example, the backbone network can be a convolutional neural network, and the input of the backbone network can be an image. After a series of convolutional layers, pooling layers, activation functions and other operations, a set of abstract feature representations is obtained.

Step S204, calling the initial bypass network to convert the characteristic data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

In the technical solution provided in the above step S204 of the present disclosure, at least one tuning module can be extracted from the backbone network, and the extracted at least one tuning module can be constructed to obtain an initial bypass network. The constructed initial bypass network can be called to convert the feature data. Among them, the tuning module can also be called a sub-module, which can be extracted from the initial backbone network and can be used to improve the accuracy of the pre-trained model prediction. The initial bypass model can also be called a tuning framework (Residual Tuning, abbreviated as Res-Tuning), which can be represented by Bypass.

Optionally, a tuning module can be extracted from the initial backbone network, and a bypass network can be constructed using the tuning module to obtain an independent bypass network and backbone network. The feature data output by the backbone network can be used as the input of the initial bypass network, and the initial bypass network can be called to convert the feature data.

Step S206: updating the parameters of the initial bypass network based on the converted characteristic data to obtain a target bypass network. In the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the characteristic data.

In the technical solution provided in the above step S206 of the present disclosure, the converted feature data can be output to the pre-trained model to output the corresponding results, and based on the output results, the parameters of the initial bypass network are adjusted to obtain the target bypass network. Among them, the target bypass network can be a new tuning network independent of the backbone network. The parameters of the initial bypass network can be used to characterize the influence of different tuning modules on the feature data, and can be weights, biases, regularization parameters, etc., which are only examples here, and no specific restrictions are made on the types of parameters. The output results can be images, text, voice and other data recognized by the pre-trained model. This is only an example, and no specific restrictions are made on the types of output results. During the training process, the data flow of the initial bypass network is independent of the data flow of the backbone network. Therefore, the data flow of the initial bypass network cannot be transmitted to the backbone network.

Since the initial backbone network usually embeds the tuning module in the backbone network, the backbone network still needs to perform redundant calculations during each training process. For example, during the back propagation process, the gradient of the backbone network still needs to be further calculated, which wastes memory and inference overhead, and further leads to the technical problem that the training process of the pre-trained model consumes a long time. To solve the above problem, in this embodiment, the tuning module is extracted from the initial backbone network, and the initial bypass model is constructed using the tuning module to obtain an independent initial bypass network and backbone network. During the training of the initial bypass network, the data stream of the initial bypass network is prohibited from being transmitted to the backbone network, so as to independently train the initial bypass network and obtain the target bypass network, thereby achieving the technical effect of reducing resource consumption in the training process of the model and improving computing efficiency, and solving the technical problems of high resource consumption and computing efficiency in the training process of the model.

Optionally, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network can be prohibited from being transmitted to the backbone network, and the parameters of the initial bypass network can be adjusted based on the converted feature data through back propagation or stochastic gradient descent to obtain the target bypass network.

For example, the tuning modules can be extracted from the initial backbone network, and the extracted tuning modules can be combined and constructed to obtain the initial bypass network. The initial backbone network with the extracted tuning modules can be used as the backbone network to obtain the feature data output by the backbone network. The initial bypass network can be called to convert the feature data, and the converted feature data can be output to the pre-trained model to output the corresponding results. Based on the output results, the parameters of the initial bypass network can be adjusted to obtain the target bypass network.

In the disclosed embodiment, a tuning module extracted from a backbone network is used to construct an initial bypass network. At the same time, during the process of adjusting the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is cut off to obtain a target bypass network (i.e., a new tuning module) that is relatively independent of the backbone network. As a result, during the training of the initial bypass network, there is no need to further calculate the parameter gradient of the backbone network, thereby saving memory and improving the training speed, thereby achieving the technical effect of reducing resource consumption in the training process of the model and improving computing efficiency, and solving the technical problems of high resource consumption and computing efficiency in the training process of the model.

The above method of this embodiment is further introduced below.

As an optional implementation, step S204, calling the initial bypass network to convert the feature data, includes: calling the tuning module in the initial bypass network to adjust the feature data; and performing weighted summation on the adjusted feature data based on the weight of the initial bypass network.

In this embodiment, the tuning module in the initial bypass network can be called to adjust the feature data, and the adjusted feature data can be weighted summed based on the weight of the initial bypass network. The weight in the initial network bypass is continuously learned, and can be used to characterize the proportion of different tuning modules, and can also be called the λ coefficient, which can be in a state of continuous learning.

Optionally, during the training process of the initial bypass network, during forward propagation, the intermediate output of the backbone network (i.e., feature data) can be used as the input of the tuning module in the initial bypass network, the tuning module in the initial bypass network can be called to adjust the feature data, and the adjusted feature data can be weighted summed based on the parameters of the initial bypass network.

In this embodiment, a framework for synchronous combined training of a backbone network and an initial bypass network is used, and the initial bypass network is constructed by combining tuning modules extracted from the initial backbone network. During the forward propagation process of the initial bypass network in training, the feature data output by the backbone network is used, and during the reverse propagation process, the initial bypass network is disconnected from the backbone network, and only the initial bypass network is passed through, so that there is no need to further calculate the gradient of the backbone network, thereby achieving memory saving and improving the training speed, thereby achieving the purpose of saving memory occupied during data processing, and obtaining a new target bypass network independent of the backbone network.

As an optional implementation, the tuning module in the initial bypass network is called to adjust the characteristic data, including: calling the first vertical tuning module corresponding to the first layer of the backbone network in the backbone network to adjust the characteristic data output by the zeroth layer of the backbone network, and calling the first horizontal tuning module associated with the first vertical tuning module to adjust the characteristic data output by the first layer of the backbone network.

In this embodiment, there is a corresponding relationship between the number of tuning modules in the initial bypass network and the number of layers of the backbone network. The first vertical tuning module corresponding to the first layer of the backbone network in the backbone network can be called to adjust the feature data output by the zeroth layer of the backbone network in the backbone network, and the first horizontal tuning module associated with the first vertical tuning module is called to adjust the feature data output by the first layer of the backbone network. Among them, the first layer of the backbone network can be a coding layer (Decoder), a convolutional layer (block), etc., which is only an example here, and there is no specific restriction on the category of the first layer of the backbone network. The zeroth layer of the backbone network can be an intermediate layer network (Middle), a convolutional layer, etc., which is only an example here, and there is no specific restriction on the category of the zeroth layer of the backbone network. The first vertical tuning module corresponds to the first layer of the backbone network, and can be a vertical tuning module, which can also be called a vertical Res-Tuner. The first horizontal tuning module is associated with the first vertical tuning module, and can be a horizontal tuning module, which can also be called a horizontal Res-Tuner.

Optionally, a tuning module (Res-Tuner) can be extracted from the initial bypass network, and a complete initial bypass network can be constructed by combining different tuning modules. The first vertical tuning module corresponding to the first layer of the backbone network in the initial bypass network can be called to perform feature number analysis on the output of the zeroth layer of the backbone network. The feature data output by the first backbone network is adjusted, and the first horizontal tuning module associated with the first vertical tuning module is called to adjust the feature data output by the first backbone network. The feature data output by the zeroth backbone network can be represented by x _0. The feature data output by the first backbone network can be represented by x ₁ .

As an optional implementation, based on the parameters of the initial bypass network, the adjusted feature data is weightedly summed, including: based on the weight of the initial bypass network, the adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module are weighted summed to obtain the first feature data.

In this embodiment, the weight of the initial bypass network can be determined. The adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module are obtained. Based on the initial bypass network weight, the adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module are weighted summed to obtain the first feature data after the weighted summation.

As an optional implementation, updating the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network includes: in response to the first layer backbone network being the last layer network in the backbone network, updating the parameters of the initial bypass network based on the first characteristic data to obtain the target bypass network.

In this embodiment, it is determined whether the first-layer backbone network is the last layer of the backbone network. If the first-layer backbone network is the last layer of the backbone network, the first feature data can be used as the output of the initial bypass network, and the first feature data can be used as the training data for back propagation to adjust the parameters of the initial bypass network to obtain the target bypass network.

Optionally, the overall structure of the pre-trained model can be divided into two parts, one part is the backbone model, the parameters of which are frozen during the training process; the other part is the trainable initial bypass network (also referred to as the bypass structure), which can be composed of several tuning modules. Each layer of the backbone network has a horizontal tuning module and a vertical tuning module, which are respectively used to receive the intermediate feature data from the layer in the backbone network and the output data of the previous layer in the bypass network, and perform weighted summation and output to the next layer, until the last layer of features is output to the pre-trained model (for example, the discriminative task network) for the output of the corresponding results, and the output results are obtained. Based on the output results, the parameters of the initial bypass network can be updated to obtain the target bypass network.

For example, training data (e.g., labeled images, speech, and other data) can be obtained, and the backbone network can be used to process the training data and output feature data. The first vertical tuning module corresponding to the first-layer backbone network in the initial bypass network can be called to adjust the feature data output by the zero-th layer backbone network, and the first horizontal tuning module corresponding to the first vertical tuning module can be called to adjust the feature data output by the first-layer backbone network. Based on the weight of the initial bypass network, a weighted sum is taken between the adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module to obtain the first feature data after the weighted sum, and it is determined whether the first-layer backbone network is the last layer of the backbone network. If the first-layer backbone network is the last layer of the backbone network, the first feature data can be used as the output of the initial bypass network. The first feature data can be input into the output layer (e.g., Head) in the pre-trained model, and the output layer adjusts the first feature data. The data is converted to obtain the final training result, and back propagation calculation can be performed based on the training result to update the parameters of the initial bypass network to obtain the target bypass network.

As an optional implementation, updating the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network includes: in response to the last layer of the network in the non-backbone network of the first layer backbone network, calling the second vertical tuning module corresponding to the second layer backbone network in the backbone network to adjust the first characteristic data, and calling the second horizontal tuning module associated with the second vertical tuning module to adjust the characteristic data of the second layer backbone network in the backbone network; based on the weight of the initial bypass network, performing weighted summation between the adjusted first characteristic data corresponding to the second vertical tuning module and the adjusted characteristic data corresponding to the second horizontal tuning module to obtain the second characteristic data; in response to the last layer of the network in the non-backbone network of the second layer backbone network, executing the following steps: The invention discloses a method for transmitting a bypass network to a third layer of a backbone network, wherein the first layer of the backbone network is determined as the second layer of the backbone network; the second vertical tuning module is called to adjust the first characteristic data, and the second horizontal tuning module is called to adjust the characteristic data of the second layer of the backbone network; based on the weight of the initial bypass network, the adjusted first characteristic data corresponding to the second vertical tuning module and the adjusted characteristic data corresponding to the second horizontal tuning module are weightedly summed to obtain the second characteristic data, until the second layer of the backbone network is the last layer of the backbone network; the parameters of the initial bypass network are updated based on the converted characteristic data to obtain the target bypass network, including: the parameters of the initial bypass network are updated based on the second characteristic data to obtain the target bypass network.

In this embodiment, a weighted sum is performed between the adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module to obtain the first feature data after the weighted sum, and it is determined whether the first-layer backbone network is the last layer of the backbone network. If the first-layer backbone network is not the last layer of the backbone network, the first feature data can be used as input data of the second vertical tuning module corresponding to the second-layer backbone network, and the second vertical tuning module corresponding to the second-layer backbone network in the backbone network can be called to adjust the first feature data, and the second horizontal tuning module associated with the second vertical tuning module can be called to adjust the feature data of the second-layer backbone network in the backbone network. Based on the weight of the initial bypass network, a weighted sum can be performed between the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module to obtain the second feature data.

Optionally, this embodiment determines whether the second-layer backbone network is the last layer of the backbone network. If the second-layer backbone network is the last layer of the backbone network, the second feature data can be output to the output layer of the pre-trained model. The output layer converts the second feature data to obtain the final output result. Back-propagation calculation can be performed based on the output result to update the parameters of the initial bypass network to obtain the target bypass network. If the second-layer backbone network is not the last layer of the backbone network, the second-layer backbone network can be determined as the first-layer backbone network, and the third-layer backbone network in the backbone network can be determined as the second-layer backbone network; the second vertical tuning module is called to adjust the first feature data, and the second horizontal tuning module is called to adjust the feature data of the second-layer backbone network; based on the weight of the initial bypass network, the corresponding adjustment of the second vertical tuning module is performed. A weighted sum is performed between the first feature data after the training and the adjusted feature data corresponding to the second level tuning module to obtain the second feature data, until the second-layer backbone network is the last layer of the backbone network, and the obtained second feature data is output to the output layer of the pre-trained model. The result of the output layer is back-propagated to update the parameters of the initial bypass network to obtain the target bypass network.

Optionally, the second characteristic data output by the bypass network may be determined based on the following formula:

in, It can be used to represent the adjusted feature data corresponding to the first-level horizontal tuning module. It can be used to characterize the output of the bypass network at layer l. Res-Tuner(x _l ) can be used to characterize the output of the horizontal tuning module at layer l. It can be used to characterize the output of the vertical tuning module of the l-1th layer, _x0 can be used to characterize the characteristic data of the output of the zeroth layer backbone network, _xl can be used to characterize the output characteristics of the lth layer, and λ can be used to characterize the weight of the initial bypass network.

Optionally, the weight of the initial bypass network may be a learnable weight coefficient, which may be continuously changed during the training of the initial bypass network.

Optionally, in each network layer corresponding to the backbone network, the initial bypass network uses a horizontal tuning module and a vertical tuning module, respectively, to process the feature data output from the lth layer of the backbone network and the first feature data of the l-1 layer of the bypass network, and the feature data of the two can be weighted and summed using weights.

Optionally, the initial input into the entire module is called layer 0.

As an optional implementation, step S206, updating the parameters of the initial bypass network based on the converted feature data to obtain the target bypass network, includes: determining the processing result obtained by the output layer in the pre-trained model processing the converted feature data; determining the difference between the processing result and the actual processing result corresponding to the processing result; adjusting the parameters of the initial bypass network based on the difference value to obtain the target bypass network.

In this embodiment, feature data after conversion of the initial bypass network is obtained, and the feature data can be processed through the output layer in the pre-trained network to obtain a processing result. The difference value between the processing result and the real processing result corresponding to the processing result can be determined. Based on the difference value between the two, the parameters of the initial bypass network can be adjusted to obtain the target bypass network. Among them, the processing result can be an image, text, sequence data, classification situation, etc., and the category of the processing result can be flexibly changed according to the use scenario of the pre-trained model. This is only an example, and no specific limitation is made to the category of the processing result. The difference value can be used to characterize the difference between the processing result and the real processing result corresponding to the processing result, and can be used to determine the loss function.

Optionally, this embodiment obtains feature data after conversion of the initial bypass network, and can process the feature data through the output layer in the pre-trained network to obtain a processing result. The loss function between the processing result and the actual processing result corresponding to the processing result can be calculated, and the gradient of the parameters of the initial bypass network is calculated based on the loss function, thereby determining the adjustment direction and size of the parameters to obtain an updated target bypass network.

For example, in order to make the model better fit the training data, back propagation can be performed based on the processing results of the output layer, and the change of the loss function can be determined according to the processing results and the actual processing results. The parameters of the initial bypass network can be adjusted based on the change of the loss function, so that the processing results of the pre-trained model are closer to the actual processing results.

Due to the large number of parameters in the large model, it is very time-consuming to directly calculate the gradient through numerical methods. In this embodiment, the gradient can be efficiently calculated through back propagation, which speeds up the training of the initial bypass network. In addition, back propagation also enables the initial bypass network to learn the characteristics and laws of the data, and continuously optimizes the performance of the pre-trained model by updating the parameters of the initial bypass network, thereby improving the accuracy and generalization ability of the pre-trained model.

As an optional implementation, the backbone network and at least one tuning module are connected based on residuals.

In this embodiment, the initial bypass network and the backbone network can be connected by residual connection, and the initial bypass network can include at least one horizontal tuning module or a vertical tuning module, so the backbone network and at least one tuning module can be connected based on residual. It should be noted that there is no specific restriction on the number of tuning modules in the initial bypass network.

Optionally, the tuning modules in the initial bypass network can be connected based on residual connections and act in parallel in the backbone network.

Since the initial backbone network constructed by embedding the tuning module in the backbone network needs to process the training data during the back propagation training of the tuning module, there is a problem of occupying memory during the training process. In this embodiment, the tuning module is extracted from the initial backbone network, and the tuning modules are connected through residual connections to construct an initial bypass network, and the initial bypass network can be connected to the backbone network in parallel. During the back propagation process, it is prohibited to transmit the data stream from the initial bypass network to the backbone network, thereby achieving the effect of reducing memory usage.

It should be noted that the above-mentioned residual connection can be changed according to actual conditions. For example, it can also be a full connection, a skip connection, etc. Different units can also be combined or replaced inside a single tuning module. No specific restrictions are made here on the connection method between the tuning modules and the components of the tuning modules.

As an optional implementation, during the process of transmitting the characteristic data to the initial bypass network, the state of the parameters of the backbone network is a locked state.

In this embodiment, during the model training process, the state of the parameters of the backbone network is a locked state, that is, the parameters of the backbone network are not updated. The locked state can also be called a frozen state, which can be used to characterize that the parameters of the backbone network do not change.

In this embodiment, the backbone network keeps parameters frozen, and at the same time, data no longer flows to the backbone network during the back propagation process, thereby achieving the purpose of saving memory.

In the embodiment of the present disclosure, a tuning module extracted from the backbone network is used to construct an initial bypass network. At the same time, during the process of adjusting the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is cut off to obtain a target bypass network (new tuning module) relatively independent of the backbone network, so that the initial bypass network During the network training process, there is no need to further calculate the parameter gradients of the backbone network, which saves memory and improves the training speed, thereby achieving the technical effect of reducing resource consumption during the model training process and solving the technical problems of high resource consumption and low computing efficiency during the model training process.

The disclosed embodiment is directed to an artificial intelligence generated content (AIGC) system and also provides a data processing method for a pre-trained model. FIG3 is a flow chart of a data processing method for a pre-trained model according to an embodiment of the disclosed embodiment. As shown in FIG3, the method may include the following steps.

Step S302, responding to the query information received in the generative interactive interface, wherein the query information at least includes: keywords of the text generation information.

In the technical solution provided in the above step S302 of the present disclosure, when the generative interaction interface receives the inquiry information, the inquiry information received in the generative interaction interface can be responded to. Among them, the generative interaction interface can be an interface that generates a dialogue using a natural speech generation model, and can be an operation interface of a mobile terminal, a client, etc. This is only an example, and there is no specific restriction on the type of generative interaction interface. The inquiry information may at least include: keywords of the text generation information. The inquiry information may be an instruction issued by the client, and may be an instruction to ask questions in a dialogue or interaction scenario to obtain guidance behavior, etc.

Optionally, the user may enter a question or request in the generative interactive interface, "Hello, I would like to know how to unsubscribe from my service?", and the query information may include the keyword "how" of the text generation information. The query information received in the generated dialogue interface may be responded to.

Step S304, calling the backbone network of the pre-trained model to at least analyze the text generation information and output text feature data, wherein the backbone network comes from the initial backbone network.

In the technical solution provided in the above step S304 of the present disclosure, in response to the query information received in the generative interactive interface, the backbone network of the pre-trained model can be called to at least analyze the text generation information and output text feature data. Among them, the backbone network can come from the initial backbone network. The text generation information can be used to determine the question raised by the query information. The text feature data can be used to characterize the text content, for example, it can be a vector, texture feature, etc. This is only for example, and no specific limitation is made on the type of feature data.

Step S306, calling a target bypass network to convert the text feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

In the technical solution provided in the above step S306 of the present disclosure, the target bypass network can be called to convert the text feature data.

Step S308: generating at least one response result matching the query information based on the converted text feature data.

In the technical solution provided in the above step S308 of the present disclosure, the pre-trained model can process the converted text feature data to generate a response result that matches the inquiry information.

Step S310, displaying the reply result in the generative interactive interface.

In the technical solution provided in the above step S310 of the present disclosure, the reply result can be displayed in the generative interactive interface. The reply result can be an image, text, voice, etc., which is only an example and does not specifically limit the content of the reply result.

For example, a generative interactive interface receives a query message such as “What is the recipe for pasta?” In response to the query message received by the generative interactive interface, the backbone network in the pre-trained model can be called to at least analyze the text generation information “recipe for pasta” in the query message, output text feature data, and call the target bypass network to convert the text feature data. Based on the converted text feature data, at least one reply result matching the query message is generated, and the reply result can be displayed in the generative interactive interface.

As an optional implementation, the method also includes: in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the impact of the tuning module on the text feature data.

In this embodiment, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters in the initial bypass network can be used to characterize the influence of the tuning module on the text feature data.

Optionally, the backbone network keeps parameters frozen, and the initial bypass network uses a tuning module extracted from the backbone network, while cutting off the data flow from the tuning module to the backbone network to form an initial bypass network that is relatively independent of the backbone network.

In this embodiment, a query information received in a generative interactive interface is responded to, wherein the query information includes at least: keywords of text generation information; a backbone network of a pre-trained model is called to at least analyze the text generation information and output text feature data, wherein the backbone network comes from an initial backbone network; a target bypass network is called to convert the text feature data, wherein the target bypass network is obtained by updating the parameters of an initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted text feature data, at least one reply result matching the query information is generated; and the reply result is displayed in the generative interactive interface, thereby achieving the technical effect of improving the efficiency of obtaining feedback results in a generative dialogue product and solving the technical problems of high resource consumption and computational efficiency in the training process of the model.

As an optional embodiment, in the application scenario of discriminative tasks, this embodiment can extract the tuning module from the initial backbone network to obtain the tuning module and the backbone network. The extracted tuning modules can be combined to obtain the initial bypass network, and the parameters in the initial bypass network can be updated to obtain the target bypass network. In the scenario of discriminative tasks, the backbone network in the pre-trained model can be obtained to process the image to be classified and obtain feature data. The trained target bypass network is called to convert the feature data. The feature data converted by the target bypass model can be input into the output layer, and the output layer processes the feature data to obtain the category to which the image to be classified belongs.

Optionally, in a discriminative task, the overall structure of the pre-trained model (e.g., discriminative task network) can be divided into two parts. One part is the backbone network, which is frozen during the training process. The other part is The trainable initial bypass network (also called bypass structure) can be composed of several tuning modules, where each network layer of the corresponding backbone network has a horizontal tuning module and a vertical tuning module, which are respectively used to receive the feature data output from the layer in the backbone network and the output of the previous layer in the initial bypass network, and the two are weightedly summed and output to the next layer.

Optionally, until the converted feature data of the last layer is output to the output layer in the pre-trained model to output the corresponding result, a back propagation process can be performed based on the output of the corresponding result, and during the back propagation process, the back propagation path from the bypass network to the backbone network will be disconnected to obtain the trained target bypass grid. When the image to be classified is obtained, the backbone model can be obtained to process the image to be classified and obtain the feature data. The target bypass network can be called to convert the feature data, and the converted feature data can be output to the output layer in the pre-trained model to determine the category to which the image to be classified belongs.

For example, given an image to be classified, the feature data of the image to be classified can be determined through the backbone network in the pre-trained model, the target bypass network can be called to convert the feature data, and based on the converted feature data, the category of the image to be classified is determined to be an image containing kittens.

As an optional implementation, step S404, calling the target bypass network to convert the text feature data, includes: calling the first vertical tuning module corresponding to the first decoding layer in the backbone network to convert the text feature data output by the middle layer of the backbone network, and calling the first horizontal tuning module associated with the first vertical tuning module to convert the text feature data output by the first decoding layer.

In this embodiment, the first vertical tuning module corresponding to the first decoding layer in the backbone network can be called to convert the text feature data output by the middle layer of the backbone network, and the first horizontal tuning module associated with the first vertical tuning module can be called to convert the text feature data output by the first decoding layer.

Optionally, in a generative task (e.g., image generation), the overall structure of the pre-trained model may include a backbone network whose parameters are frozen during training and a target bypass network whose parameters are updated, wherein the backbone network may be a deep learning model (U-Net) structure based on stable diffusion (StableDiffusion). The target bypass network may be a trainable bypass structure, which may be composed of several tuning modules.

For example, the backbone network may include three encoding layers, one middle layer and four decoding layers. Each decoding layer corresponds to a horizontal tuning module and a vertical tuning module, and the horizontal tuning module and the vertical tuning module can be used to receive text feature data output from the middle layer or the decoding layer and the text feature data converted from the previous layer in the target bypass network. The first vertical tuning module corresponding to the first decoding layer can be used to receive the text feature data of the middle layer of the backbone network and convert the text feature data of the middle layer. The first horizontal tuning module associated with the first vertical tuning module can obtain the text feature data output by the first decoding layer and convert the text feature data output by the first decoding layer.

As an optional implementation, based on the converted text feature data, generating at least one reply result matching the query instruction includes: based on the weight of the target bypass network, performing a weighted summation between the text feature data adjusted by the first vertical tuning module and the text feature data adjusted by the first horizontal tuning module to obtain The converted text feature data; based on the converted text feature data, determining a response result.

In this embodiment, based on the weight of the target bypass network, a weighted sum can be performed on the text feature data adjusted by the first vertical tuning module and the text feature data adjusted by the first horizontal tuning module to obtain converted text feature data, and a response result can be determined based on the converted text feature data.

Optionally, each corresponding decoding layer of the backbone network corresponds to a horizontal tuning module and a vertical tuning module. The horizontal tuning module and the vertical tuning module can respectively receive the text feature data output from the middle layer or the decoding layer in the backbone network and the text feature data output from the previous layer in the target bypass network, and can perform weighted summation on the two and output them to the vertical tuning module corresponding to the next layer of the backbone network, until the weighted summation result of the horizontal tuning module and the vertical tuning module corresponding to the last decoding layer of the backbone network is output to the generative task network for outputting the corresponding result, so as to obtain the response result.

For example, the backbone network may include three encoding layers, one intermediate layer and three decoding layers. Each decoding layer corresponds to a horizontal tuning module and a vertical tuning module, which can be used to receive text feature data output from the intermediate layer or the decoding layer and text feature data converted from the previous layer in the target bypass network. The first vertical tuning module corresponding to the first decoding layer can be used to receive the text feature data of the intermediate layer of the backbone network and convert the text feature data of the intermediate layer. The first horizontal tuning module associated with the first vertical tuning module can obtain the text feature data output by the first decoding layer and convert the text feature data output by the first decoding layer. Based on the weight of the target bypass network, a weighted sum can be performed between the text feature data adjusted by the first vertical tuning module and the text feature data adjusted by the first horizontal tuning module. The text feature data obtained by weighted summing the text feature data adjusted by the first vertical tuning module and the text feature data adjusted by the first horizontal tuning module can be input into the second vertical tuning module corresponding to the second decoding layer to obtain the text feature data converted by the second vertical tuning module, and the text feature data output by the second decoding layer is output to the second horizontal tuning module to obtain the corresponding text feature data adjusted by the second horizontal tuning module, and the text feature data adjusted by the second horizontal tuning module and the text feature data adjusted by the second vertical tuning module can be weighted summed. The text feature data obtained by weighted summing the text feature data adjusted by the second horizontal tuning module and the text feature data adjusted by the second vertical tuning module are continuously transmitted to the third vertical tuning module corresponding to the third decoding layer, and the text feature data output by the third decoding layer is transmitted to the third horizontal tuning module corresponding to the third decoding layer, and the text feature data adjusted by the third horizontal tuning module and the text feature data adjusted by the third vertical tuning module can be weighted summed to obtain the text feature data finally converted by the target bypass network. The text feature data can be processed based on the output layer in the pre-trained model to obtain the response result.

As an optional implementation, step S406, based on the converted text feature data, determines the reply result, including: using the converted text feature data as the output data of the last decoding layer in the pre-trained model; converting the output data into the reply result.

In this embodiment, the target bypass model can be used as the last encoding layer in the pre-trained model. The converted text feature data is used as the output data of the last decoding layer in the pre-trained model. Based on the output data, the output layer can convert the output data to obtain the response result.

Optionally, the target bypass network in this embodiment can also be used as a branch in a pre-trained model (e.g., a controllable generative task), that is, a coding module for conditional input. The controllable generative task can be a task generated based on controllable conditional data.

In this embodiment, the controllable coding part can also use the target bypass network as a branch to achieve the purpose of reducing memory consumption.

In this embodiment, feature data obtained by processing an image to be classified by a backbone network in a pre-trained model is obtained, wherein the backbone network comes from an initial backbone network; a target bypass network is called to convert the feature data, wherein the target bypass network is obtained by updating parameters of an initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, the category to which the image to be classified belongs is determined, thereby achieving a technical effect of reducing resource consumption in the training process of the model, and solving the technical problems of high resource consumption and computational efficiency in the training process of the model.

The embodiment of the present disclosure is directed to generating task scenarios, such as generating images, videos or texts according to conditional data, and also provides another method for processing data of a pre-trained model. FIG4 is a flow chart of another method for processing data of a pre-trained model according to an embodiment of the present disclosure. As shown in FIG4, the method may include the following steps:

Step S402, obtaining feature data obtained by processing the conditional data by the backbone network in the pre-trained model, wherein the backbone network comes from the initial backbone network, and the conditional data is used to determine the generation condition of the target data.

In the technical solution provided in the above step S402 of the present disclosure, conditional data can be obtained. The backbone network processes the conditional data to obtain feature data. Among them, the conditional data can be used to determine the generation conditions of the target data, for example, it can be "drawing a blue dwarf cat", it can be a line drawing, a depth map, a posture and other contents, and can include text conditions, image conditions, etc. This is only for example explanation, and no specific restrictions are made on the content of the conditional data. The backbone network can be a conditional two-dimensional deep learning model (Conditional 2D U-Net), which can include an encoding layer (Encoder), a decoding layer (Docoder) and a middle layer (Middle).

For example, the conditional data "write a set of codes for filtering data" can be obtained. Keywords in the conditional data can be extracted and converted, and feature data corresponding to the keywords can be obtained. It should be noted that the content of the above conditional data and the determination of the feature data are only for illustration and are not specifically limited here.

Step S404, calling a target bypass network to convert the characteristic data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

In the technical solution provided in the above step S404 of the present disclosure, a tuning module can be extracted from the backbone network, an initial bypass network can be constructed based on at least one tuning module, and the parameters of the initial bypass network can be updated based on the training data to obtain a target bypass network. The trained target bypass network can be called to convert the feature data. The training data can be data with an identifier, which can be pre-acquired and used to convert the initial bypass network. The data for network training, for example, can be image data, text data, voice data, etc., which is only for example and does not limit the type of training data. The parameters of the initial bypass network can be weights, bias data, etc., which is only for example and does not limit the type of parameters.

Optionally, this embodiment extracts a tuning module from the initial backbone network to obtain a tuning module and a backbone network. The extracted tuning modules can be combined to obtain an initial bypass network, and the parameters in the initial bypass network can be updated to obtain a target bypass network. In the scenario of generating tasks, the conditional data of the backbone network in the pre-trained model can be obtained to process the feature data. The trained target bypass network is called to convert the feature data.

Step S406: generating target data corresponding to the conditional data based on the converted feature data, wherein the type of the target data includes at least one of the following: text information, image information, video information and voice information.

In the technical solution provided in the above step S406 of the present disclosure, the feature data converted by the target bypass model can be input into the output layer, and the output layer processes the feature data to generate target data corresponding to the conditional data. The target data can be an animation, story, landscape, face image, dialogue, etc. generated based on the conditional data, and the type of the target data can include at least one of the following: text information, image information, video information, and voice information. This is only for example, and the category of the target data is not specifically limited.

Optionally, the pre-trained model can be a conditional pre-trained transformer model (Chat Conditional Pretrained Transformer, referred to as ChatCPT), which can be used to generate dialogue responses and conduct dialogue interactions. When the conditional data is the dialogue context, the inquiry information can be determined based on the dialogue context. The backbone network in the pre-trained model can be called to process the conditional data, and the feature data can be obtained. The target bypass network is called to transform the feature data; based on the converted feature data, a dialogue response corresponding to the conditional data (i.e., target data) can be generated. Alternatively, when the conditional data is a question raised by the user, the pre-trained model can be used to generate an answer corresponding to the user's question information. Alternatively, when the conditional data is a long text message and the user wants to simplify the text message, a concise and general summary content can be generated through the pre-trained model. Alternatively, when the conditional data is information and guidance provided by the user, the pre-trained model can assist in writing, and the conditional data can be processed to obtain articles, storylines, dialogue scripts, etc. corresponding to the field data.

It should be noted that the above is only an example of the use scenario of the pre-trained model, and no specific limitation is made here. The method of processing the conditional data through the pre-trained model to obtain the target data should be within the scope of protection of this disclosure.

In an embodiment of the present disclosure, feature data is obtained by processing conditional data by a backbone network in a pre-trained model, wherein the backbone network comes from an initial backbone network, and the conditional data is used to determine the generation condition of target data; a target bypass network is called to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, target data corresponding to the conditional data is generated, wherein the type of the target data includes at least one of the following: text information, image information, video information, and voice information, Thereby achieving the technical effect of reducing resource consumption in the model training process, and solving the technical problems of high resource consumption and computational efficiency in the model training process.

According to an embodiment of the present disclosure, a data processing method for a pre-trained model in a virtual reality scenario such as a virtual reality VR device and an augmented reality AR device is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

FIG5 is a flowchart of another method for processing data of a pre-training model according to an embodiment of the present disclosure. As shown in FIG5 , the method may include the following steps.

Step S502: input the speech to be converted on a virtual reality VR device or an augmented reality AR device.

Step S504, using the backbone model in the pre-trained model to extract feature data from the speech to be converted, wherein the backbone network comes from the initial backbone network.

Step S506, calling the target bypass network in the pre-trained model to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

Step S508: determining the image information corresponding to the speech to be converted based on the converted feature data.

Step S510: activating the VR device or AR device using the image information, and displaying the image information in the VR device or AR device.

Optionally, in this embodiment, the data processing method of the pre-trained model can be applied to a hardware environment composed of a server and a virtual reality device. The VR device or AR device is controlled to perform a human-computer interaction operation corresponding to the positioning information. The server can be a server corresponding to the media file operator. The network includes but is not limited to: a wide area network, a metropolitan area network or a local area network. The virtual reality device is not limited to: a virtual reality helmet, virtual reality glasses, a virtual reality all-in-one machine, etc.

Optionally, the virtual reality device may include: a memory, a processor, and a transmission device. The memory is used to store an application, which can be used to execute: inputting a speech to be converted on a virtual reality VR device or an augmented reality AR device; extracting feature data from the speech to be converted using a backbone model in a pre-trained model, wherein the backbone network comes from an initial backbone network; calling a target bypass network in the pre-trained model to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; determining image information corresponding to the speech to be converted based on the converted feature data; using the image information to activate the VR device or AR device, and displaying the image information in the VR device or AR device.

It should be noted that the above-mentioned processing method of the pre-trained model applied in the VR device or AR device in this embodiment may include the method of the embodiment shown in Figure 5, so as to control the VR device or AR device to perform human-computer interaction operations corresponding to the positioning information.

Optionally, the processor of this embodiment can call the application stored in the memory to execute the above steps through a transmission device. The transmission device can receive media files sent by the server through the network, and can also be used for data transmission between the processor and the memory.

Optionally, in a virtual reality device, a head-mounted display with eye tracking is provided, wherein the screen in the HMD is used to display the displayed video images, the eye tracking module in the HMD is used to obtain the real-time movement path of the user's eyes, the tracking system is used to track the user's position information and motion information in the real three-dimensional space, and the computing processing unit is used to obtain the user's real-time position and motion information from the tracking system, and calculate the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of view direction in the virtual three-dimensional space, etc.

In the embodiments of the present disclosure, a virtual reality device may be connected to a terminal, and the terminal and the server are connected via a network. The virtual reality device is not limited to: a virtual reality helmet, virtual reality glasses, a virtual reality all-in-one machine, etc. The terminal is not limited to a PC, a mobile phone, a tablet computer, etc. The server may be a server corresponding to a media file operator, and the network includes but is not limited to: a wide area network, a metropolitan area network or a local area network.

Figure 6 is a schematic diagram of a data processing result of a pre-trained model according to an embodiment of the present disclosure. As shown in Figure 6, the speech to be converted "Draw a deer" is displayed on the presentation screen of the VR device or AR device, and the backbone model in the pre-trained model can be used to extract feature data from the speech to be converted, and the target bypass network in the pre-trained model is called to convert the feature data. Based on the converted feature data, the image information corresponding to the speech to be converted can be determined; the image information can be used to activate the VR device or AR device, and the image information "deer" can be displayed in the VR device or AR device.

Through the above steps, the speech to be converted is input on the virtual reality VR device or the augmented reality AR device; the backbone model in the pre-trained model is used to extract feature data from the speech to be converted, wherein the backbone network comes from the initial backbone network; the target bypass network in the pre-trained model is called to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, the image information corresponding to the speech to be converted is determined; the image information is used to activate the VR device or AR device, and the image information is displayed in the VR device or AR device, thereby solving the technical problems of high resource consumption and computing efficiency in the training process of the model, and achieving the technical effect of reducing resource consumption in the training process of the model.

According to an embodiment of the present disclosure, there is also provided a data processing method for a pre-trained model that can be applied to an artificial intelligence generated content (AIGC) system. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

FIG7 is a flowchart of another method for processing data of a pre-training model according to an embodiment of the present disclosure. As shown in FIG7 , the method may include the following steps.

Step S702: input multimodal information in the dialogue interface, wherein the type of the multimodal information includes at least one of the following: inquiry information including character information, video frame information including frame image information, and audio information.

In the technical solution provided in the above step S702 of the present disclosure, multimodal information input in the dialogue interface can be obtained, wherein the type of multimodal information can include at least one of the following: query information containing character information, video frame information containing frame image information, and audio information. It should be noted that this is only an example and does not specifically limit the type of multimodal information. The dialogue interface can be the display interface of the terminal where the artificial intelligence content generation system is located. This is only an example. The dialogue interface that can be used for the dialogue system should be within the protection scope of the present disclosure and is not specifically limited here.

Step S704, calling the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, wherein the backbone network comes from the initial backbone network.

In the technical solution provided in the above step S704 of the present disclosure, the AIGC system can obtain multimodal information input in the dialogue interface, and call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data.

Step S706, calling the target bypass network to convert the characteristic data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

Step S708, generating reply information corresponding to the multimodal information based on the converted feature data, wherein the type of the reply information includes at least one of the following: text information, image information, video information and voice information.

In the technical solution provided in the above step S708 of the present disclosure, the pre-trained model processes the multimodal information to obtain converted feature data. Based on the converted feature data, reply information corresponding to the multimodal information can be generated, wherein the type of the reply information includes at least one of the following: text information, image information, video information and voice information. It should be noted that there is no specific restriction on the type of reply information here.

For example, to obtain the multimodal information input by the user in the dialogue interface: "Give examples of usage scenarios of neural networks", the trained pre-trained model can be called to process the multimodal information to obtain response information corresponding to the multimodal information: "The usage scenarios of neural networks can be image usage scenarios, voice usage scenarios, and code direction usage scenarios."

As an optional implementation, step S706, calling the target bypass network to convert the characteristic data, includes: calling the tuning module in the target bypass network to adjust the characteristic data; and performing weighted summation on the adjusted characteristic data based on the weight of the target bypass network.

In this embodiment, the multimodal information input by the user in the dialogue interface can be obtained, and the backbone network in the pre-trained model can be called to analyze and process the multimodal information to obtain feature data. The feature data can be input into the trained target bypass network, and the tuning module in the target bypass network adjusts the feature data, and the base The adjusted feature data is weighted and summed in the target bypass network to obtain the final converted feature data.

As an optional implementation, step S708, based on the converted feature data, generates reply information corresponding to the multimodal information, including: processing the converted feature data based on the output layer in the pre-trained model to obtain reply information corresponding to the multimodal information.

In this embodiment, the converted feature data may be processed based on the output layer in the pre-trained model to convert the feature data into reply information corresponding to the multimodal information. The reply information may be displayed in the dialogue interface.

In this embodiment, multimodal information is input in the dialogue interface, wherein the type of the multimodal information includes at least one of the following: inquiry information containing character information, video frame information containing frame image information, and audio information; the backbone network in the pre-trained model is called to at least analyze and process the multimodal information to obtain feature data, wherein the backbone network comes from the initial backbone network; the target bypass network is called to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, reply information corresponding to the multimodal information is generated, wherein the type of the reply information includes at least one of the following: text information, image information, video information, and voice information, thereby solving the technical problems of high resource consumption and computational efficiency in the training process of the model, and achieving the technical effect of reducing resource consumption in the training process of the model.

It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present disclosure.

According to an embodiment of the present disclosure, an embodiment of a data processing system for a pre-trained model is also provided. FIG8(a) is a schematic diagram of a data processing system for a pre-trained model according to an embodiment of the present disclosure. As shown in FIG8(a), The data processing system 800 of the pre-trained model may include: a client 802 and a server 804.

The client 802 is configured to display a dialogue interface and capture multimodal information input in the dialogue interface, wherein the type of the multimodal information includes at least one of the following: query information containing character information, video frame information containing frame image information, and audio information.

Optionally, the multimodal information is input in the dialogue interface of the client 802 .

The server 804 is configured to call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, and call the target bypass network to convert the feature data, and generate reply information corresponding to the multimodal information based on the converted feature data, wherein the backbone network comes from the initial backbone network, the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network, and the type of the reply information includes at least one of the following: text information, image information, video information and voice information.

Optionally, the client 802 may transmit the acquired multimodal information to the server 804. The server 804 may call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, and call the target bypass network to convert the feature data, and generate reply information corresponding to the multimodal information based on the converted feature data.

Optionally, the server 804 may transmit the reply information to the dialogue interface of the client 802 for display.

In this embodiment, a dialogue interface is displayed through a client, and multimodal information input in the dialogue interface is captured, wherein the type of multimodal information includes at least one of the following: query information including character information, video frame information including frame image information, and audio information; through a server, a backbone network in a pre-trained model is called to at least analyze and process the multimodal information to obtain feature data, and a target bypass network is called to convert the feature data, and based on the converted feature data, reply information corresponding to the multimodal information is generated, wherein the backbone network comes from an initial backbone network, the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network, and the type of reply information includes at least one of the following: text information, image information, video information, and voice information, thereby achieving a technical effect of reducing resource consumption in the training process of the model, and solving the technical problems of high resource consumption and computational efficiency in the training process of the model.

At present, parameter-efficient transfer learning methods based on large-scale pre-trained basic models have achieved great success in various downstream applications. Existing tuning methods usually introduce lightweight and trainable structures to perform fine-tuning for specific tasks while freezing the basic model. However, most existing parameter-efficient transfer learning methods change the intermediate state of the pre-trained model, so more memory and time are required for training, and redundant calculations of different tasks need to be performed through the backbone during multi-task reasoning, resulting in the problem of high consumption and low computational efficiency of the training process of the pre-trained model.

As an optional embodiment, FIG8(b) is a schematic diagram of a tuning module according to an embodiment of the present disclosure. As shown in FIG8(b), the original tuning method will adjust different optimization methods in the original training architecture according to different specific tasks. Some parts are light-weighted (such as prompt, prefix, adapter, etc.), but this method embeds the parameter-efficient tuning module deeply into the original backbone network, where the original backbone network (initial backbone network) can include a feedforward network and multi-head attention. During training, different tasks need to be calculated through the initial backbone network, resulting in a waste of memory and inference overhead.

To solve the above problems, this embodiment proposes an efficient tuning method for parameters and memory based on a pre-trained model. This method constructs a bypass network (also called a tuning framework) with efficient parameters and memory, extracts the tuning module from the backbone network, and constructs forward and reverse propagation links independent of the backbone network. While maintaining the advantage of efficient parameters, it reduces the memory overhead during training and redundant calculations during reasoning multi-tasks, thereby achieving the purpose of reducing resource consumption during model training and improving computing efficiency.

The following is a further introduction to an efficient tuning method for parameters and memory based on a pre-trained model proposed in an embodiment of the present disclosure.

Figure 8(c) is a schematic diagram of another tuning module according to an embodiment of the present disclosure. As shown in Figure 8(c), this embodiment separates the tuning module from the initial backbone network to form an independent sub-module. The separated sub-modules can act in parallel on the backbone network in the form of residuals and can be freely combined to form an independent initial bypass network.

It should be noted that the separated sub-modules can act on the backbone network in the form of residuals, or in other connection modes such as skip connections. No specific restrictions are made on the connection mode here.

Optionally, during the training process, the initial bypass network can use the intermediate output (feature data) of the backbone network during forward propagation, but during reverse propagation, it is disconnected from the backbone network, and the data flow only passes through the bypass network. Therefore, during reverse propagation, further calculation of the gradient of the backbone network can be avoided, thereby saving memory and improving training speed, thereby achieving the purpose of reducing resource consumption and improving computing efficiency during model training, and solving the problem of high resource consumption and low computing efficiency during model training.

As shown in Figure 8(c), the pre-trained model can be composed of a backbone network and an initial bypass network with residual connections, which is both parameter efficient and memory efficient. The backbone network can include multiple convolutional layers.

In this embodiment, the overall structure of the pre-trained model can be divided into two parts, one part is the backbone model, the parameters of the backbone model are frozen during the training process; the other part is the trainable initial bypass network (also called bypass structure), which can be composed of several tuning modules. The data flow from the tuning module to the backbone network can be cut off to form a new tuning module that is relatively independent of the backbone network. Different tuning modules (Res-Tuner) can be combined and designed to construct a complete initial bypass network. Based on the output results, the parameters of the initial bypass network can be updated to obtain the target bypass network.

In this embodiment, since the data flow between the initial bypass network and the backbone network is cut off during reverse propagation, the data flow of the tuning module in the initial bypass network does not flow to the backbone network, thereby saving memory and achieving the purpose of reducing memory loss.

Optionally, the output of the initial bypass network can be determined by the following formula:

in, It can be used to characterize the adjusted feature data corresponding to the first-level horizontal tuning module. It can be used to characterize the output of the bypass network at layer l. Res-Tuner(x _l ) can be used to characterize the output of the horizontal tuning module at layer l. It can be used to characterize the output of the vertical tuning module of the l-1th layer, _x0 can be used to characterize the characteristic data of the output of the zeroth layer backbone network, _xl can be used to characterize the output characteristics of the lth layer, and λ can be used to characterize the weight of the initial bypass network.

Optionally, as shown in FIG8(c), each network layer corresponding to the backbone network has a corresponding horizontal tuning module and a vertical tuning module in the initial bypass network. The horizontal tuning module and the vertical tuning module can be used to process the output features from the lth layer of the backbone network and the feature outputs from the l-1th layer of the initial bypass network, respectively, and the outputs of the two can be weighted and summed using a learnable weight (λ coefficient).

Optionally, the weight coefficients may be continuously adjusted through a back-propagation algorithm.

For example, if the backbone network has 12 network layers, the feature data can be input into the initial bypass network after all the input data are converted by the 12 network layers. However, in order to improve the data processing efficiency, the feature data can be transmitted from the first network layer of the backbone network to the first stage of the initial bypass; and then the feature data can be transmitted from the second network layer of the backbone network to the second stage of the initial bypass, and so on.

As an optional embodiment, the pre-trained model can be applied to the discriminative task.

FIG9 is a schematic diagram of a pre-training model according to an embodiment of the present disclosure. As shown in FIG9 , in a discriminant task, the overall structure of the pre-training model can be divided into two parts. One part is a backbone network, and the backbone network is frozen during the training process, wherein the backbone network may include multiple training layers (e.g., training layer_1, training layer_2...training layer_N), and the above training layer may be a convolutional layer. The other part is a trainable initial bypass network (also referred to as a bypass structure), and the initial bypass network may be composed of several tuning modules, wherein each network layer of the corresponding backbone network has a horizontal tuning module and a vertical tuning module, which are respectively used to receive the feature data output from the layer in the backbone network and the output of the previous layer in the initial bypass network, and the two are weighted and summed, and output to the next layer, until the converted feature data of the last layer is output to the output layer in the pre-training model for the output of the corresponding result, and the reverse propagation process can be performed based on the output of the corresponding result, and in the process of reverse propagation, the reverse propagation path from the bypass network to the backbone network will be disconnected to obtain the trained target bypass grid. When the image to be classified is obtained, the backbone model can be used to process the image to be classified to obtain feature data. The target bypass network can be called to convert the feature data, and the converted feature data can be output to the output layer in the pre-trained model to determine the category to which the image to be classified belongs.

As an optional embodiment, the pre-trained model can be applied to the generative task.

FIG. 10 is a schematic diagram of another pre-training model according to an embodiment of the present disclosure. As shown in FIG. 10 , when generating When performing a task (e.g., image generation) using a pre-trained model, the overall structure of the pre-trained model may include a backbone network whose parameters are frozen during training and a target bypass network whose parameters are updated, wherein the backbone network may be a deep learning model (U-Net) structure based on stable diffusion, and may include a coding layer of 64*64, a coding layer of 32*32, a coding layer of 16*16, a coding layer of 8*8, an intermediate layer of 8*8, a decoding layer of 8*8, a decoding layer of 16*16, a decoding layer of 32*32, and a decoding layer of 64*64. The target bypass network may be a trainable bypass structure, and may be composed of several tuning modules.

Each corresponding decoding layer of the backbone network corresponds to a horizontal tuning module and a vertical tuning module. The horizontal tuning module and the vertical tuning module can respectively receive feature data output from the intermediate layer or decoding layer in the backbone network and feature data output from the previous layer in the target bypass network, and can perform weighted summation on the two and output them to the vertical tuning module corresponding to the next layer of the backbone network, until the weighted summation result of the horizontal tuning module and the vertical tuning module corresponding to the last decoding layer of the backbone network is output to the generative task network for output of the corresponding result to obtain the target data.

Optionally, the target bypass model can be used as the last encoding layer in the pre-trained model, and the converted feature data can be used as the output data of the last decoding layer in the pre-trained model. Based on the output data, the output layer can convert the output data to obtain the target data.

Optionally, the target bypass network in this embodiment can also be used as a branch in a controllable generative task (pre-training model), that is, it can be an encoding module for conditional input.

Controllable generative tasks are tasks that are generated based on controllable conditional data. In this embodiment, the controllable encoding part can also use the target bypass network as a branch to achieve the purpose of reducing memory consumption.

For example, the backbone network may include three encoding layers, one intermediate layer, and three decoding layers. Each decoding layer corresponds to a horizontal tuning module and a vertical tuning module, and the horizontal tuning module and the vertical tuning module can be used to receive feature data output from the intermediate layer or the decoding layer and feature data converted from the previous layer in the target bypass network. The first vertical tuning module corresponding to the first decoding layer can be used to receive feature data of the intermediate layer of the backbone network and convert the feature data of the intermediate layer. The first horizontal tuning module associated with the first vertical tuning module can obtain feature data output by the first decoding layer and convert feature data output by the first decoding layer. Based on the weight of the target bypass network, a weighted sum can be performed between the feature data adjusted by the first vertical tuning module and the feature data adjusted by the first horizontal tuning module.

Optionally, the feature data obtained by weighted summing the feature data adjusted by the first vertical tuning module and the feature data adjusted by the first horizontal tuning module can be input into the second vertical tuning module corresponding to the second decoding layer to obtain the feature data converted by the second vertical tuning module, and the feature data output by the second decoding layer is output to the second horizontal tuning module to obtain the corresponding feature data adjusted by the second horizontal tuning module, and the feature data adjusted by the second horizontal tuning module and the feature data adjusted by the second vertical tuning module can be weighted summed. The feature data obtained by weighted summing the feature data adjusted by the second horizontal tuning module and the feature data adjusted by the second vertical tuning module is continuously transmitted to the third vertical tuning module corresponding to the third decoding layer. In the tuning module, the feature data output by the third decoding layer is transmitted to the third horizontal tuning module corresponding to the third decoding layer, and the feature data adjusted by the third horizontal tuning module and the feature data adjusted by the third vertical tuning module can be weighted summed to obtain the feature data finally converted by the target bypass network. The feature data can be processed based on the output layer in the pre-trained model to obtain the target data.

In the disclosed embodiment, a parameter- and memory-efficient initial bypass network is constructed with the help of a tuning module separated from the initial backbone, and the weights in the initial bypass network are updated to obtain a target bypass network. Compared with the original tuning method, this method can greatly reduce memory consumption and the cost of multi-task reasoning, and is applicable to both discriminative and generative tasks.

In the disclosed embodiment, a tuning module extracted from a backbone network is used to construct an initial bypass network. Meanwhile, during the process of adjusting the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is cut off to obtain a target bypass network (new tuning module) independent of the backbone network. This eliminates the need to further calculate the parameter gradient of the backbone network during the training of the initial bypass network, thereby saving memory and improving the training speed, thereby achieving the technical effect of reducing resource consumption and improving computing efficiency during the training process of the model, and solving the technical problems of high resource consumption and computing efficiency during the training process of the model.

It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure, such as weather forecast results and other data, are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present disclosure.

According to an embodiment of the present disclosure, there is also provided a device for updating parameters in a pre-trained model for implementing the method for updating parameters in a pre-trained model shown in FIG. 2 .

FIG11 is a schematic diagram of a device for updating parameters in a pre-training model according to an embodiment of the present disclosure. As shown in FIG11 , the device 1100 for updating parameters in the pre-training model may include: a first acquisition component 1102, a first A component 1104 is called and a component 1106 is updated.

The first acquisition component 1102 is configured to acquire feature data output by a backbone network of a pre-trained model, wherein the backbone network comes from an initial backbone network.

A first calling component 1104 is configured to call an initial bypass network to convert feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network;

The update component 1106 is configured to update the parameters of the initial bypass network based on the converted characteristic data to obtain a target bypass network, wherein in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the characteristic data.

It should be noted that the first acquisition component 1102, the first calling component 1104 and the update component 1106 correspond to steps S202 to S206 in Example 1, and the three components and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned components can be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned components can also be run in the computer terminal provided in Example 1 as part of the device.

According to an embodiment of the present disclosure, another data processing device for a pre-trained model for implementing the data processing method for a pre-trained model shown in FIG. 3 is also provided.

Figure 12 is a schematic diagram of a data processing device for a pre-trained model according to an embodiment of the present disclosure. As shown in Figure 12, the data processing device 1200 for the pre-trained model may include: a first processing component 1202, a second processing component 1204, a first conversion component 1206, a first generation component 1208 and a display component 1210.

The first processing component 1202 is configured to respond to query information received in the generative interactive interface, wherein the query information at least includes: keywords of the text generation information.

The second processing component 1204 is configured to call a backbone network of a pre-trained model to at least analyze text generation information and output text feature data, wherein the backbone network comes from an initial backbone network.

The first conversion component 1206 is configured to call a target bypass network to convert text feature data, wherein the target bypass network is obtained by updating parameters of an initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from an initial backbone network.

A first generating component 1208 is configured to generate at least one response result matching the query instruction based on the converted text feature data;

The display component 1210 is configured to display the reply result in the generative interactive interface.

It should be noted that the first processing component 1202, the second processing component 1204, the first conversion component 1206, the first generation component 1208 and the display component 1210 correspond to steps S302 to S310 in Example 1. The five components and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in Example 1. It should be noted that the above components can be hardware components or software components stored in a memory and processed by one or more processors. The above components can also be part of the device and can be run on the computer provided in Example 1. in a computer terminal.

According to an embodiment of the present disclosure, another data processing device for a pre-trained model for implementing the data processing method for a pre-trained model shown in FIG. 4 is also provided.

Figure 13 is a schematic diagram of another data processing device for a pre-trained model according to an embodiment of the present disclosure. As shown in Figure 13, the data processing device 1300 for the pre-trained model may include: a second acquisition component 1302, a second conversion component 1304 and a second generation component 1306.

The second acquisition component 1302 is configured to obtain feature data obtained by processing the conditional data by the backbone network in the pre-trained model, wherein the backbone network comes from the initial backbone network, and the conditional data is used to determine the generation conditions of the target data.

A second conversion component 1304 is configured to call a target bypass network to convert the feature data, wherein the target bypass network is obtained by updating parameters of an initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network;

The second generating component 1306 is configured to generate target data corresponding to the conditional data based on the converted feature data, wherein the type of the target data includes at least one of the following: text information, image information, video information and voice information.

It should be noted that the second acquisition component 1302, the second conversion component 1304 and the second generation component 1306 correspond to steps S402 to S406 in Example 1, and the three components and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned components can be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned components can also be run in the computer terminal provided in Example 1 as part of the device.

According to an embodiment of the present disclosure, another data processing device for a pre-trained model for implementing the data processing method for a pre-trained model shown in FIG. 5 is also provided.

Figure 14 is a schematic diagram of another data processing device of a pre-trained model according to an embodiment of the present disclosure. As shown in Figure 14, the data processing device 1400 of the pre-trained model may include: an input component 1402, a third processing component 1404, a third conversion component 1406 and a third generation component 1408.

The input component 1402 is configured to input multimodal information in the dialogue interface, wherein the type of the multimodal information includes at least one of the following: query information including character information, video frame information including frame image information, and audio information.

The third processing component 1404 is configured to call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, wherein the backbone network comes from the initial backbone network.

The third conversion component 1406 is configured to call a target bypass network to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network.

The third generating component 1408 is configured to generate an answer corresponding to the multimodal information based on the converted feature data. The reply information includes at least one of the following types: text information, image information, video information and voice information.

It should be noted that the above-mentioned input component 1402, the third processing component 1404, the third conversion component 1406 and the third generation component 1408 correspond to steps S702 to S708 in Example 1, and the four components and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned components can be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned components can also be run in the computer terminal provided in Example 1 as part of the device.

In the above-mentioned device, a tuning module extracted from the backbone network is used to construct an initial bypass network. At the same time, during the adjustment of the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is cut off to obtain a target bypass network (new tuning module) that is relatively independent of the backbone network. As a result, during the training of the initial bypass network, there is no need to further calculate the parameter gradient of the backbone network, thereby achieving memory savings and improved training speed, thereby reducing resource consumption during the training process of the model and solving the technical problems of high resource consumption and computational efficiency in the training process of the model.

The embodiment of the present disclosure may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the method for updating parameters in the pre-trained model: obtaining feature data output by the backbone network of the pre-trained model, wherein the backbone network comes from the initial backbone network; calling the initial bypass network to convert the feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; updating the parameters of the initial bypass network based on the converted feature data to obtain a target bypass network, wherein, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the feature data.

Optionally, Fig. 15 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure. As shown in Fig. 15 , the computer terminal A may include: one or more (only one is shown in the figure) processors 1502 , a memory 1504 , and a transmission device 1506 .

Among them, the memory can be used to store software programs and modules, such as the program instructions/modules corresponding to the method and device for updating parameters in the pre-trained model in the embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the method for updating parameters in the pre-trained model mentioned above. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected via a network. To computer terminal A. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the feature data output by the backbone network of the pre-trained model, wherein the backbone network comes from the initial backbone network; call the initial bypass network to convert the feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; update the parameters of the initial bypass network based on the converted feature data to obtain the target bypass network, wherein in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the feature data.

Optionally, the processor may also execute program codes of the following steps: calling a tuning module in the initial bypass network to adjust the characteristic data; and performing weighted summation on the adjusted characteristic data based on the weight of the initial bypass network.

Optionally, the processor may also execute program codes of the following steps: calling a tuning module in the initial bypass network to adjust the characteristic data; and performing weighted summation on the adjusted characteristic data based on the weight of the initial bypass network.

Optionally, the processor may also execute the following program code: based on the weight of the initial bypass network, perform weighted summation of the adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module to obtain the first feature data.

Optionally, the processor may also execute program code of the following steps: in response to the first-layer backbone network being the last layer of network in the backbone network, updating parameters of the initial bypass network based on the first characteristic data to obtain a target bypass network.

Optionally, the processor may also execute the program code of the following steps: in response to the last layer of network in the non-backbone network of the first layer backbone network, calling the second vertical tuning module corresponding to the second layer backbone network in the backbone network to adjust the first feature data, and calling the second horizontal tuning module associated with the second vertical tuning module to adjust the feature data of the second layer backbone network in the backbone network; based on the weight of the initial bypass network, performing weighted summation on the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module to obtain the weighted summed second feature data; in response to the last layer of network in the non-backbone network of the second layer backbone network, The following steps are performed: the second-layer backbone network is determined as the first-layer backbone network, and the third-layer backbone network in the backbone network is determined as the second-layer backbone network; the second vertical tuning module is called to adjust the first feature data, and the second horizontal tuning module is called to adjust the feature data of the second-layer backbone network; based on the weight of the initial bypass network, the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module are weighted summed to obtain the weighted summed second feature data, until the second-layer backbone network is the last layer of the backbone network; the parameters of the initial bypass network are updated based on the converted feature data to obtain the target bypass network, The method comprises: updating the parameters of the initial bypass network based on the second characteristic data to obtain the target bypass network.

Optionally, the processor may also execute the program code of the following steps: determining a processing result obtained by processing the converted feature data by the output layer in the pre-trained model; determining a difference value between the processing result and a true processing result corresponding to the processing result; and adjusting the parameters of the initial bypass network based on the difference value to obtain a target bypass network.

As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: respond to the query information received in the generative interactive interface, wherein the query information at least includes: keywords of the text generation information; call the backbone network of the pre-trained model to at least analyze the text generation information and output text feature data, wherein the backbone network comes from the initial backbone network; call the target bypass network to convert the text feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted text feature data, generate at least one reply result that matches the query instruction; and display the reply result in the generative interactive interface.

Optionally, the processor may also execute program code of the following steps: in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the characteristic data.

As another optional example, the processor may call the information and application stored in the memory through the transmission device to perform the following steps: obtaining feature data obtained by processing the conditional data by the backbone network in the pre-trained model, wherein the backbone network comes from the initial backbone network, and the conditional data is used to determine the generation conditions of the target data; calling the target bypass network to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, generating target data corresponding to the conditional data, wherein the type of the target data includes at least one of the following: text information, image information, video information and voice information.

Optionally, the processor may also execute the program code of the following steps: calling the first vertical tuning module corresponding to the first decoding layer in the backbone network to convert the text feature data output by the middle layer of the backbone network, and calling the first horizontal tuning module associated with the first vertical tuning module to convert the text feature data output by the first decoding layer.

Optionally, the processor may also execute the program code of the following steps: based on the weight of the target bypass network, performing weighted summation on the text feature data adjusted by the first vertical tuning module and the text feature data adjusted by the first horizontal tuning module to obtain converted text feature data; and determining a response result based on the converted text feature data.

Optionally, the processor may also execute the following program code: using the converted text feature data as output data of the last decoding layer in the pre-trained model; and converting the output data into a response result.

As another optional example, the processor can call the information and application stored in the memory through the transmission device. The program is used to execute the following steps: input multimodal information in a dialogue interface, wherein the type of the multimodal information includes at least one of the following: text information containing character information, video frame information containing frame image information, and audio information; call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, wherein the backbone network comes from the initial backbone network; call the target bypass network to convert the feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; based on the converted feature data, generate reply information corresponding to the multimodal information, wherein the type of the reply information includes at least one of the following: text information, image information, video information, and voice information.

Optionally, the processor may also execute program codes of the following steps: calling a tuning module in the target bypass network to adjust the characteristic data; and performing weighted summation on the adjusted characteristic data based on the weight of the target bypass network.

Optionally, the processor may also execute the following program code: processing the converted feature data based on the output layer in the pre-trained model to obtain response information corresponding to the multimodal information.

By adopting the embodiment of the present disclosure, a tuning module extracted from the backbone network is used to construct an initial bypass network. At the same time, during the process of adjusting the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is truncated to obtain a target bypass network (new tuning module) independent of the backbone network. As a result, during the training of the initial bypass network, there is no need to further calculate the parameter gradient of the backbone network, thereby achieving memory savings and improved training speed, thereby achieving the technical effect of reducing resource consumption during the training process of the model, and solving the technical problems of high resource consumption and computing efficiency during the training process of the model.

Those skilled in the art will appreciate that the structure shown in FIG. 15 is for illustration only, and the computer terminal A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. FIG. 15 does not limit the structure of the above-mentioned computer terminal A. For example, the computer terminal A may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG. 15 , or may have a configuration different from that shown in FIG. 15 .

A person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

The embodiment of the present disclosure further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method for updating parameters in the pre-trained model provided in the first embodiment.

Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

Optionally, in this embodiment, the computer readable storage medium is configured to store a computer readable storage medium for executing the above-mentioned processor. The information stored in the memory and the program code executed by the application program can be called up through the transmission device.

In the disclosed embodiment, a tuning module extracted from a backbone network is used to construct an initial bypass network. At the same time, during the process of adjusting the parameters of the initial bypass network, the data flow from the initial bypass network to the backbone network is cut off to obtain a target bypass network (new tuning module) independent of the backbone network. As a result, during the training of the initial bypass network, there is no need to further calculate the parameter gradient of the backbone network, thereby achieving memory savings and improved training speed, thereby achieving the technical effect of reducing resource consumption during the training process of the model, and solving the technical problems of high resource consumption and computational efficiency in the training process of the model.

An embodiment of the present disclosure may provide an electronic device, which may include a memory and a processor.

Figure 16 is a block diagram of an electronic device according to a method for updating parameters in a pre-trained model according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and/or required herein.

As shown in FIG. 16 , the device 1600 includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1602 or a computer program loaded from a storage unit 1608 to a random access memory (RAM) 1603. In the RAM 1603, various programs and data required for the operation of the device 1600 can also be stored. The computing unit 1601, the ROM 1602, and the RAM 1603 are connected to each other via a bus 1604. An input/output (I/O) interface 1605 is also connected to the bus 1604.

A number of components in the device 1600 are connected to the I/O interface 1605, including: an input unit 1606, such as a keyboard, a mouse, etc.; an output unit 1604, such as various types of displays, speakers, etc.; a storage unit 1608, such as a disk, an optical disk, etc.; and a communication unit 1609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1609 allows the device 1600 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

The computing unit 1601 may be a variety of general and/or special processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1601 performs the various methods and processes described above, such as a method for updating parameters in a pre-trained model. For example, in some embodiments, the method for updating parameters in a pre-trained model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1608. In some embodiments, part or all of the computer program may be transmitted to a computer system via ROM 1602 and/or a communication unit. The computer program is loaded and/or installed on the device 1600 by the computing unit 16010. When the computer program is loaded into the RAM 1603 and executed by the computing unit 1601, one or more steps of the method for updating the parameters in the pre-trained model described above may be performed. Alternatively, in other embodiments, the computing unit 1601 may be configured to execute the method for updating the parameters in the pre-trained model by any other appropriate means (e.g., by means of firmware).

Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that are executable and/or interpreted on a programmable system including at least one programmable processor that may be a special purpose or general purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions/operations specified in the flow chart and/or block diagram. The program code may be executed on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a cathode ray tube or liquid crystal display, monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or path ball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); And the input from the user can be received in any form (including acoustic input, voice input or tactile input).

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area networks, wide area networks, and the Internet.

A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

It should be noted that the serial numbers of the above-mentioned embodiments of the present disclosure are only for description and do not represent the advantages or disadvantages of the embodiments.

In the above embodiments of the present disclosure, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

In the several embodiments provided in the present disclosure, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute the methods of each embodiment of the present disclosure. The aforementioned storage medium includes: a USB flash drive, a read-only memory, a random access memory, a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

The above are only preferred embodiments of the present disclosure. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present disclosure. These improvements and modifications should also be regarded as the protection scope of the present disclosure.

Industrial Applicability

The solution provided by the embodiment of the present disclosure can be applied to the process of model training to obtain feature data output by a backbone network of a pre-trained model, wherein the backbone network comes from an initial backbone network; call an initial bypass network to convert the feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; update the parameters of the initial bypass network based on the converted feature data to obtain a target bypass network, wherein, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the feature data, thereby solving the technical problems of high resource consumption and computational efficiency in the model training process.

Claims (20)

A method for updating parameters in a pre-trained model, comprising: Acquire feature data output by a backbone network of a pre-trained model, wherein the backbone network comes from an initial backbone network; Calling an initial bypass network to convert the feature data, wherein the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial trunk network; Based on the converted characteristic data, the parameters of the initial bypass network are updated to obtain a target bypass network, wherein, in the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the trunk network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the characteristic data. The method according to claim 1, wherein calling the initial bypass network to convert the feature data comprises: Calling the tuning module in the initial bypass network to adjust the characteristic data; Based on the weight of the initial bypass network, weighted summation is performed on the adjusted feature data. The method according to claim 2, wherein calling the tuning module in the initial bypass network to adjust the characteristic data comprises: The first vertical tuning module corresponding to the first layer of the backbone network in the backbone network is called to adjust the characteristic data output by the zeroth layer of the backbone network in the backbone network, and the first horizontal tuning module associated with the first vertical tuning module is called to adjust the characteristic data output by the first layer of the backbone network. The method according to claim 3, wherein the weighted summing of the adjusted characteristic data based on the weight of the initial bypass network comprises: Based on the weight of the initial bypass network, a weighted sum is performed on the adjusted feature data corresponding to the first vertical tuning module and the adjusted feature data corresponding to the first horizontal tuning module to obtain first feature data. The method according to claim 4, wherein updating the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network comprises: In response to the first layer of backbone network being the last layer of network in the backbone network, the parameters of the initial bypass network are updated based on the first characteristic data to obtain the target bypass network. The method according to claim 4, wherein updating the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network comprises: In response to the first layer of backbone network not being the last layer of network in the backbone network, calling a second vertical tuning module corresponding to a second layer of backbone network in the backbone network to adjust the first feature data, and calling a second horizontal tuning module associated with the second vertical tuning module to adjust the backbone network Adjusting the characteristic data of the second layer backbone network in the network; Based on the weight of the initial bypass network, performing a weighted summation between the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module to obtain second feature data; In response to the second layer backbone network being not the last layer network in the backbone network, performing the following steps: Determine the second-layer backbone network as the first-layer backbone network, and determine the third-layer backbone network in the backbone network as the second-layer backbone network; Calling the second vertical tuning module to adjust the first characteristic data, and calling the second horizontal tuning module to adjust the characteristic data of the second layer backbone network; Based on the weight of the initial bypass network, weighted sum is performed between the adjusted first feature data corresponding to the second vertical tuning module and the adjusted feature data corresponding to the second horizontal tuning module to obtain the second feature data, until the second-layer backbone network is the last layer of the backbone network; Updating the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network includes: updating the parameters of the initial bypass network based on the second characteristic data to obtain the target bypass network. The method according to claim 1, wherein updating the parameters of the initial bypass network based on the converted characteristic data to obtain the target bypass network comprises: Determine a processing result obtained by processing the converted feature data by an output layer in the pre-trained model; Determine a difference result between the processing result and a true processing result corresponding to the processing result; The parameters of the initial bypass network are adjusted based on the difference value to obtain the target bypass network. The method according to claim 1, wherein the backbone network and the at least one tuning module are connected based on residual. The method according to claim 1, wherein, during the process of transmitting the characteristic data to the initial bypass network, the state of the parameters of the backbone network is a locked state. A method for processing data in a pre-training model, comprising: Respond to the query information received in the generative interactive interface, wherein the query information at least includes: Keywords for text-generated information; Calling a backbone network of a pre-trained model to at least analyze the text generation information and output text feature data, wherein the backbone network comes from an initial backbone network; The target bypass network is called to convert the text feature data, wherein the target bypass network is obtained by updating the parameters of the initial bypass network, and the initial bypass network is based on at least one tuning The module is constructed, and the tuning module is extracted from the initial backbone network; Based on the converted text feature data, generating at least one reply result matching the query instruction; The response result is displayed in the generative interactive interface. The method according to claim 10, wherein In the process of updating the parameters of the initial bypass network, the data flow of the initial bypass network is independent of the data flow of the backbone network, and the parameters of the initial bypass network are used to characterize the influence of the tuning module on the text feature data. The method according to claim 11, wherein calling the target bypass network to convert the text feature data comprises: The first vertical tuning module corresponding to the first decoding layer in the backbone network is called to convert the text feature data output by the middle layer of the backbone network, and the first horizontal tuning module associated with the first vertical tuning module is called to convert the text feature data output by the first decoding layer. The method according to claim 12, wherein generating at least one reply result matching the query instruction based on the converted text feature data comprises: Based on the weight of the target bypass network, weighted summing is performed on the text feature data adjusted by the first vertical tuning module and the text feature data adjusted by the first horizontal tuning module to obtain text feature data after weighted summing; The reply result is determined based on the converted text feature data. The method according to claim 11, wherein determining the reply result based on the converted text feature data comprises: Using the converted text feature data as output data of the last decoding layer in the pre-training model; The output data is converted into the response result. A data processing method for a pre-training model, comprising: Inputting multimodal information in the dialogue interface, wherein the type of the multimodal information includes at least one of the following: text information including character information, video frame information including frame image information, and audio information; Calling a backbone network in a pre-trained model to at least analyze and process the multimodal information to obtain feature data, wherein the backbone network comes from an initial backbone network; Calling a target bypass network to convert the characteristic data, wherein the target bypass network is obtained by updating parameters of an initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network; Based on the converted feature data, a reply message corresponding to the multimodal information is generated, wherein the type of the reply message includes at least one of the following: text information, image information, video information and voice information. information. The method according to claim 15, wherein calling the target bypass network to convert the feature data comprises: Calling the tuning module in the target bypass network to adjust the characteristic data; Based on the weight of the target bypass network, a weighted sum is performed on the adjusted characteristic data. The method according to claim 15, wherein generating reply information corresponding to the multimodal information based on the converted feature data comprises: The converted feature data is processed based on the output layer in the pre-trained model to obtain the response information corresponding to the multimodal information. A data processing system for a pre-trained model, comprising: The client is configured to display a dialogue interface and capture multimodal information input in the dialogue interface, wherein the type of the multimodal information includes at least one of the following: inquiry information including character information, video frame information including frame image information, and audio information; The server is configured to call the backbone network in the pre-trained model to at least analyze and process the multimodal information to obtain feature data, and call the target bypass network to convert the feature data, and generate reply information corresponding to the multimodal information based on the converted feature data, wherein the backbone network comes from the initial backbone network, the target bypass network is obtained by updating the parameters of the initial bypass network, the initial bypass network is constructed based on at least one tuning module, and the tuning module is extracted from the initial backbone network, and the type of the reply information includes at least one of the following: text information, image information, video information and voice information. An electronic device comprises: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method described in any one of claims 1 to 17. A computer program product comprises a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 17 is implemented.