WO2024017287A1 - Model training method and apparatus therefor - Google Patents

Abstract

A model training method, which relates to the field of artificial intelligence, and comprises: acquiring a first text and a second text, the first text and the second text being obtained via extraction from within an original code text, wherein the first text is a program code, the second text is an attribute description corresponding to the program code, and the first text and the second text comprise a known text and a prediction text; and according to the known text, by means of a natural language model, predicting a text of a text position corresponding to the prediction text to obtain a text prediction result, wherein the prediction text and the text prediction result are used for updating the natural language model. According to the present application, when a training sample synthesized by a program is constructed, a code text and a corresponding attribute description are extracted from within an original program code and serve as a pair of samples, thus reducing training difficulty and improving the prediction precision of a model.

Description

A model training method and its device

This application claims priority to the Chinese patent application submitted to the China Patent Office on July 20, 2022, with the application number 202210857821.1 and the invention title "A model training method and its device", the entire content of which is incorporated into this application by reference. middle.

Technical field

The present application relates to the field of artificial intelligence, and in particular, to a model training method and its device.

Background technique

A language model refers to a model that can predict unknown words in a sentence based on a part of given semantic fragments. For example: given a natural language sequence fragment "Huawei __ is very good.", the language model can generate unknown words based on the fragment. For example, in this example, the language model can generate the word "mobile phone" based on the given fragment, and then obtain a sentence For "Huawei mobile phones are very good.".

Language model pre-training refers to training the corresponding language model through massive language sequence corpus, so that the language model has the ability to predict the occurrence probability of a certain language unit at a certain position.

Program synthesis refers to using a certain software program to automatically synthesize a specified program language sequence with specified functions or structures. In short, it is a program that generates a program. Generally speaking, it is divided into two categories: program synthesis (code generation) and code completion. kind. Program synthesis, that is, starting from scratch, can generate corresponding code according to a certain natural language description or a preset function description. Code completion, that is, based on the existing code sequence, continue to generate the following code sequence.

With the development of deep learning technology, the use of deep learning for sequence generation has been widely used in natural language sequence generation scenarios. In particular, the emergence of pre-trained language generation model technology has led to a qualitative improvement in language generation capabilities. In recent years, generative models such as generative pre-training (GPT), T5, and BART have gradually been introduced into code generation or completion. This technology uses Transformer as the basic component, builds a decoder or encoder-decoder model architecture, and then uses a large amount of code data for training to obtain the ability to output the corresponding code context based on the above, thereby completing or generating code.

The existing technology follows the generation method of natural language processing and trains based on the original code. Currently, this method can continue to write the code based on the existing context. However, for the scenario of function-level code generation, the quality of the code generation has yet to be improved. promote.

Contents of the invention

This application provides a model training method that can reduce the difficulty of training and improve the quality of code generation.

In a first aspect, this application provides a model training method, including: obtaining a first text and a second text, where the first text and the second text are extracted from the original code text; wherein, the The first text is the program code, and the second text is the attribute description corresponding to the program code; the first text and the second text include known text and predicted text; according to the known text, through natural A language model predicts the text corresponding to the text position of the predicted text to obtain a text prediction result; the predicted text and the text prediction result are used to update the natural language model.

In a possible implementation, the attribute description includes a function description of the program code or an implementation principle of the program code (or it is also called an explanation of the program code).

In a possible implementation, the original code text can be existing programming language corpus and natural language corpus. Optionally, existing programming language corpus and natural language corpus can be obtained through the Internet (or other methods). The corpus includes code files of mixed programming language and natural language, code files of programming language, and natural language files. The collected natural language can be any human language, and the programming language can be any programming language, which is not limited by this application.

In a possible implementation, the first text and the second text are extracted from the original code text; wherein the first text is the program code, and the second text is the program code. Corresponding attribute description. The second text can be in natural language, and the second text can be in any language used for human communication, and the program code can be in any programming language, which is not limited by this application.

In a possible implementation, after identifying the first text and the second text, a correspondence relationship between the first text and the second text can be established to form a "natural language description-function implementation code" (Description-Function code) ) sentence pair form sample. Optionally, you can use a character combination before each set of natural language-program language to indicate that the following sequence is a text language sequence or a specific code language sequence, and add a sample terminator after the sample to indicate the first The text and the second text are a pair of samples.

In the embodiment of the present application, when constructing a program-synthesized training sample, the code text and corresponding attribute description are extracted from the original program code and used as a pair of samples. Compared with the prior art, the original program file is directly used as a training sample. , helping the natural language model to establish an association between the code text and the corresponding attribute description during training, without having to learn the ability to identify this association from samples during the training process, reducing the training difficulty and improving the prediction of the model Accuracy.

In one possible implementation, in order to improve the quality of the training corpus, function-level code snippets (such as code programs) can be removed based on the statistical characteristics of the code corpus, and the corpus with too short natural language description part and the excessive code part in the code corpus can be removed. Long corpus or corpus with a large length difference between the two to improve the quality of the training corpus.

In a possible implementation, the attribute description includes a function description of the program code or an implementation principle of the program code.

In a possible implementation, the first text is identified from the original code text through a program code identification method, and the second text is identified from the original code text through an attribute description identification method.

In one possible implementation, the function code segment in the original code text can be identified as the first text: for example, the original code text can be identified using certain rules (for example, a program code identification method such as a code abstract syntax tree (AST) can be used) function code snippet in .

In a possible implementation, certain rules can be used, such as annotation rules, language character rules and other attribute description recognition methods for identification and extraction.

For example, abstract syntax tree (AST) analysis of the code can be performed, the code can be analyzed to obtain the syntax tree of the corresponding code, and the natural language description corresponding to the corresponding function-level code fragment can be intercepted through the syntax tree analysis.

For example, natural language processing automatic identification methods can be used to extract code segments, such as sequence annotation methods, and a function annotator is trained to annotate natural language descriptions corresponding to all function-level codes.

In a possible implementation, the program code is the complete code of one or more functions.

In a possible implementation, in order to train a model that can be used to perform program generation functions (that is, the natural language model in the embodiment of this application), some text units in the training sample can be used as known text, and the other part of the text As the text to be predicted, the text at the text position where the text to be predicted is predicted based on the known text.

Since the first text and the second text are different types of text (the first text is a computer compiled language and the second text is a natural language), the text prediction process can be performed with different training objectives for the first text and the second text.

For example, for the first text, one of the four training objectives of CLM, MLM, MCLM and NoLM can be used to perform the text prediction process.

For example, for the second text, one of the three training objectives of CLM, MLM and MCLM can be used to perform the model training process.

When optimizing the natural language description part, you can choose one of four training objectives: CLM, MLM, MCLM, and NoLM. When optimizing the program language function code part, you can choose one of the three training objectives: CLM, MLM, and MCLM. Note that the NoLM training objective does not need to be included here. The training objectives of the natural language description part and the programming language function code part can be freely combined.

In a possible implementation, when predicting the text in the first text, the text position corresponding to the predicted text may be determined from the first text in a first manner; When predicting text in two texts, the text position corresponding to the predicted text can be determined from the second text through a second method; the first method and the second method are different.

In a possible implementation, the first method and the second method are respectively one of the following prediction methods:

Sample mask positions from the text positions of part or all of the first text or the second text, and use the mask positions as the text positions corresponding to the predicted text (that is, MLM);

Use the following text position of the known text as the text position corresponding to the predicted text; wherein the known text is all the texts that have been predicted by the natural language model (that is, CLM);

The following text position of the known text is used as the text position corresponding to the predicted text; wherein the known text is the unmasked text among all the texts that have been predicted by the natural language model (that is, MCLM ).

In a possible implementation, without predicting the text in the second text (that is, NoLM), the text position corresponding to the predicted text can be determined through one of the following prediction methods:

Sample mask positions from the text positions of part or all of the first text, and use the mask positions as the text positions corresponding to the predicted text (that is, MLM);

Use the following text position of the known text as the text position corresponding to the predicted text; wherein the known text is all the texts that have been predicted by the natural language model (that is, CLM);

The following text position of the known text is used as the text position corresponding to the predicted text; wherein the known text is the unmasked text among all the texts that have been predicted by the natural language model (that is, MCLM ).

About CLM:

In the feedforward process of model training, the natural language model can predict texts sequentially along the preset context prediction direction. In other words, all words that have been predicted by the natural language model can be used as known text, and have been The known text is used as the context of the position text to predict the text location of the text. For example, prediction can be made in the order from the context to the context (that is, from the beginning of the text to the end of the text), each time It is possible to predict a text unit and continue to predict based on the already predicted text unit in the next prediction.

About MLM:

In the feedforward process of model training, the text units in the text (for example, the second text or the first text) can be masked (for example, they can be randomly masked), and the masked text units can be used as predicted texts. The masked text unit can be used as known text, and the natural language model can sequentially calculate the text location of the masked text unit based on the unmasked text unit (or the predicted masked text unit). Text prediction.

About MCLM:

In the feedforward process of model training, the natural language model can predict text sequentially along the preset context prediction direction. The difference from CLM is that in CLM, all words that have been predicted by the natural language model can be used as Known text, and the known text is used as the upper text of the position text to predict the text position of the predicted text. In MCLM, the words that have been predicted by the natural language model in MCLM can be masked, and natural language can be used The unmasked text units in the words that have been predicted by the model are used to predict the text location of the predicted text.

About NoLM:

In the feedforward process of model training, the second text may not be predicted, but the second text may be directly used as the known text, and the first text may be used as a training sample to predict text units in the first text.

In a possible implementation, the four optional training objectives (CLM, MLM, MCLM, and NoLM) can be applied to the natural language description or function code of the "Description-Function pair" (Description-Function) respectively, and the combination 12 types of training objectives are formed.

In one possible implementation, in order to improve the quality of the training corpus, function-level code snippets (such as code programs) can be removed based on the statistical characteristics of the code corpus, and the corpus with too short natural language description part and the excessive code part in the code corpus can be removed. Long corpus or corpus with a large length difference between the two to improve the quality of the training corpus.

In a possible implementation, since training samples are usually difficult to obtain, that is, the number of training samples (text pairs composed of the first text and the second text) is small, data enhancement can be performed on the obtained training samples to obtain More training samples.

In a possible implementation, the first text is obtained based on the first original text extracted from the original code text; wherein the first text is a partial text unit in the first original text. Obtained by modification.

In a possible implementation, the modification includes: deletion, addition or replacement.

In a possible implementation, the replacement is specifically a replacement with the same semantics. For example, synonymous segments may be generated based on back translation technology, or synonymous segments may be generated based on synonymous sentence generation technology.

For example, you can randomly replace variable names in the code text, replace equivalent (loop) code segments, and randomly insert invalid code segments to obtain a large amount of high-quality automatically generated training data, ultimately improving model performance.

In a possible implementation, the second text is obtained based on the second original text extracted from the original code text; wherein the second text is a partial text unit in the second original text. Obtained by modification.

In a possible implementation, the modification includes: deletion, addition or replacement.

In a possible implementation, the replacement is specifically a replacement with the same semantics. For example, synonymous fragments can be generated based on back translation technology, Alternatively, synonymous fragments may be generated based on synonymous sentence generation technology.

For example, you can augment the natural language description by replacing synonyms, transforming sentence patterns, or using paraphrase generation models or methods in natural language processing technology, data augmentation methods, and data noise methods to obtain a large number of information. High-quality automatically generated training data ultimately improves model performance.

In a possible implementation, training samples can be processed differently at different stages of model training. For example, in the initial iterative stage of model training, the first text and the second text obtained in the above way can be used, and in the model training In the fine-tuning stage, text adapted to the application field of the program code (also including program code and corresponding code attribute descriptions) can be extracted from the original code text as a training sample.

In a possible implementation, the first text or the second text is text extracted from the original code text and adapted to the application field of the program code. In a possible implementation, the adaptation to the application field of the program code includes: text whose length or the length of a single line of text is less than a threshold; wherein the threshold is related to the application field of the program code. Alternatively, information in other dimensions besides length can also be used to quantify and apply the program code.

In one possible implementation, the samples may be filtered based on their similarity to data samples from the application domain of the function-level code generator. The similarity calculation method can be the length relationship between natural language descriptions, the length relationship between program language function codes, or other relationships.

In the second aspect, this application provides a model training device, including:

An acquisition module is used to acquire a first text and a second text. The first text and the second text are extracted from the original code text; wherein the first text is program code, and the second text is extracted from the original code text. The text is an attribute description corresponding to the program code; the first text and the second text include known text and predicted text;

A prediction module, configured to predict the text corresponding to the text position of the predicted text through a natural language model based on the known text, and obtain a text prediction result; the predicted text and the text prediction result are used to update the Natural language model.

In a possible implementation, the attribute description includes a function description of the program code or an implementation principle of the program code.

In a possible implementation, the device further includes:

A text position determination module, configured to determine the text position corresponding to the predicted text from the first text in a first manner when predicting the text in the first text;

When predicting the text in the second text, the text position corresponding to the predicted text is determined from the second text through the second method; the first method and the second method are different.

In a possible implementation, the first method and the second method are respectively one of the following prediction methods:

Sample mask positions among text positions of part or all of the first text or second text, and use the mask positions as the text positions corresponding to the predicted text;

Use the following text position of the known text as the text position corresponding to the predicted text; wherein the known text is all the text that has been predicted by the natural language model;

The following text position of the known text is used as the text position corresponding to the predicted text; wherein the known text is the unmasked text among all the texts that have been predicted by the natural language model.

In a possible implementation, the text position determination module is also configured to determine the text position corresponding to the predicted text through one of the following prediction methods without predicting the text in the second text:

Sample mask positions from text positions of part or all of the first text, and use the mask positions as text positions corresponding to the predicted text;

Use the following text position of the known text as the text position corresponding to the predicted text; wherein the known text is all the text that has been predicted by the natural language model;

The following text position of the known text is used as the text position corresponding to the predicted text; wherein the known text is the unmasked text among all the texts that have been predicted by the natural language model.

In one possible implementation,

The first text is obtained based on the first original text extracted from the original code text; wherein the first text is obtained by modifying some text units in the first original text; or,

The second text is obtained based on the second original text extracted from the original code text; wherein the second text is obtained by modifying some text units in the second original text.

In a possible implementation, the modification includes: deletion, addition or replacement.

In a possible implementation, the replacement is specifically a replacement with the same semantics.

In a possible implementation, the first text or the second text is text extracted from the original code text and adapted to the application field of the program code.

In a possible implementation, the application fields adapted to the program code include:

The text length or the length of a single line of text is less than a threshold; wherein the threshold is related to the application field of the program code.

In a third aspect, embodiments of the present application provide a model training device, which may include a memory, a processor, and a bus system. The memory is used to store programs, and the processor is used to execute programs in the memory to perform the above-mentioned first aspect. and any optional methods.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium that stores a computer program that, when run on a computer, causes the computer to execute the first aspect and any one of the above-mentioned aspects. Optional method.

In a fifth aspect, embodiments of the present application provide a computer program that, when run on a computer, causes the computer to execute the above-mentioned first aspect and any optional method thereof.

In a sixth aspect, the present application provides a chip system, which includes a processor to support an execution device or a training device to implement the functions involved in the above aspects, for example, sending or processing data involved in the above methods; Or, information. In a possible design, the chip system further includes a memory, and the memory is used to store necessary program instructions and data for executing the device or training the device. The chip system may be composed of chips, or may include chips and other discrete devices.

Description of drawings

Figure 1A is a schematic structural diagram of the main framework of artificial intelligence;

Figures 1B and 1C are schematic diagrams of the application system framework of the present invention;

Figure 1D is a schematic diagram of an optional hardware structure of the terminal;

Figure 2 is a schematic structural diagram of a server;

Figures 3 to 5 are schematic diagrams of a system architecture of this application;

Figure 6 shows the process of a cloud service;

Figure 7 shows the process of a cloud service;

Figure 8 is a schematic diagram of a system architecture of this application;

Figure 9 is a schematic flowchart of a model training method provided by an embodiment of the present application;

Figures 10A to 10D are diagrams of the text;

Figure 11 is a schematic diagram of the architecture of a model training method provided by an embodiment of the present application;

Figures 12 and 13 are a schematic of the model;

Figure 14 is a schematic structural diagram of a model training device provided by an embodiment of the present application;

Figure 15 is a schematic structural diagram of an execution device provided by an embodiment of the present application;

Figure 16 is a schematic structural diagram of the training equipment provided by the embodiment of the present application;

Figure 17 is a schematic structural diagram of a chip provided by an embodiment of the present application.

Detailed ways

The embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention. The terms used in the embodiments of the present invention are only used to explain specific embodiments of the present invention and are not intended to limit the present invention.

The embodiments of the present application are described below with reference to the accompanying drawings. Persons of ordinary skill in the art know that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

The terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms so used are interchangeable under appropriate circumstances, and are merely a way of distinguishing objects with the same attributes in describing the embodiments of the present application. Furthermore, the terms "include" and "having" and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, product or apparatus comprising a series of elements need not be limited to those elements, but may include not explicitly other elements specifically listed or inherent to such processes, methods, products or equipment.

As used herein, the terms "substantially", "about" and similar terms are used as terms of approximation, not as terms of degree, and are intended to take into account measurements or values that would be known to one of ordinary skill in the art. The inherent bias in calculated values. In addition, the use of "may" when describing embodiments of the present invention refers to "one or more possible embodiments." As used herein, the terms "use", "using", and "used" may be deemed to be the same as the terms "utilize", "utilizing", and "utilize", respectively. Synonymous with "utilized". Additionally, the term "exemplary" is intended to refer to an example or illustration.

First, the overall workflow of the artificial intelligence system is described. Please refer to Figure 1A. Figure 1A shows a structural schematic diagram of the artificial intelligence main framework. The following is from the "intelligent information chain" (horizontal axis) and "IT value chain" ( The above artificial intelligence theme framework is elaborated on the two dimensions of vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has gone through the condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (providing and processing technology implementation) to the systematic industrial ecological process.

(1)Infrastructure

Infrastructure provides computing power support for artificial intelligence systems, enables communication with the external world, and supports it through basic platforms. Communicate with the outside through sensors; computing power is provided by smart chips (hardware acceleration chips such as CPU, NPU, GPU, ASIC, FPGA, etc.); the basic platform includes distributed computing framework and network and other related platform guarantees and support, which can include cloud storage and Computing, interconnection networks, etc. For example, sensors communicate with the outside world to obtain data, which are provided to smart chips in the distributed computing system provided by the basic platform for calculation.

(2)Data

Data from the upper layer of the infrastructure is used to represent data sources in the field of artificial intelligence. The data involves graphics, images, voice, and text, as well as IoT data of traditional devices, including business data of existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.

(3)Data processing

Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

Among them, machine learning and deep learning can perform symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc. on data.

Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formal information to perform machine thinking and problem solving based on reasoning control strategies. Typical functions are search and matching.

Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

(4) General ability

After the data is processed as mentioned above, some general capabilities can be formed based on the results of further data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, and image processing. identification, etc.

(5) Intelligent products and industry applications

Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Its application fields mainly include: intelligent terminals, intelligent transportation, Smart healthcare, autonomous driving, smart cities, etc.

This application can be applied to the field of natural language processing in the field of artificial intelligence. Taking natural language processing as an example, multiple application scenarios that have been implemented into products will be introduced below.

First, the application scenarios of this application are introduced. This application can be, but is not limited to, applied to applications that include program synthesis functions or code completion functions based on code attribute descriptions (hereinafter may be referred to as program synthesis applications for short) or cloud-side servers. The cloud services provided are introduced below:

1. Program synthesis applications

The product form of the embodiment of the present application may be a program synthesis application program. Program synthesis applications can run on terminal devices or cloud-side servers.

In a possible implementation, the program synthesis application can implement program synthesis or code completion tasks based on code attribute descriptions (or prompts), where the program synthesis application can respond to input The code attribute description (or can be called a prompt) is used to perform the task of program synthesis or code completion, and the predicted text (that is, the generated code program) is obtained. The generated code program can conform to the code attribute description.

In a possible implementation, the user can open a program synthesis application installed on the terminal device and enter an attribute description of the code (or it can be called a prompt). The program synthesis application can use the program provided by the embodiment of the present application. The natural language model trained by the method performs program synthesis or code completion on the attribute description of the code, and presents the predicted text (that is, the generated code program) to the user (the presentation method can be, but is not limited to, displaying, saving, or uploading to the cloud side, etc.).

In one possible implementation, the user can open a program synthesis application installed on the terminal device and enter the attribute description of the code. The program synthesis application can send the attribute description of the code to the server on the cloud side. The server performs program synthesis or code completion on the attribute description of the code using the natural language model trained by the method provided in the embodiments of this application, and sends the predicted text (that is, the generated code program) back to the terminal device, and the terminal device can The predicted text (that is, the generated code program) is presented to the user (the presentation method can be but is not limited to display, saving, uploading to the cloud, etc.).

Next, the program synthesis application in the embodiment of this application will be introduced from the perspective of functional architecture and product architecture that implements the functions.

Referring to Figure 1B, Figure 1B is a schematic diagram of the functional architecture of a program synthesis application in an embodiment of the present application:

In a possible implementation, as shown in FIG. 1B , the program synthesis application 102 can receive input parameters 101 (for example, attribute descriptions containing codes) and generate predictive text (that is, a generated code program) 103 . Program synthesis application 102 may be executed, for example, on at least one computer system and may include computer code that, when executed by one or more computers, causes the computers to perform operations performed by the present application. The natural language model trained by the method provided in the example.

Referring to Figure 1C, Figure 1C is a schematic diagram of the entity architecture of running a program synthesis application in an embodiment of the present application:

Referring to Figure 1C, Figure 1C shows a schematic diagram of a system architecture. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers (one server is used as an example for illustration in FIG. 1C), and the server 200 may provide program synthesis function services for one or more terminals.

Among them, the terminal 100 can be installed with a program synthesis application, or open a web page related to the program synthesis function. The above-mentioned application program and web page can provide an interface, and the terminal 100 can receive relevant parameters input by the user on the program synthesis function interface. The above parameters are sent to the server 200. The server 200 can obtain the processing results based on the received parameters and return the processing results to the terminal 100.

It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without requiring the cooperation of the server, which is not limited by the embodiments of this application.

Next, the product form of the terminal 100 in Figure 1C is described;

The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR)/virtual reality (VR) device, a notebook computer, or an ultra mobile personal computer (ultra mobile personal computer). - mobile personal computer (UMPC), netbook, personal digital assistant (personal digital assistant, PDA), etc., the embodiments of this application do not impose any restrictions on this.

FIG. 1D shows an optional hardware structure diagram of the terminal 100.

Referring to FIG. 1D , the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), Microphone 162 (optional), processor 170, external interface 180, power supply 190 and other components. Those skilled in the art can understand that Figure 1D is only an example of a terminal or a multi-function device, and does not constitute a limitation to the terminal or multi-function device. It may include more or fewer components than shown in the figure, or combine certain components. Or different parts.

The input unit 130 may be used to receive input numeric or character information and generate key signal input related to user settings and function control of the portable multi-function device. Specifically, the input unit 130 may include a touch screen 131 (optional) and/or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operations on or near the touch screen using fingers, knuckles, stylus, or any other suitable objects), and drive the corresponding according to the preset program. Connect the device. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute commands from the processor 170; the touch signal at least includes contact point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, touch screens can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 may also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys 132, switch keys 133, etc.), trackball, mouse, joystick, etc.

The input device 132 may receive an attribute description of the input code and the like.

The display unit 140 may be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file display, and/or playback of any kind of multimedia files. In this embodiment of the present application, the display unit 140 may be used to display the interface of a program synthesis application, the generated predictive text (that is, the generated code program), etc.

The memory 120 can be used to store instructions and data. The memory 120 can mainly include a storage instruction area and a storage data area. The storage data area can store various data, such as multimedia files, text, etc.; the storage instruction area can store operating systems, applications, at least Software units such as instructions required for a function, or their subsets or extensions. Non-volatile random access memory may also be included; providing the processor 170 with management of hardware, software and data resources in the computing processing device and supporting control software and applications. It is also used for storage of multimedia files, as well as storage of running programs and applications.

The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100, and executes various functions of the terminal 100 by running or executing instructions stored in the memory 120 and calling data stored in the memory 120. functions and process data to provide overall control of the terminal device. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, where the application processor mainly processes operating systems, user interfaces, application programs, etc. , the modem processor mainly handles wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 170 . In some embodiments, the processor and memory can be implemented on a single chip, and in some embodiments, they can also be implemented on separate chips. The processor 170 can also be used to generate corresponding operation control signals, send them to corresponding components of the computing processing device, read and process data in the software, especially read and process the data and programs in the memory 120, so that the Each functional module performs a corresponding function, thereby controlling the corresponding components to act according to the instructions.

The memory 120 can be used to store software codes related to the data processing method, and the processor 170 can execute the steps of the data processing method of the chip, and can also schedule other units (such as the above-mentioned input unit 130 and the display unit 140) to implement corresponding functions. .

The radio frequency unit 110 (optional) can be used to send and receive information or receive and send signals during calls. For example, after receiving downlink information from the base station, it is processed by the processor 170; in addition, the designed uplink data is sent to the base station. Typically, RF circuits include but are not limited to antennas, at least one amplifier, transceivers, couplers, low noise amplifiers (LNA), duplexers, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (Code Division) Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

In this embodiment of the present application, the radio frequency unit 110 can send the attribute description of the code to the server 200, and receive the predicted text (that is, the generated code program) sent by the server 200.

It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

The terminal 100 also includes a power supply 190 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that functions such as charging, discharging, and power consumption management can be implemented through the power management system.

The terminal 100 also includes an external interface 180, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect the terminal 100 to communicate with other devices, or can be used to connect a charger to charge the terminal 100. .

Although not shown, the terminal 100 may also include a flash light, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described again here. Some or all of the methods described below may be applied in the terminal 100 as shown in FIG. 1D.

Next, the product form of the server 200 in Figure 1C will be described;

Figure 2 provides a schematic structural diagram of a server 200. As shown in Figure 2, the server 200 includes a bus 201, a processor 202, a communication interface 203 and a memory 204. The processor 202, the memory 204 and the communication interface 203 communicate through the bus 201.

The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one thick line is used in Figure 2, but it does not mean that there is only one bus or one type of bus.

The processor 202 may be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP). any one or more of them.

Memory 204 may include volatile memory, such as random access memory (RAM). The memory 204 may also include non-volatile memory (non-volatile memory), such as read-only memory (ROM), flash memory, mechanical hard drive (hard drive drive, HDD) or solid state drive (solid state drive). ,SSD).

Among them, the memory 204 can be used to store software codes related to the data processing method, and the processor 202 can execute the steps of the data processing method of the chip, and can also schedule other units to implement corresponding functions.

It should be understood that the terminal 100 and the server 200 may be centralized or distributed devices, and the processors (such as the processor 170 and the processor 202) in the terminal 100 and the server 200 may be hardware circuits (such as application specific integrated circuits) application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processing (DSP), microprocessor or microcontroller, etc.), Or a combination of these hardware circuits. For example, the processor can be a hardware system with the function of executing instructions, such as CPU, DSP, etc., or a hardware system without the function of executing instructions, such as ASIC, FPGA, etc., or the above-mentioned processor without the function of executing instructions. A combination of a hardware system with the instruction function and a hardware system with the function of executing the instruction.

It should be understood that the steps related to the model reasoning process in the embodiments of this application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and server is not limited to the processor-memory architecture introduced above. The system architecture provided by the embodiment of the present application will be introduced in detail below with reference to Figure 5 .

Figure 5 is a schematic diagram of the system architecture provided by the embodiment of the present application. As shown in Figure 5, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550 and a data collection system 560.

The execution device 510 includes a computing module 511, an I/O interface 512, a preprocessing module 513 and a preprocessing module 514. The target model/rule 501 may be included in the calculation module 511, and the preprocessing module 513 and the preprocessing module 514 are optional.

The execution device 510 may be the above-mentioned terminal device or server that runs the program synthesis application.

Data collection device 560 is used to collect training samples. The training sample can be a program file (including program code and attribute description of the program code), etc. After collecting the training samples, the data collection device 560 stores the training samples into the database 530 .

The training device 520 can maintain the training samples in the database 530 and the neural network to be trained (such as the natural language model in the embodiment of the present application, etc.) to obtain the target model/rule 501.

It should be noted that in actual applications, the training samples maintained in the database 530 are not necessarily collected from the data collection device 560, and may also be received from other devices. In addition, it should be noted that the training device 520 may not necessarily train the target model/rules 501 based entirely on the training samples maintained by the database 530. It may also obtain training samples from the cloud or other places for model training. The above description should not be used as a guarantee for this application. Limitations of Examples.

The target model/rules 501 trained according to the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in Figure 5. The execution device 510 can be a terminal, such as a mobile phone terminal, a tablet computer, and a notebook. Computers, augmented reality (AR)/virtual reality (VR) equipment, vehicle-mounted terminals, etc., and can also be servers, etc.

Specifically, the training device 520 can transfer the trained model to the execution device 510 .

In Figure 5, the execution device 510 is configured with an input/output (I/O) interface 512 for data interaction with external devices. The user can input data to the I/O interface 512 through the client device 540 (for example, this Attribute description of the code in the application embodiment, etc.).

The preprocessing module 513 and the preprocessing module 514 are used to perform preprocessing according to the input data received by the I/O interface 512. It should be understood that there may be no preprocessing module 513 and 514 or only one preprocessing module. When the preprocessing module 513 and the preprocessing module 514 do not exist, the computing module 511 can be directly used to process the input data.

When the execution device 510 preprocesses input data, or when the calculation module 511 of the execution device 510 performs calculations and other related processes, the execution device 510 can call data, codes, etc. in the data storage system 550 for corresponding processing. , the data, instructions, etc. obtained by corresponding processing can also be stored in the data storage system 550.

Finally, the I/O interface 512 provides the processing results (such as predictive text (that is, the generated code program), etc.) to the client device 540 and thereby to the user.

In the situation shown in FIG. 5 , the user can manually set the input data, and the "manually set input data" can be operated through the interface provided by the I/O interface 512 . In another case, the client device 540 can automatically send input data to the I/O interface 512. If requiring the client device 540 to automatically send the input data requires the user's authorization, the user can set corresponding permissions in the client device 540. The user can view the results output by the execution device 510 on the client device 540, and the specific presentation form may be display, sound, action, etc. The client device 540 can also be used as a data collection terminal to collect the input data of the input I/O interface 512 and the output results of the output I/O interface 512 as new sample data, and store them in the database 530. Of course, it is also possible to collect without going through the client device 540. Instead, the I/O interface 512 directly uses the input data input to the I/O interface 512 and the output result of the output I/O interface 512 as a new sample as shown in the figure. The data is stored in database 530.

It is worth noting that Figure 5 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 5, the data The storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the above execution device 510 may be deployed in the client device 540.

From the inference side of the model:

In this embodiment of the present application, the computing module 511 of the above-mentioned execution device 520 can obtain the code stored in the data storage system 550 to implement the steps related to the model inference process in the embodiment of this application.

In the embodiment of the present application, the computing module 511 of the execution device 520 may include hardware circuits (such as application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processing (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 can be a hardware system with the function of executing instructions, such as a CPU, DSP, etc. , or it is a hardware system that does not have the function of executing instructions, such as ASIC, FPGA, etc., or it is a combination of the above-mentioned hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions.

Specifically, the calculation module 511 of the execution device 520 can be a hardware system with the function of executing instructions. The steps related to the model reasoning process provided by the embodiment of the present application can be software codes stored in the memory. The calculation module 511 of the execution device 520 The software code can be obtained from the memory, and the obtained software code can be executed to implement the steps related to the model inference process provided by the embodiments of the present application.

It should be understood that the computing module 511 of the execution device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to the model inference process provided by the embodiments of the present application can also be implemented by The computing module 511 of the execution device 520 is implemented by a hardware system that does not have the function of executing instructions, which is not limited here.

From the training side of the model:

In the embodiment of the present application, the above-mentioned training device 520 can obtain the code stored in the memory (not shown in Figure 5, which can be integrated with the training device 520 or deployed separately from the training device 520) to implement the model training in the embodiment of the present application. Related steps.

In the embodiment of the present application, the training device 520 may include hardware circuits (such as application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (digital signal processing, DSP, microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 can be a hardware system with the function of executing instructions, such as a CPU, DSP, etc., or a combination of other hardware circuits. A hardware system with the function of executing instructions, such as ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without the function of executing instructions and a hardware system with the function of executing instructions.

It should be understood that the training device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to model training provided by the embodiments of the present application can also be implemented by the training device 520 that does not have the function of executing instructions. It is implemented by the hardware system that executes the instruction function, which is not limited here.

2. Program synthesis functional cloud services provided by the server:

In a possible implementation, the server can provide program synthesis function services for the client side through an application programming interface (API).

Among them, the terminal device can send relevant parameters (such as attribute descriptions containing codes) to the server through the API provided by the cloud. The server can obtain processing results (such as predicted text (that is, the generated code program)) based on the received parameters. ) and return the processing results to the terminal.

The description of the terminal and the server can be as described in the above embodiment, and will not be described again here.

Figure 6 shows the process of using a program provided by a cloud platform to synthesize functional cloud services.

1. Activate and purchase the content review service.

2. Users can download the software development kit (SDK) corresponding to the content review service. Usually the cloud platform provides multiple development versions of SDK for users to choose according to the needs of the development environment, such as JAVA version of SDK and python version. SDK, PHP version SDK, Android version SDK, etc.

3. After users download the corresponding version of the SDK locally according to their needs, they can import the SDK project into the local development environment and configure and debug it in the local development environment. The local development environment can also develop other functions, forming a collection of programs. Application of synthetic functional abilities.

4. During the use of the program synthesis function application, when the program synthesis function is needed, the API call of the program synthesis function can be triggered. When the application triggers the program synthesis function, it initiates an API request to the running instance of the program synthesis function class service in the cloud environment. The API request carries the attribute description of the code, and the running instance in the cloud environment performs the attribute description of the code. Process and obtain the processing results (such as predictive text (that is, the generated code program), etc.).

5. The cloud environment returns the processing results to the application, thus completing a program synthesis function service call.

3. Model training cloud services provided by the server:

In a possible implementation, the server can provide a model of the program synthesis function adapted to the domain of the code program based on training data provided by the client (which may include program code and attribute descriptions of the code, for example).

In a possible implementation, the server can provide model training services to the client side through an application programming interface (API).

Among them, the terminal device can send relevant parameters (such as the attribute description of the code) to the server through the API provided by the cloud. The server can obtain the processing result based on the received parameters and send the processing result (such as the code program adapted to the code). model of the domain's program synthesis function, etc.) is returned to the terminal.

Figure 7 shows the process of using a model training cloud service provided by a cloud platform.

The description of the terminal and the server can be as described in the above embodiment, and will not be described again here.

4. Federal Training

Federated learning is divided into two steps: model delivery and model upload. The central node delivers the model to the terminal device through the network; each terminal device uses local data to calculate the gradient of the model; each distributed node encrypts the gradient and uploads it to the central node; The central node summarizes the gradients of each terminal distributed node and uses the parameter average algorithm to update the parameters of the central node model.

Referring to Figure 8, Figure 8 is a schematic diagram of the architecture of a model training method provided by an embodiment of the present application. As shown in Figure 8, the architecture provided by an embodiment of the present application includes: a central node on the cloud side, which may be a server on the cloud side, for example. A1, A2, ... are distributed nodes of type A, such as mobile phone products held by users. B1, B2, ... are distributed nodes of type B, such as personal computers held by users. With the consent of the administrator of the distributed node (such as users of mobile phones and computers), the administrator of the distributed node voluntarily shares the data generated in the process of daily use of the device while protecting privacy, and joins the model training plan. , the device becomes a distributed node in the architecture. The system in this embodiment can also include more types of distributed nodes, such as smart watches and so on. To protect data privacy, distributed nodes do not upload data to the central node and only save data locally. Distributed nodes are connected to cloud servers through communication networks. The central node on the cloud side can run large models, while each distributed node can only run small models due to hardware capacity limitations, and A and B can have different data processing capabilities.

In order to better understand the solutions of the embodiments of the present application, possible application scenarios of the embodiments of the present application are briefly introduced below with reference to Figures 2 to 4.

Figure 3 shows a natural language processing system, which includes user equipment and data processing equipment. Among them, user equipment includes smart terminals such as mobile phones, personal computers, or information processing centers. User equipment is the initiator of natural language data processing. As the initiator of language question and answer or query requests, users usually initiate requests through user equipment.

The above-mentioned data processing equipment may be a cloud server, a network server, an application server, a management server, and other equipment or servers with data processing functions. The data processing equipment receives query statements/voice/text, etc. from the smart terminal through an interactive interface, and then performs language data processing in the form of machine learning, deep learning, search, reasoning, decision-making, etc. through the memory for storing data and the processor for data processing. , and feedback the processing results to the user device. The memory in the data processing device can be a general term, including local storage and a database that stores historical data. The database can be on the data processing device or on other network servers.

In the natural language processing system shown in Figure 3, the user device can receive the user's instructions. For example, the user device can receive a piece of text input by the user, and then initiate a request to the data processing device, so that the data processing device can respond to the piece of text obtained by the user device. The text executes natural language processing applications (such as natural language generation, text classification, text reasoning, named entity recognition, translation, etc.), thereby obtaining the processing results of the corresponding natural language processing application for the text (such as predicted word results, classification results) , inference results, named entity recognition results, translation results, etc.).

Among them, in the embodiment of the present application, the user equipment can receive the user's instructions. For example, the user equipment can receive a piece of text input by the user (such as an attribute description of the code), and then initiate a request to the data processing device, so that the data processing device targets the user equipment. The obtained piece of text is executed with a natural language processing application (such as program synthesis, etc.), thereby obtaining the processing result of the corresponding natural language processing application for the piece of text (such as predictive text (that is, a generated code program), etc.).

Text (such as the first text or the second text) in Figure 3, the data processing device can process the above text data through the natural language model trained by the method provided in the embodiment of the present application.

Figure 4 shows another natural language processing system. In Figure 4, the user device directly serves as a data processing device. The user device can directly receive input from the user and process it directly by the hardware of the user device itself. The specific process is as follows Figure 3 is similar, please refer to the above description, and will not be repeated here.

FIG. 4 is a schematic diagram of a natural language processing related device 300 provided by an embodiment of the present application.

The user equipment in the above-mentioned Figures 3 and 4 can be the local device 301 or the local device 302 in Figure 4, and the data processing device in Figure 3 can be the execution device 310 in Figure 4, wherein the data storage system 350 can To store the data to be processed by the execution device 310, the data storage system 350 can be integrated on the execution device 310, or can be set up on the cloud or other network servers.

The processors in Figures 3 and 4 can perform data training/machine learning/deep learning through neural network models or other models, and use the data to finally train or learn the model (such as the natural language model natural language in the embodiment of the present application Model, etc.) execute natural language processing applications (such as program synthesis, etc.) against text data (such as the attribute description text (such as the first text or the second text) of the code described in the embodiments of this application) to obtain corresponding processing results. (For example, predictive text (that is, the generated code program) predictive text, etc.).

Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms involved in the embodiments of the present application and related concepts such as neural networks are first introduced below.

(1) Neural network

The neural network can be composed of neural units. The neural unit can refer to an operation unit that takes xs (ie, input data) and intercept 1 as input. The output of the operation unit can be:

Among them, s=1, 2,...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

(2)Transformer layer

The neural network includes an embedding layer and at least one transformer layer. At least one transformer layer can be N transformer layers (N is an integer greater than 0), where each transformer layer includes successively adjacent attention layers, summation and normalization. (add&norm) layer, feed forward layer and summation and normalization layer. In the embedding layer, the current input is embedded to obtain multiple embedding vectors; in the attention layer, P input vectors are obtained from the upper layer of the first transformer layer, and any of the P input vectors are The first input vector is the center. Based on the correlation between each input vector within the preset attention window range and the first input vector, the intermediate vector corresponding to the first input vector is obtained. In this way, P input vectors are determined. Corresponding P intermediate vectors; in the pooling layer, the P intermediate vectors are merged into Q output vectors, where the multiple output vectors obtained by the last transformer layer in the transformer layer are used as features of the current input express.

(3) attention mechanism

The attention mechanism imitates the internal process of biological observation behavior, that is, a mechanism that aligns internal experience and external sensation to increase the precision of observation in some areas, and can use limited attention resources to quickly filter out high-value information from a large amount of information. . The attention mechanism can quickly extract important features of sparse data and is therefore widely used in natural language processing tasks, especially machine translation. The self-attention mechanism is an improvement of the attention mechanism, which reduces the dependence on external information and is better at capturing the internal correlation of data or features. The essential idea of the attention mechanism can be rewritten as the following formula:

Among them, Lx=||Source|| represents the length of Source. The meaning of the formula is to imagine that the constituent elements in Source are composed of a series of data pairs. At this time, given a certain element Query in the target Target, by calculating the Query and Based on the similarity or correlation of each Key, the weight coefficient of each Key's corresponding Value is obtained, and then the Value is weighted and summed to obtain the final Attention value. So essentially the Attention mechanism is a weighted summation of the Value values of the elements in the Source, and Query and Key are used to calculate the weight coefficient of the corresponding Value. Conceptually, Attention can be understood as selectively filtering out a small amount of important information from a large amount of information and focusing on this important information, while ignoring most of the unimportant information. The process of focusing is reflected in the calculation of the weight coefficient. The greater the weight, the more focused it is on its corresponding Value value. That is, the weight represents the importance of the information, and the Value is its corresponding information. The self-attention mechanism can be understood as internal Attention (intra attention). The Attention mechanism occurs between the Target element Query and all elements in the Source. The self-attention mechanism refers to between the internal elements of the Source or between the internal elements of the Target. The Attention mechanism that occurs can also be understood as the attention calculation mechanism in the special case of Target=Source. The specific calculation process is the same, but the calculation object has changed.

(4) Natural language processing (NLP)

Natural language is human language, and natural language processing (NLP) is the processing of human language. Natural language processing is the process of systematically analyzing, understanding and extracting information from text data in an intelligent and efficient way. By using NLP and its components, we can manage very large pieces of text data, or perform a large number of automated tasks, and solve a variety of problems, such as automatic summarization (automatic summarization), machine translation (MT), Named entity recognition (NER), relationship extraction (RE), information extraction (IE), sentiment analysis, speech recognition (speech recognition), question answering system (question answering), topic segmentation, etc. .

(5) Pre-trained language model (pre-trained language model)

The pre-trained language model is a natural language sequence encoder that encodes each word in the natural language sequence into a vector representation to perform prediction tasks. Its training consists of two phases. In the pre-training stage, the model performs language model training on large-scale unsupervised text to learn a word representation. In the finetuning stage, the model uses the parameters learned in the pre-training stage for initialization, and performs fewer steps of training on downstream tasks such as text classification and sequence labeling. The semantic information obtained by pre-training can be successfully transferred to downstream tasks.

(6) Autoregressive language model

An autoregressive language model refers to a model that can predict the next possible following word (such as "nice") based on a given context (such as "mobile phone is very"). This model usually predicts the following word on the right given the left context. word, but could also be a word somewhere in the middle given the context predictions on the left and right.

(7) Programming language: a formal language used to define the execution flow of computer instructions.

(8) Program synthesis: The process of generating a program based on prompts (such as attribute descriptions of codes, etc.).

(9) Code completion: The process of generating a complete code paragraph based on a given code fragment.

(10) Function: A complete code paragraph in a programming language that defines some specific functions.

(11) Abstract syntax tree (AST): An abstract syntax structure that expresses programming language code in the form of a tree.

(12) Causal language modeling (CLM): Model training is performed by predicting the next character of the character sequence.

(13) Masked language modeling (MLM): Model training is performed by predicting the original characters at the masked positions in the character sequence.

(14) Masked causal language modeling (MCLM): Model training is performed by predicting the original characters at the masked positions in the character sequence based on the previous text.

(15) No language modeling (NoLM): No loss calculation is performed on characters within this range, and characters within this range do not participate in model training.

(16)Back propagation algorithm

The convolutional neural network can use the error back propagation (BP) algorithm to modify the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, forward propagation of the input signal until the output will produce an error loss, and the parameters in the initial super-resolution model are updated by back-propagating the error loss information, so that the error loss converges. The backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

(17)Loss function

In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value that we really want to predict, we can compare the predicted value of the current network with the really desired target value, and then based on the difference between the two to update the weight vector of each layer of the neural network according to the difference (of course, there is usually an initialization process before the first update, that is, preconfiguring parameters for each layer in the deep neural network). For example, if the predicted value of the network If it is high, adjust the weight vector to make its prediction lower, and continue to adjust until the deep neural network can predict the really desired target value or a value that is very close to the really desired target value. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value". This is the loss function (loss function) or objective function (objective function), which is used to measure the difference between the predicted value and the target value. Important equations. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the deep neural network becomes a process of reducing this loss as much as possible.

(18)Back propagation algorithm

The neural network can use the error back propagation (BP) algorithm to modify the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forward propagation of the input signal until the output will produce an error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss converges. The backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

Program synthesis refers to using a certain software program to automatically synthesize a specified program language sequence with specified functions or structures. In short, it is a program that generates a program. Generally speaking, it is divided into two categories: program synthesis (code generation) and code completion. kind. Program synthesis, that is, starting from scratch, can generate corresponding code according to a certain natural language description or a preset function description. Code completion, that is, based on the existing code sequence, continue to generate the following code sequence.

Language model refers to the modeling of language sequences, such as natural language or programming language. Nowadays, most neural networks are used for language model modeling, that is, the occurrence probability of language sequences is predicted through the neural network model to generate language. Take the model as an example, which is to predict the probability of a certain word, word and other language unit appearing at each sequence position. For example, the famous GPT language model is to build a decoder network by building multiple neural network computing units transformer. The decoder is based on the existing In the above, the words that may appear in the language unit at the next position are predicted through iteration until the prediction of the entire sequence is completed, that is, decoding is completed.

Language model pre-training refers to training the corresponding language model through massive language sequence corpus, so that the language model has the ability to predict the occurrence probability of a certain language unit at a certain position.

With the development of deep learning technology, the use of deep learning for sequence generation has been widely used in natural language sequence generation scenarios. In particular, the emergence of pre-trained language generation model technology has led to a qualitative improvement in language generation capabilities. In recent years, generative models such as generative pre-training (GPT), T5, and BART have gradually been introduced into code generation or completion. This technology uses Transformer as the basic component, builds a decoder or encoder-decoder model architecture, and then uses a large amount of code data for training to obtain the ability to output the corresponding code context based on the above, thereby completing or generating code.

The existing technology follows the natural language processing generation method and trains based on the original code. Currently, this method can continue to write the code based on the existing context. However, for the scenario of function-level code generation, the quality of the code generation has yet to be improved. promote.

In order to solve the above problems, embodiments of the present application provide a model training method. The model training method of the embodiment of the present application will be introduced in detail below with reference to the accompanying drawings.

Referring to Figure 9, Figure 9 is a flow diagram of a model training method provided by an embodiment of the present application. As shown in Figure 9, a model training method provided by an embodiment of the present application may include steps 901 to 902. The following are respectively These steps are described in detail.

901. Obtain the first text and the second text. The first text and the second text are extracted from the original code text; wherein the first text is the program code and the second text is the program code. Attribute description corresponding to the program code; the first text and the second text include known text and predicted text;

In a possible implementation, in order to obtain the first text and the second text as training samples, the original corpus (such as the original code text in the embodiment of the present application) can be obtained, and the original code text can be extracted from the original code text. First text and second text.

About the original code text:

Among them, the original code text can be existing programming language corpus and natural language corpus. Optionally, existing programming language corpus and natural language corpus can be obtained through the Internet (or other methods). The corpus includes code files of mixed programming language and natural language, code files of programming language, and natural language files. The collected natural language can be any human language, and the programming language can be any programming language, which is not limited by this application.

In one possible implementation, the original code text can be deduplicated from data, files, or function codes, and then duplicate files or duplicate function codes in the corpus can be deduplicated, thereby reducing the total amount of corpus and improving processing and Training efficiency.

Regarding the first and second texts:

In a possible implementation, the first text and the second text are extracted from the original code text; wherein the first text is the program code, and the second text is the program code. Corresponding attribute description. The second text can be in natural language, and the second text can be in any language used for human communication, and the program code can be in any programming language, which is not limited by this application.

In a possible implementation, the attribute description includes a function description of the program code or an implementation principle of the program code (or it is also called an explanation of the program code). For example, refer to Figure 10A. The box shown in Figure 10A is the attribute description, and the rest is the program code, which includes two functions: "euclidean_distance" and "classifier". Refer to Figure 10B, which shows a code text of the function "classifier" and the corresponding attribute description. Refer to Figure 10C, which shows a code text of the function "classifier". Figure 10D shows an attribute of the function "classifier". Description.

In one possible implementation, the function code segment in the original code text can be identified as the first text: for example, the original code text can be identified using certain rules (for example, a program code identification method such as a code abstract syntax tree (AST) can be used) function code snippet in .

In one possible implementation, the natural language in the original code text can be recognized as the second text (associated with the recognized first text): for example, certain rules can be used (for example, a natural language recognition method can be used) to Identifies the attribute description of the code in the original code text.

In a possible implementation, after identifying the first text and the second text, a correspondence relationship between the first text and the second text can be established to form a "natural language description-function implementation code" (Description-Function code) ) sentence pair form sample. Optionally, you can use a character combination before each set of natural language-program language to indicate that the following sequence is a text language sequence or a specific code language sequence, and add a sample terminator after the sample to indicate the first The text and the second text are a pair of samples.

In the embodiment of the present application, when constructing a program-synthesized training sample, the code text and corresponding attribute description are extracted from the original program code and used as a pair of samples. Compared with the prior art, the original program file is directly used as a training sample. , helping the natural language model to establish an association between the code text and the corresponding attribute description during training, without having to learn the ability to identify this association from samples during the training process, reducing the training difficulty and improving the prediction of the model Accuracy.

In a possible implementation, the first text (or program language function-level code fragment) and its corresponding second text (or natural text) can be extracted from the original corpus or the filtered and cleaned original corpus. Language description), constructing "natural language description-programming language function code pair". Optionally, the function-level code fragments of the programming language and their corresponding natural language descriptions can be extracted through some rule methods. Optionally, the function-level code fragments of the programming language and their corresponding natural language descriptions can be extracted through the abstract syntax tree (AST). ,Optionally, program language function-level code snippets ,and their corresponding natural language descriptions can be ,extracted through other automated methods. Optionally, the programming language function code and natural language description can be extracted from the code file of mixed programming language and natural language. Optionally, the programming language function code can be extracted from the code file of the programming language. Extract natural language descriptions from natural language documents. Optionally, the "natural language description-programming language function code pair" can be constructed through the correspondence between the programming language code fragments and the natural language description in the mixed programming language and natural language code files. Optionally, the "natural language description-programming language function code pair" can be constructed through other programming language codes. The correspondence relationship between the fragments and the natural language description constructs a "natural language description-programming language function code pair".

In one possible implementation, in order to improve the quality of the training corpus, function-level code fragments (such as code programs) can be normalized, standardized code syntax, or unified code style. For example, certain rules (such as code abstract syntax tree (AST)) can be used to parse the code, extract pure code parts, and unify styles such as spaces and indentations to normalize the data.

In a possible implementation, in order to improve the quality of the training corpus, function-level code snippets (such as code programs) can be removed based on the statistical characteristics of their code corpus, and the corpus with too short natural language description part and the excessive code part in the code corpus can be removed. Long corpus or corpus with a large length difference between the two to improve the quality of the training corpus.

In a possible implementation, since training samples are usually difficult to obtain, that is, the number of training samples (text pairs composed of the first text and the second text) is small, data enhancement can be performed on the obtained training samples to obtain More training samples.

In a possible implementation, the first text is obtained based on the first original text extracted from the original code text; wherein the first text is a partial text unit in the first original text. Obtained by modification.

In a possible implementation, the modification includes: deletion, addition or replacement.

In a possible implementation, the replacement is specifically a replacement with the same semantics. For example, synonymous segments may be generated based on back translation technology, or synonymous segments may be generated based on synonymous sentence generation technology.

For example, you can randomly replace variable names in the code text, replace equivalent (loop) code segments, and randomly insert invalid code segments to obtain a large amount of high-quality automatically generated training data, ultimately improving model performance.

In a possible implementation, the second text is obtained based on the second original text extracted from the original code text; wherein the second text is a partial text unit in the second original text. Obtained by modification.

In a possible implementation, the modification includes: deletion, addition or replacement.

In a possible implementation, the replacement is specifically a replacement with the same semantics. For example, synonymous segments may be generated based on back translation technology, or synonymous segments may be generated based on synonymous sentence generation technology.

For example, you can augment the natural language description by replacing synonyms, transforming sentence patterns, or using paraphrase generation models or methods in natural language processing technology, data augmentation methods, and data noise methods to obtain a large number of information. High-quality automatically generated training data ultimately improves model performance.

In a possible implementation, training samples can be processed differently at different stages of model training. For example, in the initial iterative stage of model training, the first text and the second text obtained in the above way can be used, and in the model training In the fine-tuning stage, text adapted to the application field of the program code (also including program code and corresponding code attribute descriptions) can be extracted from the original code text as a training sample.

In a possible implementation, the first text or the second text is text extracted from the original code text and adapted to the application field of the program code. In one possible implementation, the adaptation to the application field of the program code includes: text whose length or the length of a single line of text is less than a threshold; wherein the threshold is related to the application field of the program code. Alternatively, information in other dimensions besides length can also be used to quantify and apply the program code.

In one possible implementation, the samples may be filtered based on their similarity to data samples from the application domain of the function-level code generator. The similarity calculation method can be the length relationship between natural language descriptions, the length relationship between program language function codes, or other relationships.

In a possible implementation, in order to train a model that can be used to perform program generation functions (that is, the natural language model in the embodiment of this application), some text units in the training sample can be used as known text, and the other part of the text As the text to be predicted, the text at the text position where the text to be predicted is predicted based on the known text.

Since the first text and the second text are different types of text (the first text is a computer compiled language and the second text is a natural language), the text prediction process can be performed with different training objectives for the first text and the second text.

For example, for the first text, one of the four training objectives of CLM, MLM, MCLM and NoLM can be used to perform the text prediction process.

For example, for the second text, one of the three training objectives of CLM, MLM and MCLM can be used to perform the model training process.

When optimizing the natural language description part, you can choose one of four training objectives: CLM, MLM, MCLM, and NoLM. When optimizing the program language function code part, you can choose one of the three training objectives: CLM, MLM, and MCLM. Note that the NoLM training objective does not need to be included here. The training objectives of the natural language description part and the programming language function code part can be freely combined.

In a possible implementation, when predicting the text in the first text, the text position corresponding to the predicted text may be determined from the first text in a first manner; When predicting text in two texts, the text position corresponding to the predicted text can be determined from the second text through a second method; the first method and the second method are different.

In a possible implementation, the first method and the second method are respectively one of the following prediction methods:

Sample mask positions from the text positions of part or all of the first text or the second text, and use the mask positions as the text positions corresponding to the predicted text (that is, MLM);

Use the following text position of the known text as the text position corresponding to the predicted text; wherein the known text is all the texts that have been predicted by the natural language model (that is, CLM);

The following text position of the known text is used as the text position corresponding to the predicted text; wherein the known text is the unmasked text among all the texts that have been predicted by the natural language model (that is, MCLM ).

In a possible implementation, without predicting the text in the second text (that is, NoLM), the text position corresponding to the predicted text can be determined through one of the following prediction methods:

Sample mask positions from the text positions of part or all of the first text, and use the mask positions as the text positions corresponding to the predicted text (that is, MLM);

Use the following text position of the known text as the text position corresponding to the predicted text; wherein the known text is all the texts that have been predicted by the natural language model (that is, CLM);

The following text position of the known text is used as the text position corresponding to the predicted text; wherein the known text is the unmasked text among all the texts that have been predicted by the natural language model (that is, MCLM ).

Next, CLM, MLM, MCLM and NoLM are introduced respectively.

About CLM:

In the feedforward process of model training, the natural language model can predict texts sequentially along the preset context prediction direction. In other words, all words that have been predicted by the natural language model can be used as known text, and have been The known text is used as the context of the position text to predict the text location of the text. For example, prediction can be made in the order from the context to the context (that is, from the beginning of the text to the end of the text), each time You can predict a text unit and continue to predict based on the already predicted text unit in the next prediction. For example, the text can be "the cat sat on the mat". In the first prediction, you can predict " The text at the text position of "the" can be predicted by the initial identification position and "the" in the next prediction. The text at the text position of "cat" can be predicted by the initial identification position, "the" in the next prediction. ” and “cat” to predict the text at the text position of “sat”.

The training objectives of CLM can be used for training when processing the first text or the second text.

For example, when using the second text for model training, you can use the CLM training target to train the natural language model. After predicting the second text, you can use the first text as a non-CLM training target for model training (in this case, the first text is used as the training target). The second text can be used as known text to participate in the process of using the first text for model training).

For example, when using the second text for model training, you can use non-CLM training targets to train the natural language model. After predicting the second text, you can use the CLM training targets to use the first text for model training (at this time The second text can be used as a known text to participate in the process of using the first text for model training).

About MLM:

In the feedforward process of model training, the text units in the text (for example, the second text or the first text) can be masked (for example, they can be randomly masked), and the masked text units can be used as predicted texts. The masked text unit can be used as known text, and the natural language model can sequentially calculate the text location of the masked text unit based on the unmasked text unit (or the predicted masked text unit). Text prediction, for example, the text can be "the cat sat on the mat". After masking the text, you can get "the_sat_the mat", where the symbol "_" here means masking, not referring to the underline in . In the first prediction, the text at the text position where one of the "_" is located can be predicted through the initial identification bit and the unmasked text unit.

The training target of MCL can be used for training when processing the first text or the second text.

For example, when using the second text for model training, you can use the MCL training target to train the natural language model. After predicting the second text, you can use the first text as a non-MCL training target for model training (in this case, the first text is used for model training). The second text can be used as known text to participate in the process of using the first text for model training).

For example, when using the second text for model training, you can use non-MCL training targets to train the natural language model. After predicting the second text, you can use the MCL training targets to use the first text for model training (at this time The second text can be used as a known text to participate in the process of using the first text for model training).

About MCLM:

In the feedforward process of model training, the natural language model can predict text sequentially along the preset context prediction direction. The difference from CLM is that in CLM, all words that have been predicted by the natural language model can be used as Known text, and the known text is used as the upper text of the position text to predict the text position of the predicted text. In MCLM, the words that have been predicted by the natural language model in MCLM can be masked, and natural language can be used The unmasked text units in the words that have been predicted by the model are used to predict the text position of the predicted text. For example, it can be performed in the order from the top to the bottom (that is, from the beginning of the text to the end of the text). predict.

For example, predictions can be made in the order from top to bottom (that is, from the beginning of the text to the end of the text). One text unit can be predicted at a time, and predictions can be continued based on the already predicted text units in the next prediction. , for example, the text can be "the cat sat on the mat". In the first prediction, the text at the text position of "the" can be predicted through the initial identification bit, and the initial identification bit and "the" can be masked. code, if "the" is masked, in the next prediction, the text at the text position of "cat" can be predicted through the initial identification position.

The training objectives of MCLM can be used for training when processing the first text or the second text.

For example, when using the second text for model training, you can use the MCLM training target to train the natural language model. After predicting the second text, you can use the first text as a non-MCLM training target for model training (in this case, the second text The text can be used as known text to participate in the process of using the first text for model training).

For example, when using the second text for model training, you can use a non-MCLM training target to train the natural language model. After predicting the second text, you can use the MCLM training target to use the first text for model training (at this time The second text can be used as a known text to participate in the process of using the first text for model training).

About NoLM:

In the feedforward process of model training, the second text can not be predicted, but the second text can be directly used as the known text, and the first text can be used as a training sample to predict text units in the first text. For example, you can use Training targets such as CLM, MLM, and MCLM use the first text as a training sample to predict text units in the first text.

For example, the second text can be used as a known text to participate in the process of using the first text for model training. CLM, MLM, MCLM, etc. can be used as training targets, and the first text can be used for model training (in this case, the second text can be used as a known text). Zhiwen participates in the process of model training using the first text).

In a possible implementation, the four optional training objectives (CLM, MLM, MCLM, and NoLM) can be applied to the natural language description or function code of the "Description-Function pair" (Description-Function) respectively, and the combination Form 12 types of training objectives, such as:

(1) Full causal language modeling (Description-CLM combined with Function-CLM): CLM training objectives are applied to the natural language description part, and CLM training objectives are applied to the function code part. At the same time, it trains natural language description understanding ability and function code generation ability.

(2) Description mask language modeling and code causal language modeling (Description-MLM combined with Function-CLM): the natural language description part applies MLM training objectives, and the function code part applies CLM training objectives. At the same time, it trains natural language description understanding ability and function code generation ability.

(3) Description mask causal language modeling and code language modeling (Description-MCLM+Function-CLM): The natural language description part applies MCLM training objectives, and the function code part applies CLM training objectives. At the same time, it trains natural language description understanding ability and function code generation ability.

(4) Description-NoLM+Function-CLM: Description-NoLM+Function-CLM: The natural language description part has no training objectives, and the function code part applies CLM training objectives. Only train function code generation capabilities.

By analogy, it can also include:

(5) Description-CLM combined with Function-MLM;

(6) Description-MLM combined with Function-MLM;

(7) Description-MCLM combined with Function-MLM;

(8) Description-NoLM combined with Function-MLM;

(9) Description-CLM combined with Function-MCLM;

(10) Description-MLM combined with Function-MCLM;

(11) Description-MCLM combined with Function-MCLM;

(12) Description-NoLM combined with Function-MCLM;

902. According to the known text, use the natural language model to predict the text corresponding to the text position of the predicted text to obtain a text prediction result; the predicted text and the text prediction result are used to update the natural language model .

In one possible implementation, in the process of iterative training of the natural language model, the known text can be processed through the natural language model, the text corresponding to the text position of the predicted text can be predicted, the text prediction result is obtained, and based on the text prediction The difference between the result and the corresponding ground truth determines the loss, which can be used to update the natural language model.

In one implementation, the known text can be embedded through an embedding layer to obtain an embedding vector.

The embedding layer can be called the input embedding layer. The current input can be each text unit in the known text (or it can be called a known data unit). After obtaining the current input, the embedding layer can embed each known data unit in the current input to obtain the embedding vector corresponding to each known data unit.

In some embodiments, the position vector of each known data unit can also be obtained, and the position vector is used to indicate the position of the known data unit; wherein the position is used to represent the position of the known data unit in the text (for example, the first text Or the position in the second text), specifically, the position is used to indicate the relative positional relationship between the known data unit and other known data units, as well as the known data unit and each text unit in the predicted text. .

In one implementation, the embedding layer may include an input embedding layer and a positional encoding layer. In the input embedding layer, word embedding processing can be performed on each known data unit in the current input to obtain the embedding vector of each known data unit. At the position coding layer, the position of each known data unit in the current input can be obtained, and then a position vector is generated for the position of each known data unit.

In some examples, the position of each known data unit in the text (eg, the first text or the second text) may be the absolute position of each known data unit in the text (eg, the first text or the second text). Taking the current input as "What number should I pay back the Huabei?" for example, the position of "number" can be represented as the first digit, the position of "number" can be represented as the second digit,... In some examples, the position of each known data unit in the text (eg, the first text or the second text) may be the relative position of each known data unit in the text (eg, the first text or the second text). Still taking the current input as "what number should I pay back Huabei" as an example, the position of "what number" can be expressed as before "number", and the position of "number" can be expressed as after "what number" and before "should",... …. When the embedding vector and position vector of each known data unit in the current input are obtained, the position vector of each known data unit and the corresponding embedding vector can be fused to obtain the embedding vector of each known data unit, that is, the current Enter multiple corresponding embedding vectors. It should be understood that the fusion method may be to add the embedding vector and the position vector, or to use other operations so that the embedding vector can carry a known data unit in the text (such as the first text or the second text) and the one The information on the position of the data unit in the text (for example, the first text or the second text) is known, and the specific fusion method is not limited here. Multiple embedding vectors can be represented as embedding matrices with preset dimensions. The number of the plurality of embedding vectors can be set to M, and the default dimension is H dimension, then the plurality of embedding vectors can be expressed as an M×H embedding matrix.

In one implementation, the position of the predicted text in the text (for example, the first text or the second text) can also be embedded through the embedding layer to obtain a method for representing the text (for example, the first text or the second text). The embedding vector of the position of the predicted text in the text (for example, the first text or the second text) in the second text), and the embedding vector can be used as the input of the subsequent natural language model. Wherein, the position is used to indicate the relative position relationship between each text unit in the predicted text and each known data unit in the text (such as the first text or the second text). For a description of the position, please refer to the above embodiment. Regarding the description of the location in , the similarities will not be repeated here.

In one implementation, the natural language model may include an encoder and a decoder. For example, the encoder and decoder may include multiple transformer layers.

The core feature of the transformer layer lies in its unique attention mechanism. When processing natural language, such as a sentence, the transformer model uses this attention mechanism to assign different attention coefficients to the embedding vectors of each word in the sentence, thereby more comprehensively considering the impact of the context on each word in the sentence. The specific transformer layer can include sequentially adjacent multi-head attention layers, addition and normalization (add&norm) layers, feed forward layers, and addition and normalization layers. Among them, the attention layer is connected to the embedding layer, and multiple embedding vectors are obtained from the embedding layer as input vectors. Based on the correlation between the embedding vectors in the multiple embedding vectors, each embedding vector is synthesized to obtain multiple output vectors. Output to the subsequent transformer layer. The transformer layer takes the output of the previous layer as an input vector and performs similar operations to the previous transformer layer.

The encoder can obtain multiple output vectors, and the decoder can use multiple first output vectors and the embedded vectors as inputs to obtain a vector representation of the predicted text. It should be understood that the vector representation of the predicted text can be further passed through a classifier. (For example, support vector machine, softmax classifier, K-nearest neighbor algorithm, etc. can be used) to restore the predicted text.

In one possible implementation, the loss can be constructed based on the predicted text as well as the text prediction structure, and the natural language model is updated based on the loss.

In a possible implementation, the loss can be constructed based on the predicted text and the text prediction structure, and the loss or the gradient obtained based on the loss is transferred to other devices (such as a server on the cloud side), and the server on the cloud side executes the loss based on the loss or the gradient obtained based on the loss. Gradient to update the actions of the natural language model.

Next, combined with the software module, an application example in the embodiment of this application is introduced:

Referring to Figure 11, the present invention aims at the generation of function-level code. It generally includes a two-stage training strategy, as well as a design of selecting different loss function training targets for code and text respectively, ensuring that the first stage of the model lays the foundation and the second stage is advanced. Training to improve abilities. In the data preparation stage, a targeted function-level code corpus preparation mechanism is also designed based on the different goals of the two stages, ensuring that it can well improve the excellent performance of the trained generative model in function code generation capabilities.

As shown in Figure 11, it can specifically include:

The first data module: provides the most original data and collects a large amount of the most original data containing code, including but not limited to code project files, independent code snippets, script files, etc.

The second data module: based on the first data module, provides the data required for training the first function-level code generation training module. This module is optionally composed of the following devices:

1. First data filter: Preliminarily filter the data in the first data module, optionally removing data with too long files or too long single lines of code, removing data in other than target code languages or natural language languages, etc.

2. Function code segment extractor: Based on the corpus data obtained by the first data filter, identify all function code segments in the corpus:

Optionally use certain rules, such as function definition rules in different languages, for identification and extraction.

Optionally use existing tools to perform code abstract syntax tree (AST) analysis, analyze the code to obtain the syntax tree of the corresponding code, and intercept the corresponding function-level code fragments through syntax tree analysis.

Optionally, use natural language processing automatic identification methods to extract code segments, such as sequence annotation methods, and train a function annotator to annotate all function-level codes.

3. NL-PL sample constructor identifies the natural language description corresponding to the function code fragment (if any):

Optionally use certain rules, such as annotation rules, language character rules, etc. for identification and extraction.

Optionally use existing tools to perform code abstract syntax tree (AST) analysis, analyze the code to obtain the syntax tree of the corresponding code, and intercept the natural language description corresponding to the corresponding function-level code fragment through syntax tree analysis.

Optionally, use natural language processing automatic identification methods to extract code segments, such as sequence annotation methods, and train a function annotator to annotate the natural language descriptions corresponding to all function-level codes.

The extracted code-level function and its description form a sentence pair sample of "Natural Language Description-Function Implementation Code" (Natural Language–Program Language). Optionally, you can use a combination of characters representing the language before different natural languages and programming languages to indicate that the following sequence is a text language sequence or a specific code language sequence, and add a sample terminator after the sample.

4. Data normalizer: Optionally normalize function-level code fragments, standardize code syntax, unify code style, and use optional certain rules and optional code abstract syntax tree (AST) to parse the code. Extract the pure code part and unify spaces, indentation and other styles to normalize the data to improve the quality of the corpus.

The third data module: Based on the second data module, it provides the data required for training the second function-level code generation training module, optional targeted filtering, and optional data enhancement. This module is optionally composed of the following devices:

1. Second data filter: Optionally, based on the statistical characteristics of the code corpus, remove the corpus in which the natural language description part is too short, the code part is too long, and the length difference between the two is too large. To improve the quality of training corpus.

2. Function code augmenter: Optionally, through random replacement of variable names in the code, equivalent replacement of (cyclic) code segments, and random insertion of invalid code segments, a large amount of high-quality automatically generated training data can be obtained. Finally, Improve model performance.

3. Natural language augmenter: Optionally replace synonyms and sentence patterns in natural language descriptions, or use paraphrase generation models or methods in natural language processing technology, data augmentation methods, and data noise methods to enhance natural language amplification. The language description is augmented to obtain a large amount of high-quality automatically generated training data, ultimately improving model performance.

The first function-level code generation training module: uses the data generated by the second data module to perform function code generation training. The training can use natural language to generate the pattern lines of common pattern decoders, such as GPT. Correspondingly, the first function-level code generation training module also includes a training target combiner for selecting and determining training targets. The optional 4 training objectives (objectives, CLM, MLM, MCLM, NoLM) are applied to the natural language description or function code of the "Description-Function pair" (Description-Function) respectively, and combined to form 12 training objectives.

The second function-level code generation training module: uses the data generated by the third data module to further train the function code generation based on the model trained by the first function-level code generation training module. The training can use natural language to generate common A module line with a decoder, such as GPT. Correspondingly, the first function-level code generation training module also includes a training target combiner for selecting and determining training targets. The optional 4 training objectives (objectives, CLM, MLM, MCLM, NoLM) are applied to the natural language description or function code of the "Description-Function pair" (Description-Function) respectively, and combined to form 12 training objectives.

The structure of the first function-level code generator can be any generative neural network model, such as Transformer decoder and RNN, with its model parameters randomly initialized.

The first function-level code generator uses the corpus output by the second data module as training data.

The first function-level code generator uses the first combination optimizer to update model parameters. The first combination optimizer optimizes the natural language description part and the programming language function code part respectively.

When optimizing the natural language description part, the first combination optimizer can choose four training objectives: CLM, MLM, MCLM, and NoLM. When optimizing the program language function code part, the first combination optimizer can choose three training objectives: CLM, MLM, and MCLM. Note that there is no NoLM training objective here. The training objectives of the natural language description part and the programming language function code part can be freely combined.

Step 1: Obtain training data for the function-level code generator

The first data module acquires existing programming language corpus and natural language corpus. Source code files can be crawled from the Github website, including natural language descriptions and programming language codes.

Step 2: Prepare training data for the first function-level code generator

The second data module constructs the data required for training the first function-level code generator, optionally performs data filtering and cleaning, and optionally performs data enhancement.

The second data module extracts the program language function-level code fragments and their corresponding natural language descriptions from the corpus output by the first data module or the filtered and cleaned corpus of the second data module, and constructs a "natural language description-program language function code pair" ". Optionally, extract program language function-level code snippets and their corresponding natural language descriptions through some rule methods, optionally extract program language function-level code snippets and their corresponding natural language descriptions through abstract syntax trees (AST), optionally Extract program language function-level code snippets and their corresponding natural language descriptions through other automated methods. Optionally extract the programming language function code and natural language description from the mixed programming language and natural language code files; optionally extract the programming language function code from the programming language code file; optionally extract the programming language function code from the natural language file. Natural language description. Optionally, a "natural language description-programming language function code pair" can be constructed through the corresponding relationship between the program language code snippets and the natural language description in the mixed programming language and natural language code files, and optionally through the correspondence between the program language code snippets and the natural language description. The corresponding relationship of description constructs "natural language description-programming language function code pair".

The "natural language description" and "programming language function code" in the above example can be constructed as a "natural language description-programming language function code pair".

Step 3: Train the first function-level code generator

The structure of the first function-level code generator can be any generative neural network model, or a Transformer decoder, for example:

The first function-level code generator uses the corpus output by the second data module as training data. No annotation information is required, only the unannotated corpus output by the second data module is required. For example, place the "natural language description" part of the example function "classifier" between <descr> and <python> in Figure 12, and place the "programming language function code" part of the example function "classifier" between After <python> in Figure 12, this is used as the training input of the first function-level code generator.

The first function-level code generator uses the first combination optimizer to update model parameters. The first combination optimizer optimizes the natural language description part and the programming language function code part respectively. When optimizing the natural language description part, the first combination optimizer can choose four training objectives: CLM, MLM, MCLM, and NoLM. When optimizing the program language function code part, the first combination optimizer can choose three training objectives: CLM, MLM, and MCLM. Note that there is no NoLM training objective here. The training objectives of the natural language description part and the programming language function code part can be freely combined. Take NoLM as the "natural language description" part and CLM as the "programming language function code" part as an example:

As shown in Figure 13, Loss calculation can not be performed on the "Natural Language Description" part in Figure 13, and CLM Loss calculation can be performed on the "Programming Language Function Code" part.

Step 4: Prepare training data for the second function-level code generator

The third data module constructs the data required for training the second function-level code generator. The third data module performs filtering and optional data enhancement on the corpus output by the second data module.

The third data module filters the corpus output by the second data module to make the training data closer to the application field of the second function-level code generator. The filtering is based on similarity between the data sample output by the second data module and the data sample in the application domain of the second function-level code generator. You can compare the length of the "natural language description" part of the two data sets, for example, count the maximum length L _max and minimum length L _min of the "natural language description" in the application field data of the second function-level code generator, and select only the second The samples whose "natural language description" length l in the data output by the data module conforms to L _min ≤ l ≤ L _max are used as the output of the third data module.

Step 5: Train the second function-level code generator

The structure of the second function-level code generator is consistent with the structure of the first function-level code generator, and the second function-level code generator is initialized with model parameters of the first function-level code generator.

The second function-level code generator uses the corpus output by the third data module as training data. No annotation information is required, only the unannotated corpus output by the third data module is required.

The second function-level code generator uses the second combinatorial optimizer to update model parameters. The second combination optimizer optimizes the natural language description part and the programming language function code part respectively. When optimizing the natural language description part, the second combination optimizer can choose four training objectives: CLM, MLM, MCLM, and NoLM. When optimizing the program language function code part, the second combination optimizer can choose three training objectives: CLM, MLM, and MCLM. Note that there is no NoLM training objective here. The training objectives of the natural language description part and the programming language function code part can be freely combined.

Step 6: Optimize the inference parameters of the second function-level code generator

When using the second function-level code generator to generate function-level code, it can optionally be generated based on random sampling (sampling) generation or beam search (beam search) generation, or other strategy generation. During this code generation process, there are optional inference parameters that affect the randomness of the generated results, such as temperature. Optionally, a part of the data output by the third data module is reserved as a development set (development set), which does not participate in the training of the second function-level code generator. Based on this development set, the aforementioned inference parameters can be adjusted to improve the performance of the third function-level code generator. Correctness of code generated by a two-function-level code generator.

This embodiment is based on the data disclosed by Github, uses the first, second, and third data modules to collect and process the training data of the first and second function-level code generators, and uses the first and second combined optimizers to perform train. The corresponding "program language function code" (Python) can be generated for the input "natural language description". After comparative testing, using the code generator obtained by the present invention can achieve a Pass@1 score of 17.07% on the public HumanEval test task, significantly exceeding the currently public best score (13.17%).

Compared with the existing technology, the function-level code generator designed in this example based on pre-training and natural language description can generate program language function fragments corresponding to the functions based on natural language input, and the function-level code is more accurate. In addition, this solution uses public Github data to train the model, making data acquisition easy and sustainable expansion.

Referring to Figure 14, Figure 14 is a schematic structural diagram of a model training device provided by an embodiment of the present application. As shown in Figure 14, a model training device 1400 provided by an embodiment of the present application includes:

The acquisition module 1401 is used to acquire a first text and a second text, where the first text and the second text are extracted from the original code text; wherein the first text is a program code, and the third text is a program code. The second text is the attribute description corresponding to the program code; the first text and the second text include known text and predicted text;

For a specific description of the acquisition module 1401, please refer to the introduction of step 901 in the above embodiment, and will not be described again here.

The prediction module 1402 is used to predict the text corresponding to the text position of the predicted text through the natural language model according to the known text, and obtain a text prediction result; the predicted text and the text prediction result are used to update the Describe the natural language model.

For a specific description of the prediction module 1402, please refer to the introduction of step 902 in the above embodiment, and will not be described again here.

In a possible implementation, the attribute description includes a function description of the program code or an implementation principle of the program code.

In a possible implementation, the device further includes:

A text position determination module, configured to determine the text position corresponding to the predicted text from the first text in a first manner when predicting the text in the first text;

When predicting the text in the second text, the text position corresponding to the predicted text is determined from the second text through the second method; the first method and the second method are different.

In a possible implementation, the first method and the second method are respectively one of the following prediction methods:

Sample mask positions among text positions of part or all of the first text or second text, and use the mask positions as the text positions corresponding to the predicted text;

Use the following text position of the known text as the text position corresponding to the predicted text; wherein the known text is all the text that has been predicted by the natural language model;

The following text position of the known text is used as the text position corresponding to the predicted text; wherein the known text is the unmasked text among all the texts that have been predicted by the natural language model.

In a possible implementation, the text position determination module is also configured to determine the text position corresponding to the predicted text through one of the following prediction methods without predicting the text in the second text:

Sample mask positions from text positions of part or all of the first text, and use the mask positions as text positions corresponding to the predicted text;

Use the following text position of the known text as the text position corresponding to the predicted text; wherein the known text is all the text that has been predicted by the natural language model;

The following text position of the known text is used as the text position corresponding to the predicted text; wherein the known text is the unmasked text among all the texts that have been predicted by the natural language model.

In one possible implementation,

The first text is obtained based on the first original text extracted from the original code text; wherein the first text is obtained by modifying some text units in the first original text; or,

The second text is obtained based on the second original text extracted from the original code text; wherein the second text is obtained by modifying some text units in the second original text.

In a possible implementation, the modification includes: deletion, addition or replacement.

In a possible implementation, the replacement is specifically a replacement with the same semantics.

In a possible implementation, the first text or the second text is text extracted from the original code text and adapted to the application field of the program code.

In a possible implementation, the application fields adapted to the program code include:

The text length or the length of a single line of text is less than a threshold; wherein the threshold is related to the application field of the program code.

Next, an execution device provided by an embodiment of the present application is introduced. Please refer to Figure 15. Figure 15 is a schematic structural diagram of an execution device provided by an embodiment of the present application. The execution device 1500 can be embodied as a virtual reality VR device, a mobile phone, Tablets, laptops, smart wearable devices, monitoring data processing equipment or servers, etc. are not limited here. Specifically, the execution device 1500 includes: a receiver 1501, a transmitter 1502, a processor 1503 and a memory 1504 (the number of processors 1503 in the execution device 1500 can be one or more, one processor is taken as an example in Figure 15) , wherein the processor 1503 may include an application processor 15031 and a communication processor 15032. In some embodiments of the present application, the receiver 1501, the transmitter 1502, the processor 1503, and the memory 1504 may be connected through a bus or other means.

Memory 1504 may include read-only memory and random access memory and provides instructions and data to processor 1503 . A portion of memory 1504 may also include non-volatile random access memory (NVRAM). The memory 1504 stores processor and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, where the operating instructions may include various operating instructions for implementing various operations.

The processor 1503 controls the execution of operations of the device. In specific applications, various components of the execution device are coupled together through a bus system. In addition to the data bus, the bus system may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are called bus systems in the figure.

The methods disclosed in the above embodiments of the present application can be applied to the processor 1503 or implemented by the processor 1503. The processor 1503 may be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the above method can be completed by instructions in the form of hardware integrated logic circuits or software in the processor 1503 . The above-mentioned processor 1503 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field programmable Gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 1503 can implement or execute each method, step and logical block diagram disclosed in the embodiment of this application. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field. The storage medium is located in the memory 1504. The processor 1503 reads the information in the memory 1504 and completes the steps related to the model inference process in the above method in combination with its hardware.

The receiver 1501 may be configured to receive input numeric or character information and generate signal inputs related to performing relevant settings and functional controls of the device. The transmitter 1502 can be used to output numeric or character information through the first interface; the transmitter 1502 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1502 can also include a display device such as a display screen .

The embodiment of the present application also provides a training device. Please refer to Figure 16. Figure 16 is a schematic structural diagram of the training device provided by the embodiment of the present application. Specifically, the training device 1600 is implemented by one or more servers. The training device 1600 There may be relatively large differences due to different configurations or performance, which may include one or more central processing units (CPU) 1616 (for example, one or more processors) and memory 1632, one or more storage applications Storage medium 1630 for program 1642 or data 1644 (eg, one or more mass storage devices). Among them, the memory 1632 and the storage medium 1630 may be short-term storage or persistent storage. The program stored in the storage medium 1630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processor 1616 may be configured to communicate with the storage medium 1630 and execute a series of instruction operations in the storage medium 1630 on the training device 1600 .

The training device 1600 may also include one or more power supplies 1626, one or more wired or wireless network interfaces 1650, one or more input and output interfaces 1658; or, one or more operating systems 1641, such as Windows ServerTM, Mac OS XTM , UnixTM, LinuxTM, FreeBSDTM and so on.

In the embodiment of this application, the central processor 1616 is used to perform actions related to model training in the above embodiment.

An embodiment of the present application also provides a computer program product that, when run on a computer, causes the computer to perform the steps performed by the foregoing execution device, or causes the computer to perform the steps performed by the foregoing training device.

Embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a program for performing signal processing. When the program is run on a computer, it causes the computer to perform the steps performed by the aforementioned execution device. , or, causing the computer to perform the steps performed by the aforementioned training device.

The execution device, training device or terminal device provided by the embodiment of the present application may specifically be a chip. The chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor. The communication unit may be, for example, an input/output interface. Pins or circuits, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register, cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as Read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (random access memory, RAM), etc.

Specifically, please refer to Figure 17. Figure 17 is a structural schematic diagram of a chip provided by an embodiment of the present application. The chip can be represented as a neural network processor NPU 1700. The NPU 1700 serves as a co-processor and is mounted to the main CPU (Host). CPU), tasks are allocated by the Host CPU. The core part of the NPU is the arithmetic circuit 1703. The arithmetic circuit 1703 is controlled by the controller 1704 to extract the matrix data in the memory and perform multiplication operations.

In some implementations, the computing circuit 1703 internally includes multiple processing units (Process Engine, PE). In some implementations, arithmetic circuit 1703 is a two-dimensional systolic array. The arithmetic circuit 1703 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 1703 is a general-purpose matrix processor.

For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit obtains the corresponding data of matrix B from the weight memory 1702 and caches it on each PE in the arithmetic circuit. The operation circuit takes matrix A data and matrix B from the input memory 1701 to perform matrix operations, and the partial result or final result of the matrix is stored in an accumulator (accumulator) 1708 .

The unified memory 1706 is used to store input data and output data. The weight data directly passes through the storage unit access controller (Direct Memory Access Controller, DMAC) 1705, and the DMAC is transferred to the weight memory 1702. Input data is also transferred to unified memory 1706 via DMAC.

BIU is the Bus Interface Unit, that is, the bus interface unit 1710, which is used for the interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1709.

The bus interface unit 1710 (Bus Interface Unit, BIU for short) is used to fetch the memory 1709 to obtain instructions from the external memory, and is also used for the storage unit access controller 1705 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1706 or the weight data to the weight memory 1702 or the input data to the input memory 1701 .

The vector calculation unit 1707 includes multiple arithmetic processing units. If necessary, the output of the arithmetic circuit 1703 is further processed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. Mainly used for non-convolutional/fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

In some implementations, vector calculation unit 1707 can store the processed output vectors to unified memory 1706 . For example, the vector calculation unit 1707 can apply a linear function; or a nonlinear function to the output of the operation circuit 1703, such as linear interpolation on the feature plane extracted by the convolution layer, or a vector of accumulated values, to generate an activation value. In some implementations, vector calculation unit 1707 generates normalized values, pixel-wise summed values, or both. In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit 1703, such as for use in a subsequent layer in a neural network.

The instruction fetch buffer 1709 connected to the controller 1704 is used to store instructions used by the controller 1704;

The unified memory 1706, the input memory 1701, the weight memory 1702 and the fetch memory 1709 are all On-Chip memories. External memory is private to the NPU hardware architecture.

The processor mentioned in any of the above places can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above programs.

In addition, it should be noted that the device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physically separate. The physical unit can be located in one place, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between modules indicates that there are communication connections between them, which can be specifically implemented as one or more communication buses or signal lines.

Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by software plus necessary general hardware. Of course, it can also be implemented by dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, Special components, etc. to achieve. In general, all functions performed by computer programs can be easily implemented with corresponding hardware. Moreover, the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special-purpose circuits. circuit etc. However, for this application, software program implementation is a better implementation in most cases. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or that contributes to the existing technology. The computer software product is stored in a readable storage medium, such as a computer floppy disk. , U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including several instructions to cause a computer device (which can be a personal computer, training device, or network device, etc.) to execute the steps described in various embodiments of this application. method.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, the computer instructions may be transferred from a website, computer, training device, or data The center transmits to another website site, computer, training equipment or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store, or a data storage device such as a training device or a data center integrated with one or more available media. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, solid state disk (Solid State Disk, SSD)), etc.

Claims (25)

A model training method, characterized by including: Obtain the first text and the second text, the first text and the second text are extracted from the original code text; wherein, the first text is the program code, and the second text is the program Attribute description corresponding to the code; the first text and the second text include known text and predicted text; According to the known text, the text corresponding to the text position of the predicted text is predicted through a natural language model to obtain a text prediction result; the predicted text and the text prediction result are used to update the natural language model. The method according to claim 1, wherein the attribute description includes a function description of the program code or an implementation principle of the program code. The method according to claim 1 or 2, characterized in that the first text is identified from the original code text through a program code identification method, and the second text is obtained from the original code text through an attribute description identification method. identified in . The method according to any one of claims 1 to 3, characterized in that the program code is the complete code of one or more functions. The method according to any one of claims 1 to 4, characterized in that the method further includes: When predicting text in the first text, determine the text position corresponding to the predicted text from the first text in a first manner; When predicting the text in the second text, the text position corresponding to the predicted text is determined from the second text through the second method; the first method and the second method are different. The method according to claim 5, characterized in that the first method and the second method are respectively one of the following prediction methods: Sample mask positions among text positions of part or all of the first text or second text, and use the mask positions as the text positions corresponding to the predicted text; Use the following text position of the known text as the text position corresponding to the predicted text; wherein the known text is all the text that has been predicted by the natural language model; The following text position of the known text is used as the text position corresponding to the predicted text; wherein the known text is the unmasked text among all the texts that have been predicted by the natural language model. The method according to any one of claims 1 to 4, characterized in that the method further includes: Without predicting the text in the second text, determine the text position corresponding to the predicted text through one of the following prediction methods: Sample mask positions from text positions of part or all of the first text, and use the mask positions as text positions corresponding to the predicted text; Use the following text position of the known text as the text position corresponding to the predicted text; wherein the known text is all the text that has been predicted by the natural language model; The following text position of the known text is used as the text position corresponding to the predicted text; wherein the known text is the unmasked text among all the texts that have been predicted by the natural language model. The method according to any one of claims 1 to 7, characterized in that, The first text is obtained based on the first original text extracted from the original code text; wherein the first text is obtained by modifying some text units in the first original text; or, The second text is obtained based on the second original text extracted from the original code text; wherein the second text is obtained by modifying some text units in the second original text. The method according to claim 8, characterized in that the modification includes: deletion, addition or replacement. The method according to claim 9, characterized in that the replacement is specifically a replacement with the same semantics. The method according to any one of claims 1 to 10, characterized in that the first text or the second text is text extracted from the original code text and adapted to the application field of the program code. The method according to any one of claims 1 to 11, characterized in that the application field adapted to the program code includes: The text length or the length of a single line of text is less than a threshold; wherein the threshold is related to the application field of the program code. A model training device, characterized by including: An acquisition module is used to acquire a first text and a second text. The first text and the second text are extracted from the original code text; wherein the first text is program code, and the second text is extracted from the original code text. The text is an attribute description corresponding to the program code; the first text and the second text include known text and predicted text; A prediction module, configured to predict the text corresponding to the text position of the predicted text through a natural language model based on the known text, and obtain a text prediction result; the predicted text and the text prediction result are used to update the Natural language model. The device according to claim 13, wherein the attribute description includes a function description of the program code or an implementation principle of the program code. The device according to claim 13 or 14, characterized in that the device further includes: A text position determination module, configured to determine the text position corresponding to the predicted text from the first text in a first manner when predicting the text in the first text; When predicting the text in the second text, the text position corresponding to the predicted text is determined from the second text through the second method; the first method and the second method are different. The device according to claim 15, wherein the first method and the second method are respectively one of the following prediction methods: Sample mask positions among text positions of part or all of the first text or second text, and use the mask positions as the text positions corresponding to the predicted text; Use the following text position of the known text as the text position corresponding to the predicted text; wherein the known text is all the text that has been predicted by the natural language model; The following text position of the known text is used as the text position corresponding to the predicted text; wherein the known text is the unmasked text among all the texts that have been predicted by the natural language model. The device according to claim 13 or 14, characterized in that the text position determination module is further configured to determine the text position through one of the following prediction methods without predicting the text in the second text. The text position corresponding to the predicted text: Sample mask positions from text positions of part or all of the first text, and use the mask positions as text positions corresponding to the predicted text; Use the following text position of the known text as the text position corresponding to the predicted text; wherein the known text is all the text that has been predicted by the natural language model; The following text position of the known text is used as the text position corresponding to the predicted text; wherein the known text is the unmasked text among all the texts that have been predicted by the natural language model. The device according to any one of claims 13 to 17, characterized in that: The first text is obtained based on the first original text extracted from the original code text; wherein the first text is obtained by modifying some text units in the first original text; or, The second text is obtained based on the second original text extracted from the original code text; wherein the second text is obtained by modifying some text units in the second original text. The device according to claim 18, wherein the modification includes: deletion, addition or replacement. The device according to claim 19, wherein the replacement is specifically a replacement with the same semantic meaning. The device according to any one of claims 13 to 20, wherein the first text or the second text is text extracted from the original code text and adapted to the application field of the program code. The device according to any one of claims 13 to 21, characterized in that the application field adapted to the program code includes: The text length or the length of a single line of text is less than a threshold; wherein the threshold is related to the application field of the program code. A computer storage medium, characterized in that the computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to execute claims 1 to 12 The operation of any of the methods. A computer program product, characterized by comprising computer-readable instructions, which when run on a computer device, cause the computer device to execute the method according to any one of claims 1 to 12. A system includes at least one processor and at least one memory; the processor and the memory are connected through a communication bus and complete communication with each other; The at least one memory is used to store code; The at least one processor is configured to execute the code to perform the method according to any one of claims 1 to 12.