RU2824338C2 - Multistage training of machine learning models for ranking search results - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to a system and a method for training a machine learning model to rank digital objects of the use stage. Method includes obtaining by processor a first plurality of training digital objects, wherein each training digital object from the first plurality of training digital objects is associated with a parameter of past user actions indicating user actions of past users with said training digital object; training at the first stage of training based on the first set of training digital objects of the machine learning model to determine the parameter of predicted user actions for the digital object of the use stage, wherein the predicted user actions parameter indicates user actions of future users with the digital object of the use stage; obtaining by the processor a second set of training digital objects, wherein each training digital object from the second set of training digital objects is connected (a) with a training search query used to generate a training digital object from a second plurality of training digital objects, and (b) with a first label indicating the degree of relevance of the object from the second plurality of training digital objects to the training search query; training at the second training stage following the first training stage, based on the second plurality of training digital objects of the machine learning model, determining a synthesized label of the digital object of the use stage, indicating the degree of relevance of the digital object of the use stage to the search request of the use stage; application by a processor of a machine learning model with respect to a first plurality of training digital objects to augment an object from the first plurality of training digital objects with a synthesized label and thus forming a first augmented plurality of training digital objects; and training, based on the first augmented set of training digital objects of the machine learning model, to determine the relevance parameter of the digital object of the use stage, which indicates the degree of relevance of the digital object of the use stage to the search request of the use stage, wherein the training digital object from the first plurality of training digital objects contains an indication of the digital document associated with the document metadata, and based on the first plurality of training digital objects, training the machine learning model at the first training stage further includes: converting the document metadata into a text representation thereof containing tokens; preprocessing the text representation for masking several masked tokens therein; and training, based on the first plurality of training digital objects, of the machine learning model to determine a token from a plurality of masked tokens based on the context provided by neighboring tokens, wherein the relevance parameter of the digital object of the use stage further indicates a semantic relevance parameter indicating the degree of semantic relevance of the search query of the use stage to the content of the digital object of the use stage.

EFFECT: high relevance of search results generated by a search engine in response to a user request, due to accurate ranking of search results on a SERP page, performed by a machine learning model.

23 cl, 6 dwg

Description

Field of technology to which the invention relates

[01] The present technology relates to machine learning methods, in particular to methods and systems for training and applying transformer-based machine learning models for ranking search results.

State of the art

[02] Web search is a major task that involves processing billions of user queries every day. Modern web search engines typically rank search results according to their relevance to the search query and other criteria. Determining the relevance of search results to a query often involves the use of machine learning algorithms trained to use several manually defined features to evaluate various relevance scores. Such relevance determination can be viewed, at least in part, as a language understanding problem, since the relevance of a document to a search query has at least some bearing on the semantic understanding of the query and the search results, even in cases where the query and results do not share words or where the results are images, music, or other non-textual results.

[03] Recent developments in neural natural language processing include the use of transformer machine learning models, as described in Vaswani et al., “Attention Is All You Need,” Advances in neural information processing systems, pages 5998–6008, 2017. A transformer is a deep learning model (i.e., an artificial neural network or other machine learning model containing multiple layers) that uses an attention mechanism to assign greater importance to some parts of the input data than others. In natural language processing, attention is used to assign context to words in the input data, where the same word may have different meanings in different contexts. Transformers are capable of processing many words or tokens of natural language in parallel, enabling parallel learning.

[04] Transformers are also the basis for other advances in natural language processing, including pre-trained systems that can be trained in advance using a large dataset and then “refined” for use in specific applications. Examples of such systems include the Bidirectional Encoder Representations from Transformers (BERT) model described in Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” Proceedings of NAACL-HLT 2019, pages 4171–4186, 2019, and the Generative Pre-trained Transformer (GPT) model described in Radford et al., “Improving Language Understanding by Generative Pre-Training,” 2018.

[05] Although transformers have made significant advances in natural language processing tasks, there may be some challenges when using them in practice to rank search results. For example, many large search relevance datasets contain non-textual data, such as information about the links a user clicked, which can be useful in training a ranking model.

Disclosure of invention

[06] Various embodiments of the present technology provide methods for efficiently training transformer models on query metadata and search relevance data, such as click data, in a pre-training phase. These models can then be refined using smaller crowdsourced relevance data sets for use in ranking search results. The described technology improves the efficiency of systems used to rank search results to potentially serve tens of millions of active users and process thousands of queries per second.

[07] According to one aspect of the present technology, a computer method for training a machine learning model to rank digital objects of the use stage generated using a search query of the use stage is implemented. The method is performed by a processor and includes receiving by the processor a first plurality of training digital objects, wherein an object from the first plurality of training digital objects is associated with a parameter of past user actions indicating user actions of past users with an object from the first plurality of training digital objects. The method at the first stage of training the machine learning model further includes training, based on the first plurality of training digital objects, to determine a parameter of predicted user actions for the digital object of the use stage, wherein the parameter of predicted user actions indicates user actions of future users with the digital object of the use stage. The method also includes receiving by the processor a second plurality of training digital objects, wherein the object from the second plurality of training digital objects is associated with (a) a training search query used to form the object from the second plurality of training digital objects, and (b) with a first label generated by the evaluator, indicating the degree of relevance of the object from the second plurality of training digital objects to the training search query from the point of view of the human evaluator who assigned the first label generated by the evaluator. The method at the second training stage following the first training stage also includes training a machine learning model based on the second plurality of training digital objects to determine a synthesized evaluator label for the digital object of the use stage, indicating the degree of relevance of the digital object of the use stage to the search query of the use stage from the point of view of the human evaluator in the case of providing the digital object of the use stage to the human evaluator. The method also includes the application by the processor of the machine learning model in relation to the first plurality of training digital objects to supplement the object from the first plurality of training digital objects with the synthesized label of the evaluator and thus forming the first supplemented plurality of training digital objects. The method also includes training the machine learning model based on the first supplemented plurality of training digital objects to determine the relevance parameter of the digital object of the use stage, indicating the degree of relevance of the digital object of the use stage to the search query of the use stage.

[08] In some embodiments of the invention, an object from the first plurality of training digital objects comprises an indication of a digital document associated with the document metadata. Based on the first plurality of training digital objects, training a machine learning model in a first training stage further comprises: converting the document metadata into a text representation thereof comprising tokens; pre-processing the text representation to mask a plurality of masked tokens therein; and based on the first plurality of training digital objects, training the machine learning model to determine a token from the plurality of masked tokens based on the context provided by neighboring tokens. The relevance parameter of the usage stage digital object further indicates a semantic relevance parameter indicating the degree of semantic relevance of the usage stage search query to the content of the usage stage digital object. In some embodiments of the invention, the document metadata comprises at least one of the following: a training search query associated with an object from the first plurality of training digital objects, a title of the digital document, content of the digital document, and a web address associated with the digital document.

[09] In some embodiments of the method, it further includes determining a parameter of past user actions associated with an object from the first plurality of training digital objects, based on data about "clicks" (presses, for example, selecting a search result) of past users. In some embodiments of the invention, the data about "clicks" comprises data about at least one "click" of at least one past user made in response to sending a training search query associated with an object from the first plurality of training digital objects.

[010] In some embodiments of the method, before training the machine learning model to determine the relevance parameter of the digital object of the use stage, it further includes receiving by the processor a third plurality of training digital objects, wherein the object from the third plurality of training digital objects is associated with (a) a training search query used to generate the object from the third plurality of training digital objects, and (b) with a second label generated by the evaluator indicating the degree of relevance of the object from the third plurality of training digital objects to the training search query from the point of view of the human evaluator who assigned the second label generated by the evaluator. In these embodiments, the method also includes training, at a third training stage following the second training stage, on the basis of the third plurality of training digital objects, the machine learning model to determine a refined synthesized label of the evaluator of the digital object of the use stage indicating the degree of relevance of the digital object of the use stage to the search query of the use stage from the point of view of the human evaluator in the case of providing the digital object of the use stage to the human evaluator. The method also includes the processor applying a machine learning model to the first augmented plurality of training digital objects to augment an object from the first augmented plurality of training digital objects with the refined synthesized label of the evaluator and thus forming a second augmented plurality of training digital objects. In these embodiments of the method, it further includes training the machine learning model to determine the relevance parameter of the digital object of the use stage based on the second augmented plurality of training digital objects. In some embodiments of the invention, the plurality of the first plurality of training digital objects, the second plurality of training digital objects and the third plurality of training digital objects is at least partially different from any other plurality of the first plurality of training digital objects, the second plurality of training digital objects and the third plurality of training digital objects. In some embodiments of the invention, the plurality of the first plurality of training digital objects, the second plurality of training digital objects and the third plurality of training digital objects is larger in size than a subsequent plurality of the first plurality of training digital objects, the second plurality of training digital objects and the third plurality of training digital objects.

[011] In some embodiments of the method, after training the machine learning model to determine the relevance parameter of the digital object of the use stage, it further includes receiving by the processor a third plurality of training digital objects, wherein the object from the third plurality of training digital objects is associated with (a) a training search query used to generate the object from the third plurality of training digital objects, and (b) with a second label generated by the evaluator indicating the degree of relevance of the object from the third plurality of training digital objects to the training search query from the point of view of the human evaluator who assigned the second label generated by the evaluator. The method also includes training the machine learning model based on the third plurality of training digital objects to determine a refined relevance parameter of the digital object of the use stage indicating the degree of relevance of the digital object of the use stage to the search query of the use stage. In some embodiments of the invention, the plurality of the first plurality of training digital objects, the second plurality of training digital objects, and the third plurality of training digital objects is at least partially different from any other plurality of the first plurality of training digital objects, the second plurality of training digital objects, and the third plurality of training digital objects. In some embodiments of the invention, the plurality of the first plurality of training digital objects, the second plurality of training digital objects, and the third plurality of training digital objects is larger in size than a subsequent plurality of the first plurality of training digital objects, the second plurality of training digital objects, and the third plurality of training digital objects. In some embodiments of the invention, the third plurality of training objects is identical to the second plurality of training digital objects.

[012] In some embodiments of the invention, in the first training stage, the machine learning model is trained to determine a rough initial estimate of the relevance parameter of the digital object of the use stage. In each subsequent training stage, the machine learning model is trained to improve the rough initial estimate. In some embodiments of the invention, the improvement of the rough initial estimate is determined using a metric based on a normalized discounted cumulative indicator.

[013] In some embodiments of the invention, the machine learning model comprises at least one learning model. In some embodiments of the invention, the at least one learning model is a transformer-based learning model.

[014] In some embodiments of the invention, the machine learning model comprises at least two learning models. The first model of the two learning models is trained to determine the synthesized evaluator label for the digital object of the use stage in order to form a first augmented set of training digital objects. The second model of the two learning models is trained to determine the relevance parameter of the digital object of the use stage based on the first augmented set of training digital objects. In some embodiments of the invention, the first model of the two learning models differs from the second model. In some embodiments of the invention, the first model of the two learning models is a transformer-based learning model.

[015] In some embodiments of the method, it further includes ranking the digital objects of the use stage based on the relevance parameters associated therewith. In some embodiments of the method, it further includes ranking the digital objects of the use stage based on the relevance parameters associated therewith, including using another learning model trained to rank the digital objects of the use stage using the relevance parameters generated by the machine learning model as input features. In some embodiments of the invention, the other learning model is a learning model based on CatBoost decision trees.

[016] According to another aspect of the present technology, a system is implemented for training a machine learning model to rank use-stage digital objects generated using a use-stage search query. The system comprises a processor and memory associated with the processor and containing a machine learning training module executed by the processor. The machine learning training module comprises instructions, when executed, the processor performs the following actions: obtaining a first plurality of training digital objects, wherein an object from the first plurality of training digital objects is associated with a past user action parameter indicating user actions of past users with an object from the first plurality of training digital objects; in a first training stage, training the machine learning model based on the first plurality of training digital objects to determine a predicted user action parameter for a use-stage digital object indicating user actions of future users with the use-stage digital object; obtaining a second plurality of training digital objects, wherein an object from the second plurality of training digital objects is associated with (a) a training search query used to generate an object from the second plurality of training digital objects, and (b) with a first label generated by the evaluator indicating the degree of relevance of an object from the second plurality of training digital objects to the training search query from the point of view of a human evaluator who assigned the first label generated by the evaluator; in a second training stage following the first training stage - training a machine learning model based on the second plurality of training digital objects to determine a synthesized label of the evaluator of the use-stage digital object indicating the degree of relevance of the digital object of the use-stage to the search query of the use-stage from the point of view of the human evaluator in the case of providing the digital object of the use-stage to the human evaluator; applying the machine learning model with respect to the first plurality of training digital objects to supplement the object from the first plurality of training digital objects with the synthesized label of the evaluator and thereby forming the first supplemented set of training digital objects; and, based on the first augmented set of training digital objects, training a machine learning model to determine the relevance parameter of the digital object of the use stage, indicating the degree of relevance of the digital object of the use stage to the search query of the use stage.

[017] In some embodiments of the invention, an object from the first plurality of training digital objects comprises an indication of a digital document associated with document metadata. The machine learning training module also comprises instructions, when executed, for training a machine learning model in a first training stage based on the first plurality of training digital objects by: converting the document metadata into a text representation thereof comprising tokens; pre-processing the text representation to mask a plurality of masked tokens therein; and, based on the first plurality of training digital objects, training the machine learning model to determine a token from a plurality of masked tokens based on the context provided by neighboring tokens. In these embodiments of the invention, the relevance parameter of the use stage digital object further indicates a semantic relevance parameter indicating the degree of semantic relevance of the use stage search query to the content of the use stage digital object.

[018] In some embodiments of the invention, the machine learning training module further comprises instructions that, when executed, the processor, prior to training the machine learning model to determine the relevance parameter of the use-stage digital object, performs the following actions: obtaining a third plurality of training digital objects, wherein an object from the third plurality of training digital objects is associated with (a) a training search query used to generate an object from the third plurality of training digital objects, and (b) with a second evaluator-generated label indicating the degree of relevance of an object from the third plurality of training digital objects to the training search query from the point of view of a human evaluator who assigned the second evaluator-generated label; training, in a third training stage following the second training stage, based on the third plurality of training digital objects, the machine learning model to determine a refined synthesized evaluator label of the use-stage digital object indicating the degree of relevance of the use-stage digital object to the use-stage search query from the point of view of the human evaluator in the case of providing the use-stage digital object to the human evaluator; applying a machine learning model to a first augmented set of training digital objects to augment an object from the first augmented set of training digital objects with a refined synthesized label of the evaluator and thereby forming a second augmented set of training digital objects; and training the machine learning model, based on the second augmented set of training digital objects, to determine the relevance parameter of the digital object of the use stage.

Brief description of the drawings

[019] These and other features, aspects and advantages of the present technology are explained in the following description, in the appended claims and in the following drawings.

[020] Fig. 1 is a diagram of an example computer system for use in some embodiments of systems and/or methods according to the present technology.

[021] Fig. 2 is a block diagram of the architecture of a machine learning model according to various embodiments of the present technology.

[022] Fig. 3 illustrates a structure of data sets that may be used to pre-train and fine-tune a machine learning model for use in ranking search results according to various embodiments of the present technology.

[023] Fig. 4 is a flow chart of the pre-training and fine-tuning steps performed to train a machine learning model to generate relevance scores, according to various embodiments of the present technology.

[024] Fig. 5 is a block diagram of a computer method for training a machine learning model according to various embodiments of the present technology.

[025] Fig. 6 is a block diagram of the use of a fully trained machine learning model for ranking search results according to various embodiments of the present technology.

Implementation of the invention

[026] Various exemplary embodiments of the present technology are described more fully below with reference to the accompanying drawings. However, the present technology can be implemented in many different forms and should not be considered as limited to the exemplary embodiments described herein. The absolute and relative sizes of layers and regions may be exaggerated in the drawings for clarity. The same numerals refer to the same elements throughout.

[027] The examples and conventional language provided herein are intended to provide a better understanding of the principles of the present technology, and not to limit its scope to such specifically provided examples and conditions. It is obvious that those skilled in the art are able to devise various methods and devices that are not explicitly described or shown, but implement the principles of the present technology within its spirit and scope.

[028] To facilitate better understanding, the following description may contain simplified embodiments of the present technology. Those skilled in the art will understand that other embodiments of the present technology may be significantly more complex.

[029] In some cases, useful examples of modifications of the present technology are given. They facilitate understanding, but also do not define the scope or boundaries of the present technology. The list of modifications provided is not exhaustive and a person skilled in the art can develop other modifications within the scope of the present technology. In addition, if in some cases modifications are not described, this does not mean that they are impossible and/or that the description contains the only possible embodiment of a particular element of the present technology.

[030] It is obvious that although the numerals "first", "second", "third", etc. are used herein to describe various elements, these elements should not be limited by such numerals. Such numerals are used only to indicate the difference between the elements. Thus, the first element discussed below can be called the second element without going beyond the scope of the present technology. In the context of this document, the term "and/or" corresponds to any element and all combinations of elements from the corresponding listed elements.

[031] It should be understood that when an element is said to be connected or related to another element, it may be connected or related to the other element directly, or there may be intermediate elements. If an element is said to be directly connected or related to another element, there are no intermediate elements. Other words used to describe the relationship between elements should be understood in a similar way (e.g., "between" and "directly between", "adjacent" and "directly adjacent", etc.).

[032] The terminology used herein is intended to describe specific exemplary embodiments only, but not to limit the scope of protection of the present technology. The singular words used herein also include the plural words, unless the context clearly indicates otherwise. It should be understood that the terms "comprises" and/or "comprising" as used herein correspond to the presence of the stated features, parts, steps, operations, elements and/or components, but do not exclude the presence or addition of one or more other features, parts, steps, operations, elements, components and/or groups thereof.

[033] The functions of the various elements shown in the drawings, including any functional unit designated as a "processor", may be implemented using specialized hardware as well as hardware capable of executing software. If a processor is used, these functions may be performed by a single dedicated processor, a single shared processor, or a plurality of separate processors, some of which may be shared. In some embodiments of the present technology, the processor may be a general-purpose processor, such as a central processing unit (CPU), or a specialized processor, such as a digital signal processor (DSP). Furthermore, the explicit use of the term "processor" should not be construed as referring exclusively to hardware capable of executing software, and may include, but is not limited to, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a read-only memory (ROM) for storing software, a random access memory (RAM), and a non-volatile memory. Other hardware, general-purpose and/or custom, may also be included.

[034] Software modules or simply modules or blocks that are intended to be implemented as software may be represented here as any combination of flow chart elements or other elements indicating the execution of process steps and/or containing a textual description. Such modules may be executed by hardware, shown explicitly or implied. In addition, it should be understood that a module may, among other things, contain, for example, computer software logic, computer software instructions, application software, a stack, firmware, hardware circuitry, or a combination thereof that provide the required capabilities.

[035] As used herein, the term "database" means any structured collection of data, regardless of its specific structure, database management software, or computer hardware for storing, using, or otherwise enabling the use of that data. A database may be located on the same hardware as the process that stores or uses the information stored in the database, or the database may be located on separate hardware, such as a dedicated server or multiple servers.

[036] The present technology may be implemented as a system, a method, and/or a computer product. The computer program product may comprise a computer-readable storage medium (or several storage media) storing computer-readable program instructions that, when executed, cause a processor to implement aspects of the present technology. The computer-readable storage medium may, for example, be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of a computer-readable storage medium includes a portable computer disk, a hard disk, RAM, ROM, flash memory, an optical disk, a memory card, a floppy disk, a medium with mechanical or visual encoding (e.g., a punch card or a bar code), and/or any combination thereof. In the context of this document, a computer-readable storage medium shall be considered as a computer-readable physical storage medium. It should not be considered as a changing signal such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g. light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[037] It should be understood that the machine-readable program instructions may be loaded from a machine-readable storage medium into the corresponding computing or processing devices or into an external computer or external storage device via a network, such as via the Internet, a local area network, a wide area network and/or a wireless network. A network interface in the computing or processing device may receive the machine-readable program instructions via the network and forward the machine-readable program instructions for storage in a machine-readable storage medium in the corresponding computing or processing device.

[038] Machine-readable program instructions for performing operations according to the present technology may be assembler instructions, machine instructions, firmware instructions, configuration data for integrated circuits, or other source code or object code written using any combination of programming languages. Machine-readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network.

[039] The description of the principles, aspects and embodiments of the present technology, as well as specific examples thereof, are intended to cover their structural and functional equivalents, regardless of whether they are currently known or will be developed in the future. For example, those skilled in the art should understand that any block diagrams described herein correspond to conceptual representations of illustrative circuit diagrams implementing the principles of the present technology. It should also be understood that any block diagrams, process diagrams, state transition diagrams, pseudocodes and the like correspond to various processes that can be represented in computer-readable software instructions. These computer-readable software instructions can be provided to a processor or other programmable data processing device to form a machine so that the instructions, when executed by the computer processor or other programmable data processing device, create means for implementing the functions or actions indicated in the block diagram and/or a block or blocks of the block diagram. These machine-readable program instructions may also be stored in a machine-readable storage medium that may cause a computer, programmable data processing device, and/or other devices to operate in a specific manner so that the machine-readable storage medium with the instructions stored therein comprises an article containing instructions implementing aspects of the functions or actions indicated in the flow charts, process diagrams, state transition diagrams, pseudocodes, etc.

[040] These computer-readable program instructions may also be loaded into a computer, other programmable data processing device, or other devices to initiate a sequence of operational steps to be performed in the computer, other programmable device, or other devices to form a computer process so that the instructions executed in the computer, other programmable device, or other devices implement the functions or actions specified in flow charts, process diagrams, state transition diagrams, pseudocodes, etc.

[041] In some alternative embodiments of the invention, the functions indicated in flow charts, process diagrams, state transition diagrams, pseudocodes, and the like may be implemented in an order different from that indicated in the drawings. For example, two blocks shown in a flow chart as sequential may actually be performed simultaneously, or these blocks may sometimes be performed in the opposite order - depending on the function being implemented. It should also be noted that each function indicated in the drawings, and combinations of such functions, may be implemented by systems based on specialized hardware that perform the indicated functions or actions, or by combinations of specialized hardware and computer instructions.

[042] Next, taking into account the above principles, some non-limiting examples are considered, illustrating various options for implementing aspects of this technology.

Computer system

[043] Fig. 1 shows a computer system 100. The computer system 100 may be a multi-user computer, a single-user computer, a laptop, a tablet computer, a smartphone, an embedded management system, or any other computer system that is currently known or will be developed in the future. In addition, it should be understood that some or all of the elements of the computer system 100 may be virtualized and/or cloud-based. As shown in Fig. 1, the computer system 100 includes one or more processors 102, memory 110, a data storage interface 120, and a network interface 140. These system elements are interconnected via a bus 150, which may include one or more internal and/or external buses (not shown) (such as a PCI bus, a USB bus, an IEEE 1394 FireWire bus, a SCSI bus, a Serial-ATA bus, etc.), to which various hardware elements are electronically connected.

[044] Memory 110, which may be RAM or any other type of memory, may contain data 112, an operating system 114, and a program 116. Data 112 may be any data corresponding to input or output data of any program in computer system 100. Operating system 114 is an operating system such as MICROSOFT WINDOWS or LINUX. Program 116 may be any program or set of programs containing program instructions that can be executed by a processor to control actions performed by computer system 100. For example, program 116 may be a machine learning training module that trains a machine learning model, as described below. Program 116 may also be a system that uses a trained machine learning model to rank search results, as described below.

[045] The data storage interface 120 is used to connect storage devices, such as the storage device 125, to the computer system 100. The storage device 125 of one type is a solid-state drive, in which an integrated circuit block may be used for persistent storage of data. The storage device 125 of another type is a hard disk drive, such as an electromechanical device that uses a magnetic storage device to store and retrieve digital data. The storage device 125 may also be an optical disk drive, a memory card reader, such as an SD card, or a flash memory device, which can be connected to the computer system 100, for example, via a universal serial bus (USB).

[046] In some embodiments of the invention, computer system 100 may employ well-known virtual memory mechanisms to allow programs of computer system 100 to operate as if they had access to a large contiguous address space, rather than accessing multiple smaller memory areas such as memory 110 and storage device 125. Thus, although data 112, operating system 114, and programs 116 are shown as being located in memory 110, one skilled in the art will appreciate that these elements need not be contained entirely within memory 110 at any one time.

[047] The processors 102 may comprise one or more microprocessors and/or other integrated circuits. The processors 102 execute software instructions stored in the memory 110. When the computer system 100 is started, the processors 102 may first perform a boot procedure and/or execute software instructions that form the operating system 114.

[048] Network interface 140 is used to connect computer system 100 to other computer systems or network devices (not shown) via network 160. Network interface 140 may include a combination of hardware and software that provides communication via network 160. In some embodiments of the invention, network interface 140 may be a wireless network interface. Software of network interface 140 may include software that uses one or more network protocols to communicate via network 160. For example, network protocols may include Transmission Control Protocol/Internet Protocol (TCP/IP).

[049] It should be understood that the computer system 100 is merely an example and that the described technology may be used with computer systems or other computer devices of different configurations.

Machine learning model architecture

[050] Fig. 2 is a block diagram of a machine learning model architecture 200 according to various embodiments of the present technology. The machine learning model architecture 200 is based on the BERT machine learning model, as described, for example, in the above-mentioned work (Devlin et al.). Like the BERT model, the machine learning model architecture 200 comprises a transformer stack 202 of transformer blocks, including, for example, transformer blocks 204, 206, and 208.

[051] Each of the transformer blocks 204, 206 and 208 comprises a transformer encoder block, such as described in the work (Vaswani et al.) cited above. Each of the transformer blocks 204, 206 and 208 comprises a multi-headed attention layer 220 (shown for illustration only in the transformer block 204) and a feedforward neural network layer 222 (also shown for illustration only in the transformer block 204). The transformer blocks 204, 206 and 208 typically have the same structure but different weights (after training). The multi-headed attention layer 220 implements dependencies between the input data of the transformer block, which, for example, can be used to provide contextual information for each element of the input data based on each other element of the input data of the transformer block. In the feedforward neural network layer 222, such dependencies are typically absent, so the input data of the feedforward neural network layer 222 can be processed in parallel. It should be understood that although only three transformer blocks (transformer blocks 204, 206, and 208) are shown in Fig. 2, in actual embodiments of the present technology, the transformer stack 202 may contain many more such transformer blocks. For example, in some embodiments of the invention, 12 transformer blocks may be used in the transformer stack 202.

[052] The input data 230 of the transformer stack 202 comprises tokens such as the token 232 [CLS] and the tokens 234. The tokens 234 may, for example, represent words or parts of words. The token 232 [CLS] is used as a representation for classifying the entire set of tokens 234. Each token 234 and the token 232 [CLS] is represented by a vector. In some embodiments of the invention, the length of each of these vectors may, for example, correspond to 768 floating point values. It should be understood that a variety of compression methods may be used to efficiently reduce the sizes of the tokens. In various embodiments of the invention, a fixed number of tokens 234 may be used as input data 230 of the transformer stack 202. For example, in some embodiments of the invention, 1024 tokens may be used, and in other embodiments of the invention, the transformer stack 202 may receive 512 tokens (in addition to the [CLS] token 232). Input data 230 that are shorter than this fixed number of tokens 234 may be padded to the fixed length by adding padding tokens.

[053] In some embodiments of the invention, input data 230 may be generated using a tokenizer 238 from a digital object 236, such as an element from a training set. The architecture of the tokenizer 238 typically depends on the digital object 236 used as input to the tokenizer 238. For example, known encoding methods, such as byte pair encoding, may be used to generate input data 230 in the tokenizer 238, and pre-trained neural networks may also be used.

[054] The output data 250 of the transformer stack 202 comprises the output data 252 [CLS] and the vector output data 254, including the vector output data for each token 234 of the input data 230 of the transformer stack 202. The output data 250 may then be sent to the task module 270. In some embodiments of the invention, as shown in Fig. 2, the task module uses only the output data 252 [CLS], which represents the entire set of output data 254. This may be most useful when the task module 270 is used as a classifier, either to output a label or value that characterizes the entire input digital object 236, such as to generate a relevance score or a “click” probability (clicks, such as selecting a search result) on a document. In some embodiments of the invention (not shown in Fig. 2), all or some of the output data 254 and possibly the output data 252 [CLS] may be used as input to the task module 270. This is most useful when the task module 270 is used to generate labels or values for individual input tokens 234, such as to predict a masked or missing token or to recognize a named entity. In some embodiments of the invention, the task module 270 may comprise a feedforward neural network (not shown) that generates a task-specific output 280, such as a relevance score or a click probability. Other models may also be used in the task module 270. For example, the task module 270 may be a transformer or other type of neural network. Additionally, the task-specific output can be used as input to other models, such as the CatBoost model, as described in Dorogush et al., “CatBoost: gradient boosting with categorical features support,” NIPS, 2017.

[055] It should be understood that the architecture described with reference to Fig. 2 is simplified for better understanding. For example, in practical embodiments of the machine learning model architecture 200, each of the transformer blocks 204, 206 and 208 may include layer normalization operations, the task module 270 may contain a softmax normalization function, etc. Those skilled in the art will understand that these operations are widely used in neural networks and deep learning models, such as the machine learning model architecture 200.

Pre-training and fine-tuning

[056] According to various embodiments of the present technology, a machine learning model, the architecture of which is shown in Fig. 2, can be trained using pre-training and fine-tuning processes, as described below. Fig. 3 shows data sets that can be used to pre-train and fine-tune a machine learning model for use in ranking search results.

[057] The data sets comprise a Docs data set 302, which is a large collection of unlabeled documents 303 with a maximum token length 304 of 1024. The Docs data set 302 is used for pre-training using Masked Language Modeling (MLM) (see below). Pre-training on the Docs data set 302 is used to provide a kind of base language model that helps improve the quality of subsequent training and ensure the stability of training. In some embodiments, the Docs data set 302 may contain approximately 600 million training digital objects (i.e., unlabeled documents with a maximum token length of 1024).

[058] The data sets also comprise a "Clicks" data set 310, the elements 311 of which comprise a user query 312 and a document 314 from the search results for the user query 312 and are marked up using "click" information 316 indicating whether the user selected the document 314. The query 312, in addition to the query text, comprises query metadata 313, which may, for example, comprise the geographic region from which the query was sent. Similarly, the document 314 comprises the document text and document metadata 315, which may comprise the document title and the web address of the document (for example, in the form of a URL).

[059] In some embodiments of the invention, the "click" information 316 may be pre-processed to indicate the user's selection of a document only in the case of a "long click," where the user dwells on the selected document for an "extended" time. "Long clicks" are a widely used indicator of the relevance of a search result to a query, since they indicate that the user was able to find relevant information in the document, rather than simply "clicking" (tapping on) the document and quickly returning to the search results. For example, in some embodiments of the invention, a "long click" may indicate that the user dwelled on the document for at least 120 seconds.

[060] Because the Clicks data set 310 is based on information regularly collected as a result of users using the search engine, it is extremely large. For example, in some embodiments of the invention, the Clicks data set 310 may contain approximately 23 billion training digital objects (i.e., elements containing a query and a document and labeled using the click information). Because of its size, the Clicks data set 310 forms the main part of the training pipeline and is used in pre-training, as described below.

[061] The data sets also contain relevance data sets 350 used for fine-tuning, as described below. In some embodiments of the invention, relevance data sets 350 contain a Rel-Big data set 352, a Rel-Mid data set 354, and a Rel-Small relevance data set 356. Items 357, 358, and 359 of these data sets contain queries 360, 362, and 364, respectively, and documents 370, 372, and 374, respectively. Items from relevance data sets 350 are labeled using relevance scores 380, 382, and 384, respectively. Relevance scores 380, 382, and 384 are based on human rater input about how relevant documents are to the search query. This human rater input may be provided using crowdsourcing or other means of collecting data from people about document relevance to the query.

[062] Since the ratings 380, 382, and 384 are based on data entered by human raters, the relevance data sets 350 may require more time and resources to collect than other data sets used to train the machine learning model. Therefore, the relevance data sets 350 are much smaller than the other data sets and are used for fine-tuning rather than pre-training. For example, in some embodiments, the Rel-Big data set 352 may contain approximately 50 million training digital objects (i.e., items), the Rel-Mid data set 354 may contain approximately 2 million training digital objects, and the Rel-Small data set 356 may contain approximately 1 million training digital objects. In general, the 350 relevance datasets vary in size, age, and proximity to the latest ways of calculating relevance scores: the 352 "Rel-Big" dataset is the largest and oldest (in terms of data age and ways of calculating relevance scores), and the 356 "Rel-Small" dataset is the smallest and newest.

[063] Fig. 4 is a flow chart 400 of the pre-training and fine-tuning steps performed to train a machine learning model to generate relevance scores, according to various embodiments of the present technology. In a first step 402, the machine learning model is pre-trained using the "Docs" data set 302 (see Fig. 3) and the MLM task.

[064] The MLM task is based on one of two unsupervised learning tasks used in the BERT model, which is used to learn text representations from collections of unlabeled documents (it should be noted that the other unsupervised learning task used in the BERT model is the next sentence prediction task, which is not typically used in embodiments of the present technology). To pre-train on the MLM task, one or more tokens from the input to the machine learning model are masked by replacing them with a special token [MASK] (not shown). The machine learning model is trained to predict the probability of a masked token corresponding to tokens in a token dictionary. This is done based on the outputs (each element of which is a vector) corresponding to the masked tokens from the last layer of the transformer stack (see above) of the machine learning model. Since the actual masked tokens are known (i.e. they are the control information), the cross-entropy loss, which is a measure of the deviation of the predicted probabilities from the actual masked tokens (here called MLM loss), is calculated and used to adjust the weights in the machine learning model to reduce the loss.

[065] In the second pre-training step 404, the training digital objects from the Clicks dataset 310 (see Fig. 3) are used to pre-train the machine learning model. This is accomplished by tokenizing the query, including the query metadata, and the document, including the document metadata. The tokenized query and document are used as input to the machine learning model, with one or more tokens masked as in the first step. In this way, the query metadata and the document metadata, containing information such as the web address of the document and the geographic region of the query, are fed directly to the machine learning model along with the query text and the natural language document.

[066] To transform the query and the document, including metadata, of the training digital object from the Clicks data set into tokens, a pre-built dictionary of tokens suitable for the natural language text and the types of metadata used in the Clicks data set may be used. In some embodiments, this may be accomplished using the WordPiece byte pair encoding scheme used in the BERT model with a sufficiently large dictionary size. For example, in some embodiments, the dictionary size may correspond to approximately 120,000 tokens. In some embodiments, text pre-processing may be performed, such as converting all words to lowercase and Unicode NFC normalization. A WordPiece byte pair encoding scheme that may be used in some embodiments of the invention to construct a token dictionary is described, for example, in Rico Sennrich et al., “Neural Machine Translation of Rare Words with Subword Units,” Proceedings of the ^54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, 2016.

[067] In the second pre-training step 404, the machine learning model is trained using the MLM loss as described above with masked tokens. The machine learning model is also configured with a neural network-based classifier as a task module (as described with reference to Fig. 2) that predicts the probability of a "click" for a document. In some embodiments of the invention, the predicted probability of a "click" may be determined based on the output of the [CLS]. Since the training digital objects from the "Clicks" data set contain information about whether the user has selected the document or not, this control information can be used, for example, to determine a cross-entropy loss (called a "click" prediction loss) representing the distance or difference between the predicted probability of a "click" and the control information. The "click" prediction loss can be used to adjust the weights in the machine learning model for the purpose of training the model.

[068] Although the Clicks dataset collected from transaction logs can be used as a surrogate for relevance, it may not properly reflect the actual relevance of a document to a query. This problem is addressed in the fine-tuning step 406 by using the relevance datasets (described above) to train a machine learning model on documents manually labeled by human raters for their relevance to a query.

[069] In some embodiments of the invention, the fine-tuning step 406 is first performed using the Rel-Big data set (as described above with reference to Fig. 3), which is not only the largest but also the oldest of the relevance data sets. Queries and documents are tokenized as described above and provided to a machine learning model as input. The machine learning model uses a neural network task module to generate a predicted relevance score. In some embodiments of the invention, the task module can determine the predicted relevance score based on the output of the [CLS]. The Rel-Big data set contains a relevance score determined by a human assessor, which can be used as control information when training the machine learning model. This control information can be used, for example, to determine a cross-entropy loss that represents the distance or difference between the predicted relevance score and the control information, which can be used to adjust the weights in the machine learning model.

[070] In some embodiments of the invention, re-labeling a large Clicks data set and re-training a model using the re-labeled data set may be used in fine-tuning to improve the performance of a machine learning model. This may be accomplished by using a machine learning model trained as described above to generate predicted relevance scores for data objects from the Clicks data set to efficiently re-label the data objects from the Clicks data set to generate an augmented Clicks data set with synthesized rater labels. The augmented Clicks data set may then be used to re-train the machine learning model to predict relevance scores using the synthesized rater labels as control information.

[071] It should be understood that a similar approach, in which the first model is used to augment or label a data set that is then used to train a second model, can be used to effectively transfer the knowledge embedded in the first model to the second model. In effect, the first model becomes a "teacher" for the second model. Such transfer methods can be used with different model architectures such that the architecture of the first model is different from the architecture of the second model. For example, the second model may be a smaller neural network than the first model and may provide substantially similar or even more accurate results, for example using fewer layers, and thus may be faster to execute at the use stage.

[072] In some embodiments of the invention, such fine-tuning may be repeated using other data sets from among the relevance data sets. For example, a machine learning model may first be tuned using the Rel-Big data set, then refined using the Rel-Mid data set, and then refined again using the Rel-Small data set. In some embodiments of the invention, all or some of these machine learning model refinement steps may also include re-labeling the Clicks data set (or other large data set) and re-training the machine learning model, as described above.

[073] A machine learning model using this multi-stage approach can be thought of as providing a rough initial estimate of the relevance of a document to a query after initial training using the Clicks dataset, and improving this rough initial estimate at each subsequent fine-tuning stage. A metric commonly used in ranking problems, such as a metric based on the normalized discounted cumulative score, can be used to determine the improvements over the initial estimate at each fine-tuning stage.

[074] Fig. 5 shows a flow chart 500 of a computer method for training a machine learning model according to various embodiments of the present technology. The flow chart 500 includes a first pre-training stage 570, a second pre-training stage 572, and a fine-tuning stage 574.

[075] In block 502 of the first stage 570 of pre-training, the processor receives a set of unlabeled digital natural language documents. In block 504, the processor converts the digital documents from the set of unlabeled digital natural language documents into a set of tokens, and then one or more tokens are masked.

[076] In block 506, a machine learning model is trained using the masked set of tokens as input. The output of the machine learning model corresponding to the masked tokens is used along with the actual masked tokens to determine a loss (e.g., a cross-entropy loss) used to adjust the weights of the machine learning model. It should be understood that blocks 504 and 506 may be repeated for all unlabeled digital natural language documents or for a subset thereof. In some embodiments of the invention, the first pre-training stage 570 may be omitted or training may begin with a second pre-training stage, for example, using a "regular" pre-trained BERT model.

[077] In block 508 of the second pre-training stage 572, the processor receives a first set of training digital objects. The training digital objects from the first set of training digital objects are associated with a parameter of past user actions. The parameter of past user actions represents a user action of a past user with the training digital object, such as a "click" on a digital document associated with the training digital object that matches a query associated with the training digital object. In some embodiments of the invention, the training digital object is associated with a query containing a query text and query metadata, with a document containing a document text and document metadata, and with a past user action. The query metadata may, for example, contain a geographic region from which the query is sent. The document metadata may, for example, contain a web address of the document, such as a URL of the document, and a title of the document. In some embodiments of the invention, the query, including its metadata, may be included in the document metadata.

[078] In block 510, the processor converts the query and the digital document associated with the training digital object, including metadata associated with the query and the digital document, into tokens and one or more tokens are masked to form input tokens. Such tokenization can be performed using a pre-created dictionary of tokens, which in some embodiments of the invention can be defined using byte pair encoding.

[079] In block 512, a machine learning model is trained to determine a predicted user action parameter, such as the probability of a user "clicking" on a document, indicating that the user considers the document relevant to the query. This is performed using the predicted user action parameter and the past user action parameter to determine a loss used to adjust the weights in the machine learning model. In some embodiments of the invention, the machine learning model may be further trained on the input tokens to predict masked tokens based on the context provided by neighboring tokens. The output of the machine learning model corresponding to the masked tokens is used together with the actual masked tokens to determine a loss used to adjust the weights of the machine learning model. By training on these masked tokens, the predictions generated by the machine learning model may contain information indicative of a semantic relevance parameter indicative of the degree of semantic relevance of the search query to the content of the input digital object. It should be understood that blocks 510 and 512 may be repeatedly executed for all objects from the set of training digital objects or for a subset thereof.

[080] In block 514 of fine-tuning step 574, the processor receives a second set of training digital objects, a training digital object from which is associated with a search query that may contain metadata, with a digital document that may contain metadata, and with a label generated by the evaluator. The label generated by the evaluator indicates the degree of relevance of the training digital object (in particular, the digital document in some embodiments of the invention) to the search query from the point of view of the human evaluator who assigned the label generated by the evaluator.

[081] In block 516, the processor trains a machine learning model to determine a synthesized rater label for the training digital object. The synthesized rater label is a prediction of the machine learning model regarding how relevant the training digital object is to the search query. The training may be performed by providing the machine learning model with a tokenized representation of the training digital object (containing the search query and the document) and using the machine learning model to generate a synthesized rater label. The synthesized rater label and the rater-generated label generated by the human rater are used to determine a loss that may be used to adjust the weights in the machine learning model to fine-tune the machine learning model. It should be understood that block 516 may be repeated for all objects from the second set of training digital objects or for a subset thereof.

[082] In block 518, the machine learning model is further fine-tuned by the processor, applying the machine learning model to all objects from the first set of training digital objects or a subset thereof in order to supplement the first set of training digital objects with the synthesized labels of the evaluator and to form a first supplemented set of training digital objects. In block 520, the machine learning model is fine-tuned using the first supplemented set of training digital objects for training the machine learning model, substantially as described above with reference to block 516.

[083] It should be understood that the fine-tuning step 574 may be repeated in whole or in part with other sets of training digital objects containing the evaluator-generated labels in order to subsequently refine the machine learning model. For example, in some embodiments of the invention, after performing the fine-tuning described above, the processor receives a third set of training digital objects, a training digital object from which is associated with a search query that was used to generate the training digital object and may contain metadata, with a digital document that may contain metadata, and with a label generated by the evaluator. As before, the label generated by the evaluator indicates the degree of relevance of the training digital object (in particular, the digital document in some embodiments of the invention) to the search query from the point of view of the human evaluator who assigned the label generated by the evaluator. This additional set of training digital objects may be different from any other set of digital training objects used in training, as described above, or, for example, may be the same as the second set of training digital objects. Additionally, the size of the third set of training digital features may differ from the size of the other sets of training digital features used to train and/or fine-tune the machine learning model.

[084] The machine learning model is fine-tuned using an additional set of training digital objects to train the machine learning model, substantially as described above with reference to block 516. After this additional training, the model may be used to generate a refined relevance label.

[085] Fig. 6 is a flow chart 600 of using a fully trained machine learning model to rank search results. In block 602, the processor receives a set of use-stage digital objects. Each use-stage digital object is associated with a search query (including metadata) entered by the user and with a digital document (including metadata) provided in response to this query. For example, if the search engine found 75 documents matching the query, then the set of use-stage digital objects will contain 75 use-stage digital objects, each of which will contain the query (including metadata) and one of the documents (including metadata).

[086] In block 604, the processor tokenizes a use-stage digital object from the set of use-stage digital objects and uses the resulting tokens as input to a use-stage machine learning model. The use-stage machine learning model generates a relevance parameter for the use-stage digital object. The relevance parameter is a prediction of the use-stage machine learning model for the relevance of a use-stage digital object (e.g., a document associated with the use-stage digital object) to a query. The use-stage digital object is labeled using the relevance parameter. Block 604 may be repeated for all objects in the set of use-stage digital objects or for a subset thereof to generate a labeled set of use-stage digital objects.

[087] In block 606, the labeled set of digital objects of the use stage is ranked by their relevance parameter. In some embodiments of the invention, this may be performed using another machine learning model that is pre-trained to rank the labeled set of digital objects of the use stage using their relevance parameters as input features. In some embodiments of the invention, this other machine learning model may be a CatBoost decision tree learning model.

[088] It should also be understood that, although the embodiments of the invention presented herein are described with reference to specific features and structures, various modifications and combinations thereof may be implemented without departing from the scope of such technologies. For example, various optimizations applicable to neural networks, including transformers and/or the BERT model, may be similarly applied to the present technology. In addition, optimizations that speed up the determination of relevance at the use stage may also be applied. For example, in some embodiments of the invention, the transformer model may be split such that some transformer blocks are divided between query processing and document processing, so that document representations can be pre-computed offline and stored in a document search index. The description and drawings should be considered only as illustrative of the discussed embodiments or implementations of the invention, the principles of which are defined by the appended claims, which cover any modifications, changes, combinations and equivalents within the scope of the present invention.

Claims (56)

1. A computer-implemented method for training a machine learning model to rank use-stage digital objects generated using a use-stage search query, executed by a processor and comprising: - receiving by the processor a first plurality of training digital objects, wherein each training digital object from the first plurality of training digital objects is associated with a parameter of past user actions indicating user actions of past users with said training digital object; - training, at the first stage of training, based on the first set of training digital objects of the machine learning model, to determine the parameter of predicted user actions for the digital object of the use stage, wherein the parameter of predicted user actions indicates the user actions of future users with the digital object of the use stage; - receiving by the processor a second plurality of training digital objects, wherein each training digital object from the second plurality of training digital objects is associated with (a) a training search query used to form a training digital object from the second plurality of training digital objects, and (b) with a first label indicating the degree of relevance of an object from the second plurality of training digital objects to the training search query; - training at the second stage of training, following the first stage of training, based on the second set of training digital objects of the machine learning model, to determine the synthesized label of the digital object of the use stage, indicating the degree of relevance of the digital object of the use stage to the search query of the use stage; - applying a machine learning model by the processor to the first plurality of training digital objects to supplement an object from the first plurality of training digital objects with a synthesized label and thereby forming a first supplemented plurality of training digital objects; and - training, based on the first augmented set of training digital objects, of a machine learning model to determine the relevance parameter of a digital object of the use stage, indicating the degree of relevance of the digital object of the use stage to the search query of the use stage, wherein the training digital object from the first set of training digital objects contains an indication of a digital document associated with the document metadata, and training of the machine learning model based on the first set of training digital objects at the first training stage additionally includes: - transformation of document metadata into their text representation containing tokens; - pre-processing the text representation to mask multiple masked tokens in it; and - training, based on the first set of training digital objects, a machine learning model to determine a token from several masked tokens based on the context provided by neighboring tokens, wherein the relevance parameter of the digital object of the use stage additionally indicates the semantic relevance parameter indicating the degree of semantic relevance of the search query of the use stage to the content of the digital object of the use stage. 2. The method according to claim 1, characterized in that the document metadata contains at least one of the following: a training search query associated with an object from the first plurality of training digital objects, a title of the digital document, content of the digital document, and a web address associated with the digital document. 3. The method according to claim 1, characterized in that it further includes determining a parameter of past user actions associated with an object from the first plurality of training digital objects, based on data on clicks of past users. 4. The method according to claim 3, characterized in that the click data comprises data about at least one click of at least one past user made in response to sending a training search query associated with an object from the first plurality of training digital objects. 5. The method according to paragraph 1, characterized in that before training the machine learning model to determine the relevance parameter of the digital object of the use stage, it additionally includes: - receiving by the processor a third plurality of training digital objects, wherein the object from the third plurality of training digital objects is associated (a) with a training search query used to form the object from the third plurality of training digital objects, and (b) with a second label indicating the degree of relevance of the object from the third plurality of training digital objects to the training search query; - training at the third stage of training, following the second stage of training, based on the third set of training digital objects of the machine learning model, to determine a refined synthesized label of the digital object of the use stage, indicating the degree of relevance of the digital object of the use stage to the search query of the use stage; - applying a machine learning model by the processor to the first augmented set of training digital objects to augment an object from the first augmented set of training digital objects with a refined synthesized label and thereby forming a second augmented set of training digital objects; and - training a machine learning model to determine the relevance parameter of a digital object of the use stage based on the second augmented set of training digital objects. 6. The method according to claim 5, characterized in that the set of the first set of training digital objects, the second set of training digital objects and the third set of training digital objects is at least partially different from any other set of the first set of training digital objects, the second set of training digital objects and the third set of training digital objects. 7. The method according to claim 5, characterized in that the set of the first set of training digital objects, the second set of training digital objects, and the third set of training digital objects is larger in size than the subsequent set of the first set of training digital objects, the second set of training digital objects, and the third set of training digital objects. 8. The method according to paragraph 1, characterized in that after training the machine learning model to determine the relevance parameter of the digital object of the use stage, it additionally includes: - receiving by the processor a third plurality of training digital objects, wherein the object from the third plurality of training digital objects is associated (a) with a training search query used to form the object from the third plurality of training digital objects, and (b) with a second label indicating the degree of relevance of the object from the third plurality of training digital objects to the training search query; - training, based on the third set of training digital objects, a machine learning model to determine the corresponding refined parameter of relevance of the digital object of the use stage, indicating the degree of relevance of the digital object of the use stage to the search query of the use stage. 9. The method according to claim 8, characterized in that the set of the first set of training digital objects, the second set of training digital objects and the third set of training digital objects is at least partially different from any other set of the first set of training digital objects, the second set of training digital objects and the third set of training digital objects. 10. The method according to claim 8, characterized in that the set of the first set of training digital objects, the second set of training digital objects, and the third set of training digital objects is larger in size than the subsequent set of the first set of training digital objects, the second set of training digital objects, and the third set of training digital objects. 11. The method according to claim 8, characterized in that the third set of training objects is identical to the second set of training digital objects. 12. The method according to paragraph 1, characterized in that at the first stage of training, the machine learning model is trained to determine a rough initial estimate of the relevance parameter of the digital object of the use stage, and at each subsequent stage of training, the machine learning model is trained with the aim of improving the rough initial estimate. 13. The method according to claim 10, characterized in that the improvement of the rough initial estimate is determined using a metric based on a normalized discounted cumulative indicator. 14. The method according to claim 1, characterized in that the machine learning model contains at least one learning model. 15. The method according to claim 14, characterized in that at least one learning model is a learning model based on a transformer. 16. The method according to claim 1, characterized in that the machine learning model contains at least two learning models, wherein the first model of the two learning models is trained to determine a synthesized label for a digital object of the use stage for the purpose of forming a first augmented set of training digital objects, and the second model of the two learning models is trained to determine the relevance parameter of a digital object of the use stage based on the first augmented set of training digital objects. 17. The method according to paragraph 16, characterized in that the first model of the two training models differs from the second model. 18. The method according to claim 16, characterized in that the first model of the two training models is a transformer-based training model. 19. The method according to paragraph 1, characterized in that it additionally includes ranking digital objects of the use stage according to the relevance parameters associated with them. 20. The method according to claim 1, characterized in that it additionally includes ranking digital objects of the use stage based on the relevance parameters associated with them, including the use of another learning model trained to rank digital objects of the use stage using the relevance parameters generated by the machine learning model as input features. 21. The method according to claim 20, characterized in that the other learning model is a learning model based on CatBoost decision trees. 22. A system for training a machine learning model to rank digital objects of the use stage, generated using a search query of the use stage, comprising: - processor; - memory associated with the processor; and - a machine learning training module located in memory, executed by a processor and containing commands, whereby the processor, when executing these commands, is capable of performing the following actions: - obtaining a first plurality of training digital objects, wherein the training digital object from the first plurality of training digital objects is associated with a parameter of past user actions indicating user actions of past users with the training digital object from the first plurality of training digital objects; - training, at the first stage of training, based on the first set of training digital objects of the machine learning model, to determine the parameter of predicted user actions for the digital object of the use stage, wherein the parameter of predicted user actions indicates the user actions of future users with the digital object of the use stage; - obtaining a second set of training digital objects, wherein the training digital object from the second set of training digital objects is associated with (a) a training search query used to form an object from the second set of training digital objects, and (b) with a first label indicating the degree of relevance of the object from the second set of training digital objects to the training search query; - training at the second stage of training, following the first stage of training, based on the second set of training digital objects of the machine learning model, to determine the synthesized label of the digital object of the use stage, indicating the degree of relevance of the digital object of the use stage to the search query of the use stage; - applying a machine learning model to a first set of training digital objects to supplement an object from the first set of training digital objects with a synthesized label and thereby forming a first supplemented set of training digital objects; and - training, based on the first augmented set of training digital objects, a machine learning model to determine the relevance parameter of a digital object of the use stage, indicating the degree of relevance of the digital object of the use stage to the search query of the use stage, wherein the training digital object from the first set of training digital objects contains an indication of a digital document associated with the document metadata, the training machine learning module contains additional commands, and the processor, when executing these commands, is capable of training the machine learning model at the first training stage based on the first set of training digital objects by: - converting document metadata into its text representation containing tokens; - pre-processing the text representation to mask multiple masked tokens in it; and - training, based on a first set of training digital objects, a machine learning model to determine a token from several masked tokens based on the context provided by neighboring tokens, wherein the relevance parameter of the digital object of the use stage additionally indicates the semantic relevance parameter indicating the degree of semantic relevance of the search query of the use stage to the content of the digital object of the use stage. 23. The system according to item 22, characterized in that the machine learning training module contains additional commands, and the processor, when executing these commands, is capable of performing the following actions before training the machine learning model to determine the relevance parameter of the digital object of the use stage: - obtaining a third set of training digital objects, wherein an object from the third set of training digital objects is associated (a) with a training search query used to form an object from the third set of training digital objects, and (b) with a second label indicating the degree of relevance of an object from the third set of training digital objects to the training search query; - training at the third stage of training, following the second stage of training, based on the third set of training digital objects of the machine learning model, to determine a refined synthesized label of the digital object of the use stage, indicating the degree of relevance of the digital object of the use stage to the search query of the use stage; - applying a machine learning model to a first augmented set of training digital objects to augment an object from the first augmented set of training digital objects with a refined synthesized label and thereby forming a second augmented set of training digital objects; and - training a machine learning model to determine the relevance parameter of a digital object of the use stage based on the second augmented set of training digital objects.

Images