RU2538303C1 - Method for automatic semantic comparison of natural language texts - Google Patents

Abstract

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to information technology. The disclosed method includes presenting two texts to be compared in digital form for subsequent processing; indexing the texts to obtain elementary units of first to fifth levels; detecting the frequency of occurrence of elementary units of the fourth level, each being a semantically significant object or attribute, and the frequency of occurrence of semantically significant relationships linking semantically significant objects, as well as the semantically significant objects and attributes; storing the formed elementary units of the second to fifth levels, and the obtained indices together with links to specific sentences of said text; forming from a triad, which are elementary units of the fifth level, a semantic network; ranking the elementary units of the fourth level according to semantic weight by comparing the semantic weight of each of them with a predetermined threshold and removing elementary units of the fourth level having a semantic weight below the threshold; detecting for two compared texts the degree of crossing of their semantic networks.

EFFECT: faster process of comparing texts.

4 cl, 2 dwg, 26 tbl

Description

FIELD OF THE INVENTION

The present invention relates to the field of information technology, and in particular to a method for automated semantic comparison of texts in natural language.

State of the art

There are various ways of automated semantic (i.e. semantic) comparison of texts in natural languages.

The semantic comparison of texts in a natural language cannot be carried out “head-on”, since in this case it is not necessary to compare specific words, but the meaning behind the whole sentences and even paragraphs or sections. Therefore, usually semantic comparison of texts is preceded by semantic indexing of these texts, which is carried out in various ways. Moreover, the elimination of the semantic ambiguity of these texts is important.

Such methods of semantic indexing of texts for their subsequent comparison with the elimination of semantic ambiguity are described, for example, in RF patent No. 2242048 (publ. 10.12.2004), in US patents No. 6871199 (publ. 22.03.2005), 7024407 (publ. 04.04. 2006) and 7383169 (publ. 06/03/2008), in applications for US patent No. 2007/0005343 and 2007/0005344 (both publ. 04.01.2007), 2008/0097951 (publ. 24.04.2008), in Japanese applications laid out No. 05-128149 (publ. 05/25/1993), 06-195374 (publ. 07/15/1994), 10-171806 (publ. 06/26/1998) and 2005-182438 (publ. 07/07/2005), in EPO application no. 0853286 (published on July 15, 1998).

Closest to the claimed invention can be considered a method of automated semantic indexing of text in a natural language, disclosed in the patent of the Russian Federation No. 2399959 (publ. 09/20/2010). In this method, text in digital form is segmented into elementary units of the first level (words); form for each elementary unit of the first level (words) an elementary unit of the second level (normalized word form); segment text in digital form into sentences corresponding to sections of indexed text; reveal in the text in the process of linguistic analysis elementary units of the third level (stable phrases); in the process of multi-stage semantic-syntactic analysis by referring to linguistic and heuristic rules preformed in the database in a predefined linguistic environment, elementary units of the fourth level (semantically significant object and its attribute) and semantically significant relationships between the identified semantically significant are revealed in each of the generated sentences objects, as well as between semantically significant objects and attributes; form within this text for each of the identified semantically significant relationships, a set of elementary units of the fifth level (triads); index on a set of formed triads all semantically significant objects connected by semantically significant relations, as well as individually attributes and all triads of the form “semantically significant object - semantically significant relation - semantically significant object”, as well as all triads of the form “semantically significant object - semantically significant relation - attribute ”; save the generated triads and the resulting indexes in a database together with a link to the source text from which these triads are formed.

The disadvantage of this method is the lack of ranking of the formed elementary units of the fourth level in terms of their relevance to the text, which leads to an unreasonably large amount of calculations associated with the need to use the entire generated index for further processing.

Disclosure of invention

The purpose of the present invention is to expand the arsenal of methods for semantic comparison of texts in natural languages by accelerating the process of comparing texts.

Achieving this goal and obtaining the indicated technical result is provided in the present invention by means of an automated semantic comparison of texts in natural language, which consists in the following: represent two compared texts in digital form for subsequent automatic and (or) automated processing; indexing these texts in digital form, obtaining: elementary units of the first level, including at least words, elementary units of the second level, each of which is a normalized word form, elementary units of the third level, each of which is a stable phrase in the mentioned text, elementary units of the fourth level, each of which is a semantically significant object and attribute, and elementary units of the fifth level, each of which constitutes a triad of either two semantically significant objects and a semantically significant relationship between them, or from a semantically significant object and attribute and the semantically meaningful relationship that connects them; identify the frequency of occurrence of elementary units of the fourth level and the frequency of occurrence of the mentioned semantically significant relationships; save in the database the formed elementary units of the second, third, fourth and fifth levels with the detected frequencies of occurrence of elementary units of the fourth level and semantically significant relations, as well as the resulting indices, together with links to specific sentences of this text; form a semantic network of triads in such a way that the first elementary unit of the fourth level of the subsequent triad is associated with the same second elementary unit of the fourth level of the previous triad; during the iterative procedure, renormalization of the frequencies of occurrence into the semantic weight of the elementary units of the fourth level, which are the vertices of the semantic network, is carried out in such a way that the elementary units of the fourth level connected in the network with a large number of elementary units of the fourth level with a high frequency of occurrence increase their semantic weight, and other elementary units of the fourth level evenly lose it; rank elementary units of the fourth level by semantic weight by comparing their semantic weight with a predetermined threshold value; remove elementary units of the fourth level having a semantic weight below a threshold value; save in memory the remaining elementary units of the fourth level with a weight above the threshold, as well as semantically significant relations between the remaining elementary units of the fourth level; for the two texts being compared, the degree of intersection of their semantic networks both over the vertices and the connections between these vertices is taken into account, taking into account the semantic weights of the vertices of the semantic networks and the weight characteristics of their connections, and the degree of intersection of the semantic networks of the two compared texts is a value characterizing the semantic similarity of these texts .

A feature of the method of the present invention is that indexing is carried out in the process of performing the following steps: segment the text in digital form into elementary units of the first level, including at least words; Segment digitally into sentences according to graphematic rules; form for each elementary unit of the first level, which is a word, based on morphological analysis, elementary units of the second level, including a normalized word form; calculate the frequency of occurrence of each elementary unit of the first level for two or more adjacent units of the first level in this text and combine among the elementary units of the first level the sequence of words following each other in this text into elementary units of the third level, which are stable combinations of words, if for every two or more consecutive words in a given text the difference in the calculated frequencies of occurrence of these words for the first occurrence of a given sequence of words s and for several subsequent occurrences for each pair of words, the sequences remain unchanged; identify in the process of multi-stage semantic-syntactic analysis by referring to linguistic and heuristic rules pre-generated in the database in a predefined linguistic environment in each of the generated sentences, semantically significant objects and attributes - elementary units of the fourth level; for each elementary unit of the fourth level, the identity is fixed by reference between the corresponding semantically significant object, as well as the attribute and the corresponding anaphoric link, if any, in the indexed text, replacing each anaphoric link with the corresponding antecedent; store in memory every semantically significant object and attribute; in the process of multistage semantic-syntactic analysis, they identify the semantically significant relations between the identified units of the fourth level - semantically significant objects, as well as between semantically significant objects and semantically meaningful objects and attributes assign to each semantically significant relation the corresponding type from the subject ontology stored in the database on the subject matter of the subject area to which the indexed text belongs; identify throughout the text the frequency of occurrence of elementary units of the fourth level and the frequency of occurrence of the mentioned semantically significant relationships; store in memory each identified semantically significant relation together with the type assigned to it; form within the given text for each of the identified semantically significant relations, linking both the corresponding semantically significant objects and the semantically significant object and its attribute, a multitude of triads that are elementary units of the fifth level; index on a set of formed triads individually all associated semantically significant relationships semantically significant objects with their frequencies of occurrence, all attributes with their frequencies of occurrence and all formed triads.

Another feature of the method of the present invention is that the degree of intersection of two semantic networks belonging to the two texts being compared is calculated as the sum of the coincidences of elementary units of the fifth level of these two semantic networks.

At the same time, the stages are carried out in which: one of the two semantic networks is selected as the basic network, in which, after ranking and removing vertices with semantic weights, more vertices remain below the threshold value than in the other, which is chosen as the comparison one; find for each vertex of the core network in the compared network a vertex that is the same elementary unit of the fourth level, i.e. the same semantically significant object or the same attribute; calculating for each vertex found in each of the base and compared networks the values of all triads connected with a given vertex as the area of triangles whose sides correspond to the components of each of these triads, and the angle between the sides is proportional to the weight of the semantically significant relationship of this triad; choose for each pair of triads associated with a pair of specific vertices in the base and the compared networks, the smaller of the calculated values as the degree of intersection of the triads in the base and the compared networks; summarize for each of the vertices associated with a given vertex, all selected calculated values, obtaining the degree of intersection for a given pair of vertices of the base and compared networks; normalize the found amount to the number of semantically significant objects and attributes associated with a given vertex in that of the base and compared networks, which contains more vertices associated with this vertex; summarize the normalized sums for all the vertices of that of the base and compared networks, which contains more vertices; normalize the amount received to the number of elementary units of the fourth level remaining in this network, obtaining the degree of intersection of the semantic networks of the texts being compared.

Brief Description of the Drawings

The present invention is further explained by the description of a specific example of its implementation and the accompanying drawings.

Figure 1 shows the conditional block diagram explaining the claimed method.

2 is a flowchart illustrating a preferred method for indexing text.

DETAILED DESCRIPTION OF THE INVENTION

The method of the present invention can be implemented in almost any computing environment, for example, on a personal computer connected to external databases. The steps of the method are illustrated in FIG.

All further explanations are given as applied to the Russian language, which is one of the most highly inflected languages, although the proposed method is applicable to the semantic comparison of texts in any natural languages.

First of all, each of the texts subject to semantic comparison must be submitted in electronic form for subsequent automated processing. This step in Fig. 1 is conventionally indicated by the reference number 1 and can be performed in any known manner, for example, by scanning the text and then recognizing it using well-known means such as ABBYY FineReader. If the text is sent for indexing from an electronic network, for example, from the Internet, then the stage of its submission in electronic form is performed in advance, before this text is posted on the network.

Professionals should be clear that the operations of this and subsequent steps are carried out with storing intermediate results, for example, in random access memory (RAM).

The text converted into electronic form is sent for processing, during which indexation is carried out. This indexing (step 2 in FIG. 1) can be done in the same way as it is disclosed, for example, in the aforementioned patent of the Russian Federation No. 2399959 or in the application for US patent No. 2007/0073533 (published on March 29, 2007). In the process of this indexing, elementary text units of different levels are obtained. Elementary units of the first level include at least words; each of the elementary units of the second level is a normalized word form; each of the elementary units of the third level is a sequence of successive words in the processed text; each of the fourth-level elementary units is a semantically significant object or attribute; each of the elementary units of the fifth level is a triad of either two semantically significant objects and a semantically significant relationship between them, or a semantically significant object and its attribute and a semantically significant relation connecting them.

It is preferable, however, to index the text using the method claimed in the patent application of the Russian Federation No. 20112150734 (priority from 11.27.2012) and illustrated in FIG. 2. In this method, the text in digital form is first segmented into elementary units of the first level, including at least words. In the mentioned patent of the Russian Federation No. 2399959, these elementary units of the first level are called tokens. A token can be any text object from the following set: words consisting of each sequence of letters and, possibly, hyphens; sequence of spaces; punctuation marks; numbers. Sometimes sequences of symbols such as A300, i150b, etc. are also included here. Tokens are always allocated according to fairly simple rules, for example, as in the aforementioned RF patent No. 2399959. In Fig.2, this step is conventionally indicated by reference numeral 21.

Following this, at step 22 (FIG. 2), the indexed text is digitally segmented into sentences corresponding to sections of the given text. Such segmentation is carried out according to graphematical rules. For example, the simplest rule for highlighting sentences is: “A sentence is a sequence of tokens, starting with a capital letter and ending with a period”.

Further, for each elementary unit of the first level (for each token), which is a word, on the basis of morphological analysis, the corresponding elementary unit of the second level is formed, which is a normalized word form, hereinafter referred to as the lemma. For example, for the word “go” the normalized word form will be “go”, for the word “beautiful” the normalized word form will be “beautiful”, and for the word “wall” the normalized word form will be “wall”. In addition, for each word form, the part of speech to which the given word belongs and its morphological characteristics are indicated. Naturally, for different parts of speech, these characteristics are different. For example, for nouns and adjectives it is a gender (masculine - feminine - average), number (singular - plural), case; for verbs it is a form (perfect - imperfect), person, number (singular - plural); etc. Thus, for a given word, its normalized word form (lemma) + morphological characteristics, including part of speech, are its morph. One and the same word can have several morphs. For example, the word "glass" has two morphs - one for a noun of the middle gender and one for a past tense verb. This step is conventionally indicated in figure 2 by reference numeral 23.

The next step, conventionally indicated in FIG. 2 by reference numeral 24, is that for each of the mentioned first level elementary units, the frequency of occurrence is counted in said text. In other words, they determine how many times each word occurs in the processed text. This operation is carried out automatically, for example, by simply calculating the frequency of occurrence of each token, either as described in the patent of the Russian Federation No. 2164450 (publ. 05/20/2001), or in US patent No. 6189002 (publ. 13.02.2001). Simultaneously with the calculation of the frequency of occurrence, for every two or more words that follow in a given text, the differences in the calculated frequencies of occurrence of these words are found in the first occurrence of this sequence of words and in their subsequent occurrences. If these differences for the first occurrence of a given sequence of words and for several subsequent occurrences of them remain unchanged, such a sequence of words following each other in this text (i.e., elementary units of the second level) is combined into elementary units of the third level, which are stable collocations.

Further, in the next step, indicated in FIG. 2 by the reference numeral 25, in order to identify semantically significant objects and attributes, a multi-stage semantic-syntactic analysis is performed. Such multistage semantic-syntactic analysis is performed by referring to the linguistic and heuristic rules generated in the database in a predetermined linguistic environment. Such a medium may be, for example, the linguistic medium mentioned in the aforementioned application for US patent No. 2007/0073533 or in the above patents of the Russian Federation No. 2242048 and the Russian Federation No. 2399959, or any other linguistic medium that defines the appropriate rules that allow to eliminate syntactic and semantic ambiguities words and expressions of real text. Linguistic and heuristic rules in the selected environment are referred to below as rules.

The identification of semantically significant objects and attributes, which are considered elementary units of the fourth level, is performed in the sentence on the set of elementary units of the first, second and (or) third levels.

For each semantically significant object or attribute, i.e. elementary units of the fourth level with the types assigned to them, find the corresponding anaphoric link (if any). For example, in the sentence “Mechanics is a part of physics that studies the laws of mechanical motion and the causes that cause or change this movement”, the anaphoric reference to the word “mechanics” will be the pronoun “which”, while the word “mechanics” will be an antecedent for this anaphora, and also, the anaphoric reference to the word “mechanical” will be the pronoun “this,” while the word “mechanical” will be an antecedent for this anaphora. This step of finding the anaphoric link is conventionally indicated in FIG. 2 by reference numeral 26. Each anaphoric link is replaced with its corresponding antecedent. After that, each identified semantically significant object and attribute is stored in the corresponding memory.

In the next step, indicated by a reference numeral 27 in FIG. 2, a multi-stage semantic-syntactic analysis is performed, with which, on the basis of elementary units of the first, second, third and fourth levels, semantically significant relations between semantically significant objects are found using the above rules, and between semantically significant objects and attributes.

At the step indicated by reference numeral 28 in FIG. 2, each semantically significant relation is assigned a corresponding type from the subject ontology stored in the database on the subject of the subject area to which the indexed text belongs. After that, each semantically significant relation is stored in the corresponding memory together with the type assigned to it and the morphological and semantic attributes found for it.

After that, at the stage indicated by reference numeral 29 in FIG. 2, the occurrence frequencies of semantically significant objects and attributes, as well as the occurrence frequencies of semantically significant relationships between semantically significant objects and between semantically significant objects and attributes throughout the entire text, are detected. This operation is performed in almost the same way as in step 24 for elementary units of the first level.

In the step indicated by reference numeral 30 in FIG. 2, stored semantically meaningful objects, as well as attributes and semantically meaningful relationships, are used to form triads. Moreover, within the limits of the text being indexed, for each of the identified semantically meaningful relationships that connect certain semantically meaningful objects and attributes, many triads of two types are formed. Each of the many triads of the first type includes a semantically meaningful relationship and two semantically meaningful objects that are linked by this semantically meaningful relationship. Each of the many triads of the second type includes a semantically significant relation, one semantically significant object, as well as its attribute, which are associated with this semantically significant relation. If two semantically significant objects are denoted by O _i and O _j , and the semantically significant relation connecting them is denoted by R _ij , then each of the triads of the first type can be conditionally represented (depicted) as O _i → R _ij → O _j . Each of the triads of the second type can be represented as O _i → R _im → A _m , where A _m are the corresponding attribute, and R _im associates the semantically significant object and the attribute is a semantically significant relation. In these entries, the indices i, j, m are integers.

Then, in the step indicated by reference numeral 31 in FIG. 2, the text is indexed. At the same time, individually associated with semantically significant relations, all semantically significant objects with their frequencies of occurrence, all attributes with their frequencies of occurrence, and all formed triads are indexed individually on the set of formed triads.

For this, on the set of triads formed, all semantically significant objects and their attributes are separately indexed with their occurrence frequencies and all triads of the form “semantically significant object - semantically significant relation - semantically significant object”, as well as all triads of the form “semantically significant object - semantically significant attitude is an attribute. ” The indices generated in step 30 of the triad and the indices obtained in step 31, together with a link to specific sentences of the source text from which these triads are generated, are stored in the database (step 32 in FIG. 2).

For specialists, it is obvious that the storage devices mentioned at separate stages can in fact be both different devices, and one storage device of sufficient volume. Similarly, the individual databases mentioned at the respective stages can be not only physically separate databases, but also the only database. Moreover, said storage devices (memories) may store the same single database, or store said databases separately. Those skilled in the art will also understand that the methods claimed in the present invention are executed in an appropriate computing environment under the control of appropriate programs that are recorded on computer-readable media intended for direct participation in a computer.

Returning to the block diagram of FIG. 1. At stage 3, the frequencies of occurrence of elementary units of the fourth level (i.e., semantically significant objects and attributes) are identified, as well as the frequencies of occurrence of semantically significant relationships are identified. Note that the formed elementary units of the fourth level are stored in the database along with the identified frequency of occurrence. In addition, the resulting indexes are stored in the database along with links to specific sentences of the text.

Then, in step 4, a semantic network is formed in the method of the present invention such that the first semantically significant object of the subsequent triad is associated with the same second semantically significant object of the previous triad. At the same time, in the course of the iterative procedure, the frequency of occurrence of semantically significant objects and attributes is renormalized into the semantic weight of semantically significant objects and attributes, which are the vertices of the semantic network. This renormalization is carried out in such a way that semantically significant objects and attributes connected in a network with a large number of semantically significant objects and attributes with a high frequency of occurrence increase their semantic weight, while other semantically significant objects and attributes evenly lose it (step 5 in FIG. one).

Next, the elementary units of the fourth level are ranked by semantic weight by comparing their semantic weight with a predetermined threshold value (step 6 in FIG. 1).

The elementary units of the fourth level with a semantic weight below the threshold are removed (step 7 in FIG. 1). The remaining elementary units of the fourth level with a weight above the threshold are stored in memory (step 8). The semantically significant relations between semantically significant objects and also between semantically significant objects and attributes remaining in the semantic network are also stored in memory.

Next, at step 9, the degree of intersection of the constructed semantic networks of the two texts to be compared both in the vertices and in the links is determined, taking into account the semantic weights of the vertices of the semantic networks and the frequency of occurrence of semantically significant relationships connecting them. This degree of intersection of the semantic networks of the texts being compared is a characteristic of the semantic similarity of these texts.

The degree of intersection of two semantic networks, formed as described above, belonging to the two texts being compared, is calculated as the sum of the coincidences of elementary units of the fifth level of these two semantic networks. In principle, this calculation can be carried out by various methods known to those skilled in the art.

и $${\overline{c}}_j$$

, где вектор $${\overline{c}}_i$$

соответствует первому семантически значимому объекту или атрибуту элементарной единицы пятого уровня, вектор $${\overline{c}}_j$$

соответствует второму семантически значимому объекту либо атрибуту элементарной единицы пятого уровня, а угол между векторами c_i, и c_j, равный w_ij, пропорционален частоте встречаемости семантически значимого отношения между первым и вторым семантически значимыми объектами или между первым семантически значимым объектом и атрибутом, нормированной на 90°: w_ij∈(0…90°).Preferably, the degree of intersection can be calculated as the sum of the intersections of the fifth level elementary units of these two networks. For this, one of the two semantic networks is selected as the basic network, in which, after ranking and deleting vertices with semantic weights below the threshold value (see step 7 in FIG. 1), more vertices remain than in the other, which is chosen as the comparison. For each vertex of the core network, a vertex is found in the compared network, which is the same elementary unit of the fourth level, i.e. the same semantically significant object or the same attribute. For each vertex found in each of the base and compared networks, the values of all triads connected with this vertex are calculated as the area of triangles whose sides correspond to the components of each of these triads. This area calculation can be performed as a 100% normalized scalar product on vectors $${\overline{c}}_i$$

and $${\overline{c}}_j$$

where is the vector $${\overline{c}}_i$$

corresponds to the first semantically significant object or attribute of an elementary unit of the fifth level, the vector $${\overline{c}}_j$$

corresponds to the second semantically significant object or attribute of an elementary unit of the fifth level, and the angle between the vectors c _i and c _j equal to w _ij is proportional to the frequency of occurrence of the semantically significant relationship between the first and second semantically significant objects or between the first semantically significant object and the attribute normalized 90 °: w _ij ∈ (0 ... 90 °).

Next, choose for each pair of triads associated with a pair of specific vertices in the base and the compared networks, the smaller of the calculated values as the degree of intersection of the triads in the base and the compared networks. All selected calculated values are summed for each of the vertices, obtaining the degree of intersection for a given pair of vertices of the base and compared networks. The found amount is normalized to the number of semantically significant objects and attributes associated with a given vertex in that of the base and compared networks, which contains more vertices. The obtained normalized sums are now summed over all the vertices of that of the base and compared networks, which contains more vertices. Finally, the resulting total amount is normalized to the number of elementary units of the fourth level remaining in this network, i.e. semantically significant objects and attributes, getting the degree of intersection of semantic networks for two compared texts.

Obviously, if there is no vertex in the compared network, the degree of intersection for this vertex is taken equal to zero.

Example

To illustrate the implementation of the claimed method of automated semantic comparison of text in a natural language, consider the following example. Let there be some Russian-language text about the rules for admission to the Nizhny Novgorod State University. N.I. Lobachevsky in 2005, presented on the website http://www.unn.ru/rus/priem.htm, and several (for example, two) other texts from the same and from other subject areas. Thus, we can assume that the conversion of texts into electronic form, indicated in FIG. 1 by reference numeral 1, has already been completed.

A typical example of such text is the following snippet:

“At the NNSU, citizens of the Russian Federation, the Republic of Belarus, the Republic of Kazakhstan, the Kyrgyz Republic and the Republic of Tajikistan who have a document on the formation of state education are admitted to the first year to study in the main educational programs of higher professional education in places financed from the federal budget sample (certificate of secondary (complete) general education; diploma of secondary vocational education; diploma of primary vocational education, if It has a record of receiving secondary (full) general education), if the formation of the first level is obtained. "

In accordance with the claimed method of automated semantic comparison of texts in a natural language, a previously created base of syntactic rules and dictionaries is used, within which text processing and construction of a semantic index will be carried out. Such databases are prepared by linguistic experts, who, based on their experience and knowledge, determine the sequence and composition of the syntactic processing of the text specific to a particular language.

Linguistic experts preliminarily construct many syntactic rules that allow using the corresponding linguistic dictionaries also previously constructed by expert linguists to automatically identify specific information in processed texts corresponding to semantically significant objects, attributes of semantically significant objects, and semantically significant relationships that may have space between semantically significant objects or between semantically significant objects Tami and attributes.

In addition to the specification of the subject area and the rules in accordance with the above methods, dictionaries of general and special vocabulary are used.

In accordance with the claimed method of automated semantic comparison of texts in natural language, the text is first segmented into elementary units — tokens (reference position 21 in FIG. 2) and morphological analysis of word tokens (reference position 23 in FIG. 2). As a result of this stage, the source text is transformed into many tokens and morphs, which are presented in Table 1 and Table 2, respectively.

Introductory words and plug-in constructions do not carry any syntactical load, therefore tokens of this type are excluded from further analysis.

Geographic tokens are considered as one word with a morph corresponding to the morph of the main word.

Further, after segmenting the text into tokens and morphological analysis of the word tokens, stable phrases are extracted (reference position 24 in FIG. 2). To do this, calculate the frequency of occurrence of words in sequences of two or more words in the text. Then, the differences in the frequencies of occurrence of words in the sequence are compared for the first appearance of a given sequence of words and for several subsequent occurrences of them.

The frequency of occurrence of words with a nerve occurrence of a sequence and with its subsequent occurrence, as well as the difference of these frequencies are presented in Table 3.

As a result of this ethane, the source text, in addition to elementary units of the first and second levels, is supplemented by a multitude of units of the third level - stable phrases. The phrases for our example are presented in Table 4.

After the above steps are completed, the processed text is fragmented into sentences (reference position 22 in FIG. 2). As a result of this step, the sets formed above are supplemented by the set of sentences presented in Table 5.

Thus, after all the above steps are completed, the processed text will be segmented into sentences, each of which is marked with sets of annotations of elementary units of the first, second and third levels.

Following this, in accordance with the claimed method of automated semantic comparison of texts in natural language, semantically significant objects and attributes (elementary units of the fourth level) are identified (reference position 25 in FIG. 2). It is made in each sentence on the set of elementary units of the first, second and (or) third levels by applying a pre-formed set of linguistic and heuristic rules using the pre-formed corresponding linguistic dictionaries.

Semantic-syntactic processing of sentences is carried out in several stages. All stages will be carried out on the text selected by the authors as an example.

1. Subdivision of sentences on punctuation marks and unions (union words and phrases) into initial fragments and determining the type of fragment based on its morphological characteristics. For this, a dictionary of unions, union words and phrases is used.

Borders of fragments are set for all punctuation marks and unions (union words and phrases) without a comma. In addition, the dictionary of unions determines whether there is such a complex union, the beginning of which is in the fragment to the left of the left, and the end is in this. In our case, such an allied phrase is “as long as”. If there is such a union, then a comma is transferred before the whole union.

The fragment type is one of the following values indicated in table 6. In the order indicated in table 6, the word form with the corresponding homonym is searched in the fragment, the remaining homonyms of the found word form are not considered.

2. Combining the original segments with simple cases of homogeneous series of adjectives, adverbs, nouns, etc. A sign of homogeneity is the presence of a compositional union (or comma), before and after which there should be word forms of one part of speech, which have homonyms that have the same morphological information. The remaining homonyms are not considered during further analysis, thus, partial removal of homonymy occurs.

In our example, fragments 6 and 7 are connected by the creative union “and”, since the tokens 38 (“Kyrgyz Republic”) and 41 (“Republic of Tajikistan”) of table 1 have homonyms of one part of speech that have the same morphological information - Rod.p. J.R. Unit Noun Inanimate. In addition, commas stand between fragments 3-6, the tokens 30, 32, 36, 38 of the table 1 included in them have homonyms for one part of speech that have the same morphological information - Rod. J.R. Unit Noun Inanimate. Therefore, we merge fragments 3-7 into one. The type of fragment obtained is 9 (Table 8).

3. Construction of simple syntactic groups corresponding to the attribute level of description (Table 9): attribute of an object / subject / action + object / subject / action, measure of an attribute of an object / subject / action + object / subject / action.

Further in sentences of the text anaphoric links are revealed and revealed. To do this, within the entire text being processed, during the stage indicated in FIG. 2 by the reference numeral 26, pronouns are found that can be anaphoric references to the corresponding words, and for pronouns that really are, they fix the identity by reference between the corresponding semantically significant object and its anaphoric reference. In our example, there are no anaphora.

4. Attachment of contact-located fragments (participial, participial revolutions, accessory definitive, etc.) and establishing a hierarchy on fragments. The participial turnover and the adjunctive definitive will be a sign of the corresponding object, the participial turnover - a sign of action.

In our example, fragments 2 and 4 (Table 8) with type 5 “financed from the federal budget” and “having a document on the formation of a state sample” are involved sentences with the main words “funded” and “having” respectively, consistent with the syntactic nouns “places” and “citizens” of the previous fragments by gender and number, therefore, the entire fragment 2 obeys the noun “places”, and fragment 4 - the noun “citizens”, being their characteristic description. Thus, the entire fragment 2 is the attribute (attribute) of the noun “place”, and fragment 4 is the attribute of the noun “citizens”.

The second column of table 11 shows the enlarged fragments of the proposal obtained after combining and investing.

5. The construction of many unique morphological interpretations of each fragment.

Within each sentence, partial removal of homonymy is carried out at the morphological level by:

1) the allocation of groups of nouns consistent with one or more adjectives / participles / pronouns-adjectives that are in homogeneous connection (the so-called attributive level described above in clause 3);

2) analysis of the location of the dash, which removes homonymy, firstly, from the word form “this”, since the dash before this wordform indicates that “this” is a particle, and secondly, from nouns before and after the dash, because . for the closest noun to the dash, only the nominative case is possible on the right, and the nominative or instrumental on the left;

3) the identification of participles following the noun, since such a turn is distinguished by commas, and the nouns included in it depend on the participle and cannot be in the nominative case. So, in our example, the word form “document” (token 44, Table 2) can only be in the accusative case.

4) identifying prepositions, while those homonyms that have a case not used with this preposition are removed from the subordinate of the preposition of the noun (the preposition management model is used). In our example:

- the preposition “on” (token 3, Table 1) before the word form “course” (token 5, Table 1) cannot manage a noun in the nominative case;

- the preposition “for” (token 6, Table 1) before the word form “training” (token 7, Table 1) cannot control a noun in the nominative or accusative case;

- the preposition “on” (token 15, Table 1) before the word form “places” (token 16, Table 1) cannot manage a noun in the nominative or genitive case;

- the preposition “for” (token 19, table 1) before the word form “account” (token 20, table 1) cannot manage a noun in the nominative case;

- the preposition “on” (token 26, table 1) before the word form “basis” (token 28, table 1) cannot control a noun in the dative case,

therefore, these homonyms are removed from consideration.

In table 2, variants of homonyms that are excluded from consideration as a result of partial removal of homonymy at the morphological level are highlighted in gray.

6. Combining fragments into simple sentences as part of a complex subject using subordinate unions. Subordinate unions act as boundaries of simple sentences (Table 11, column 3).

7. Identification of the predictive minimum (including the main semantically significant objects and the main semantically significant relationships - predicates) of a sentence by comparing its structure with a dictionary of templates of minimal structural sentence schemes, a fragment of which is shown in Table 12. The result for our example is shown in Table 13. It should be noted that fragments 2 and 4 (Table 8) are indicative descriptions of the nouns “places” and “citizens”, respectively, i.e. act as a definition, therefore, their word forms are not analyzed as members of a sentence.

8. The selection of the remaining members of a simple sentence (other semantically significant objects and attributes) and other semantically meaningful relationships is carried out by a consistent comparison of the words of the sentence with the actual structure of the verb from the dictionary of valencies of verbs. The filled valence nests for predicates of the example text are shown in Table 14.

We consider the predicate in more detail. According to the semantic classification used in the dictionary of valencies of verbs, he predicts the situation of inclusion of an object in the composition of something. Verbs of this class require filling valence cells 1 and 6 (ie, Subject and Final Locative) and have a formal expression of the form "noun in the nominative case - the verb is a prepositional group with the prepositions in, to, on, under and noun in the accusative case" . Those. part of the proposal is suitable for this scheme. In the NNSU, citizens are accepted for the first year in the field, and everything else is optional members of the proposal and contains additional information expressing the semantics of the goal (for training), an attribute of the action represented by the verb noun “training” (for basic educational programs ) Thus, the main semantically significant objects “citizens”, “course”, “place” and the main semantically significant relation “inclusion” are revealed.

9. The construction of syntactic groups inside the received simple sentences, in which the predicate actants are the main words, with the help of syntactic rules that reveal the syntactic relations between words. The constructed groups are shown in Table 15.

Thus, many other semantically significant objects and attributes are revealed, as well as other semantically related relationships. For this example, they are summarized in Table 16.

After performing the previous steps on a set of selected elementary units of the first, second, third and fourth levels, using the above-mentioned rules, semantically significant relations are found between semantically significant objects, as well as between semantically significant objects and attributes. So, for example, in the proposal “At the NNSU, citizens of the Russian Federation, the Republic of Belarus, the Republic of Kazakhstan, the Kyrgyz Republic and the Republic of Tajikistan are accepted on a competitive basis for the first year to study in the main educational programs of higher professional education in places financed from the federal budget having a document on the formation of a state sample "of the text in question using a variety of rules, the corresponding processing scheme is presented in figure 2 (stage processing 21-27), and the dictionaries used in the rule shown in Tables 6-16 are allocated named relation "a". Other semantically significant relationships are distinguished using the same set of rules. Each semantically significant relation is assigned its type. As a result, semantically significant relationships are distinguished in the source text. Many of these semantically significant relationships with the types assigned to them for the example in question are presented in Table 17.

Thus, after all the above processing steps have been completed, the source text will be marked out with a lot of annotations corresponding to semantically significant objects, their attributes and semantically significant relations between semantically significant objects, as well as between semantically significant objects and attributes.

After that, at the step indicated by reference numeral 29 in FIG. 2, the occurrence frequencies of semantically significant objects and attributes, as well as semantically significant relationships between semantically significant objects and between semantically significant objects and attributes throughout the entire text are detected. This operation is performed in almost the same way as in step 24 for elementary units of the first level. A fragment of such a frequency dictionary for our example is presented in Table 19.

The next step, indicated by reference numeral 30 in FIG. 2, is technical and is performed to form triads corresponding to stored semantically significant objects, attributes, and semantically significant relationships. A fragment of the set of such triads for our example is presented in Table 20. In fact, the generated set of triads makes up the initial data for constructing the semantic index processed in the previous stages of the text.

In the step indicated by reference numeral 31 in FIG. 2, a semantic index is constructed as follows. First, from the set of triads obtained in the previous step, subsets of triads are formed, each of which corresponds to one semantically significant object with its attributes, and each obtained subset of triads is used as an input for one of the standard indexers, for example, the well-known freely distributed indexer Lucene, the search indexer Yandex machines, Google indexer, or any other indexer, from the output of which they receive an index unique for a given subset of triads. A similar sequence of actions is performed for all subsets of triads corresponding to triads of the form “semantically significant object - semantically significant relation - semantically significant object” and triads of the form “semantically significant object - semantically significant relation - attribute”, receiving a set of corresponding unique indices, which together and make up the semantic index of the text.

In the step indicated by reference numeral 32 in FIG. 2, the indices generated in step 30 of the triad and the indices obtained in step 31, together with a link to the source text from which these triads are generated, are stored in the database.

In accordance with the method of automated semantic comparison of texts in natural language from these triads, a semantic network can be formed in such a way that the first semantically significant object of the subsequent triad is associated with the same second semantically significant object of the previous triad. An example of a fragment of such a semantic network is shown in Table 21.

Moreover, before storing the generated triads and obtained indices in the database, the iterative procedure renormalizes the frequencies of occurrence of semantically significant objects and attributes, as well as the frequencies of occurrence of semantically significant relations, into the semantic weight of semantically significant objects and attributes that are the vertices of the semantic network, thus that semantically significant objects or attributes connected in a network with a large number of semantically significant objects or attributes with a high frequency occurrences increase their semantic weight, while other semantically significant objects or attributes evenly lose it. An example of the renormalized into semantic weights of the numerical values of the weighting coefficients of the concepts of the semantic network is shown in Table 22. Other texts (in this example, two) are processed in a similar way, which should be compared with the source text.

Next, the degrees of intersection of the semantic networks of the source text and two other analyzed texts are calculated both by the vertices and by their relationships, taking into account the semantic weights of the vertices of the semantic networks and the weight characteristics of their relationships. An example of the values of the degrees of intersection of the semantic networks of three texts is given in Table 23. The degree of intersection of the source text with the first text to be compared indicates their semantic similarity, and the degree of intersection with other compared text indicates their small semantic similarity.

The degree of intersection of two semantic networks belonging to two texts is calculated as the sum of the degrees of intersection of elementary units of the fifth level of these two networks. This sum is formed over all the vertices of the network with more vertices. For each vertex of this network, there is a vertex in another network, which is the same elementary unit of the fourth level - the same semantically significant object or the same attribute. If such a vertex is not in the second network, the degree of intersection for this vertex is equal to zero. An example of the values of the degrees of intersection of the vertices of fragments of semantic networks of two texts is given in Table 24.

For each vertex of one semantic network (for each semantically significant element or attribute - elementary units of the fourth level), we calculate the degree of intersection with the corresponding vertex of another semantic network. In the given example, we consider, for example, the “learning” top (see Table 24), which is available in the semantic networks of both compared texts. This degree of intersection is calculated as the sum of the degrees of intersection of all semantically significant objects and attributes associated with this vertex. In the semantic networks of the selected texts, these are “charter”, “faculty”, “department”, “legal”, etc. in one semantic network and “faculty”, “legal”, “entrant”, “course”, etc. - in another semantic network. If no pair is found, the degree of intersection is considered equal to zero. The degree of intersection of semantic networks is thus calculated by summing the smallest degrees of intersection of two pairs of the same semantically significant concepts or attributes of the two compared networks. In this case, semantic intersections of semantic weights of each semantically significant object or attribute associated with this vertex in these two networks are calculated. These semantic intersections are calculated as normalized to 100% scalar products of the semantic weights of the first and second vertices, and the angle between them is taken proportional to the normalized at 100% frequency of occurrence of a semantically significant relationship connecting them. To the resulting amount, add the smaller of the scalar products. If there is no corresponding semantically significant object or attribute in the second network for a given vertex, the degree of intersection over this semantically significant object or attribute is equal to zero. After summing over all semantically significant objects or attributes associated with the current vertex, the resulting sum is normalized to the largest number of semantically significant objects and attributes associated with this vertex in the two networks and go to the next vertex.

The sum obtained for all vertices in one of the networks (with the largest number of vertices) is normalized to the number of elementary units of the fourth level saved after applying the processing in step 7 (see Fig. 1).

For the “learning” vertices, the normalized 100% scalar products are calculated 99 × 99 × sin (67.5 °) / 100 = 90.54 and 99 × 99 × sin (72 °) / 100 = 93.21 with the “faculty” vertices . And so for all the vertices of the semantic network, whose semantic weight exceeded the threshold value (chosen equal to 70 in this example).

The total degree of intersection of two semantic networks at the top of “training” - 122.45 for all vertices of semantic networks adjacent to it is normalized to the largest number of 10 remaining after removal of subthreshold vertices in one of the two semantic networks of compared texts.

It should be emphasized once again that although the linguistic experts preliminarily construct a lot of syntactic rules and corresponding linguistic dictionaries in the claimed method (as a result of which the definition of “automated” is used in the name of the claimed method), the semantic comparison of texts described above is carried out without operator intervention.

Thus, the present invention provides a method for semantic comparison of texts in natural language with virtually no operator. The main difference of this method from known methods is that the frequencies of occurrence of elementary units of the fourth level are calculated semantically significant objects and attributes with their subsequent renormalization into semantic weights. Combining triads of semantically meaningful objects and attributes using semantically meaningful relationships into a semantic network provides a quick comparison of texts, especially texts in highly inflected languages.

Table 1 Token text segmentation Token Number Token Start the end Token Type one AT one one word 2 NNSU 3 6 reduction 3 on 8 9 word four the first eleven 16 word 5 course word 6 for word 7 learning word 8 by word 9 the main word 10 educational word eleven programs word 12 higher word 13 professional word fourteen education word fifteen on word 16 places word 17 , prep sign eighteen funded word 19 behind word twenty score word 21 means word 22 federal word 23 the budget word 24 , prep sign 25 accepted word 26 on word 27 competitive word 28 basis word 29th citizens word thirty Russian Federation the word is geogr. title 31 , prep sign 32 The Republic of Belarus the word is geogr. title 35 , prep sign 36 Republic of Kazakhstan the word is geogr. title 37 , prep sign 38 Kyrgyz Republic the word is geogr. title 40 and word 41 Republic of Tajikistan the word is geogr. title 42 , prep sign 43 having word 44 document word 45 about word 46 education word 47 state word 48 sample word 49 (certificate of secondary (full) general education; diploma of secondary vocational education; diploma of primary vocational education, if it contains a record of secondary (complete) general education) plug-in design - plug-in offer fifty if word 51 education word 52 of this word 53 level word 54 it turns out word 55 first word 56 . prep sign

table 2 Lemmas and morphs Token Number Lemmas Morphs one AT Pretext 3 on Pretext four the first Named after M.R. Unit Numeral Ordinal Win.p. M.R. Unit Numeral Ordinal 5 course Named after M.R. Unit Noun Inanimate. Win.p. M.R. Unit Noun Inanimate. 6 for Pretext to extend Real Holy Communion Imperfect 7 training Named after S.R. Mn. Noun Inanimate. Rod.p. S.R. Unit Noun Inanimate. Win.p. S.R. Mn. Noun Inanimate. 8 by Pretext 9 basic Dat.p Mn. Adjective Tv.p. M.R. Unit Adjective Tv.p. S.R. Unit Adjective main Dat.p Mn. Adjective Tv.p. M.R. Unit Adjective Tv.p. S.R. Unit Adjective 10 educational Dat.p Mn. Adjective Tv.p. M.R. Unit Adjective Tv.p. S.R. Unit Adjective eleven program Dat.p J.R. Mn. Noun Inanimate. Rod.p. M.R. Unit Adjective 12 higher Rod.p. S.R. Unit Adjective Win.p. M.R. Unit Adjective Animation. Win.p. S.R. Unit Adjective Animation. Rod.p. M.R. Unit Adjective 13 professional Rod.p. S.R. Unit Adjective Win.p. M.R. Unit Adjective Animation. Win.p. S.R. Unit Adjective Animation. Named after S.R. Mn. Noun Inanimate. fourteen education Rod.p. S.R. Unit Noun Inanimate. Win.p. S.R. Mn. Noun Inanimate. fifteen on Pretext Named after S.R. Mn. Noun Inanimate. 16 a place Rod.p. S.R. Unit Noun Inanimate. Win.p. S.R. Mn. Noun Inanimate. eighteen to finance Named after Mn. Present Passive Communion Sov.-Nesov. view Win.p. Mn. The Real Passive Communion is Inanimate. Sov.-Nesov. view 19 behind Pretext twenty score Named after M.R. Unit Noun Inanimate. Win.p. M.R. Unit Noun Inanimate. 21 means Rod.p. S.R. Mn. Noun Inanimate. Rod.p. M.R. Unit Adjective 22 federal Rod.p. S.R. Unit Adjective Win.p. M.R. Unit Adjective Animation. Win.p. S.R. Unit Adjective Animation. 23 budget Rod.p. M.R. Unit Noun Inanimate. 25 be accepted Mn. Real 3rd person Verb Imperfect 26 on Pretext Rod.p. J.R. Unit Adjective 27 competitive Dat.p J.R. Unit Adjective Tv.p. J.R. Unit Adjective Offers J.R. Unit Adjective 28 the basis Dat.p J.R. Unit Noun Inanimate. Offers J.R. Unit Noun Inanimate. 29th citizen Named after M.R. Mn. Noun Animated. Rod.p. J.R. Unit Noun Inanimate. thirty the Russian Federation Dat.p J.R. Unit Noun Inanimate. Offers J.R. Unit Noun Inanimate. Named after J.R. Mn. Noun Inanimate. 32 Republic of Belarus Rod.p. J.R. Unit Noun Inanimate. Win.p. J.R. Mn. Noun Inanimate. 36 The Republic of Kazakhstan Named after J.R. Mn. Noun Inanimate. Rod.p. J.R. Unit Noun Inanimate. Win.p. J.R. Mn. Noun Inanimate. 38 Kyrgyz Republic Rod.p. J.R. Unit Noun Inanimate. 40 and Union 41 The Republic of Tajikistan Named after J.R. Mn. Noun Inanimate. Rod.p. J.R. Unit Noun Inanimate. Win.p. J.R. Mn. Noun Inanimate. 43 have Named after Mn. Real Active Communion Imperfect Vip.p. Mn. Real Active Communion Inanimate. Imperfect 44 document Named after M.R. Unit Noun Inanimate. Win.p. M.R. Unit Noun Inanimate. 45 about Pretext 46 education Offers S.R. Unit Noun Inanimate. Rod.p. M.R. Unit Adjective 47 state Win.p. M.R. Unit Adjective Animation. Rod.p. S.R. Unit Adjective 48 sample Rod.p. M.R. Unit Noun Inanimate. fifty if Union 51 education Named after S.R. Unit Noun Inanimate. Win.p. S.R. Unit Noun Inanimate. 52 data Rod.p. S.R. Unit Noun Inanimate. Rod.p. M.R. Unit Adjective Rod.p. S.R. Unit Adjective Win.p. M.R. Unit Adjective Animation. the Win.p. S.R. Unit Adjective Animation. Rod.p. M.R. Unit Pronoun Pronoun. Rod.p. S.R. Unit Pronoun Pronoun. Win.p. M.R. Unit Pronoun Animated. Pronoun. Rod.p. M.R. Unit Past Passive Communion Perfect give Win.p. M.R. Unit Past Passive Communion Animated. Perfect Rod.p. S.R. Unit Past Passive Communion Perfect Win.p. S.R. Unit Past Passive Communion Animated. Perfect 53 level Rod.p. M.R. Unit Noun Inanimate. 54 get out Unit Real 3rd person Verb Imperfect 55 first Adverb

Table 3 Frequencies of occurrence of the first and subsequent words of the sequence in the text, as well as differences in the frequencies of occurrence for different words in the sequence Reiteration The words Frequency Difference sequences of words in the text sequences occurrences frequencies one one national one economy one 0 2 national 2 economy 2 0 3 national 3 economy 3 0 2 one economy four written one 3 2 economy 5 written 2 3 3 economy 6 written 3 3 ... ... ... ... ...

Table 4 Stable word combinations in the text Collocation National economy Writing Economics ... ...

Table 5 Lots of text suggestions Text suggestions Level 1 Units Level 2 Units Level 3 Units Admission rules to the Nizhny Novgorod State University. N.I. Lobachevsky in the year rules, admission, in, Nizhny Novgorod, state, university, im., n., and., lobachevsky, in, year rules, admission, in, Nizhny Novgorod, state, university, lobachevsky, in, year Nizhny Novgorod State University N.I. Lobachevsky Approved by Acting Rector of UNN, prof. R.G. Strongin May approved, acting, about, rector, UNN, prof., river, city, strongin, May, city Approved, Rector, UNN, Strongin, May Rector of UNN prof. R.G. Strongin ...

Table 6 Tin Fragment Personal verb Brief communion Short adjective Predicative word Participle Communion Infinitive Introductory word Other one 2 3 four 5 6 7 8 9

Table 7 Initial Fragmentation Results No. of fragments. Offer snippets Fragment Type one At the NNSU, the first year for training in the main educational programs of higher professional education in the field 9 2 funded by the federal budget 5 3 citizens of the Russian Federation are accepted on a competitive basis one four The Republic of Belarus 9 5 Republic of Kazakhstan 9 6 Kyrgyz Republic 9 7 and the Republic of Tajikistan 9 8 having a state education certificate 5 9 if education of this level is obtained for the first time one

Table 8 Fragmentation of a sentence after combining homogeneous series No. of fragments. Offer snippets Fragment Type one At the NNSU, the first year for training in the main educational programs of higher professional education in the field 9 2 funded by the federal budget 5 3 citizens of the Russian Federation, the Republic of Belarus, the Republic of Kazakhstan, the Kyrgyz Republic and the Republic of Tajikistan are accepted on a competitive basis one four having a state education certificate 5 5 if education of this level is obtained for the first time one

Table 9 Description attribute level elements Offer Components Morphological features Object / Subject Noun pronoun Act Verb Item Feature Full adjective, ordinal, pronoun-adjective, consistent with the object / subject by gender, number and case Sign of action Adverb Attribute measure Adverb, adverbial numeral

Table 10 Syntactic groups corresponding to the attribute level of description Syntax Group Elements Token Numbers Syntax group feature of the object + object 4 + 5 The first course feature of the object + object 9 + 10 + 11 major educational programs feature of the object + object 12 + 13 + 14 higher vocational education feature of the object + object 22 + 23 federal budget feature of the object + object 27 + 28 competitive basis feature of the object + object 47 + 48 state model feature of the object + object 52 + 53 of this level action + sign of action 54 + 55 it turns out for the first time

Table 11 Received simple sentences as a result of enlargement of fragments No simple offer Enlarged Fragments Simple sentences one At the NNSU, the first year for training in the main educational programs of higher professional education in places funded from the federal budget At the NNSU, citizens of the Russian Federation, the Republic of Belarus, the Republic of Kazakhstan, the Kyrgyz Republic, and the Republic of Tajikistan who have a state-approved document citizens of the Russian Federation, the Republic of Belarus, the Republic of Kazakhstan, the Kyrgyz Republic and the Republic of Tajikistan are accepted on a competitive basis, having a document on the formation of a state standard 2 if education of this level is obtained for the first time if education of this level is obtained for the first time

Table 12 Minimum block diagrams of sentences (fragment) MCC Examples of offers N1 V (f) The Rooks Have Arrived. Things are done by people. N1 Cop (f) Adj1 The night was quiet (quiet, quiet). N1 Cop (f) Adj5 The night is quiet (quiet). N1 Cop (f) Adj (f) The night was quieter than the day. N1 Cop (f) N1 He (was) a student. N1 Cop (f) N5 He was a student. Cop (f) N1 It will be raining. It was winter. Whisper. Timid breathing. Silence. ... ... Explanation of table 12: V (f) - conjugated forms of the verb (not infinitive); Cop (f) - conjugated forms of a bunch of official words to be, become, appear; Inf - infinitive of a verb or connective; N1, N5 - nominative, instrumental case of the substantive; Adj1, Adj5 - nominative, instrumental case of adjectives and passive participles; Adj (f) - short forms and corporate bodies of adjectives and passive participles.

Sentences with the pattern Cop (f) N1 can be called, i.e. the link verb is not present there explicitly. In this case, we assume that the predicate is zero, denoted as NULL.

Table 13 The predictive minimum of simple sentences that make up the complex sentence of the source text No simple offer Simple sentences MCC Template Predictive minimum (Subject-Predicate) one At the NNSU, citizens of the Russian Federation, the Republic of Belarus, the Republic of Kazakhstan, the Kyrgyz Republic, and the Republic of Tajikistan who have a state-approved document N1 V (f) - Noun in the nominative case + Conjugate form of the verb citizens are accepted 2 if education of this level is obtained for the first time. N1 V (f) - Noun in the nominative case + Conjugate form of the verb education is obtained

Table 14 Fill valencies for example text predicates No simple offer Predicate 1. Subject 2. Object 3. Addressee 4. Tool 5-7. Locatives one be accepted citizens - - - in UNN, for the course, for places - 2 get out education - - - - Note to the table: 5 - the initial locative, 6 - the final locative, 7 - the middle locative.

Table 15 Syntactic groups derived from source text using syntax rules No simple offer Offer snippets Syntactic groups where actants and predicate are the main words Name of groups and rules one At the NNSU, the first year for training in the main educational programs of higher professional education in places funded from the federal budget on the first course Object + Object Tag to places funded by the federal budget Object + Object Tag citizens of the Russian Federation, the Republic of Belarus, the Republic of Kazakhstan, the Kyrgyz Republic and the Republic of Tajikistan are accepted on a competitive basis, having a document on the formation of a state standard citizens of the Russian Federation Genitive determination in postposition one citizens of the Republic of Belarus Genitive determination in postposition citizens of the Republic of Kazakhstan Genitive determination in postposition citizens of the Kyrgyz Republic Genitive determination in postposition citizens of the Republic of Tajikistan Genitive determination in postposition citizens with a document on education of the state sample Object + Object Tag 2 if education of this level is obtained for the first time level education Object + Object Tag Genitive determination in postposition of this level Object + Object Tag it turns out for the first time Action + sign of action

Table 16 Many semantically significant objects and attributes (fragment) Simple sentence Semantically significant objects Attributes At the NNSU, citizens of the Russian Federation, the Republic of Belarus, the Republic of Kazakhstan, the Kyrgyz Republic, and the Republic of Tajikistan who have a state-approved document NNSU - course the first places funded by the federal budget citizens of the Russian Federation, citizens of the Republic of Belarus, citizens of the Republic of Kazakhstan, citizens of the Kyrgyz Republic, citizens of the Republic of Tajikistan having a state education certificate if education of this level is obtained for the first time level education -

Table 17 Relations between semantically significant objects, and between semantically significant objects and attributes Semantically significant object 1 Semantically significant object 2 Semantically significant relationship Type of semantically significant relationship one citizens NNSU be accepted include 2 citizens course be accepted include 3 citizens a place be accepted include ...

Table 18 The frequency of occurrence of semantically significant objects and attributes. Semantically significant object or attribute Frequency of occurrence one citizens 7 2 NNSU 38 3 course 7 four a place 41 ...

Table 19 The frequency of occurrence of semantically significant relationships between semantically significant objects, and between semantically significant objects and attributes Semantically significant object 1 - semantically significant object 2 Semantically significant relationship The frequency of the semantically significant relationship one citizens - UNN be accepted 56 2 citizens - course be accepted fifty 3 citizens - place be accepted 53 ...

Table 20 Many triads (fragment). Triads one citizens - NGSU 2 citizens - course 3 citizens - place four NSU - place 5 NGSU - citizens 6 NSU - course 7 place - applicant ...

Table 21 Semantic network of triads (fragment). Main word Attitude Subordinate Word one citizens include NGSU 2 citizens include course 3 citizens include a place four NGSU whole part a place 5 NGSU include citizens 6 NGSU whole part course 7 a place claim enrollee ...

Table 22 The semantic weight of semantically significant words and attributes Semantically significant object or attribute Semantic weight one citizens 98 2 NNSU 98 3 course 95 four a place 99 ...

Table 23 The degrees of intersection of semantic networks of the source text with the networks of two other texts Text 1 Text 2 Text 3 Text 1 100.00 8.35 2.91 Text 2 8.35 100.00 1.15 Text 3 2.91 1.15 100.00

Table 24 The degree of intersection of two fragments of two semantic networks fragment of the first network fragment of the second network degree of intersection of second objects or attributes vertex vertex associated with the first semantic weight, relationship weight vertex associated with the first semantic weight, relationship weight training 122.45 / 10 = 12.24 one university 100.98 0 2 student 99.91 0 3 teacher 99.95 0 four budget 99.53 0 5 rector 99.55 0 6 employee 99.42 0 7 charter 99.30 0 8 faculty 99.75 faculty 99.80 90.54 9 the department 78.65 0 10 legal 71.30 legal 72.40 31.91 enrollee 99.75 0 enrollment 99.32 0 course 98.41 0 degree of intersection of peaks “training” 122.45 course faculty 99 0 ...

Table 25 The degree of intersection of semantic networks of two texts first network second network vertex vertex associated with the first vertex vertex associated with the first total weight one state one state 14.25 2 budget 2 citizen 0 3 teacher 3 commission 0 four training four Informatics 0 5 rector 5 rector 19.42 6 employee 6 written 0 7 student 7 student 18.31 8 faculty 8 faculty 15,20 9 the department 9 0 ... ... 60 legal 79 legal 13.18 Amount 659.91 Normalized amount 660/79 = 8.35

Table 26 The degree of intersection of the top "learning" of two semantic networks first network second network first peak second peak the weight relationship weight scalar product first peak second peak the weight relationship weight scalar product training 99 training 99 faculty 99 75 90.54 faculty 99 80 93.21 legal 71 thirty 31.91 legal 72 40 41.89 degree intersection "faculty" 90.54 degree of intersection "legal" 31.91 degree of intersection 122.45 / 10 = 12.24

Claims (4)

1. The method of automated semantic comparison of texts in natural language, which consists in the fact that:
- represent two compared texts in digital form for subsequent automatic and (or) automated processing;
- carry out the indexing of these texts in digital form, receiving:
- elementary units of the first level, including at least words,
- elementary units of the second level, each of which is a normalized word form,
- elementary units of the third level, each of which is a stable phrase in the said text,
- elementary units of the fourth level, each of which is a semantically significant object and attribute, and
- elementary units of the fifth level, each of which is a triad of either two semantically significant objects and a semantically significant relationship between them, or from a semantically significant object and attribute and a semantically meaningful relation connecting them;
- identify the frequency of occurrence of elementary units of the fourth level and the frequency of occurrence of the mentioned semantically significant relationships;
- store in the database the formed elementary units of the second, third, fourth and fifth levels with the identified frequencies of occurrence of elementary units of the fourth level and semantically significant relationships, as well as the resulting indices, together with links to specific sentences of this text;
- form a semantic network from said triads in such a way that the first elementary unit of the fourth level of the subsequent triad is associated with the same second elementary unit of the fourth level of the previous triad;
- during the iterative procedure, the mentioned frequencies of occurrence are renormalized into the semantic weight of elementary units of the fourth level, which are the vertices of the semantic network, so that elementary units of the fourth level, connected in a network with a large number of other elementary units of the fourth level with a high frequency of occurrence, increase their semantic weight, and other elementary units of the fourth level evenly lose it;
- rank elementary units of the fourth level by semantic weight by comparing the semantic weight of each of them with a predetermined threshold value and delete elementary units of the fourth level having semantic weight below the threshold value;
- retain in memory the remaining elementary units of the fourth level with a semantic weight above the threshold, as well as semantically significant relations between the remaining elementary units of the fourth level;
- reveal the degree of intersection of their semantic networks for the two texts being compared both by vertices and the relationships between these vertices, taking into account the semantic weights of the vertices of the semantic networks and the weight characteristics of their connections, the degree of intersection of the semantic networks of the two texts being compared is a value characterizing the semantic similarity of these texts. 2. The method according to claim 1, wherein said indexing is carried out in the process of performing the following steps:
- segment the text in digital form into elementary units of the first level, including at least words;
- segment text in digital form into sentences according to graphematical rules;
- form for each elementary unit of the first level, which is a word, on the basis of morphological analysis, elementary units of the second level, including a normalized word form;
- calculate the frequency of occurrence of each elementary unit of the first level for two or more adjacent units of the first level in this text and combine among the mentioned elementary units of the first level sequences of words following one after another in this text into elementary units of the third level, which are stable combinations of words , if for every two or more consecutive words in a given text the difference is in the calculated frequencies of occurrence of these words for the first occurrence of a given sequence word integrity and for several subsequent occurrences for each pair of words, the sequences remain unchanged;
- identify in the process of multi-stage semantic-syntactic analysis by referring to linguistic and heuristic rules pre-generated in the database in a predefined linguistic environment in each of the generated sentences, semantically significant objects and attributes - units of the fourth level;
- for each elementary unit of the fourth level, the identity is fixed by reference between the corresponding semantically significant object, as well as the attribute and the corresponding anaphoric link, if any, in the indexed text, replacing each anaphoric link with the corresponding antecedent;
- store in memory each semantically significant object and attribute;
- identify in the process of multi-stage semantic-syntactic analysis by referring to linguistic and heuristic rules pre-generated in the database in a predefined linguistic environment in each of the generated sentences, semantically significant relationships between the identified units of the fourth level - semantically significant objects, as well as between semantically significant objects and their attributes;
- assign to each semantically significant relation the corresponding type from the subject ontology stored in the database on the subject of the subject area to which the indexed text belongs;
- identify throughout the text the frequency of occurrence of elementary units of the fourth level and the frequency of occurrence of the mentioned semantically significant relationships;
- store in memory each identified semantically significant relation together with the type assigned to it;
- form within the given text for each of the identified semantically significant relations that connect both the corresponding semantically significant objects and the semantically significant object and its attribute, a multitude of triads, which are elementary units of the fifth level;
- index on the set of formed triads individually all related semantically meaningful relationships semantically significant objects with their frequencies of occurrence, all attributes with their frequencies of occurrence and all formed triads. 3. The method according to claim 1, wherein said degree of intersection of two semantic networks belonging to the two compared texts is calculated as the sum of the coincidences of the elementary units of the fifth level of these two semantic networks. 4. The method according to claim 3, in which:
- choose one of the two semantic networks as the basic network, in which, after ranking and removing vertices with semantic weights, more vertices remain below the threshold value than the other, which is chosen as the comparison;
- find for each vertex of said core network in said comparative network a vertex that is the same elementary unit of the fourth level, i.e. the same semantically significant object or the same attribute;
- calculate for each vertex found in each of the aforementioned base and compared networks the values of all the mentioned triads associated with a given vertex as the area of triangles whose sides correspond to the components of each of these triads, and the angle between the sides is proportional to the weight of the semantically significant relationship of this triad;
- choose for each pair of said triads associated with a pair of specific vertices in said basic and compared networks, the smaller of said calculated values as the degree of intersection of said triads in said basic and compared networks;
- summarize for each of the vertices associated with a given vertex, all selected calculated values, obtaining the degree of intersection for a given pair of vertices of the mentioned base and compared networks;
- normalize the found amount to the number of the mentioned semantically significant objects and attributes associated with a given vertex in that of the mentioned base and compared networks, which contains more vertices associated with this vertex;
- summarize the normalized sums for all the vertices of that of the mentioned base and compared networks, which contains more vertices;
- normalize the received amount to the number of elementary units of the fourth level remaining in this network, obtaining the mentioned degree of intersection of the semantic networks of the mentioned compared texts.

Images