RU2273879C2 - Method for synthesis of self-teaching system for extracting knowledge from text documents for search engines - Google Patents

Abstract

FIELD: computer science, information-searching and intellectual systems engineering.

SUBSTANCE: result is achieved due to realization of self-teaching mechanism in form of stochastically indexed artificial intelligence system; morphological and syntax analysis is performed, and also stochastic indexing of text documents on given topic for forming semantic analysis databases; user request is transformed in stochastically integrated form to multiple new requests equivalent to source request, and selection of stochastically indexed text document fragments is performed with all word combinations of transformed request, of which stochastically indexed semantic structure is formed, short response of system on basis of structure is formed and relevance of short system response to query is checked by means of their comparison.

EFFECT: possible automatic forming of knowledge by extracting them from text documents represented in digital form at different languages, and possible intellectual processing of text information and user requests to extract knowledge on any foreign language.

2 cl, 1 dwg, 3 tbl

Description

Technical field

The invention relates to the field of computer technology, information retrieval and intelligent systems.

The invention is intended for use in the creation of information retrieval and other information and intelligent systems based on the Internet.

State of the art

Currently, the Internet has accumulated a huge amount of information on various subject areas and topics. This information contains and constantly updates comprehensive information and knowledge. However, access to them from the multimillion-dollar user audience is difficult. This is due to the lack of effectiveness of modern methods of extracting information for search engines. Known methods for extracting information for search engines Yandex, Yahoo, Rambler. Known methods provide the issuance of text documents at the request of the user from the Internet.

The main disadvantages of the known methods of extracting information from these systems are:

- the complexity of formalized query languages;

- lack of apparatus for semantic analysis of the content of text documents and their relevance to the questions asked;

- the impossibility of accurately determining the presence in the search document of the information specified in the user’s request, as well as the selection of specific information and knowledge necessary for the user from voluminous information sources.

Due to these shortcomings, when implementing information retrieval procedures, along with useful information, a lot of superfluous, “noise” information is transmitted, which is poorly selected by modern search engines. This significantly increases the search time for the necessary information, loads the channels and servers of the system by transmitting and processing search noise.

The main problem is that in doing so, the user, asking the system, receives large amounts of information, often not containing the necessary information. There is a need to familiarize yourself with each document received to determine if it contains the required data. This leads to unjustified time and intellectual costs. The inability to obtain in real time from the vast arrays of the Internet the specific information and knowledge that the user needs to solve problems of a different nature, significantly reduces both the value of the information and the effectiveness of the search engines working with it.

A known method of extracting knowledge and information at the request of a user from knowledge bases, which is implemented in an intelligent information-logical computing system described in the monograph: Nasypny V.V. The development of the theory of building open systems based on information technology of artificial intelligence. M., 1994 .-- 248 p. (S.85-112). The specified method, based on stochastic information technology, provides the ability to effectively search for knowledge and its processing using inference in real time. This is due to the fact that, in contrast to the existing methods of knowledge processing, which are used in modern artificial intelligence systems, this method provides a linear dependence of the search time and logical processing on the amount of knowledge needed to form an answer. However, this method does not make it possible to extract knowledge from text documents, which is explained by its orientation to the processing of formalized information of knowledge bases carried out by experts and knowledge engineers. This makes it impossible to use this method to extract knowledge from text documents of modern information retrieval systems.

There is also a method of extracting knowledge from text documents described in the work: Nasypny V.V., Nasypnaya G.A. Building an intelligent information retrieval system. M .: Prometheus, 2001 .-- 27 p. The method is based on stochastic intelligent information technology, which provides real-time morphological, syntactic and semantic analysis of large amounts of textual information. This system can function in conjunction with existing information retrieval systems as an intellectual add-on above them, as well as create new generation search engines with their own standards for stochastic indexing of text documents, information exchange protocols and user request processing. The main advantages of this method in comparison with the methods implemented in modern search engines are:

- processing user requests in natural language;

- search and issuance of documents that reliably contain complete information relevant to the user's request;

- selection of fragments of text in accordance with the user's request, containing information and knowledge on various subject areas necessary to solve specific problems.

The main disadvantage of this method is that the knowledge bases of intelligent systems designed for morphological, syntactic, semantic analysis of the text are filled by experts and require long time and technological costs. Therefore, the creation of such systems for extracting knowledge from text documents in the interests of users of developed countries that have national Internet subsystems with information in the language of a given country requires a long time. As a result of this, the indicated method cannot be used to create multilingual systems for extracting knowledge from texts on the basis of the Internet. This significantly complicates the transition to the knowledge industry, which would be based on textual information from national search engines and provide a qualitatively new information service in various fields - industrial, scientific, educational, cultural and everyday activities of a person, taking into account the modern requirements of a civilized society.

Other disadvantages of this method include the lack of the ability to automatically analyze new words that are not part of the dictionaries. If they appear in text documents, the participation of experts is required in determining which part of the speech the new word refers to and its morphological characteristics. This makes it impossible to automatically configure the knowledge extraction system to process text documents on specified new topics. We also note that in order to ensure the efficiency of knowledge extraction, complex processing of text fragments from various documents is required, based on the analysis of semantic relationships with the help of logical inference between these fragments, as well as on equivalent transformations of the sentence of the given text. This function is also not implemented in the considered method.

Disclosure of invention

The objective of the invention is to provide a method for the synthesis of a self-learning system for extracting knowledge from text documents for search engines for use in creating a global knowledge industry based on the Internet that does not have the above disadvantages. The achieved result is:

- the ability to automatically generate knowledge by extracting them from text documents presented in various languages in electronic form to fill knowledge bases;

- automatic analysis of new words and dictionary updates;

- equivalent conversions of user requests and sentences of text documents, providing increased efficiency of knowledge extraction;

- self-study of these systems the rules of grammatical and semantic analysis;

- intellectual processing of textual information and user requests in order to extract knowledge in a given foreign language.

The specified technical result is achieved by the fact that in the method of synthesis of a self-learning system for extracting knowledge in a given language from text documents of search engines

provide a self-learning mechanism in the form of a stochastically indexed artificial intelligence system based on the use of unique binary combinations of stochastic information indices,

provide automatic training of the system for the rules of grammatical and semantic analysis by using equivalent transformations of stochastically indexed text fragments, inference and the formation of related semantic structures and stochastic indexing for presentation in the format of production rules,

perform morphological analysis and stochastic indexing of linguistic texts in electronic form with simultaneous automatic training of the system to the rules of morphological analysis,

perform morphological and syntactic analysis, as well as stochastic indexing of text documents on a given topic in electronic form in a given language with simultaneous automatic training of the system in the rules of parsing,

perform semantic analysis of stochastically indexed text documents on a given topic in electronic form with simultaneous automatic training of the system in the rules of semantic analysis,

form a user request in a natural given language and submit it in electronic form after stochastic indexing in the form of an interrogative sentence,

Convert a user’s request in stochastically indexed form into many new queries equivalent to the original query,

in accordance with the user's request, a preliminary selection is made of stochastically indexed fragments of text documents in electronic form, containing all the phrases of the converted request in aggregate,

form a stochastically indexed semantic structure using the indicated fragments of text documents,

on the basis of this structure, using a logical conclusion that ensures the connection of stochastically indexed elements of various texts, and equivalent text conversion form a short response of the system,

check the relevance of the received short response of the system to the request by forming on its basis an interrogative sentence, comparing the received interrogative sentence with the request,

if the question and offer received are identical, they decide on the relevance of the short response of the system to the request and present it in a given language.

The specified technical result is achieved by the fact that in the method of synthesis of a self-learning system for extracting knowledge in any of the given foreign languages from text documents of search engines

provide a self-learning mechanism in the form of a stochastically indexed artificial intelligence system based on the use of unique combinations of binary signals of stochastic information indices for stochastic indexing and searching for fragments of linguistic texts in a given base language containing a description of grammatical and semantic analysis procedures and automatic training of the system in grammar and semantic rules analysis by equivalent transforms are stochastically indexed fragments of text, inference, and the formation of their associated semantic structures of the stochastic index for submission to the rules of output formats,

perform morphological analysis and stochastic indexing of linguistic texts in a given base language in electronic form with simultaneous automatic training of the system for the rules of morphological analysis, the formation of a database of stochastically indexed dictionaries and the formation of index tables of linguistic texts for each of the specified foreign languages, as well as the knowledge base of morphological analysis, containing the resulting production rules for a given base language and each of the specified foreign x languages

perform morphological and syntactic analysis, as well as stochastic indexing of text documents on a given topic in each of the given foreign languages in electronic form from a search system with their presentation in the form of tables of indexes of text documents on a given topic and writing to the database of stochastically indexed texts with simultaneous automatic learning system rules for parsing using stochastically indexed linguistic texts in a given base language and form vaniem knowledge base to parse the language and each of the specified languages,

perform semantic analysis of stochastically indexed text documents on a given topic in a given base language in electronic form with simultaneous automatic training of the system for the rules of semantic analysis and the formation of a semantic analysis knowledge base for the base language and each of the specified foreign languages,

form a user’s request in a natural predetermined foreign language and submit it in electronic form after stochastic indexing in the form of an interrogative sentence, including an interrogative phrase and phrases that determine the semantics of the request,

convert the user’s request in stochastically indexed form into many new queries equivalent to the original query in a given foreign language,

in accordance with the user's request, a preliminary selection of stochastically indexed fragments of text documents in a given foreign language in electronic form, containing all the phrases of the converted request in total, is performed,

form a stochastically indexed semantic structure based on the indicated fragments of text documents,

on the basis of the generated stochastically indexed semantic structure with the help of logical inference, providing the connection of stochastically indexed elements of various texts, and equivalent text conversion, a short system response is formed containing phrases in stochastically indexed form that determine the semantics of the query, as well as a group of response words corresponding to the interrogative phrase request

the relevance of the short answer of the system to the query is checked by replacing the group of response words with the corresponding interrogative phrase in a stochastically indexed form, obtaining a stochastically indexed interrogative sentence, comparing the received interrogative sentence with the query and if the received interrogative sentence and query are identical, they decide on the relevance of the short answer of the system to the query and represent it in a given foreign language.

In the event of an unsuccessful attempt to generate an interrogative sentence identical to the user's request, they request new text documents from the search engine to search for an answer relevant to the user's request.

Additionally, at the user's request, a complete answer can be generated containing more detailed information or a set of specific knowledge, while using logical inference to form a stochastically indexed semantic structure and the necessary equivalent transformations of this set of text fragments to obtain a stochastically indexed new text that reveals the contents with possible detail short answer received earlier.

At the same time, the system is automatically trained in the rules of morphological analysis by highlighting in a stochastically indexed text a specific set of word forms for each word, obtaining stochastic indices of the word base and a given set of its endings or prepositions, random access by indicated indices to stochastically indexed linguistic texts, extracting fragments from them, connecting the specified set of word endings or prepositions with the part of speech corresponding to the given word, as well as with a complete set rum of endings or prepositions obtained by declension or conjugation, converting these fragments into the format of production rules by stochastic indexing, while ensuring the correctness of each rule by independently generating it based on several fragments from the corresponding linguistic texts, and obtaining a table of indexes of production rules for the base knowledge of morphological analysis.

In addition, when stochastic indexing of linguistic texts after determining the part of the speech of each word using the rules of the knowledge base of morphological analysis, the database of the stochastically indexed dictionary is filled with stochastic indices of the basis of each word and a complete set of its endings or prepositions, and stochastic conversion is performed when forming tables of text indices information and obtaining unique binary combinations of indices of word stems, their endings, prepositions, sentences, ab atsev titles and text which are placed in the table base stochastically indexed text index with providing connectivity between said indices defined in the source code and its restoration by providing an index table.

In addition, automatic training of the system for the rules of parsing is carried out by searching in stochastically indexed linguistic texts for fragments describing the order of parsing sentences, while the logical conclusion is realized to obtain a stochastically indexed semantic structure that determines the relationship of syntactic elements and structures with given parts of speech, and the formation of production rules that determine the syntactic analysis of sentences according to morphological characteristics word ticks, while ensuring the correctness of each rule by independently forming it on the basis of several fragments from the corresponding linguistic texts, the obtained rules are entered into the syntactic analysis knowledge base, as they are filled, they are stochastically indexed and presented in the form of an index table.

In addition, automatic training of the system for the rules of semantic text analysis is carried out by forming a query on the tables of indexes of linguistic texts on stochastic indices of the stems of words and parts of speech, not precisely defined members of the sentence, and receiving an answer in the form of a fragment of text describing the semantic characteristics that the words should have for their correspondence to this particular member of the sentence, and to the answer received, using the stochastic index of the base of the word and the required semant characteristics, refer to tables of indexes of explanatory dictionaries and general and thematic encyclopedias, while using logical inference they try to form a stochastically indexed semantic structure that connects this word and the required semantic characteristics, in the positive case, they consider that the specified sentence member is defined exactly, and the text fragment relevant to the request is transformed into a production rule, while ensuring the correctness of each rule by its independent Formations based on several fragments from the corresponding linguistic texts, which are included in the knowledge base of semantic analysis, stochastically index this database, represent it in the form of an index table and are used in the semantic analysis of words as members of a sentence and relations between words expressed by phrases.

After the formation of the index table of each text and the completion of its morphological, syntactic and semantic analysis, stochastic indices of the names of the parts of speech, sentence members and questions that correspond to each word in the sentences are formed, and the indicated indices are written in the cells of the index table of this text, which allows when searching for fragments of text, automatically determine to which part of speech each sentence belongs to a sentence member, and form questions to it.

Then, after receiving all the text index tables, a text index table is formed on this topic, the rows of which are named by non-repeating stochastic indexes of word stems, and each column corresponds to a stochastic index of a specific text, while stochastic paragraph indexes in which in this text are written in the table cells contains a word with the corresponding base index, the resulting index table on this topic is used for preliminary search of fragments containing a certain aggregate s query phrases.

In this case, equivalent conversions of the user's initial request are carried out using synonyms that are close in meaning of the words, as well as replacing parts of speech and sentence members while preserving the semantic content of the initial request based on the application of stochastically indexed rules of morphological, syntactic and semantic analysis to obtain equivalent phrase structures of the interrogative sentence request and save semantic communication between them.

The set of semantically related fragments of text containing all the words of the user's query is formed by referring to the table of text indices for a given topic using stochastic indices of the indicated word stems, selecting stochastic paragraph indices and texts corresponding to them containing all the query phrases in aggregate, referring to the indicated indices to an index table for each of the selected texts, inference from index tables and equivalent text transformations to form stochastically ind pensate semantic structure linking group codes words response corresponding to an interrogative request phrase and all query phrase defining query semantics, and included in the pre-selected paragraph.

At the same time, the stochastically indexed semantic structure that is formed in the logical inference process and corresponds to the user's request is taken as the basis for generating, using the resulting set of fragments of text, a question sentence identical to the user's request, which is formed by equivalent conversion of stochastic indices of the base words of the query and their endings with using knowledge base rules to provide the required semantic characteristics of each word combining the text fragment included in the query, as well as using inference on transitive dependencies between phrases to combine them into a single interrogative sentence, identical to the user's query, which contains a group of response words corresponding to the interrogative phrase of the query.

The correctness of the short answer can be ensured by the formation of several identical stochastically indexed semantic structures of the mentioned answer based on different, pre-selected stochastically indexed fragments of text documents.

In the process of searching and generating an answer using tables of indexes of text documents, self-learning of the system is carried out by forming indexed text elements linking the request and the relevant short answer to obtain a knowledge base containing elements of the type "request-response", which is stochastically indexed, presented in the form of a table indices and are used in grammatical and semantic analysis of text sentences, as well as in the formation of answers to repeated user requests contained in indicated indexed knowledge base.

At the same time, to formulate a complete answer containing knowledge relevant to the user's request, based on a short answer using logical inference on the index tables used to obtain the text fragment, a stochastically indexed semantic structure is formed that connects the group of response words with stochastic indices of sentence word strings supporting transitive dependence, providing in its entirety a full disclosure of the content of the short answer within the framework of the generated text fragment, Then, with the help of equivalent sentence transformations, based on the indicated stochastically indexed semantic structure, a single linked text of the complete answer is obtained.

An equivalent transformation of stochastically indexed text fragments is performed by presenting each sentence as a set of stochastically indexed phrases that are transformed using the rules of knowledge bases of morphological, syntactic and semantic analysis by equivalent conversion of stochastic indices of the stems of cognate words, their endings and prepositions to form new parts of speech or members of the proposal ensuring the invariability of the connection of the specified words REPRESENTATIONS within stochastically indexed semantic structure of each sentence and harmonization of these proposals with each other in the formation of these new text fragment.

When a new word appears in the indexed text during stochastic indexing of text documents that is not contained in the dictionary of stochastically indexed words and in linguistic texts, a single-root word with the specified new word is found in this dictionary, and the rules for the equivalent conversion of found in dictionary of the same root word into a new word, while by the type of equivalent conversion, the part of speech to which the new word and all its word belong is determined forms obtained by declension or conjugation, and in the absence of cognates in the dictionary, a certain set of word forms of the new word is selected from the text, based on the prepositions or endings of which using the stochastically indexed dictionary or rules of production of morphological analysis determine the part of speech to which it refers, and the full a set of its word forms obtained by declension or conjugation.

Moreover, to simultaneously extract knowledge from text documents in given foreign languages, the system is first automatically taught the rules of morphological, syntactic, semantic analysis for a given base language, a stochastically indexed dictionary base and morphological, syntactic, semantic analysis knowledge bases are generated using stochastically indexed linguistic texts in a given base language, using the generated databases automatic generation of queries for automatically teaching the system to any of the given foreign languages, at the same time, preliminary selection of fragments of linguistic texts in the base language containing the knowledge necessary to study the given foreign language, equivalent conversions of the indicated texts, the formation of stochastically indexed semantic structures and logical inference on given structures for the formation of answers relevant to automatic s to aprosu, who use morphological, syntactic and semantic analysis for any of the given foreign languages to form knowledge bases for extracting knowledge from text documents in a given foreign language.

Brief Description of the Drawings

The invention is illustrated by the example illustrated in FIG. 1, which shows a block diagram of an intelligent self-learning system for extracting knowledge from text documents for search engines; as well as the following tables:

Table 1- frame proposals

Table 2 - Text indices,

Table 3 - Text indices on this topic.

Preferred Embodiment

The following are definitions of terms used in the present description.

A knowledge base is one or several specially organized files that store a systematic set of concepts, rules, and facts related to a certain subject area.

Interrogative phrase - a phrase with a interrogative pronoun or adverb as the interrogative word associated with the main word of the phrase (name or verb).

Grammatical analysis - morphological and syntactic analysis.

Knowledge - new textual information that is not explicitly contained in text documents, which is automatically generated by the system using equivalent transformations and inference in the form of an answer, relevant to the user's request and aimed at solving his tasks in accordance with the request.

Linguistic texts - educational, methodical, scientific, reference (explanatory dictionaries, encyclopedias) and other texts intended for the study of this language.

The logical conclusion is a method of processing knowledge that imitates the process of human reasoning, which, based on individual language units, allows you to synthesize a semantic structure with a specific semantic content.

Morphological analysis is the analysis of the words of a sentence to determine the morphological composition with the subsequent refinement of the characteristics of individual words related to a particular part of speech, first indicating the constant morphological characteristics of the word, which are independent of its position in the sentence, then analyzing the grammatical form of the word related with its declination or conjugation.

The basis of the word is the part of the word expressing its lexical meaning, while in the inflected and conjugated words there is a basis and an ending, and the rest of the words contain only the basis.

Search system - a system that automatically searches for information on keywords, topics, etc.

Production rules - a form of knowledge representation in the form of a complex sentence "If (condition), then (conclusion)", in which the condition contains various phrases, including predicative and other types of relations between objects of the subject area, united by logical connectives "and", and the conclusion contains a phrase or set of phrases that determine the semantic consequence that is true, or an action that is activated if all the phrases of the condition are true.

Relevance - a measure that determines how fully a particular document meets the criteria specified in the user’s request.

The semantic structure is a form of communication of individual linguistic units of various sentences, taking into account the types of relations between them, expressing a certain semantic content of the analyzed text.

Semantic analysis is the analysis of the meaning, meaning of individual language units: words, sentences of a sentence, their correlation with certain types of relations between objects of the subject area and phenomena of reality.

Syntactic analysis is the analysis of sentence words to determine the syntactic composition with subsequent refinement of the characteristics of individual words, phrases, their types, forms of communication between words in a phrase and sentence, the structure of sentences, structural types of sentences.

An artificial intelligence system is a software and hardware system that contains, as a basis, a subsystem of logical inference, a knowledge base, and also depending on the class, other hardware and software of artificial intelligence and designed to support human intellectual activity or its replacement in a number of control processes.

Declension of a word is a change of nouns by cases (for most names and by numbers), and for adjectives and other coordinated words also by gender.

A phrase is a syntactic unit formed by combining two or more words on the basis of a subordinate connection - coordination, control or adjacency - and those lexical and grammatical relations that are generated by this connection.

Wordform - a given word in a given grammatical form.

Conjugation of a word is a change of a verb in persons, numbers, tenses and moods, and in the past tense and in the subjunctive mood in the singular also by gender.

An equivalent transformation is the replacement of individual linguistic units with others with the provision of their connection within the framework of the semantic structure of a sentence or in a certain set of text sentences that can express the same semantic content.

Let us consider in more detail the implementation of the proposed method on the example of the construction and operation of an intelligent self-learning system for extracting knowledge for search engines (ISIS), presented in Figure 1. The mentioned stochastically indexed artificial intelligence system includes:

- multilingual linguistic processor (1);

- a subsystem of stochastic indexing of text documents and highlighting fragments of texts (2);

- a subsystem for managing the self-learning and knowledge extraction mode (3);

- interpreter of stochastically indexed texts and production rules (4);

- a subsystem of equivalent text transformations (5);

- logical inference subsystem (6);

- a database of stochastically indexed dictionaries of the base and new words (7);

- a database of stochastically indexed linguistic texts (8);

- knowledge base "request-response" (9);

- a database of stochastically indexed text documents on given topics (10);

- a database of stochastically indexed dictionaries of foreign words (11);

- knowledge base of morphological analysis (12);

- knowledge base of parsing (13);

- knowledge base of semantic analysis (14);

- the base of stochastically indexed phrases (15).

The indicated system is based on the use of stochastic conversion and indexing of symbolic information, the formation of index tables of production rules for controlling the self-learning mode and text indices. It provides access by stochastic indices to fragments of textual information, logical inference and equivalent text transformations using stochastically indexed rules to extract knowledge from selected fragments of text and present them in the form of production rules or in the form of answers to user requests.

The creation of ISSIS involves the development of a self-learning mechanism for the system of rules for morphological, syntactic and semantic analysis of textual information based on linguistic texts. These texts contain dictionaries of common words, thematic dictionaries, dictionaries of synonyms, explanatory dictionaries, educational and methodical texts on the grammar of given languages, etc.

The user communicates with the system through a multilingual linguistic processor (1). It provides the input of queries in a natural language and the issuance of answers generated by the system. Moreover, the exchange of information between the user and the system can be carried out in predetermined languages. In addition, the linguistic processor (1), at the command of the subsystem (3) for managing the self-learning and knowledge extraction mode, provides interaction with the search system connected to the AISS. The purpose of this interaction is to enter, at the request of subsystem (3), new text documents from a search engine in a given language on a specific topic for their subsequent processing. The multilingual linguistic processor (1) also provides input into the system of linguistic texts in a given language in electronic form.

Morphological analysis of linguistic texts and automatic training of the system for the rules of morphological analysis are performed by the command of the subsystem (3) for managing the self-learning and knowledge extraction process in the process of forming the basic dictionary and writing it to the database (7) of stochastically indexed dictionaries of the base and new words. These functions are carried out simultaneously with the indexing of linguistic texts using the subsystem (2) of stochastic indexing of text documents and the selection of text fragments.

To form a stochastically indexed basic dictionary, a dictionary of common words is used in electronic form, which is entered into the system through a multilingual linguistic processor (1) and determined from the given word forms of this dictionary, the speech part of each word, its basis, and the corresponding sets of endings. The basis of this word is stochastically indexed using the subsystem (2) of stochastic indexing of text documents and extracting text fragments and written into the database (7) of stochastically indexed dictionaries of the base and new words in the table of the stochastically indexed base dictionary in the column of the word base indices.

As a result of the described word processing of the specified dictionary in the multilingual linguistic processor (1), stochastic indices of the stems of all words and the stems themselves are obtained, as well as a certain set of endings that are entered into the database (7) of stochastically indexed dictionaries of the base and new words.

The stochastically indexed basic dictionary recorded in the database (7) of stochastically indexed dictionaries of the basic and new words has several table formats, each of which corresponds to a certain part of speech. The heading of the tables contains graphs that include the names of morphological characteristics (gender, number, case, person, time, etc.), as well as questions that correspond to the word forms of a given word obtained when it is declined or conjugated. Moreover, each basis corresponds to a line containing the endings of the indicated word forms of the given word. Note that at the beginning of filling a stochastically indexed basic dictionary, only a few word forms of each word are known, namely those that are listed in the dictionary of common words. Finding the remaining word forms and their endings to fill in the tables of a stochastically indexed basic dictionary is performed in the automatic training mode of the system for the rules of morphological analysis after the initial indexing of the corresponding linguistic texts.

The basis of this mechanism is the introduction of a new method for stochastic indexing of text documents, which is implemented in subsystem (2) of stochastic indexing of text documents and highlighting text fragments. The procedure is based on the functions of stochastic transformation of symbolic information and the formation of stochastic indices in the form of unique binary combinations of the foundations of words, sentences, paragraphs and titles of text documents, including bibliographic data. At the same time, along with the stochastic transformation of symbolic information, the formation of stochastic indices {I _ξi ^(u) } of word stems, sentences {I _ξi ^(p) } , paragraphs {I _ξi ^(a) } and the name of the text I _ξi ^(t) , which is in processing, the frames of each sentence are filled in (Table 1) and the table of indices for this text is formed (Table 2).

The specified frame (Table 1), which is formed in the subsystem (2) of stochastic indexing of text documents and extracting text fragments, contains ten levels (lines) of slots (cells). These levels of slots are filled in during the stochastic indexing of the text, as well as when performing morphological, syntactic and semantic analysis of each sentence.

In stochastic indexing of linguistic texts, stochastic indices of the stems of the words {I _ξij ^(u) } and their endings are written in slots of the first level. Slots of the second level contain words in the order in this sentence with the number i. In this case, prepositions, particles, unions and punctuation marks are entered in the slots of those words with which they are associated. To fill the slots of the third level, stochastic indices of the stems {I _ξij ^(u) } and their endings recorded in the slots of the first level are used.

On the basis of the indexes of words, access is made to the rows of the corresponding tables of the stochastically indexed basic dictionary, named by identical indexes to determine the part of speech to which the given word belongs. The specified information from the database (7) of stochastically indexed dictionaries of the base and new words is recorded in the slots of the third level of the sentence frame corresponding to the words of the slots of the second level.

Writing into the slots of the third level of the frame the characteristics of the parts of speech, as well as filling in the slots of the fourth to tenth levels, is carried out in the process of further morphological and syntactic analysis of the text, which is carried out simultaneously with teaching the system the rules of morphological and syntactic analysis. This process will be discussed below.

Based on the receipt of text sentence frames with the first four levels of slots filled in the subsystem (2) of stochastic indexing of text documents and the selection of text fragments, an index table of this text is formed.

Table 2 of text indices is a table whose rows are named by stochastic indices {I _ξi ^(u) } of word stems, columns are indicated by paragraph indices {I _ξj ^(a) } in the order they appear in the text, and the cells located at the intersection of the corresponding columns and rows contain the indices of the lists {I _ξij ^(s) } . Moreover, the information contained in each list, named {I _ξij ^(s) }, is recorded in a separate file and in the general case should include the following data:

{I _ξi ^(p) } - the sentence index in which the given word is included;

N _i ⁽ⁿ⁾ is the number of the sentence the given word is in;

(u _i u _j ) - the ending that the given word has in the sentence (I _ξi ^(p) N _i ⁽ⁿ⁾ ) ;

I _ξj-1 ^(u) is the index of the previous word in the sentence or paragraph of the text; _moreover , if I _ξj ^(u) is the first word in the sentence (paragraph), then a dot is placed after the index I _ξj-1 ^(u) . I _ξj-1 ^(u) may correspond to the word ending the previous sentence in the framework of this paragraph or the previous paragraph. If a comma is placed after I _ξj-1 ^(u) , then this means that I _ξj ^(u) can begin the participial or participial turn, the subordinate clause or the simple sentence in the compound;

I _{ξ (j + 1)} ^(u) is the index of the next word in the sentence, paragraph, text, and if

I _ξj ^(u) is the final word in the sentence (paragraph), then I _ξj-1 ^(u) is _{preceded by} a period.

I _ξj-1 ^(u) may correspond to a word starting a new sentence of a given paragraph or a subsequent paragraph. If a comma is placed before I _ξj-1 ^(u) , then this means that I _ξj ^(u) can complete the participle, participial turns or a simple sentence as part of a complex one;

I _ξj ^(vu) is the index of the question to the given word, as to the sentence member;

I _ξj ^(pu) - index of the name of the sentence member to which the given word corresponds;

I _ξj ^(vpru) - the index of the question, which corresponds to the participle, participial sentences or subordinate clause that begins I _ξj ^(u) ;

I _ξj ^(pru) - the index of the name of the sentence member, to which the participle, participial sentences or the subordinate clause starting I _ξj ^(u) _correspond .

The indicated indices and symbols correspond to the word with the basis I _ξi ^(u) as part of one of the sentences I _ξj ^{(p) of the} paragraph I _ξj ^(a) and have a given format that determines the location of the indices and symbols in this group. If there are no individual indices, then a space sign is put in their place. If the given word I _ξi ^(u) is included in n sentences {I _ξi ^(p) } of the paragraph I _ξi ^(a) , then the indicated groups in the list will also be n .

Note that the first six indices of the list I _ξij ^(s) are formed during stochastic indexing of the text. Moreover, from the basis index I _ξi ^(u), by referring to a stochastically indexed basic dictionary, it is always possible to determine to which part of speech the indicated word belongs. The remaining data of the list I _ξij ^{(s) are} determined after filling in the levels of four to ten frames of text sentences in the process of further morphological and syntactic analysis, which are implemented simultaneously with the system’s self-learning rules for grammar analysis of sentences.

After stochastic indexing of all linguistic texts, including texts containing descriptions of the grammatical analysis of sentences, they are written into the database (8) of stochastically indexed linguistic texts and proceed to the formation of the rules for morphological analysis of the text simultaneously with filling in the database (7) of stochastically indexed dictionaries of the base and new words .

For this purpose, from each table of a stochastically indexed basic dictionary that contains the basics of words related to a given part of speech, a stochastic index of the basics of each word and a given set of its endings or prepositions is selected. Then random access is made at the indicated indices to the base (8) of stochastically indexed linguistic texts to extract fragments from them that connect the index of the speech part and the specified set of word endings or prepositions with the corresponding complete set of endings, prepositions or questions obtained in declension or conjugation. After that, this fragment of the text enters the interpreter (4) of stochastically indexed texts and production rules, which form a stochastically indexed semantic structure in the form of a combination of phrases of each sentence included in this fragment:

where I _ξi ^(u) I _ξj ^(u) are the stochastic indices of the main and dependent stems of the words of the given phrase, I _ξi ^(r) I _ξj ^(r) are the stochastic indices of the parts of speech of the main and dependent words of the given stratum, I _ξi ^(z) I _ξj ^(z) are the stochastic indices, respectively, of the morphological characteristics of the speech parts of the main and dependent words of a given phrase, and the → sign defines the relationship between the main and dependent words of a given phrase.

The main connecting link of each stochastically indexed semantic structure represented by expression (1) is a verb that defines the semantics of relationships within a given structural scheme. The connection between different stochastically indexed semantic structures (1) that are included in different sentences is carried out if they have identical phrases, their synonyms, the repetition of the main words or the use in the second sentence of a pronoun corresponding to one of the phrases of the first sentence, as well as a pronoun in combination with the main word. Accordingly, sentences or parts of sentences are found in which the stochastically indexed semantic structure containing the indexed source data of the request is appropriately linked to the stochastically indexed semantic structure with indexed response data. At the same time, to determine the semantics of verbs, stochastic indices of their stems are accessed to the base (8) of stochastically indexed linguistic texts to access the synonym dictionary index tables.

If the first and second structural schemes are interconnected by a phrase containing a definable part of speech, and the meanings of the verbs associated with this part of speech are identical or synonymous with the verbs of the request and the intended answer, then these structural schemes enter the subsystem (5) of equivalent text transformations. In subsystem (5), these two semantic structural schemes are transformed into a single stochastically indexed semantic structure of the production rule, which contains a condition that includes a request and a conclusion (response). The indicated stochastically indexed semantic structure has the following general form:

where I _ξi ^(su) - is the stochastic index of the corresponding phrase

I _ξi ^(su) : (I _ξi ^(u) I _ξi ^(r) I _ξi ^(z) ) → (I _ξj ^(su) (I _ξj ^(r) (I _ξj ^(z) ) from expression (1), and the sequence ⇒ is interpreted in the usual logical sense as a sign of the logical following of the conclusion located on the right side of the expression (2) from the condition on the left side of the expression (2), if all the word combinations of the condition are true (correspond to the initial data of the query). the rule is ensured by independent formation of the identical stochastically indexed semantic structures (2) based on non how many fragments from the corresponding linguistic texts.

Each production rule, formed in the subsystem (5) of equivalent transformations in the form of expression (2), enters the interpreter (4) of stochastically indexed text and production rules, where this expression (2) is converted to a text format of production rules, presented in the form " If (condition), then (conclusion). " The resulting rule is indexed into the knowledge base (11) of morphological analysis. The synthesis of knowledge bases containing stochastically indexed rules will be described below.

When forming the rules for morphological analysis of the text simultaneously with filling in the database (7) of stochastically indexed dictionaries of the basic and new words, the first stochastically indexed semantic structure (1) (structural diagram) contains stochastic indices of the word’s base, indicating part of speech and a given set of its endings or prepositions. The second structural diagram (1) is associated with the first through an identical index of a part of speech and defines a complete set of endings, prepositions, questions obtained by declension or conjugation of a given part of speech.

By applying the procedure described above to the tables of indexes of dictionaries of synonyms corresponding to the linguistic texts of the base (8) of stochastically indexed linguistic texts, the correspondence of the semantics of the verbs of the first and second semantic structures to the query and the expected answer is determined. Then the phrase connecting the first and second structures is determined. If the result is positive, two parts of the indicated fragment of the text enter the subsystem (5) of equivalent text transformations, then to the interpreter (4) of stochastically indexed texts and production rules. As a result, this fragment is converted into a format of production rules, presented in the form "If (condition), then (conclusion)". In this case, the condition of the rule includes phrase indices connecting the part of speech and a given set of word endings or prepositions located in the dictionary format and determining the changes in the word form when this word is declined or conjugated. The conclusion contains a complete set of endings, prepositions and questions obtained by declension or conjugation of a given word as an appropriate part of speech. The generated production rule is recorded in the knowledge base (11) of morphological analysis. After completion of the formation of the rules that determine the parts of speech, at the command of the subsystem (3) for controlling the self-learning and knowledge extraction mode, they proceed to the synthesis of the rules for equivalent transformations of cognates. Here we use the general rule for transforming parts of speech, previously recorded in the knowledge base (11) of morphological analysis, based on the use of tables of a stochastically indexed basic dictionary and the selection of the corresponding fragments of linguistic texts that describe the formation order of one part of speech based on another one-root part of speech:

" If you want to convert one part of speech into another,

first we highlight the basis of the first part of speech,

we turn to the format of a stochastically indexed basic dictionary,

we are looking for the second part of speech, the basis of which has a common part, including the root (maybe two, possibly with a prefix, possibly with alternation, addition, with the exception of individual vowels or consonants), with the basis of the first part of speech,

after highlighting the root, using the basis of these parts of speech, select their suffixes,

then, by referring to stochastic indices of the basics of words of parts of speech to tables of indices of linguistic texts, we select a fragment that describes the corresponding method of converting one part of speech to another, and check according to the format of the dictionary how the basis of the second part of speech is formed in relation to the basis of the first ( replacing, discarding, adding suffixes),

then we determine whether this method of replacing a part of speech corresponds to the required method of forming a second part of speech from the first part of speech,

in the positive case, we accept the second part of speech as newly formed. "

In the process of converting specific words using a general rule, an appropriate particular rule is formed on its basis, indicating the transformed parts of speech, suffixes and the method of forming one part of speech from another. This occurs in the interpreter (4) of stochastically indexed texts and production rules and in the subsystem (5) of equivalent text transformations. By the above-described procedure, this fragment is converted first into a single stochastically indexed semantic structure of the production rule (2), and then into the format of production rules presented in the form "If (condition), then (conclusion)". After stochastic indexing, these rules are entered into the knowledge base (11) of morphological analysis.

If, when indexing the next text document, a new word appears, the basis of which is not contained in the base dictionary, then they proceed to the procedure for determining the part of speech of a new word and its endings when declension or conjugation.

To begin the process of determining which part of speech a new word refers to, at least two different word forms of this word are extracted from the text, by comparing them, the unchanging part, which is supposedly the basis of the new word, and its end are determined. After that, it is determined whether the words in the basic dictionary format have a common root (possibly with a prefix) with the new word. The root is the common, indivisible part of the basics of related words (containing at least two letters, including one vowel), which, when adding prefixes, suffixes and endings, is used to form one-root parts of speech. In accordance with this, a common root is extracted by comparing the foundations of a new word and the foundations of words from the format of the basic dictionary until they find the common indivisible part of the two compared bases - the new word and the next word from the basic dictionary.

After that, a reference is made to the knowledge base (12) of morphological analysis to select a rule that allows you to determine which part of the speech a new word refers to. For this purpose, use the corresponding rule of equivalent transformations.

In order to use the rules of equivalent transformations to determine the part of speech of a new word, it is believed that the second part of speech in the general rule of equivalent transformations above refers to a new word and is unknown, while the first part of speech having a common root with it is found in the base dictionary and therefore known. Then they check whether it is possible, using the transformations described in the rule, to obtain from the basis of a known part of speech the basis of a new word whose part of speech is unknown. In this case, a family of specific rules is used, obtained on the basis of a general rule and contained in the knowledge base (12) of morphological analysis, which allows you to convert a known (first) part of speech into other parts of speech. If as a result of using one of the rules it is possible to obtain the basis of a new word, the part of the speech to which it refers will become known - it will correspond to the second part of the speech indicated in the rule. Moreover, using the rules of production of the knowledge base (12) of morphological analysis, it is possible to determine in more detail the characteristics of each part of speech. For example, if in the morphological analysis of texts in Russian, the rules of the knowledge base (12) of morphological analysis allow us to determine not only the part of the speech of a new word, but also the end of the name (noun, adj.) In the im.p., singular, then therefore, they provide an opportunity to clarify what type of declension (1, 2, 3) a new word refers to. For nouns, adjectives, ordinal numbers, some types of pronouns, as well as participles, this allows you to accurately determine the complete set of their endings obtained by declension. In this case, for the indicated parts of speech, it is enough to find the corresponding word in the dictionary format, which has in im. unit the same ending as in the new word. A complete set of endings of these parts of speech will correspond to a set of endings of a new word, which are written in the format of the dictionary of new words along with its basis. After that, the stochastic index of the base is formed, and all the obtained characteristics of the new word are recorded in the format of the dictionary of new words.

If the new word is a verb, then after highlighting its foundation as described above and referring to the knowledge base (12) of morphological analysis, using the appropriate rule, determine its part of speech and find the infinitive. By the suffix of this infinitive (-ty or -ty), referring to the format of the basic dictionary, they find a verb that has the same suffix (-ty or -ty) in undefined form . In this case, the complete set of endings of a given verb, obtained after its conjugation and recorded in the dictionary format, is presumably selected as the complete set of endings of the new verb. For a more accurate determination of what type of conjugation (1, 2) this verb belongs and, accordingly, to clarify the complete set of its endings in the process of indexing the text, they find a sentence in which this verb is presented in the form of 3rd L. plural To do this, find a sentence in which there is a subject expressed by a noun (pronoun) in plural, which coordinates with the predicate, expressed by this verb with a personal ending -ut / -yut (1 conjugation) or -at / -yat (2 conjugation ) By the personal ending of the marked verb in the format of the basic dictionary, the verb is found having the same ending in it in 3rd L. plural In this case, the complete set of endings of a given verb is taken as a complete set of endings of a new verb and recorded together with its basis in the format of a dictionary of new words. After receiving the stochastic index of the basis of the new verb, all of the specified information is recorded in the format of the dictionary of new words.

In the process of indexing the text when various word forms of new words that are not contained in the database (7) of stochastically indexed dictionaries of the base and new words appear, by comparing the indicated word forms in the subsystem (2) of stochastic indexing of text documents and extracting text fragments, the basis of the new word is selected and a specific set of its endings. Then, a stochastic index of the basis of the new word is formed and, together with its endings, it is entered into the dictionary format of the new words in the database (7) of stochastically indexed dictionaries of the base and new words. After processing a given set of word forms of a given word and, accordingly, filling out a dictionary format with various types of its endings, an appeal is made to the indexed base dictionary table. This dictionary after filling contains the indices and foundations of commonly used words, as well as all kinds of endings of various parts of speech and their types related to this word, which are obtained by declining or conjugating with the characteristics of parts of speech. The query to the dictionary contains the stochastic index of the stem of the given word, the stem itself, as well as all kinds of endings that had word forms of this word when processing text documents. In the database (7) of stochastically indexed dictionaries of the base and new words at the endings of a given word, using the dictionary format, there is a word having the same endings among a complete set of endings. This means that the new word refers to the same part of speech as the word in the dictionary with identical endings. After determining the part of speech to which the new word belongs, all the information included in the request is entered into the dictionary of new words in the established format. At the same time, in the interpreter (4) of stochastically indexed texts and production rules and in the subsystem (5) of equivalent text transformations as described above, this fragment is converted first into a single stochastically indexed semantic structure (2) production rules, and then into the production rules format, presented in the form "If (condition), then (conclusion)".

As a result, a production rule is formed, the condition of which includes a given set of endings for a given word, and the conclusion contains the name of the speech part of a given word having the endings given in the condition, as well as a complete set of endings located in the dictionary format that determine changes in the word form when this the words. In addition, the conclusion contains questions about the word forms of a given part of speech when it is declined or conjugated, which are arranged in the order determined by the format of the dictionary.

Thus, in the process of processing texts containing new words that are presented in their various word forms, they automatically determine their part of speech, fill out the dictionary format for new words in the database (7) of stochastically indexed dictionaries of the base and new words, and also teach the system rules morphological analysis. These rules are entered into the knowledge base (12) of morphological analysis. As the knowledge base (12) is filled and its stochastic indexing is described below, along with the format of the stochastically indexed basic dictionary, it is used to determine which part of the speech the new word and its characteristics belong to if it is not contained in the new word dictionary format.

After completion of morphological analysis and stochastic indexing of linguistic texts, formation of a knowledge base (12) of morphological analysis, base (8) of stochastically indexed linguistic texts, and also base (7) of stochastically indexed dictionaries of the base and new words, they pass to stochastic indexing of texts on a given topic with simultaneous automatic training of the system for the rules of parsing.

The system is automatically taught the rules of parsing by the command of the subsystem (3) for managing the self-learning and knowledge extraction mode by searching the database (8) for stochastically indexed linguistic texts of fragments that determines the order of parsing sentences. First, the above-described order converts these fragments into a set of stochastically indexed semantic structures of production rules, which in the general case are of the form of expression (2).

After that, in the inference subsystem (6) using the obtained stochastically indexed semantic structures (2) production rules, which describe the order of parsing sentences, the logical conclusion is implemented to obtain stochastically indexed semantic structures of the new production rules. These semantic structures associate syntactic elements with predetermined parts of speech during the formation of production rules that determine the parsing of sentences according to the morphological characteristics of words. The obtained rules are entered into the knowledge base (12) of parsing, as it is filled, it is stochastically indexed and presented in the form of an index table.

As noted above, the parsing of the text begins with determining the order of its implementation, which is described in educational and methodical text documents on the grammar of this language. In this case, to extract knowledge from these texts that determine the order of parsing, the subsystem (3) for controlling the self-learning mode and the extraction of knowledge initially forms a request to the database (8) of stochastically indexed linguistic texts for accessing the index tables of educational and methodical texts. For this query containing the phrase "Procedure for parsing" in a given language, paragraphs that include this phrase and terms that determine the sequence of this parsing will be found in the indicated texts.

After processing, as described above, a fragment of the text obtained from the corresponding educational and methodical publications for the Russian language, for example, the following production rule can be formed: “If you need to parse the sentence, its order will be as follows: phrase (composing or subordinate connection) , simple sentence (subject, predicate, definition, addition, circumstance), type of simple sentence (narrative, interrogative, incentive), sentence structure (two compound or one-compound, not widespread or widespread), predicate (simple, compound verb, compound nominal), sentence with homogeneous members, sentence with separate members, sentence with direct speech, compound sentence, compound sentence with one subordinate clause, compound sentence with several subordinate clauses, an all-union compound sentence, a complex sentence with different types of communication. " After the formation of this rule in the form of expression (2) based on the indices {I _ξi ^(su) } of phrases, the stochastic index of the production rule I _ξi ^{(pp) is formed} in the form of a unique binary combination of a given length:

where F is the stochastic transformation function of the production rule.

Then, the contents of each of the terms given in the conclusion of the production rule (3) are disclosed one by one by generating the corresponding queries to the database (8) of stochastically indexed linguistic texts. As a result, a set of rules {I _ξij ^(pp) } will be generated that define each of the syntactic terms that are contained in the rule I _ξi ^(pp) . In this case, using the relationships between the production rules, which include identical syntactic terms in the condition or in conclusion, a logical conclusion is realized in subsystem (6). As a result, the following sequence of logical relationships between production rules will be formed:

Here, the indices {I _ξij ^(pp) } denote the set of rules corresponding to a certain level of parsing, which is specified in the rule I _ξi ^(pp) . For example, it can be a phrase (composing or subordinate connection), simple sentence (subject, predicate, definition, addition, circumstance), type of simple sentence (narrative, interrogative, incentive), etc.

Thus, a deductive inference is implemented in the system, the purpose of which is to associate syntactic terms with certain parts of speech of words, their characteristics and conduct sequential parsing in accordance with the above rule. For example, for the Russian language in the process of the indicated logical conclusion for the term “subject” the following text fragment can be found: “The subject in the sentence can be expressed by the following words: noun in the im.p., pronoun in the im.p., infinitive, whole phrase " The resulting fragment of the text enters the interpreter (4), the subsystem (5) of the equivalent text transformations and the subsystem (6) of the logical inference. As a result of the above transformations using expression (2), we obtain a set of production rules that connect the morphological characteristics of words with the names of the sentence members:

“If the sentence contains a word that is a noun in the im.p., then this word is presumably a subject.”

"If the sentence contains a word that is a pronoun in the im.p., then this word is presumably a subject."

"If the sentence has a word that is an infinitive, then that word is supposedly subject."

"If the sentence contains words related to the whole phrase, then these words are presumably subject."

In the process of extracting text fragments to form production rules that define phrases and individual sentence members, the morphological characteristics of the sentence words are used as initial information. Based on these initial data, fragments of the text are distinguished in which the indicated data are connected by identical phrases with the proposed answer, which has the name of the sentence member. These phrases correspond to a word with initial morphological characteristics.

Therefore, the marked fragment of the text that defines the relationship between the word with the given morphological characteristics and the sentence member can be translated into a stochastically indexed semantic structure (2) with the provision of the order of its correctness described above. Then, the stochastically indexed semantic structure (2) will be presented in the format of the production rule: "If (condition), then (conclusion)". The specified procedure is carried out using the interpreter (4), subsystem (5) of equivalent text transformations and production rules. In this case, the initial morphological characteristics of the word are included in the rule condition, and the conclusion contains the name of the sentence member and the question that corresponds to it.

As a result, production rules will be formed to determine the main members of the proposal (subject and predicate), secondary members of the proposal (definition, addition, circumstance), as well as the phrases they form. When defining a predicate, it is indicated what type it belongs to: simple verb, compound verb, compound nominal. First of all, the predicative basis of the sentence is determined, in which the subject and predicate are coordinated, as well as other phrases and the corresponding types of relations. They include subject and definition, predicate and complement, predicate and circumstance, etc.

Thus, in the process of processing textual information during parsing of a sentence, the system self-learns the rules for determining the main and secondary members of a sentence. The rules thus obtained are entered into the knowledge base (13) of parsing. Then, in accordance with the order of the parsing, self-study of the system begins with the rules for determining separate members of a sentence. The initial data here are the parts of speech, the members of the sentence and their characteristics, which, after the conversion of the text, enter the conditions of the rules of production. The conclusions of these rules determine the type of group of separate members, the name of the member of the proposal and the question to which they correspond.

Thus, they describe separate agreed definitions (participles, adjectives with dependent words), isolated inconsistent definitions, isolated appendices, separate additions, isolated circumstances, etc., including questions related to them.

After that, in self-learning mode, production rules are formed that allow you to parse a simple sentence on the basis of the source data that determines which members of the sentences are the words that are included in this sentence, which phrases and separate groups of sentence members they form. As a result, production rules will be obtained that allow one to determine whether a given proposal is two-part or one-part (if it is one-part, then to which type it refers - indefinitely personal, impersonal, callative, etc.). In this case, sentences with homogeneous members, with separate members of the proposal, with direct speech are highlighted.

Then, based on the selected fragments of the text, production rules are formed for the parsing of complex sentences. The initial data included in the terms of the rules of production, here are the types and characteristics of simple offers that are part of complex offers. Moreover, the conclusions of the rules make it possible to determine the type of the given complex sentence: the compound sentence, the compound sentence with one subordinate clause, the compound sentence with several subordinate clauses, the unionless compound sentence, the complex sentence with different types of relations. In conclusion, the rules also determine which question corresponds to each of the simple sentences in this complex sentence.

All described levels of formation of production rules correspond to a sentence parsing scheme formed at the beginning of the self-learning mode by the command of the subsystem (3) for managing the self-learning mode and extracting knowledge in the form of a logical expression (4).

As a result of the implementation of the self-learning mode, the obtained production rules are recorded in the knowledge base (13) of parsing. Note that the system’s self-study of the rules for parsing sentences is performed directly in the process of processing source texts on a given topic by analyzing each sentence. This analysis allows you to fill in the levels of five to ten frames of each sentence of the text, which in turn is used to populate the index table of the text (Table 2) and the lists described above that make up the contents of each of its cells.

As the parsing knowledge base is filled, it is stochastically indexed and presented in the form of an index table. This significantly increases the efficiency of the analysis of proposals due to random access by condition indices corresponding to the production rules to obtain the desired result.

Let us consider in more detail the order of stochastic indexing of knowledge bases and their use in the process of grammatical analysis of sentences.

After receiving the completed text of the knowledge base in the form of a set of production rules presented in the form of stochastically indexed text in the format “If (condition), then (conclusion)”, each production rule enters the interpreter (4) of stochastically indexed texts and production rules. Here, a stochastically indexed semantic structure (2) is re-formed, which contains the totality of all phrases of this rule:

In addition, each phrase is associated with the index I _ξi ^(su) :

then, based on these indices, unique stochastic indices of each production rule I _ξi ^{(pp) are formed} in accordance with expression (3).

Next, the index table is generated for this knowledge base in text form, similar to how regular text documents are indexed. Moreover, the production rule with the index (I _ξi ^(pp) ) is adopted as a paragraph . In accordance with this entry in the table of production rules index is a row containing {I _ξi ^(u) } the basis of the words of the dictionary of production rules (the set of non-repeating bases of words that are part of the production rules). Each cell of the row corresponding to a specific index (I _ξi ^(u) ) contains the index I _ξi ^{(su) of the} phrase and the index (I _ξi ^(pp) ) of the rule, which includes the given word, the ending and the number of this word as part of the production rule, and also the indices (I _ξi-1 ^(u) ) and (I _{ξi + 1} ^(u) ) , respectively, of the previous and next words in this rule. This allows, as for the case with text documents, to form the text of any production rule based on the index. In this case, the expression

is written as a row in the table of the base (15) of stochastically indexed phrases.

The source data for accessing the index of the text of production rules is extracted from the frame of the parsed offer. As was presented above, this frame after morphological analysis contains four line levels, which include, respectively, indices of the stems of the words {I _ξi ^(u) } , words in the context of the sentence, parts of speech and characteristics corresponding to these words, and questions to them. It is this information in various combinations that is part of the conditions of production rules and allows, based on a logical conclusion, to conclude which member of the sentence (accurately or inaccurately) this part of speech belongs. In this case, a reference to the table of indexes of production rules is made by the indices of the foundations of words {I _ξi ^(u) } of the sentence frame, as well as by the values of {I _ξi ^(su) } of phrases of terms or conclusions of the rules.

To implement inference functions using production rules, an interpreter (4) of stochastically indexed text and production rules is used. As a result, the production rule is transformed into the form (2) of a stochastically indexed semantic structure. Moreover, according to the phrases (I _ξi ^(su) ) of the conditions of the production rules (after referring to the base (15) of the stochastically indexed phrases and determining the stochastic indices {I _ξi ^(u) } of the stems of the words of the phrases according to the indices I _ξi ^{(su), the} words search for the corresponding cells of the sentence frame and reading from them the names of words, characteristics of parts of speech or questions to them. According to the phrases {I _ξj ^(su) } of the conclusion, the corresponding cells of levels 5-10 of the sentence frame should be filled out, defining the names of the sentence members, their groups, separate members, types of simple sentences in a complex sentence with questions to them. In this case, the production rules are checked for all phrases of the condition, and if all phrases of the condition are true, combined by logical connectives "and" (in the proposal frame all the characteristics and data described in the phrases of the condition of the production rule are found), the conclusion is considered true. At the same time, the data defined in the phrases of the conclusion of the rule are entered in the corresponding cells of the frame of the proposal of levels 5-10. If the conclusion contains a preliminary result or phrase, according to which it is necessary to find logically related rules, then they are searched by referring to the index bases of the word phrases to the index table of the corresponding knowledge base. Moreover, due to random access to tables based on stochastic indices, the need for enumeration on the whole set of production rules is eliminated. As a result, the linear dependence of the inference time on the number of production rules involved in processing is ensured. The appeal to the knowledge base and the processing of production rules are intended to fill all the cells of the proposal frame with accurate data.

If in the process of parsing the individual members of the sentence are not defined accurately, then for their exact definition the system proceeds to the semantic analysis of the words of these sentences simultaneously with the implementation of the self-learning mode of the rules of semantic analysis. This applies primarily to the definition of the subject, additions and circumstances expressed by a noun with a preposition, participle turnover, etc.

To accurately determine the sentence members, a semantic analysis is used, which is based on the function developed by AISS, which provides the selection of paragraphs and sentences from the texts that describe all possible types of relations between different objects. The system’s requests for the implementation of this function can be generated automatically in the subsystem (3) for managing the self-learning and knowledge extraction mode, if as a result of parsing it will not be established exactly which member of the sentence is the speech part of the sentence being studied.

For this purpose, the subsystem (3) for controlling the self-learning and knowledge extraction mode, the subsystem (6) for logical inference, and the interpreter (4) for the text and production rules are used. Clarification of sentence members in the case of their inaccurate definition during parsing is based on the selection of sentences describing the relations between given objects from the set of texts and determining the types of relations between them. As a result of automatic generation of system queries and semantic analysis of selected sentences between given objects in the interpreter (4) of stochastically indexed text and production rules, the following types of relations can be defined:

- generic,

- aggregate (part - the whole),

- object relations,

- definitive relationships,

- circumstantial,

- permissible, unacceptable.

In turn, the circumstances of the relationship are divided into the following types:

- way of action,

- places

- time

- measures or degrees

- causes,

- goals

- conditions,

- concessions.

In the text, the indicated relations between objects are described by the predicative basis of each sentence, which consists of the subject and the predicate, as well as phrases between the various members of the sentence and, first of all, phrases describing the relationship of the predicate with the circumstance (circumstances) or with the complement (object relations). Moreover, collocations containing the predicate and related addition or circumstance play a decisive role in classifying the type of relationship. It is from the content of the two indicated members of the sentence that it is determined what type of relationship exists in the given sentence between the objects of the subject area expressed by the subject, as well as by addition or circumstance. In this case, the definitive relationship describes the properties of the subject, additions or circumstances using phrases that contain agreed or inconsistent definitions. In the process of analyzing sentence members, classifying the type of relationships described by it allows you to practically accurately determine the sentence members in the most complex cases when the parsing gives an inaccurate result.

In order to classify the type of relations in phrases, the interpreter (4), using the command of the control subsystem (3), writes stochastic indexes of typical phrases of each of the above relations from the tables of indexes of explanatory dictionaries of the database (8) of stochastically indexed linguistic texts. At the same time, in the process of semantic analysis, each of the phrases under study is correlated using the logical inference of the explanatory dictionary text index table and the formation of a stochastically indexed semantic structure with one of the phrase indices written in the interpreter (4). The inference order for the text index tables will be presented below when describing the process of forming a stochastically indexed semantic structure of the system response.

In the general case, for the semantic analysis of words and phrases of sentences, the system uses five sources of information, namely:

- a knowledge base (9), which contains text elements of the "request-response" type, formed during the operation of the AHMIS to process typical requests (this database will be described in detail below);

- base (8) of stochastically indexed linguistic texts, which contains tables of indexes of texts of explanatory dictionaries, encyclopedias and basic scientific and methodological materials of general and special purpose, allowing to extract knowledge about objects of the subject area and types of relations between them;

- the knowledge base (14) of semantic analysis, which contains the rules for accurately determining the sentence members, ensuring the equivalence of the conversion of sentence members, which are necessary for semantic analysis and assessing the relevance of the generated answers to incoming requests; it will be described in detail below;

- knowledge base (12) of morphological analysis, which contains rules for determining parts of speech and their equivalent transformations;

- knowledge base (13) of parsing, which contains rules for determining the sentence members and their equivalent transformations.

The first of these knowledge bases is formed on the basis of stochastically indexed short answers generated during the processing of user requests, and contains many text elements of the "request-response" type. This knowledge is the semantic basis of relevant responses to user requests and contains interrogative sentences. Each of these suggestions is identical to the corresponding user request, in which, after the interrogative word (or interrogative phrase), the corresponding group of response words is additionally included. This group may contain one or more phrases, be a group of separate members of a sentence, or a subordinate clause. Moreover, in each element of the specified knowledge, the question to the group of answer words is precisely defined, which allows one to classify the relations between the objects of the subject area that are presented in this sentence and, accordingly, determine which member of the sentence is the main word in the phrase of the answer.

The second base of linguistic texts is represented by a multitude of stochastically indexed texts, explanatory dictionaries, encyclopedias, basic scientific and methodological materials of both general and thematic purposes. They include a detailed description of common vocabulary, as well as special terms on the subject. These text materials, presented in the form of index tables, are used to extract knowledge from them that characterize the basic properties of various types of objects of the subject area and the relationships between them, correlating them with the above classification system.

The third knowledge base (14) of semantic analysis consists of production rules that are automatically generated and designed to solve the problems of semantic text analysis using inference and information contained in the first two knowledge bases.

Morphological and syntactic analysis knowledge bases are used for equivalent text transformations during semantic analysis. The process of equivalent transformations will be described in more detail below in the analysis of query processing functions.

To ensure rational processing of knowledge, the first base described above is presented in the form of an index table, the input of which includes the basis of words in the “request-response” knowledge. Moreover, each row of the table has cells containing the text index, index and paragraph number, on the basis of which the sentence is formed, the number of the word in its composition, the end of the word, as well as the indices of the foundations of the previous and subsequent words in the sentence. This allows, at the request of the system, to make random access using the word base indices to the corresponding rows of the table, select the required cells from them and, if necessary, restore the source text of the corresponding "request-response".

The described knowledge base allows one to determine the members of a sentence in the most complex cases when parsing a sentence. For example, to distinguish a subject from direct addition or indirect addition from a circumstance with an exact classification of its type, etc. For this purpose, the system of semantic analysis generates a corresponding request to the knowledge base. In the first case, when it is necessary to clarify the subject (for example, in sentences like Rain wet an umbrella or an umbrella wet rain ), at the request of the system, it is determined for which object the relation expressed by the predicate is permissible. It is obvious that the object corresponding to the permissible ratio is accepted as the subject.

In the case when the knowledge base does not allow an answer to the specified query, the question will be addressed to the tables of text indexes on this issue for searching for a phrase containing the required relationship between objects on the whole set of text documents of the second knowledge base on this topic.

In the second case, based on the system’s request to the knowledge base, it should be determined which question the sentence member answers, which can be attributed both to the supplement and to the circumstance, and thereby precisely determine which member of the sentence the given word is. For this purpose, the query of the system, addressed to the knowledge base, indicates the required word and the proposed question to it. If, at the same time, the knowledge base contains the corresponding “request-response”, in which the main word and the question in the phrase match, respectively, with the content of the system’s request, this means that the analyzed sentence member accurately answers this question. Therefore, the specified result of processing the system’s request allows you to determine exactly which member of the sentence is the word contained in it. For example, if a sentence like Man is walking in a park or a Man is walking in a suit to analyze which sentence member (circumstance or addition) are phrases in a park or in a suit is analyzed , two system requests are generated. The first query contains a question word where? and the phrase in the park , because as a result of parsing, an inaccurate conclusion was made that the park is a circumstance of the place. In the second case, the following system request is generated: in what? - in a suit . If as a result of processing the system request a positive answer is given to each of them, this means that the first phrase is exactly a circumstance, and the second is an addition. If a system request is generated containing an erroneous statement (for example , where? - in a suit ), then the answer will be negative. This means that the phrase in the suit is not a circumstance.

The described method for generating queries to the first knowledge base of the semantic analysis system can be used in more complex cases of syntactic analysis of sentences. For example, when determining what type of circumstance is the participle turnover (participle), or when specifying the type of subordinate clause. For this purpose, a special request is generated containing the given participle turnover or subordinate clause, on the basis of which, up to synonyms, their analogues are searched on a set of knowledge of the "request-response" type. If the indicated analogs are contained in the group of response words of this database, then using the text index table they will be extracted from it. This will allow you to determine the question to which the determined adverbial turnover or subordinate clause corresponds and, therefore, accurately identify what type they belong to.

If the first knowledge base does not contain the requested analogues, then the second and third knowledge bases in combination with the inference subsystem (6) are used to accurately determine the sentence members. As noted above, the third knowledge base is composed of production rules, which allow using semantic analysis to clarify the names of sentence members, participles or types of subordinate clauses in complex sentences in order to formulate relevant questions for them.

One of the main options for conducting semantic analysis using this knowledge base is to translate, using production rules, semantic definitions specific to each sentence member into a set of phrases containing the defined word and some base word. This base word is semantically associated only with the given member of the sentence and unambiguously corresponds to it (cannot be used with other members of the sentence). When forming the word combination described in the production rules from the source text being analyzed, it is often necessary to carry out equivalent transformations of the source text based on the rules of the knowledge bases of morphological, syntactic analysis using inference.

After obtaining the required phrase, its validity is checked by referring to the second indexed text database, which allows the selection of paragraphs and individual sentences containing the required phrases. If on a set of text documents there is one or more sentences in which this phrase is used, then the relationship between the words of this phrase is valid. Therefore, it is believed that the word under investigation refers exactly to this member of the sentence.

Instead of separate phrases, more complex constructions can be used (for example, participial, participial sentences, subordinate clauses in complex sentences). Thus, a combination of semantic knowledge expressed by specific phrases, together with determining the admissibility of the relationship between words in them on a variety of text documents, will allow you to accurately determine the sentence members if their parsing does not give an accurate result.

After the morphological, syntactic and semantic analysis of sentences of this text document is completed, based on the frames of sentences received, the index table of the text is completely filled (Table 2), including the lists {I _ξi ^(s) } that determine the content of each table cell. After that, we move on to stochastic indexing of the next text on this topic. At the same time, automatic training is implemented and the knowledge base (14) of semantic analysis is filled in with production rules formed on the basis of the corresponding text fragments as described above using a stochastically indexed semantic structure (2). Note that the correctness of each rule is ensured by independent formation of the identical stochastically indexed semantic structures (2), described above, based on several fragments from the corresponding linguistic texts. Then, the stochastically indexed semantic structure is translated into the format of production rules, presented in the form "If (condition), then (conclusion)". This occurs in the interpreter (4) of stochastically indexed texts and production rules and in the subsystem (5) of equivalent text transformations.

After processing all the submitted text documents on this topic, a table of text indices is formed on this topic (Table 3). Its lines are named by the non-repeating indices {I _ξi ^(u) } of the stems of words included in text documents. The columns of this table correspond to the stochastic indices {I _ξi ^(t) } of texts that were processed in the course of grammatical and semantic analysis. The cells of this table contain indices {I _ξi ^(s) } of lists containing index indices of paragraphs {I _ξi ^(a) } of each text I _ξi ^(t) , which include the corresponding index I _ξi ^(u) of the word base. List entries are stored in a separate file, accessed by the corresponding indices {I _ξi ^(s) } .

After the formation of the indicated index tables and filling in the knowledge bases in the ISISS self-learning mode, according to the command of the subsystem (3) for managing the self-learning and knowledge extraction subsystem, they proceed to processing the user request in order to extract knowledge from text documents relevant to this request.

In this process, equivalent transformations of both a user's request and sentences of fragments of text are widely used when extracting knowledge from them. Let us consider in more detail the order of transformations of text sentences.

In ISIS, the following levels of equivalent text transformations are provided.

The first level of transformations is implemented within groups of proposal members - phrases containing subject, predicate, addition, circumstance. In this case, there is a change in the parts of speech in order to replace agreed definitions with inconsistent ones. Transformations of terms correspond to this level, for example: a computer network - a network of computers, computer service - computer maintenance .

The second level of transformations corresponds to equivalent transformations of sentence members within simple sentences, both independent and complex. In this case, the following types of replacement of sentence members using transformations of cognate parts of speech are realized:

the subject is replaced by the predicate,

predicate - on the subject,

addition to the subject,

predicate - on circumstance, etc.

In special cases, parts of speech may not change (only cases are changed).

The third level of equivalent transforms corresponds to the transformation within complex sentences. In this case, the subordinate clause of one type can be replaced by the subordinate clause of another type or by the participle, participial turnovers. Sometimes a compound sentence is converted into a simple sentence by replacing the conjunction with the corresponding prepositions defined by the rules.

Consider the example of equivalent transformations using the replacement of sentence members in phrases, namely: replacing the agreed definition with an inconsistent and direct addition with the subject. We choose the following sentence as the initial sentence: "Software and hardware protect computer programs." In the system, the initial sentence with the index I _ξ1 ^(p) will be represented by the stochastically indexed semantic structure below:

This structure contains the following phrases of the original sentence:

I _ξ12 ^(su) : = (software and hardware),

I _ξ13 ^(su) : = (protect),

I _ξ14 ^(su) : = (computer programs).

We perform the above equivalent transformations of the sentence members. In this case, the following phrases will be formed:

I _ξ22 ^(su) : = (computer program),

I _ξ23 ^(su) : = (defend),

I _ξ24 ^(su) : = (software and hardware).

As a result of these transformations, we get a sentence equivalent to the original sentence with the index I _ξ1 ^(p) , which has the index I _ξ2 ^(p) and the following stochastically indexed semantic structure:

Based on this structure, a sentence will be formed: "Computer programs are protected by software and hardware," which is equivalent to the original one. Note that in the new sentence, the subject I _ξ22 ^(su) corresponds to the phrase direct supplement I _ξ14 ^{(su) of the} original sentence, in which the agreed definition is replaced by the inconsistent one. Moreover, the subject of the first sentence I _ξ12 ^{(su) was} transformed into an indirect complement I _ξ24 ^(su) in the second sentence, and the predicate I _ξ13 ^(su) began to take the form of the reflexive verb I _ξ23 ^(su) . The indicated transformations are most often used both for equivalent transformations of stochastically indexed text sentences, and for user queries.

The user request is formed in a natural language. The user’s request is then converted into many new queries, including a question word and phrases that define the semantics of the query, equivalent to the original query. These equivalent conversions of the user's initial request are carried out using synonyms that are close in meaning to the words, as well as replacing parts of speech and sentence members. This ensures the preservation of the semantic content of the initial query based on the application of stochastically indexed rules of morphological, syntactic and semantic analysis to obtain equivalent phrase structures of the query query query and preserve the semantic connection between them.

After that, in accordance with the next converted user request, a preliminary selection of fragments of text documents is carried out, containing all the phrases of the request in total. If this request did not provide the possibility of preliminary selection of fragments of text documents that meet the specified requirements, then a new equivalent query conversion is performed.

Consider the order of processing the request and the response generation algorithm based on various text documents, paragraphs and sentences. After the next user request arrives at the linguistic processor (1), it is entered into the subsystem (2) of stochastic indexing and selection of text fragments, where stochastic indexes of word stems are formed and their endings are highlighted. After that, a stochastically indexed request through the subsystem (3) for controlling the self-learning and knowledge extraction mode is recorded in the inference subsystem (6). Here, on the basis of the rules for the production of knowledge bases (12-13), the user is first morphologically and syntactically parsed. Get the frame of the interrogative sentence. Then, in the interpreter (4), the interrogative sentence is represented as a combination of phrases containing the main and dependent words, and the corresponding stochastic indices of the word stems

where I _ξi ^(u) I _ξj ^(u) are stochastic indices, respectively, of the main and dependent word stems of a given phrase,

I _ξi ^(r) I _ξj ^(r) - stochastic indices of the parts of speech of the main and dependent words of a given phrase,

I _ξi ^(z) I _ξj ^(z) are stochastic indices of the morphological and syntactic characteristics of the parts of speech of the main and dependent words of a given phrase, respectively.

Based on the obtained indices, a stochastically indexed semantic query structure is formed, which in the general case has the following form:

where I _ξ1 ^(su) is the index of the interrogative phrase,

I _ξ2 ^(su) - index of phrases to be, I _ξ3 ^(su) - phrases index ^{_{predicate, I ξ2 (su) → I}} ξ3 (su) - predicative basis of a proposal that links the subject and ^{_{predicate, I ξ3 (su) → I}} ξ4 ( ^{su) the} relationship between the predicate and the complement (circumstance), which determines the type of relationship in this sentence, I _ξ4 ^(su) is the index of phrases of the complement (circumstances), I _ξ5 ^(su) is the index of phrases of the circumstance (additions).

Using the obtained indexes of expressions (6, 7), by accessing the database (10) of stochastically indexed texts on given topics using tables of text indexes on a given topic (Figure 4), a set of fragments is found that includes all query phrases, including the interrogative phrase. In addition, each piece of text may consist of one or more paragraphs.

If one or several texts that meet the specified conditions are found, then they proceed to further processing the paragraphs of these texts using the index tables of each of them. Note: the presence in the index table of one of the texts of the index I _ξ1 ^{(su) of the} interrogative phrase containing the index of the question (in the list I _ξi ^{(s) of} one of the cells in the table) and the index of the main word base associated with it, indicates that in the indicated the paragraph of this text is a sentence that contains the group of response words I _ξ0 ^(su) related to the main word of the interrogative phrase: (I _ξ0 ^(su) → I _ξ1 ^(su) ) .

If at least one of the texts that meets these conditions is not found, then they switch to equivalent transformations of the user's request by replacing words that are not included in the paragraph of the text with synonyms and words that are close in meaning, and also using replacing parts of speech and sentence members without changing the meaning of the request.

Further processing of the text that meets the above conditions is performed according to the index table of the text. To this end, using the indices of the interrogative phrase I _ξ1 ^(su) by looking at the table of indexes of text from the database (10), we find a sentence containing a group of response words that corresponds to the interrogative phrase of the query and is associated with the main word of this query. If collocations

queries in different paragraphs of different texts

then a necessary condition for the formation of a unified, logically related response text is the presence of at least one of the paragraphs of the group of response words I _ξ0 ^(su) corresponding to I _ξ1 ^{(su) of the} interrogative phrase of the query and the predicative basis I _ξ2 ^(su) → I _ξ3 ^{( su)} expressions (7), which generally includes the indices of phrases of the subject and predicate. If the specified condition is fulfilled, then the selected set of paragraphs is used during further processing, since on the basis of pre-selected paragraphs you can try to form a single, logically related response text. In the opposite case, it is necessary to move on to entering and indexing new texts on this topic.

We first consider a simpler case of the formation of a relevant answer, when a fragment of text containing all phrases of a query can be formed on the basis of one or more consecutive paragraphs of a given text. In this case, first form the basis of the stochastically indexed semantic structure of the user's response in the form of the following expression:

where I _ξ0 ^(su) is the index of the group of answer words, I _ξ1 ^(su) is the index of the interrogative phrase, I _ξ2 ^(su) is the index of the subject phrase, I _ξ3 ^(su) is the predicate phrase index, I _ξ2 ^(su) → I _ξ3 ^(su) is the predicative basis of a sentence . To this end, after defining a sentence in this fragment of the text, where the group of answer words associated with the main word of the interrogative phrase (I _ξ0 ^(su) → I _ξ1 ^(su) ) is indexed, they find a sentence that includes the predicative base (I _ξ2 ^(su) → I _ξ3 ^(su) ).

Since these groups of words generally occur in different expressions, for the formation of a semantic structural scheme (8), the logical inference procedure is implemented using indexed sentences of a given text fragment. For this purpose, a sentence with number i containing a group of response words is represented as follows:

where I _ξ0 ^(su) is the index of the group of response words, I _ξ1 ^(su) is the index of the interrogative phrase, I _ξ2i ^(su) is the index of the subject phrase, I _ξ3i ^(su) is the predicate phrase index, I _ξ2i ^(su) → I _ξ3i ^(su) is the predicative basis of the sentence , I _ξ3i ^(su) → I _ξ4i ^(su) is the relationship between the predicate and the complement (circumstance), which determines the type of relationship in this sentence, I _ξ4i ^(su) is the index of the phrases of the complement (circumstance), I _ξ5i ^(su) is the index of the combination of circumstances (additions).

To implement a logical conclusion on the basis of expression (9) using a transitive dependence, a stochastically indexed semantic structure of the topic → topic type of sentence with number i is formed :

where the topic is the index I _ξ2i ^{(su) of the} phrase of the subject, and Rhema is the index I _ξ4i ^{(su) of the} phrase of the supplement (circumstance).

Moreover, the sentence with number j , containing the predicative basis of the query, has in the general case the following stochastically indexed semantic structure:

where I _ξ2 ^(su) is the phrase index of the subject query, I _ξ3 ^(su) is the phrase index of the predicate query, I _ξ2 ^(su) → I _ξ3 ^(su) is the predicative basis of the query sentence , I _ξ3 ^(su) → I _ξ4j ^{(su )} is the relationship between the predicate and the complement (circumstance), which determines the type of relationship in this sentence with the number j , I _ξ4 ^(su) is the index of the phrases of the complement (circumstances), I _ξ5j ^(su) is the index of the phrase of the circumstances (additions). Then expression (11) is transformed into the following semantic structure topic → rema of the sentence with number j :

Note that in textual information between completed sentences there is a semantic, and therefore grammatical (syntactic) connection. There are two ways of structural correlation of sentences - syntactic connection between them. The first method can be called a chain (sequential), and the second - parallel connection.

Chain connection reflects the consistent development of thought in a coherent text. The theme is the starting point, the beginning of the movement of thought, "given", Rema - the development of thought, its basis, core, "new".

The syntactic nature of the chain link is expressed in the structural correlation of two neighboring sentences. Typically, a member of a previous sentence, such as an addition, in a subsequent sentence becomes subject. The most common structural types of chain links are "complement - subject", "complement - complement", "subject - complement", "subject - subject", etc.

The structural correlation between sentences in a chain connection is expressed: a) using lexical repetition (when the corresponding members of the sentences are expressed the same way); b) through synonymous vocabulary; c) using pronouns.

Chain communication is one of the most important and most common ways of connecting independent offers.

A parallel connection, like a chain one, consists in the structural correlation of the connected sentences. However, the nature of this correlation is different. The main structural features of the parallel connection of sentences: a) the parallelism of the structure (uniformity or syntactic proximity of the connected sentences); b) parallel (similar) word order; c) the same grammatical expression of all or some members of sentences.

The semantic “entry” into both a chain and parallel link structure of a paragraph is the topic of its initial sentence in the related sentences of this paragraph or several consecutive paragraphs of text.

In accordance with this, on the basis of the elementary semantic structure of each sentence of the topic → rem type, with the help of logical inference, more complex semantic structures can be formed that determine the relationships between sentences of both sequential and parallel types. Therefore, a necessary condition for the semantic connection between the group of response words contained in the sentence with the number i and the predicative basis of the query sentence, which is part of the sentence with the number j , is the proof using the logical conclusion of their occurrence in a single semantic structure of this fragment of the text. In a stochastically indexed form, this structure may look like this:

The logical conclusion for establishing a semantic connection between the indicated groups of words is produced according to the table of text indexes of the base (10) of stochastically indexed text documents on given topics. To this end, the inference subsystem (6) and the subsystem (5) of equivalent text transformations are used. In this case, the logical conclusion begins with the sentence with number i , containing the group of answer words, which is connected with the main word of the interrogative phrase, the predicative basis of the query, and has a stochastically indexed semantic structure (9).

After presenting the named sentence in the form of a semantic structure of the topic → reme type (10), the next sentence is found from the index table in which the current sentence goes into the topic of the next sentence. To do this, use cells that correspond to the index of this paragraph I _ξj ^(a) and the phrase index I _ξ4i ^(su) , which is the complement or circumstance of the sentence with number i . These cells find the sentence number of this paragraph, in which this phrase includes the subject. Then, using the address information of the cell, the predicate index of the indicated sentence and related addition indexes or circumstances I _ξ4k ^{(su) are found} , i.e. in accordance with expression (13), remu of the next sentence, logically related to the previous one, etc. The logical conclusion continues until the next sentence defined by the relation (I _ξ2 ^(su) → I _ξ4j ^(su) ) does not contain indices (I _ξ2 ^(su) → I _ξ3 ^(su) ) that correspond to the predicative basis request.

If during the logical conclusion the index of the _mode I _ξ4n ^{(su) of the} next sentence does not coincide with the topic I _{ξ2n + 1} ^{(su) of the} subsequent sentence, then this means that the subsequent sentence uses either a synonym for the word or a pronoun. In the first case, by the indices of the word stems I _{ξ2n + 1} ^{(su) of} this phrase, they refer to the table of indexes of the synonym dictionary of the base (8) of stochastically indexed linguistic texts. Here, the foundations of the synonyms {I _ξs ^(u) } are found , from which one can form the index I _ξ4n ^{(su) of the} rema of the previous sentence. In the second case, the index I _{ξ2n + 1} ^{(su) of the} topic of the next sentence may correspond to the pronoun consistent with the phrase I _ξ4n ^(su) , which is checked according to the index table of the database dictionary (7). When the first or second condition is fulfilled, the logical conclusion continues until a sentence containing the desired phrase of the query is found, in this case (I _ξ2 ^(su) → I _ξ3 ^(su) ) of the predicative basis of the query. Thus, in the course of inference, a stochastically indexed semantic structure will be synthesized, described by expression (13).

Since in the case under consideration all phrases of the query are included in one paragraph or in a group of consecutive paragraphs of the same text, the logical inference in this fragment of the text will continue with the aim of forming a single stochastically indexed semantic structure containing all phrases of the query, including phrases of the complement I _ξ4 ^(su) and circumstances I _ξ5 ^(su) :

For this purpose, the logical inference functions described above are implemented according to the theme → rem scheme until all phrase phrases included in various sentences of this paragraph are included in the semantic structure (14). Note that the necessary condition for the synthesis of the indicated semantic structure (14) is the correspondence of the phrases of the query and the identical phrases in the text of the paragraph to the same sentence members. Therefore, if some phrases identical to the phrases of the query in the sentences of the text relate to other members of the sentence, then these sentences are subjected to equivalent transformations so that these phrases refer to the required members of the sentences. These functions are performed as described above in the subsystem (5) of equivalent text transformations.

After the formation of the semantic structure (14), they proceed to control its consistency. To this end, check the semantic correspondence of the word combinations of the predicates {I _ξ3i ^(su) } included in each of the sentences, on the basis of which the semantic structure (14) is formed, to the basic relations. These include generic relations, relations of the type "part-whole" or "cause-effect" (condition-conclusion). These relations are determined by referring to the indicated indices to the base (8) of stochastically indexed texts to search for the semantic values of the predicates {I _ξ3i ^(su) } in the tables of explanatory dictionary indexes. In this case, the identity of the semantic values of the predicates {I _ξ3i ^(su) } with the indices of the above basic relations or their synonyms recorded in the interpreter (4) is checked. If these conditions are met, the transitive dependence is maintained in the formed semantic structure (14). Therefore, any desired phrase of a query with index I _ξj ^(su) can be transferred to the generated response sentence using logical inference on the formed semantic structure of the topic → rem type after a phrase with index I _ξj-1 ^(su) . If this condition is not met, then this paragraph does not contain an answer relevant to the user's request. In this case, they proceed to the analysis of the next pre-selected paragraph or set of consecutive paragraphs.

The described inference procedure for determining the semantic connection between query phrases when they are found in various sentences of a paragraph is performed until a short answer is formed to the user in the form of a sentence containing a group of response words, an interrogative phrase, a predicative base and all other phrases that are included in the answer. In this case, the generated short answer will be presented in the form of the following stochastically indexed semantic structure:

where I _ξ0 ^(su) is the index of the group of answer words, I _ξ1 ^(su) is the index of the interrogative phrase, I _ξ2 ^(su) is the index of the subject phrase, I _ξ3 ^(su) is the predicate phrase index, I _ξ2 ^(su) → I _ξ3 ^(su) is the predicative basis of the sentence , I _ξ3 ^(su) → I _ξ4 ^(su) is the relationship between the predicate and the complement (circumstance), which determines the type of relationship in this sentence, I _ξ4 ^(su) is the index of the phrases of the complement (circumstance), I _ξ5 ^(su) is the index of the combination of circumstances (additions).

At the same time, the correctness of the short answer is ensured by forming, as described above, several identical stochastically indexed semantic structures (15) based on various pre-selected stochastically indexed fragments of text documents.

The generated expression (15) means that as a result of the logical conclusion, a short answer is received, identical to the interrogative sentence of the request. Therefore, this answer is a relevant user request. It can be issued to the user after conversion to a text form in a given language in the form of knowledge generated by the system in accordance with his request.

If it is necessary to obtain a more complete answer at the request of the user, they proceed to transform the original paragraph of the text, on the basis of which a short answer is formed, and, if necessary, the subsequent paragraphs of the text. This is done in order to obtain, on the basis of the indicated paragraphs, a unified stochastically indexed semantic structure that gives the possibility of clarifying the short answer within the given text fragment. The described functions for generating a complete response will be presented below.

If, as a result of a preliminary search on the table of text indexes, texts containing paragraphs that include all phrases of the answer are not found, then the texts obtained by the query indexes are found, fragments of which together include all the phrase phrases. If such a collection is not found, then this means that the content of the database (10) of stochastically indexed text documents does not allow forming an answer relevant to the user's request. In this case, it is necessary to proceed to entering and indexing new texts on a given topic from a search engine.

In the preliminary selection process, using the table of text indices by phrase indices S: {I _ξi ^(u) → I _ξj ^(u) } of the query, fragments are selected for each text as a set of paragraphs containing all query phrases

V: = {I _ξi ^(t) , I _ξj ^(a) },

where I _ξi ^(t) , I _ξj ^(a) are the text index and paragraph index of the given text, respectively, containing certain phrases of the user's request. If the indices I _ξi ^(su) : {I _ξi ^(u) → I _ξj ^(u) } of the query phrases are not included in their entirety in any paragraph (I _ξi ^(t) , I _ξj ^(a) ) of at least one of the texts I _ξi ^(t) , but are contained in different paragraphs of the same text or in different paragraphs of different texts V: = {I _ξi ^(t) , I _ξj ^(a) } , then based on the pre-selected paragraphs of the text fragments, it is necessary to form a single logically connected text containing all query phrases

S: = {I _ξi ^(su) }, including the interrogative phrase.

If the phrases S: = {I _ξi ^(su) }, at the same time, are included in different paragraphs of different texts V: = {I _ξi ^(t) , I _ξj ^(a) } , then the necessary condition for the formation of a single, logically related response text is the presence of at least one of the paragraphs of the indices of the group of response words I _ξ0 ^(su) , the main word of the interrogative phrase I _ξ1 ^{(su) of the} query and the predicative basis (I _ξ2 ^(su) → I _ξ3 ^(su) ) of expression (15) into which in general, the indices of phrases of the subject and predicate are included.

If the specified condition is fulfilled, then the selected set of paragraphs is used during further processing, since on the basis of pre-selected paragraphs you can try to form a single, logically related response text. In the opposite case, it is necessary to move on to entering and indexing new texts on this topic.

When the specified conditions are met, they proceed to the formation of a logically connected totality of these paragraphs. To this end, verify the following condition: each phrase is included in at least two different paragraphs:

If this condition is not met, it is checked whether in the paragraphs containing only one phrase I _ξi ^(su) , the other phrase I _ξk ^(su) , which is contained in other pre-selected paragraphs and is associated with the phrase I _ξi ^(su), one of the basic semantic relationship. To check this position, the subsystem (3) for managing the self-learning and knowledge extraction mode generates a query to search for a proposal in the database (8) of stochastically indexed linguistic texts, which includes the indicated indices related by the topic → rem relation :

The found sentence goes to the interpreter (4) of the stochastically indexed text and production rules, where it is checked whether relation (16a) corresponds to the generic, aggregate, or causal relationship.

If conditions (16) or (16a) are not fulfilled, it is considered that this fragment of the text cannot be used to form the answer.

If these conditions are met, then we go on to test the possibility of forming a single semantic structure on the basis of the selected paragraphs. To this end, using the index table of each text containing pre-selected paragraphs, lists of word indexes are first formed. These phrase indices are included in the paragraph indicated by the corresponding index:

Then, it is determined which of the paragraphs each given paragraph is associated with using identical phrase indices in the lists of said paragraphs. Based on the lists for each paragraph index, new lists are compiled, each of which contains indices of other paragraphs associated with the same paragraph with identical phrase indices. If at the same time each of the lists contains at least one index of the paragraph included in at least one of the other lists, then using direct or transitive links between the lists, they form a single list, which includes indices of all paragraphs. In this case, it is believed that the pre-selected paragraphs form a logically related collection of paragraphs in the form of a single piece of text. In the opposite case, it is believed that this set of paragraphs does not form the logical structure necessary to form a single piece of text. At the same time, it is excluded from the processing process, and they proceed to the preliminary selection of new text fragments.

After determining that the pre-selected paragraphs form a single structure of logically related paragraphs, on the basis of the corresponding index tables of each text form a single text table. At the same time, the indicated paragraphs are arranged in the sequence determined by the sequence of the phrases of the query included in them in the interrogative sentence of the query. The resulting fragment of the text goes into further processing to determine, using logical inference, the form of semantic connections between paragraph sentences containing all the phrases {I _ξi ^(u) } of the query. The purpose of the implementation of these functions is an attempt to form a stochastically indexed semantic structure, including all phrases of the query, on the basis of the received fragment of text in accordance with the algorithm described above. Then, the obtained semantic structure using equivalent transformations and inference on transitive dependencies in accordance with the algorithm described above is used to form the semantic structure (15) of the sentence containing a short answer relevant to the user's request. At the same time, the correctness of the short answer is ensured by forming, as described above, several identical stochastically indexed semantic structures (15) based on various pre-selected stochastically indexed fragments of text documents.

The short answer obtained, together with the interrogative phrase, is then recorded in the knowledge base (9) "request-response", which is used to process repeated typical user requests, as well as, as described above, in the semantic analysis of indexed texts.

If, after the formation of the semantic structure, it turns out that between the phrases {I _ξi ^(su) } of the query in this fragment of the text the required basic semantic relationships are not supported, then we proceed to search for new texts to form the user's response.

In the case of a positive result of the logical conclusion, a proposal will be generated containing a short answer relevant to the request, for delivery to the user in text form in a given language. If at the same time the user demands to give him a more complete answer, then they proceed to the formation of a complete answer based on the conversion of the previously received fragment of text in accordance with the algorithm described below.

Consider, for example, the implementation order of the short answer generation algorithm described above. Suppose that after equivalent conversions of a user’s request, he took the following form in text expression: "What program is used when the computer is not completed correctly as a result of power failure?" This made it possible to preselect the next two logically related paragraphs from different text documents containing all the phrases of the converted query together. First paragraph:

" Logical errors can occur on the hard disk. Logical errors are violations in the file structure. The Disk Check program is used to detect logical errors. Logical errors occur when the computer is not completed correctly."

The second paragraph: "As a result of the loss of voltage in the network on the hard disk, violations occur in the file structure. In this case, the" Disk Check "program is used.

In a stochastically indexed form, in which the real process of processing the request and forming a short response takes place, the text of the request has the following form:

Moreover, the following phrases correspond to stochastic indices I _ξ0j ^(su) :

I _ξ01 ^(su) : = (which program),

I _ξ02 ^(su) : = (program),

I _ξ03 ^(su) : = (used),

I _ξ04 ^(su) : = (upon incorrect completion),

I _ξ05 ^(su) : = (work with computer),

I _ξ041 ^(su) : = (as a result of disappearance),

I _ξ051 ^(su) : = (mains voltage).

The sentences of the first paragraph in stochastically indexed form will be presented as follows:

Moreover, the following phrases correspond to stochastic indices I _ξij ^(su) :

I _ξ12 ^(su) : = (logical errors),

I _ξ13 ^(su) : = (may occur),

I _ξ14 ^(su) : = (on the hard disk),

I _ξ22 ^(su) : = (logical errors)

I _ξ23 ^(su) : = (- this),

I _ξ24 ^(su) : = (violations in the file structure),

I _ξ32 ^(su) : = (Disk Check program),

I _ξ33 ^(su) : = (used),

I _ξ34 ^(su) : = (for identification),

I _ξ35 ^(su) : = (logical errors),

I _ξ42 ^(su) : = (logical errors),

I _ξ43 ^(su) : = (arise),

I _ξ44 ^(su) : = (upon incorrect completion),

I _ξ45 ^(su) : = (work with the computer).

The sentences of the second paragraph in a stochastically indexed form will have the following form:

Moreover, the following phrases correspond to stochastic indices I _ξij ^(su) :

I _ξ52 ^(su) : = (file structure violations),

I _ξ53 ^(su) : = (arise),

I _ξ54 ^(su) : = (on the hard disk),

I _ξ55 ^(su) : = (as a result of disappearance),

I _ξ551 ^(su) : = (mains voltage),

I _ξ62 ^(su) : = (Disk Check program),

I _ξ63 ^(su) : = (used),

I _ξ64 ^(su) : = (for identification),

I _ξ65 ^(su) : = (logical errors),

I _ξ651 ^(su) : = (in this case).

Based on the above stochastically indexed semantic structures, the stochastically indexed semantic structure including all I _ξij ^(su) phrases of the query will be formed as described above. The structure I _ξ3 ^(p) , which includes the group of response words I _ξ32 ^(su) , corresponding to the interrogative phrase I _ξ31 ^{(su), was} chosen as the basis . This takes into account the identity (up to the basics of words) of the following phrase indices:

As a result, the indicated stochastically indexed structure will have the following form:

Considering the above-mentioned identity of the corresponding indices and the fact that the dependencies between the indices in this semantic structure are of a genus-specific and causal nature, we obtain the following structure using inference on transitive dependencies:

As a result, a stochastically indexed semantic structure of the short answer will be formed, which in the textual representation will look like this: "The Disk Check program is used when the computer is not completed properly due to a power failure . "

The received short answer after replacing the group of answer words "Program" Disk Check " with the corresponding interrogative phrase " Which program " will be identical to the query: " Which program is used when the computer is not completed correctly due to power failure? " . This is a criterion for the relevance of the received short answer to the request, therefore, the received short answer can be given to the user.

To formulate a complete answer based on a pre-selected paragraph or a received fragment of text, only those sentences that were involved in the inference during the formation of a short answer-sentence are selected. At the same time, from the sentences of the indicated paragraphs or text fragments, the sequences due to logical connections are built. The order of logical connections is the same as in the definition of semantic connectivity between phrases of a query. These phrases, which are part of different sentences, are associated with those query phrases that are in the sentence containing a group of response words and the main word of the interrogative phrase. The order of the sentence chains is determined by the order of the corresponding phrases of the request in the previously formed short sentence - the response to the user. In the process of forming a complete answer, to ensure coordination of sentences, they can produce equivalent transformations of individual sentences by replacing parts of speech or members of sentences without changing the semantic content of these sentences. If equivalent conversions of a sentence require replacing prepositions, then they are made taking into account what characteristics parts of speech should have when combined with specific prepositions. If necessary, to harmonize nouns or adjectives, pronouns or participles with new prepositions, they can replace the cases of these parts of speech. To do this, use the appropriate rules connecting the preposition with cases in which these parts of speech are consistent with this preposition.

If the question word or phrase of the request (how? How?) Does not imply a short answer in one sentence, but a representation of a sequence of actions or descriptions of a process or phenomenon, in this case the short answer can be a sentence-sentence containing a group of response words of the type : "as follows," "in this way." In this case, the following response sentences disclose the contents of the sequence of actions or descriptions containing the answer to the user with the required completeness. In the absence of such a typical group of response words, it can be introduced in addition to form a sentence-reason. After this, the group of response words in the sentence-sentence is accepted as the initial topic of the future complete answer. Then, using a logical conclusion, a sequence of sentences of one or several paragraphs is selected, which form a set of semantically related sentences of a complete answer to this user's question. At the same time, the boundaries of the answer will be determined by a continuous chain of logically related sentences, which ends when one of the paragraphs ends, if the topic of the last sentence of this paragraph is not connected with the first sentence of the next paragraph. After the formation of a fragment of text containing a complete answer, including a sentence-sentence, it is issued to the user.

The developed method can be used to synthesize a self-learning system for extracting knowledge from text documents of search engines in a given foreign language. The system is automatically trained in the rules of morphological, syntactic and semantic analysis as described above using stochastically indexed linguistic texts in a given foreign language. The obtained rules, also presented in a given foreign language, are stochastically indexed and recorded in the corresponding knowledge bases (12-14) of morphological, syntactic and semantic analysis. At the same time, the database (7) of stochastically indexed dictionaries of the base and new words is filled, as well as the databases (10) of stochastically indexed text documents on given topics in this foreign language.

After filling in the indicated databases and knowledge as described above, the user requests are converted in this foreign language, and the preliminary selection of fragments of text documents on relevant topics is performed. Then, equivalent transformations of these fragments of text documents are carried out, the formation of stochastically indexed semantic structures and logical inference using these structures to form a short answer relevant to the request in a given foreign language.

The developed method can also be used to synthesize a self-learning system for extracting knowledge from text documents of search engines in any of a variety of specified foreign languages. For this purpose, the self-learning mechanism described above is used in the form of a stochastically indexed artificial intelligence system based on the use of unique combinations of binary signals of stochastic information indices for stochastic indexing and searching for fragments of linguistic texts in a given base language containing a description of grammatical and semantic analysis procedures. This mechanism provides automatic training of the system for the rules of grammatical and semantic analysis by equivalent transformations of stochastically indexed text fragments in any of the given foreign languages, inference and formation of related semantic structures from these text fragments, their stochastic indexing for presentation in the format of production rules.

First, using the mechanism described above, morphological analysis and stochastic indexing of linguistic texts in a given basic language in electronic form are performed with simultaneous automatic training of the system in the rules of morphological analysis. This is carried out simultaneously with the formation of a database (7) of stochastically indexed dictionaries and the formation of index tables of linguistic texts of the database (8) for each of the given foreign languages, as well as the knowledge base (12) of the morphological analysis containing the obtained production rules for the given base language and each from the given foreign languages.

After that, morphological and syntactic analysis is performed, as well as stochastic indexing of text documents on a given topic in each of the given foreign languages after receiving them in electronic form from a search system. In this case, tables of indexes of text documents are generated on a given topic and written to the database (10) of stochastically indexed texts with simultaneous automatic training of the system in the rules of parsing. The specified training is carried out as described above using stochastically indexed linguistic texts in a given base language. In this case, the formation of the knowledge base (13) of parsing for the base language and each of the specified foreign languages is carried out.

Then, a semantic analysis of stochastically indexed text documents is carried out on a given topic in a given base language in electronic form with simultaneous automatic training of the system for the rules of semantic analysis and the formation of a semantic analysis knowledge base (14) for the base language and each of the specified foreign languages.

After filling in the knowledge base (11-12), the system switches from automatic learning mode to user request processing mode. At the same time, the user’s request is formed in a naturally defined foreign language and presented in electronic form after stochastic indexing in the form of an interrogative sentence, including an interrogative phrase and phrases that determine the semantics of the request. After that, as described above, the user's request is converted in a stochastically indexed form into many new queries that are equivalent to the original query in a given foreign language. Then, in accordance with the user’s request, a preliminary selection of stochastically indexed fragments of text documents in a given foreign language in electronic form, containing together all the phrases of the converted request, is carried out. Using the indicated fragments of text documents, a stochastically indexed semantic structure is formed. Based on the generated stochastically indexed semantic structure with the help of logical inference, ensuring the connection of stochastically indexed elements of various texts, and equivalent text conversion, a short system response is formed containing phrases in stochastically indexed form that determine the query semantics, as well as a group of response words corresponding to the interrogative phrase request. This ensures the correctness of the short answer by forming several identical stochastically indexed semantic structures based on different, pre-selected stochastically indexed fragments of text documents.

Then, the relevance of the short answer of the system to the request is checked by replacing the group of response words with the corresponding interrogative phrase in a stochastically indexed form, obtaining a stochastically indexed interrogative sentence, comparing the received interrogative sentence with the query. Based on a comparison of the indicated sentences with the identity of the received interrogative sentence and the request, a decision is made on the relevance of the short response of the system to the request and presented in a given foreign language.

Let us now consider a different order of application of this method for the synthesis of a self-learning system that provides simultaneous extraction of knowledge from text documents in any of the given foreign languages. In this case, first, the system is automatically trained in the order described above for the rules of morphological, syntactic and semantic analysis using stochastically indexed linguistic texts in a given base language. Moreover, the base (8) of stochastically indexed linguistic texts includes teaching aids for studying each of the given foreign languages in the selected base language. Dictionaries are written in the database (11) of stochastically indexed dictionaries of foreign words, providing direct and reverse translation of individual words from the base language into any of the specified foreign languages. Then, a database (7) of a stochastically indexed dictionary and knowledge bases (12-14) of morphological, syntactic, semantic analysis in a given base language are formed. After that, the control subsystem (3) of the automatic learning mode automatically generates queries to the indicated databases and knowledge for preliminary selection of fragments of linguistic texts in the base language containing the knowledge necessary for studying each of the given foreign languages. Then, equivalent text transformations are made, stochastically indexed semantic structures are formed, and logical inference is made on the given structures to form answers relevant to automatic queries. These answers are used to formulate production rules for morphological, syntactic and semantic analysis of text documents for each foreign language. For example, if the base language is Russian, then when creating the knowledge base of parsing for learning English, among the automatically generated rules there may be the following:

1. If a noun without a preposition is at the beginning of a sentence,

and this noun faces the noun with the pretext of (in, from),

and this noun is followed by the verb

then the first noun is the subject.

For example: The work of the engineer is on the table.

2. If the phrase consists of a bunch verb (the verb to be in personal form) and a nominal part expressed by an adjective,

then this phrase is a compound nominal predicate.

For example: The tree is big.

The rules obtained after stochastic indexing are recorded in the knowledge base (12-14) of morphological, syntactic and semantic analysis to ensure the extraction of knowledge from text documents in a given foreign language in accordance with user requests. Moreover, the formation of a database of stochastically indexed dictionaries and tables of indexed text documents on given topics is performed using the corresponding foreign language. Note that in the process of semantic analysis of text documents on given topics in the corresponding foreign language to determine the type of semantic relation, individual phrases are translated using the base (11) of stochastically indexed dictionaries of foreign words into the base language. The specified phrase with the help of logical inference on the tables of indexes of explanatory dictionaries in the base language is correlated with one of the types of semantic relations whose indices are written in the interpreter (4) of stochastically indexed texts and production rules. This allows you to use semantic analysis to clarify the above-described order of word belonging to the sentence members, as well as to determine the type of relationship between phrases in the formation of a stochastically indexed semantic structure of the response to the request.

Using the indicated databases and knowledge of the commands of the subsystem (3), the self-learning and knowledge management regimes perform equivalent conversion of user requests in given foreign languages. Then a preliminary selection of fragments of text documents is carried out on given topics, their equivalent transformations, the formation of stochastically indexed semantic structures and logical inference on these structures. This ensures the formation of answers relevant to user requests in each of the number of specified foreign languages.

If, when processing a request, it turns out that a search engine is needed to enter new text documents in one of the foreign languages on a given topic, then the subsystem (3) for managing the self-learning and knowledge extraction mode connects a multilingual linguistic processor (1). It receives a command to enter new documents indicating the topic and the name of the language, which are presented in the base language. The multilingual linguistic processor (1), using the database (11) of stochastically indexed dictionaries of foreign words, selects the appropriate dictionary and translates the words denoting the topic name into the corresponding foreign language. According to the information received, the multilingual linguistic processor (1) generates a formalized request in a given language to a search system for entering new documents in a foreign language on a relevant topic. The indicated documents enter the subsystem (2) of stochastic indexing of text documents and extract fragments of texts for the processing described above and input into the database (10) of stochastically indexed text documents on given topics.

Industrial applicability

A method for synthesizing a self-learning system for extracting knowledge from text documents of search engines can primarily be used to create an Internet-based global knowledge industry using multilingual systems for extracting knowledge from texts. This will provide a qualitatively new information service in various fields - industrial, scientific, educational, cultural and domestic human activities, taking into account the modern requirements of the development of a civilized society. Another promising area of industrial application of this method are mobile systems (mobile Internet). This is due to the possibility of creating intelligent information retrieval systems that extract specific knowledge and information from large volumes of Internet text documents at the request of users with minimizing the time for transmission and perception by the user of the information he needs. In this case, requests can be entered by the user into the system in natural language and in speech form. An important area of industrial application of the proposed method is the creation of a new generation of intelligent teaching systems in various subjects and problem areas.

Table 1
Frame suggestions Questions for simple sentences Questions for simple sentences are formed on the basis of the syntactic analysis knowledge base The name of simple sentences in the composition of complex or complex subject Characteristics of simple sentences Questions to offer member groups Questions to the groups of members of the proposal are formed on the basis of questions to the members of the proposal, which are the basis of this group Names of proposal member groups Subject Group
predicate group
supplement group
group of circumstances
a group of separate members of the proposal,
group of introductory words, phrases and plug-ins Questions to members of the proposal According to the format of the dictionary (including prepositions) and the table for translating questions to parts of speech into questions to sentence members Names of proposal members Subject
predicate (simple verb, compound verb, compound noun),
definition (agreed, inconsistent),
addition (direct, indirect),
circumstance (mode of action, place, time, measure or degree, reason, purpose, condition, assignment) Questions for parts of speech By dictionary format Parts of speech and their characteristics By dictionary format Word In the context of the proposal Stochastic indices of word stems Calculated by a special algorithm or extracted from the dictionary format

table 2
Text indices Basic Word Indexes Paragraph Indices I _ξ1 ^(a) I _ξ2 ^(a) ... I _ξn ^(a) I _ξ1 ^(u) I _ξ11 ^(s) I _ξ12 ^(s) ... I _ξ1n ^(s) I _ξ2 ^(u) I _ξ21 ^(s) I _ξ22 ^(s) ... I _ξ2n ^(s) ... ... ... ... ... I _ξm ^(u) I _ξm1 ^(s) I _ξm2 ^(s) ... I _ξmn ^(s)

Table 3
Indexes of texts on this topic Basic Word Indexes Text Indexes I _ξ1 ^(t) I _ξ2 ^(t) ... I _ξn ^(t) I _ξ1 ^(u) I _ξ11 ^(s) I _ξ12 ^(s) ... I _ξ1n ^(s) I _ξ2 ^(u) I _ξ21 ^(s) I _ξ22 ^(s) ... I _ξ2n ^(s) ... ... ... ... ... I _ξm ^(u) I _ξm1 ^(s) I _ξm2 ^(s) ... I _ξmn ^(s)

Claims (20)

1. A method for synthesizing a self-learning system for extracting knowledge in a given language from text documents of search engines, in which: provide a self-learning mechanism in the form of a stochastically indexed artificial intelligence system based on the use of unique binary combinations of stochastic information indices, provide automatic training of the system for the rules of grammatical and semantic analysis by using equivalent transformations of stochastically indexed text fragments, inference and the formation of related semantic structures and stochastic indexing for presentation in the format of production rules, perform morphological analysis and stochastic indexing of linguistic texts in electronic form with simultaneous automatic training of the system to the rules of morphological analysis, perform morphological and syntactic analysis, as well as stochastic indexing of text documents on a given topic in electronic form in a given language with simultaneous automatic training of the system in the rules of parsing, perform semantic analysis of stochastically indexed text documents on a given topic in electronic form with simultaneous automatic training of the system in the rules of semantic analysis, form a user request in a natural given language and submit it in electronic form after stochastic indexing in the form of an interrogative sentence, Convert a user’s request in stochastically indexed form into many new queries equivalent to the original query, in accordance with the user's request, a preliminary selection is made of stochastically indexed fragments of text documents in electronic form, containing all the phrases of the converted request in aggregate, form a stochastically indexed semantic structure using the indicated fragments of text documents, on the basis of this structure, using a logical conclusion that ensures the connection of stochastically indexed elements of various texts, and equivalent text conversion form a short response of the system, check the relevance of the received short response of the system to the request by forming on its basis an interrogative sentence, comparing the received interrogative sentence with the request, if the question and offer received are identical, they decide on the relevance of the short response of the system to the request and present it in a given language. 2. A method for synthesizing a self-learning system for extracting knowledge in any of the given foreign languages from text documents of search engines, in which: provide a self-learning mechanism in the form of a stochastically indexed artificial intelligence system based on the use of unique combinations of binary signals of stochastic information indices for stochastic indexing and searching for fragments of linguistic texts in a given base language containing a description of grammatical and semantic analysis procedures and automatic training of the system in grammar and semantic rules analysis by equivalent transforms are stochastically indexed fragments of text, inference, and the formation of their associated semantic structures of the stochastic index for submission to the rules of output formats, perform morphological analysis and stochastic indexing of linguistic texts in a given base language in electronic form with simultaneous automatic training of the system for the rules of morphological analysis, the formation of a database of stochastically indexed dictionaries and the formation of index tables of linguistic texts for each of the specified foreign languages, as well as the knowledge base of morphological analysis, containing the resulting production rules for a given base language and each of the specified foreign x languages perform morphological and syntactic analysis, as well as stochastic indexing of text documents on a given topic in each of the given foreign languages in electronic form from a search system with their presentation in the form of tables of indexes of text documents on a given topic and writing to the database of stochastically indexed texts with simultaneous automatic learning system rules for parsing using stochastically indexed linguistic texts in a given base language and form vaniem knowledge base to parse the language and each of the specified languages, perform semantic analysis of stochastically indexed text documents on a given topic in a given base language in electronic form with simultaneous automatic training of the system for the rules of semantic analysis and the formation of a semantic analysis knowledge base for the base language and each of the specified foreign languages, form a user’s request in a natural predetermined foreign language and submit it in electronic form after stochastic indexing in the form of an interrogative sentence, including an interrogative phrase and phrases that determine the semantics of the request, convert the user’s request in stochastically indexed form into many new queries equivalent to the original query in a given foreign language, in accordance with the user's request, a preliminary selection of stochastically indexed fragments of text documents in a given foreign language in electronic form, containing all the phrases of the converted request in total, is performed, form a stochastically indexed semantic structure based on the indicated fragments of text documents, on the basis of the generated stochastically indexed semantic structure with the help of logical inference, providing the connection of stochastically indexed elements of various texts, and equivalent text conversion, a short system response is formed containing phrases in stochastically indexed form that determine the semantics of the query, as well as a group of response words corresponding to the interrogative phrase request the relevance of the short answer of the system to the query is checked by replacing the group of response words with the corresponding interrogative phrase in a stochastically indexed form, obtaining a stochastically indexed interrogative sentence, comparing the received interrogative sentence with the query and if the received interrogative sentence and query are identical, they decide on the relevance of the short answer of the system to the query and represent it in a given foreign language. 3. The method according to claim 1, characterized in that in case of an unsuccessful attempt to generate an interrogative sentence identical to the user's request, they request new text documents from the search engine to search for an answer relevant to the user's request, 4. The method according to claim 1, characterized in that, in addition, at the request of the user, a complete answer is formed containing more detailed information or a set of specific knowledge, while using logical inference to form a stochastically indexed semantic structure and the necessary equivalent transformations of the specified set of text fragments to obtain a stochastically indexed new text revealing with possible detail the content of the previously received short answer. 5. The method according to claim 1, characterized in that the system is automatically taught the rules of morphological analysis by highlighting in a stochastically indexed text a specific set of word forms for each word, obtaining stochastic indices of the word base and a given set of its endings or prepositions, random access to the indicated indices stochastically indexed linguistic texts, extracting fragments from them, connecting the specified set of word endings or prepositions with the corresponding part of the word speech, as well as with a full set of endings or prepositions obtained by declension or conjugation, converting these fragments into the format of production rules by stochastic indexing, while ensuring the correctness of each rule by independently forming it on the basis of several fragments from the corresponding linguistic texts, and obtaining a table of indexes of production rules for the knowledge base of morphological analysis. 6. The method according to claim 5, characterized in that when stochastically indexing linguistic texts after determining part of the speech of each word using the rules of the knowledge base of morphological analysis, the database of the stochastically indexed dictionary is filled with stochastic indices of the basis of each word and a complete set of its endings or prepositions. 7. The method according to claim 6, characterized in that for the formation of the text index tables, the information is stochastically converted and unique binary combinations of word base indices, their endings, prepositions, sentences, paragraphs and text names are placed that are stochastically indexed in the index tables texts with ensuring the connectivity between the indicated indices, defined in the source text and ensuring its recovery from the index table. 8. The method according to claim 1, characterized in that the system is automatically taught the rules of parsing by searching in stochastically indexed linguistic texts of fragments describing the order of parsing sentences, while the logical conclusion is realized to obtain a stochastically indexed semantic structure that determines the relationship of syntactic elements and structures with given parts of speech of words, and the formation of production rules that determine the parsing of sentences by fologicheskim characteristics words, while ensuring the correctness of each independent rule by its formation on the basis of several fragments from the respective linguistic texts received rules are entered into the knowledge base parsing, as filling which is exercised by the stochastic index, and represented as index table. 9. The method according to claim 1, characterized in that the system is automatically taught the rules of semantic analysis of the text by generating a query on the index tables of linguistic texts on stochastic indices of the stems of words and parts of speech, not precisely defined members of the sentence, and receiving the answer in the form of a text fragment describing the semantic characteristics that words must have in order to match this particular member of the sentence, and according to the answer received, using the stochastic index of the basis of this words and the required semantic characteristics refer to tables of indexes for explanatory dictionaries and general and thematic encyclopedias, while using logical inference they try to form a stochastically indexed semantic structure linking the given word and the required semantic characteristics, in the positive case, consider that the specified sentence member defined exactly, and the text fragment relevant to the request is converted into a production rule, while ensuring the correctness of each rule by independently forming it on the basis of several fragments from the corresponding linguistic texts, which are included in the knowledge base of semantic analysis, stochastically index this database, present it in the form of an index table and apply it in the semantic analysis of words as members of a sentence and relations between words expressed by phrases. 10. The method according to claim 9, characterized in that after the formation of the index table of each text and the completion of its morphological, syntactic and semantic analysis, stochastic indices of the names of the parts of speech, sentence members and questions to them that correspond to each word in the sentence are formed, and they record the indicated indices in the cells of the index table of the given text, which allows you to automatically determine which part of speech, each word belongs to the sentence, when searching for fragments of the text, and form millet to him. 11. The method according to claim 10, characterized in that after receiving all the text index tables, a text index table is formed on a given topic, the rows of which are named by non-repeating stochastic indexes of word stems, and each column corresponds to a stochastic index of a specific text, while in the table cells are written stochastic index indices of paragraphs in which the text contains the word with the corresponding index of the basis, the resulting index table on this topic is used to preliminary search for fragments containing aschih certain set phrases request. 12. The method according to claim 11, characterized in that the equivalent transformations of the user's initial request are carried out using synonyms that are close in meaning of the words, as well as replacing parts of speech and sentence members while maintaining the semantic content of the initial request based on the application of stochastically indexed morphological, syntactic rules and semantic analysis to obtain equivalent collocation structures of the interrogative sentence of the query and preserve the semantic connection between them. 13. The method according to p. 12, characterized in that the set of semantically related text fragments containing all the words of the user's query is formed by referring to the table of text indices for a given topic by stochastic indices of the indicated word strings, selecting stochastic paragraph indices and the texts corresponding to them, containing in aggregate all phrases of the query, accessing the indicated indices to the index table of each of the selected texts, logical inference on the index tables and equivalent text transformations in order to form a stochastically indexed semantic structure linking the indices of the response word group corresponding to the interrogative phrase, as well as all query phrases that determine the semantics of the query and are included in pre-selected paragraphs. 14. The method according to p. 13, characterized in that the stochastically indexed semantic structure corresponding to the user’s request that is successfully generated during the logical inference process is taken as the basis for generating, using the resulting set of fragments of text, an interrogative sentence identical to the user's request, which is formed by the equivalent transform stochastic indices of the stems of query words and their endings using the rules of knowledge bases to provide the required semantic characteristics of each word combination of the text fragment that is part of the query, as well as using logical inference on transitive dependencies between word combinations to combine them into a single interrogative sentence identical to the user's query, which contains a group of response words corresponding to the interrogative phrase. 15. The method according to 14, characterized in that the correctness of the short answer is ensured by the formation of several identical stochastically indexed semantic structures of the response based on various pre-selected stochastically indexed fragments of text documents. 16. The method according to p. 15, characterized in that in the process of searching and generating an answer using tables of indexes of text documents, self-learning of the system is carried out by forming indexed text elements linking the request and the relevant short answer to obtain a knowledge base containing elements of the type "request - the answer ", which is stochastically indexed, presented in the form of a table of indices and is used in the grammatical and semantic analysis of sentences of the text, as well as in the formation of answers to repeated user surveys contained in the specified indexed knowledge base. 17. The method according to clause 16, characterized in that for the formation of a complete response containing knowledge relevant to the user's request, based on a short answer using logical inference on the index tables used to obtain the text fragment, form a stochastically indexed semantic structure that connects the group response words with stochastic indices of the stems of sentence words that support transitive dependence, which together provide a complete disclosure of the content of the short answer in the framework of ovannogo text fragment and then with equivalent transformations proposals on the basis of said stochastically indexed semantic structure obtained single associated text complete response. 18. The method according to 17, characterized in that the equivalent conversion of stochastically indexed text fragments is performed by presenting each sentence as a set of stochastically indexed phrases that are converted using the rules of knowledge bases of morphological, syntactic and semantic analysis by equivalent conversion of stochastic indices of the fundamentals of the same root words, their endings and prepositions for the formation of new parts of speech or members of the sentence with the provision of it mennosti regard these phrases as part of a stochastically indexed semantic structure of each sentence and harmonization of these proposals with each other in the formation of these new text fragment. 19. The method according to p. 18, characterized in that when a new word appears in the indexed text during stochastic indexing of text documents that is not contained in the dictionary of stochastically indexed words and in linguistic texts, the root word with the specified new word is found in this dictionary, and in the knowledge base of morphological analysis find the rules for the equivalent conversion of the same root word found in the dictionary into a new word, while by the type of equivalent conversion the part of speech to which the new word belongs to and all its word forms obtained by declension or conjugation, and if there are no cognate words in the dictionary, a certain set of word forms of the new word is selected from the text, based on prepositions or endings using the stochastically indexed dictionary or production rules of morphological analysis, they determine the part of speech, to which it refers, and a complete set of its word forms obtained by declension or conjugation. 20. The method according to claim 19, characterized in that for the simultaneous extraction of knowledge from text documents in predetermined foreign languages, the system is first automatically trained in the rules of morphological, syntactic, semantic analysis for a given base language, the database is formed stochastically indexed dictionary and morphological knowledge bases , syntactic, semantic analysis using stochastically indexed linguistic texts in a given base language, using of these databases, they automatically generate queries for automatically learning the system in any of the given foreign languages, while preliminary selection of fragments of linguistic texts in the base language containing the knowledge necessary for learning a given foreign language, equivalent conversions of these texts, and the formation of stochastically indexed ones semantic structures and logical inference on given structures for the formation of answers, elevantnyh automatic queries that used to form the knowledge base of the morphological, syntactic and semantic analysis for any of the specified foreign languages, providing knowledge extraction from text documents in a given foreign language.