CN111813907A - A Question Intention Recognition Method in Natural Language Question Answering Technology - Google Patents

Abstract

The patent of the present invention proposes a question sentence intent recognition method in natural language question answering technology, the method includes: an end-to-end executable correction question generation model, the model includes four input layer, encoding layer, matching layer and decoding layer. section, generating revised questions by combining sentence generation, machine reading comprehension, and replication mechanisms; and a method for automatically generating training data for article compression. The method of the invention can improve the answering accuracy of machine reading when the input question of the existing question answering system is too short to recognize the answer, and at the same time, even if the input question is long enough, the candidate set of correction questions can be selected from the candidate set. Questions that match the questioner's intent to obtain high-precision answers in question answering systems.

Description

A Question Intention Recognition Method in Natural Language Question Answering Technology

technical field

The invention discloses a question sentence intent recognition method in natural language question answering technology, which relates to the fields of natural language processing, question answering system and artificial intelligence.

Background technique

With the development of the mobile Internet, voice devices such as smartphones, voice robots, and smart speakers continue to popularize, and there is an increasing demand for question-and-answer technology based on artificial intelligence to answer user questions. In recent years, question answering based on machine reading comprehension has attracted special attention. Machine reading comprehension is a question-and-answer technique that enables systems to read and understand natural language documents or paragraphs, and to find and extract information for answers from paragraphs, so as to be able to Answer questions.

Although the current question answering based on machine reading comprehension can achieve high answering accuracy, there are still problems in practical use. In the actual question answering system, when the questioner inputs ambiguous questions or short questions with insufficient information required by the system, it is difficult to achieve high answer accuracy. For example, in the actual scene, the interaction between the customer service machine and the questioner, a questioner who is not familiar with the relevant business knowledge sometimes asks the customer service machine some questions with unclear intentions. Therefore, it is necessary to introduce a mechanism with speculation and recognition to confirm the questioner's intention into the current question answering system based on machine reading to achieve accurate answers to vague questions.

SUMMARY OF THE INVENTION

In order to overcome the above-mentioned shortcomings of the existing machine reading comprehension question answering technology, the purpose of the present invention is to provide a question sentence intent recognition method in the natural language question answering technology.

For achieving the above object, the present invention adopts the following technical scheme:

A question sentence intent recognition method in natural language question answering technology, the method adopts an end-to-end executable correction question generation model, the model combines sentence generation model, machine reading model and replication mechanism, the method includes the following step:

Step 1, when the input question and its corresponding question paragraph are input into the generative model, the model will use the machine reading model to read and extract the relevant information in the content of the input question and paragraph;

Step 2, the model is based on the attention observation and supervision mechanism, and the sentence generation model generates a revised set of questions while extracting information;

Step 3, then, the copy mechanism will copy the input question and the important words and expressions in the paragraph into the revised set of questions, so as to generate the revised final question according to the content of the paragraph;

The structure of the correction problem generation model includes the following four parts:

Input layer: The input question sentence and question paragraph are represented as a sequence vector model of one-hot encoding;

Encoding layer: use the word embedding model to convert the sequence of paragraph tokens and question sentences of the input layer into vectors of continuous values, and at the same time consider the contextual relationship between paragraphs and question sentences to create a vector sequence;

Matching layer: captures the relationship between the words in the paragraph and the input question sentence, and models the vector sequence of the paragraph;

Decoding layer: Generates a sequence of word tokens that constitute the correction problem using a GRU-RNN gated recurrent neural network with an attention mechanism and two replication mechanisms.

Further, in the decoding layer, the GRU-RNN gated recurrent unit recurrent neural network structure is as follows:

Activation function: PReLu is selected as the activation function in GRU-RNN. This function is used as a modified linear unit, which can change the slope of all negative values in the input vector according to the specific training data, and the remaining values remain unchanged. Increase the contrast and improve model learning efficiency;

Neuron selection: Gated recurrent unit GRU is a kind of neuron in RNN. In the hidden layer, two gate structures are used to control the retention of information, which can filter out important and unimportant information;

Other parameters: The model is composed of a layer of GRU with an attention mechanism and a softmax layer, adaptive gradient descent, and a learning rate of 0.0007.

Furthermore, a method of automatically generating training data by paragraph sentence compression is adopted. Through sentence compression, short question sentences with ambiguous intentions are generated from the question bank of machine reading corpus, so as to automatically correct the questioning model.

The beneficial effects of the present invention are: introducing the mechanism with the intention of the questioner into the current question answering system based on machine reading, it can improve the answering accuracy of machine reading when the input question of the existing question answering system is too short to recognize the answer, and at the same time, even if When the input question is long enough, by giving a candidate set of correction questions, the question that matches the intention of the questioner's question can be selected from the candidate set, so that a higher-precision answer can be obtained in the question answering system.

Description of drawings

Figure 1 is a schematic diagram of the structure of a machine reading question answering system incorporating a corrected question generation model;

2 is a schematic structural diagram of a problem-correcting generation model of the present invention;

Figure 3 is the core structure diagram of the residual highway network;

Figure 4 is an expanded view of the single-layer structure of GRU-RNN.

FIG. 5 is a basic structural diagram of a GRU unit to which the present invention belongs

Detailed ways

It should be noted that the following descriptions are all exemplary and are intended to provide further explanation for the present application. Unless otherwise specified, all scientific and technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

In addition, it should be noted that the terms used here are only used to describe the specific embodiments, but not intended to limit the exemplary embodiments according to the present application. The present invention will be further described below with reference to the accompanying drawings and examples.

Referring to Figures 1 to 4, a method for recognizing the intent of a question sentence in a natural language question answering technology, the method adopts an end-to-end executable correction question generation model, which combines a paragraph generation model, a machine reading model and a copy The task of the model is to take the input question and the corresponding paragraph as input and output the corrected question with the relevant intention. The questioner can select the corrected question that conforms to his correct intention from a plurality of corrected question candidate sets. The method includes the following steps :

Step 1, as shown in Figure 1, when the input question and its corresponding question paragraph are input into the generation model, the model will use the machine reading model to read and extract the relevant answer information in the content of the input question and paragraph;

The model takes the word token sequence of the input question and the word token sequence of the paragraph as input, and estimates the possible starting and ending positions of the answer from the paragraph. The output is a word token sequence with the estimated answer range extracted from the paragraph. . The input question sentence is firstly subjected to word segmentation and part-of-speech tagging through morphological and grammatical analysis, and is expressed as a one-hot vector encoding sequence q={q ₁ , q ₂ ,...q _j }, and the one-hot vector is a V-dimensional vector, which is the same as The dimension value corresponding to the index of the word dictionary is 1, and the other dimension values are 0; the paragraph processing is similar to the input problem, the number of words in the paragraph is large, and a one-hot vector x={x ₁ ,x ₂ ,...,x _T }, the paragraph should contain relevant information that will answer the question.

Step 2, the model is based on the attention observation supervision mechanism, while extracting information, the paragraph generation model generates a revised set of questions;

In step 3, the copying mechanism will then copy the input question and the important words and expressions in the paragraph into the revised set of questions, thereby generating a revised final question based on the content of the paragraph. The revised final question is a sentence in which the content of the input question is embodied as y={y ₁ ,y ₂ ,...,y _K }.

Further, as shown in Fig. 2, the model structure for generating the correction problem specifically includes:

Input layer: The input question sentence and question paragraph are represented as a sequence vector model of one-hot encoding;

Encoding layer: use the word embedding model to convert the sequence of paragraph tokens and question sentences of the input layer into vectors of continuous values, and at the same time consider the contextual relationship between paragraphs and question sentences to create a vector sequence;

转换后的向量最终通过两层残差公路网络分别获得段落与问题向量序列，分别表示为

与

其中残差公路网络如图3所示，在刚开始输入向量x时，按照常规的神经网络进行权值相加，然后通过激活函数ReLu，在权值叠加后再把输入的信息与此时的输出叠加后再通过激活函数。The x and q of the input layer become the input of the encoding layer, which is represented as a V-dimensional one-hot encoded vector. First, the one-hot encoded vector is converted into a v-dimensional continuous-valued vector. To transform words, use the trained vector transformation matrix

The converted vector finally obtains the sequence of paragraph and question vectors through the two-layer residual highway network, which are expressed as

and

Among them, the residual highway network is shown in Figure 3. When the vector x is input at the beginning, the weights are added according to the conventional neural network, and then the activation function ReLu is used to superimpose the weights and then add the input information to the current value. The outputs are superimposed and then passed through the activation function.

与输入问题的上下文矩阵

其中d是隐藏层维数。Then, the paragraphs and questions converted into continuous-valued vector sequences are input into the GRU-RNN network. In the present invention, a layer of GRU units is used, and the corrected linear unit value of the GRU will dynamically change with the paragraphs and input questions during the training process. In addition, in order to prevent the input vector from becoming less influential during the training process, a bidirectional GRU is used to obtain the context matrix of the paragraph

Context matrix with input questions

where d is the hidden layer dimension.

The RNN state in the encoding layer is also used to determine the initial state of the RNN in the decoding layer. The initial state of the decoding layer is calculated as follows:

其中

表示输入问题上下文矩阵的最终状态。This calculation formula focuses on the context matrix of the input question, where

in

Represents the final state of the input question context matrix.

Matching layer: captures the relationship between the words in the paragraph and the input question sentence, and models the vector sequence of the paragraph;

In the matching layer, the passages in the encoding layer are matched with the input question, and the regions related to the input question are found from the passages, and the bidirectional attention flow (BiDAF) model is used in the matching layer to obtain the relationship between the passage and the input question. The relationship between the two-way attention flow will calculate the attention value from both the paragraph and the input question, and finally create a context matrix for the paragraph according to the input question.

In BiDAF, first use the context matrix H for transfer and the context matrix U for input question, the similarity matrix is calculated according to the following formula

的计算方式为：Among them, ω _s is the learning parameter, [;] represents the row-vector cascade operation. Next, based on the similarity matrix, the bidirectional attention values from the paragraph to the input question are computed in both directions. From the attention value of the paragraph to the input question, for the words in the paragraph, use the words in the input question to calculate the weighted vector, the attention vector of the t-th word in the paragraph

is calculated as:

a _t =softmax _j (S _t )

中的向量

根据段落的序列长度T与输入问题中与之紧密相关的单词加权计算得出：During the process from entering questions to paragraphs, the matrix

vector in

Calculated by weighting the sequence length T of the paragraph with the words in the input question that are closely related to it:

Then calculate the bidirectional attention vector between each word in the paragraph and the input question;

Finally, the bidirectional attention vector G is input into the first layer of bidirectional GRU, and the context matrix M is obtained.

Decoding layer: Generates a sequence of word tokens that constitute the correction problem using a GRU-RNN gated recurrent neural network with an attention mechanism and two replication mechanisms.

In this layer, corrected problems are generated based on the information in the encoding layer and the matching layer. The final integration is a network that combines an RNN-based language generation model with a replication mechanism for passage and input questions.

Further, the RNN language generation model includes:

The generative model consists of a layer of GRU and a softmax layer with an attention mechanism. When the revised word sequence y={y ₁ ,...,y _s } is input, the next word probability distribution P _g of the revised question output by the generative model can be obtained. Expressed as:

进行更新。where W _g and b _g represent learning parameters, V _g represents the number of words generated by the generative model, and V > V _g . The words generated by the generative model are only high-frequency words, and the low-frequency words are extracted by the replication mechanism, which can reduce the size of the generative model and improve the learning speed. h _s+1 is the hidden layer of the GRU, and is composed of

to update.

和编码层一样，经过一次单热编码向量化之后，词嵌入层与两个残差公路网络层将y_s转换为连续向量

接着使用e_s与前一个GRU隐藏状态h_s计算向量

用于注意力值的计算。According to the previous word y _s output by the generative model, the input vector of the GRU is determined as follows

Like the encoding layer, after a one-hot encoding vectorization, the word embedding layer and the two residual highway network layers convert y _s into continuous vectors

Then use _{es and the previous GRU hidden state h s to} calculate the vector

For the calculation of attention value.

Then calculate the attention value α _st of the paragraph and the attention value β _sj of the input question:

可以表达如下：Finally, the input value of the GRU

It can be expressed as follows:

与

为学习参数，f为PReLu非线性激活函数。in,

and

is the learning parameter, f is the PReLu nonlinear activation function.

Further, the replication mechanism includes:

This mechanism is a mechanism for generating consistent sentences in the context of sentences and dialogues by duplicating part of the input word during sentence generation, with the aim of generating revised questions based on the content of the input question or paragraph, where the input The questions and passages of are used as copy sources, and the generation probability distribution of words is obtained by the copy mechanism calculation:

P _cp (y _s+1 |y≤s,x,q)=∑ _t f(y _s+1 =x _t )α _(s+1)t ;

P _cq (y _s+1 |y≤s,x,q)=∑ _j f(y _s+1 =q _j )β _(s+1)q ;

Among them, f(y _s+1 = x _t ) is a function, and the value is 1 when y _s = x _t , otherwise it is 0, and f(y _s+1 = q _j ) is the same.

Further, the RNN-based language generation model of the decoding layer and the replication mechanism network for paragraphs and input questions include:

The weighted sum of the generation probability of each word calculated by the generative model can be used to determine the word probability distribution P of the final output word:

P(y _s+1 |y≤s,x,q)＝λ _s P _g (y _s+1 |y≤s,x,q)

+μ _s P _cp (y _s+1 |y≤s,x,q)

+υ _s P _cq (y _s+1 |y≤s,x,q);

决定：where λ _s μ _s υ _s is the weight parameter, λ _s μ _s υ _s ∈ [0,1] and λ _s + μ _s +υ _s =1, the value of the weight parameter is output by the softmax layer shown in the following equation

Decide:

λ _s =γ _s0 , μ _s =γ _s1 ,υ _s =γ _s2

Among them, Highway ₂ ( ) represents a two-layer residual highway network, and W _c and _bc represent the weights and biases in the learning parameters. Then, according to the probability distribution generated by the decoding layer, a beam search is introduced in the generation process, and a plurality of correction problem candidate sets are generated by searching the generation range through the beam width b.

其中N是批处理训练大小，i是每一批数据中第i个样本的索引值。During the training process of the generation model of the correction problem, the error L is calculated using the negative log-likelihood loss, and the error function is minimized by updating the parameters to continuously optimize the generation model. The error function is defined as:

where N is the batch training size and i is the index value of the ith sample in each batch.

Further, the GRU-RNN gated recurrent neural network structure of the decoding layer includes:

Activation function: PReLu is selected as the activation function in GRU-RNN. This function is used as a modified linear unit, which can change the slope of all negative values in the input vector according to the specific training data, and the remaining values remain unchanged. Increase the contrast and improve model learning efficiency;

Neuron selection: As shown in Figure 4, it is an expanded view of the single-layer structure of GRU-RNN. It is based on a time series length model and can effectively capture the correlation information between contexts. The gated recurrent unit GRU is one of the RNNs. Neurons control the retention of information through two gate structures in the hidden layer, and can filter out important and unimportant information. As shown in Figure 5, the GRU unit structure is as follows:

W_z，W_r，W为权重矩阵；The input of the sequence t is x _t , and the input of the previous layer of the hidden layer is h _t-1 , which represents less important information in the previous data. These two inputs pass through the reset gate r _t , the update gate z _t and the output gate The processing of h _t forms the output h _t of the unit and the output of the hidden layer

W _z , W _r , W are weight matrices;

Update gate: z _t =σ(W _z ·[h _t-1 ,x _t ])

Reset gate: r _t =σ(W _r ·[h _t-1 ,x _t ])

Hidden layer:

Output gate:

Other parameters: The model is composed of a layer of GRU with an attention mechanism and a softmax layer, adaptive gradient descent, and a learning rate of 0.0007.

Further, a method for automatically generating training data for paragraph sentence compression, comprising:

The compression method is an unsupervised sentence compression method based on syntactic dependency structure and integer programming. First, the syntax dependency structure of the input sentence is obtained, and then the words in the sentence are assigned an index number in sequence from the beginning of the sentence, and formulate based on the index number. Integer programming problem, the integer programming expression is defined as follows:

计算得出，F为训练语料库中所有单词出现的总频率，F(a_i)为单个单词的出现频率。段落句子压缩过程中，压缩后的长度由约束条件∑_i≤La_i≤l决定，本发明方法目的在于寻找出问题的最佳意图，因此要对系统中被认为是比较重要的单词进行一定的抑制，

约束就是为了防止一直选择权重最大的单词，从而确保生成意图比较精准的问题。When a _i =1, the i-th word in the sentence is selected; when a _i =0, that is, the word is not selected, and it is deleted by sentence compression. L is the length of the sentence before compression, l is the maximum number of words in the sentence after compression, parent(i) shows the word corresponding to the parent node of the ith word in terms of syntactic dependency, and w _i represents the word weight, given by

It is calculated that F is the total frequency of occurrence of all words in the training corpus, and F( _ai ) is the frequency of occurrence of a single word. In the process of paragraph and sentence compression, the compressed length is determined by the constraint condition ∑ _i≤L a _i ≤l. The purpose of the method of the present invention is to find the best intention of the problem. Therefore, certain words are considered to be important in the system. inhibition,

The constraint is to prevent the word with the largest weight from being selected all the time, so as to ensure that the generation intent is more accurate.

Although the above content has described the specific implementation method of the present invention in conjunction with the accompanying drawings, it is only a preferred embodiment of the present invention, and those skilled in the art should be clear that on the basis of the technical solution of the present invention, the Any tampering, equivalent, and deformation or modification that can be made without creative work shall be included within the scope of protection of the claims of the present invention.

Claims (3)

1. a question sentence intent recognition method in natural language question answering technology, it is characterized in that, described method adopts a kind of end-to-end executable correction question generation model, and this model is by combining sentence generation model, machine reading model and replication mechanism , the method includes the following steps: Step 1, when the input question and its corresponding question paragraph are input into the generative model, the model will use the machine reading model to read and extract the relevant information in the content of the input question and paragraph; Step 2, the model is based on the attention observation and supervision mechanism, and the sentence generation model generates a revised set of questions while extracting information; Step 3, then, the copy mechanism will copy the input question and the important words and expressions in the paragraph into the revised set of questions, so as to generate the revised final question according to the content of the paragraph; The structure of the correction problem generation model includes the following four parts: Input layer: The input question sentence and question paragraph are represented as a sequence vector model of one-hot encoding; Encoding layer: use the word embedding model to convert the sequence of paragraph tokens and question sentences of the input layer into vectors of continuous values, and at the same time consider the contextual relationship between paragraphs and question sentences to create a vector sequence; Matching layer: captures the relationship between the words in the paragraph and the input question sentence, and models the vector sequence of the paragraph; Decoding layer: Generates a sequence of word tokens that constitute the correction problem using a GRU-RNN gated recurrent neural network with an attention mechanism and two replication mechanisms. 2. the question sentence intent identification method in a kind of natural language question answering technology according to claim 1, is characterized in that, among the described decoding layer, GRU-RNN gated recurrent neural network structure is as follows: Activation function: PReLu is selected as the activation function in GRU-RNN. This function is used as a modified linear unit, which can change the slope of all negative values in the input vector according to the specific training data, and the remaining values remain unchanged. Increase the contrast and improve model learning efficiency; Neuron selection: Gated recurrent unit GRU is a kind of neuron in RNN. In the hidden layer, two gate structures are used to control the retention of information, which can filter out important and unimportant information; Other parameters: The model is composed of a layer of GRU with an attention mechanism and a softmax layer, adaptive gradient descent, and a learning rate of 0.0007. 3. The method for recognizing question intent in a natural language question answering technique according to claim 1 and 2, wherein a method for automatically generating training data by compressing a paragraph sentence is used to compress a sentence from a machine-reading corpus. A short question sentence with ambiguous intent is generated in the question bank, thereby automatically correcting the questioning model.

Images