CN116895072A - A table parsing method, device, equipment and storage medium - Google Patents

Abstract

The invention discloses a method, a device, equipment and a storage medium for analyzing a table, wherein the method comprises the following steps: receiving image data of a content-containing table; extracting geometric position features of text blocks, appearance features of cells and content features of text information from the image data respectively; separating visual features on rows and columns from the appearance features; extracting semantic features from the geometric position features and the content features respectively to serve as appearance semantic features and content semantic features; learning a first collaboration block between the representation rows and columns, a second collaboration block between the representation cells and the text boxes, and a third collaboration block between the representation text boxes and the text information according to the visual features, the appearance semantic features, and the content semantic features; fusing the first collaboration block, the second collaboration block and the third collaboration block into a collaboration graph code; and encoding and identifying structural information in the table according to the collaborative graph. The shallow image characterization information combines semantic information in the cells, so that the accuracy of identifying the table is improved.

Description

A table parsing method, device, equipment and storage medium

Technical field

The present invention relates to the technical field of computer vision, and in particular, to a table parsing method, device, equipment and storage medium.

Background technique

The archives of the power grid come from a wide range of sources and are of various types. A large number of construction archive project files will be generated in various production and construction, and most of these files store information in the form of tables. Whenever a file is reviewed, a consistency review of the division table is required, that is, the file names corresponding to the division table are divided according to the file directory table index, and then one by one in the division table is checked whether the project names marked with "√" are in the file directory. exist in.

In order to improve office efficiency, scripts are currently used to automatically perform data analysis, modification, mining, and visualization of the contents in the table, so as to divide the file names corresponding to the table according to the file directory table index and finally compare whether the project name is in file exists in the directory.

At present, it is mainly based on deep learning to identify the structure of tables, including bottom-up methods, top-down methods and image text generation methods. Among them, the main feature of the bottom-up method is to first detect table cells and text blocks, and then classify the cell relationships; the top-down method first divides the table rows and columns, and then merges the cells, etc. Operation; the image text generation method refers to directly generating sequence text corresponding to the table structure based on the table image.

However, the division tables in power grid production and construction files often have complex table structures and hundreds of cell contents. When parsing the table structure, problems such as confusing merge relationships and incorrect division of rows and columns are prone to occur, which will also lead to affected units. Incomplete case content, semantic errors, etc. The identification omission of a few cells is likely to cause large-scale table structure parsing errors, leading to incomplete identification of project names to be reviewed and compared in the division table, resulting in omissions.

Contents of the invention

The present invention provides a table parsing method, device, equipment and storage medium to solve the problem of how to improve the accuracy of the structure of the parsed table.

According to one aspect of the present invention, a table parsing method is provided, including:

Receive image data containing tables;

Extract geometric position features of text blocks, appearance features of cells, and content features of text information from the image data respectively;

Separating visual features in rows and columns from said appearance features;

Extract semantic features from the geometric position features and the content features respectively as appearance semantic features and content semantic features;

According to the visual features, the appearance semantic features and the content semantic features, learn to represent the first collaboration block between rows and columns, represent the second collaboration block between cells and text boxes, and represent between text boxes and text information. The third collaboration block;

Fusion of the first collaboration block, the second collaboration block and the third collaboration block into collaborative graph coding;

Structural information in the table is identified based on the collaboration graph encoding.

Optionally, extracting geometric position features of text blocks, appearance features of cells, and content features of text information from the image data includes:

Load the first fully connected layer, residual network, second fully connected layer, word vector model, and convolutional layer;

Perform optical character recognition on the image data to obtain text blocks, where the text blocks have position information;

Input the position information of the text block into the first fully connected layer for mapping to obtain geometric position features;

Input the image data into the residual network to extract image features;

Perform an aggregation operation on the region of interest on the image features to obtain features representing the text box corresponding to the cell;

Input the characteristics of the text box into the second fully connected layer for mapping to obtain the appearance characteristics of the cell;

Enter the text information corresponding to the text box into the word vector model for encoding to obtain a text vector;

The text vector is input into the convolution layer to perform a convolution operation to obtain the content features of the text information.

Optionally, the separation of visual features on rows and columns from the appearance features includes:

Load the separate aggregation module and long short-term memory network;

Input the appearance features into the separate aggregation module to extract fusion features on rows and columns;

Extract the first separation feature on the row and the second separation feature on the column from the fusion feature;

The first separation feature and the second separation feature are input into the long short-term memory network and fused into visual features in rows and columns.

Optionally, extracting semantic features from the geometric position features and the content features respectively as appearance semantic features and content semantic features includes:

Load the self-semantic extractor, which is provided with a multi-head attention mechanism associated through residual connections;

Construct the geometric position feature as a first directed graph;

Input the first directed graph into the self-semantic extractor, and extract semantic features through the multi-head attention mechanism as appearance semantic features;

constructing the content features as a second directed graph;

The second directed graph is input into the self-semantic extractor, and semantic features are extracted through the multi-head attention mechanism as content semantic features.

Optionally, the learning based on the visual features, the appearance semantic features and the content semantic features represents a first collaboration block between rows and columns, a second collaboration block between cells and text boxes, and a representation of text. The third collaboration block between boxes and text information includes:

Load a cross-context synthesizer, which is equipped with multiple multi-head attention mechanisms;

Input the visual features into the cross-context synthesizer, and learn to represent the first cooperative block between rows and columns through multiple multi-head attention mechanisms in parallel;

The appearance semantic features and the content semantic features are input into the cross-context synthesizer, and multiple multi-head attention mechanisms are used in parallel to learn to represent the second cooperation block between the cell and the text box and between the text box and the text information. The third collaboration block.

Optionally, identifying the structural information in the table based on the collaborative graph encoding includes:

loading a structure prediction network having a plurality of prediction blocks;

The collaborative graph encoding is sequentially input into multiple prediction blocks for processing to identify structural information in the table.

Optionally, the number of the prediction blocks is three, and each prediction block has a third fully connected layer, a fourth fully connected layer and an activation layer.

According to another aspect of the present invention, a table parsing device is provided, including:

The image data receiving module is used to receive image data containing tables;

A feature extraction module, configured to extract geometric position features of text blocks, appearance features of cells, and content features of text information from the image data;

A visual feature separation module, used to separate visual features in rows and columns from the appearance features;

A semantic feature recognition module, configured to extract semantic features from the geometric position features and the content features respectively as appearance semantic features and content semantic features;

A collaboration block generation module, configured to learn a first collaboration block between rows and columns, a second collaboration block between cells and text boxes, and a representation based on the visual features, appearance semantic features, and content semantic features. The third collaboration block between text boxes and text information;

A collaborative graph coding generation module, configured to fuse the first collaboration block, the second collaboration block and the third collaboration block into collaborative graph coding;

A structural information generation module, configured to identify the structural information in the table based on the collaborative graph encoding.

Optionally, the feature extraction module is also used to:

Load the first fully connected layer, residual network, second fully connected layer, word vector model, and convolutional layer;

Perform optical character recognition on the image data to obtain text blocks, where the text blocks have position information;

Input the position information of the text block into the first fully connected layer for mapping to obtain geometric position features;

Input the image data into the residual network to extract image features;

Perform an aggregation operation on the region of interest on the image features to obtain features representing the text box corresponding to the cell;

Input the characteristics of the text box into the second fully connected layer for mapping to obtain the appearance characteristics of the cell;

Enter the text information corresponding to the text box into the word vector model for encoding to obtain a text vector;

The text vector is input into the convolution layer to perform a convolution operation to obtain the content features of the text information.

Optionally, the visual feature separation module is also used to:

Load the separate aggregation module and long short-term memory network;

Input the appearance features into the separate aggregation module to extract fusion features on rows and columns;

Extract the first separation feature on the row and the second separation feature on the column from the fusion feature;

The first separation feature and the second separation feature are input into the long short-term memory network and fused into visual features in rows and columns.

Optionally, the semantic feature recognition module is also used to:

Load the self-semantic extractor, which is provided with a multi-head attention mechanism associated through residual connections;

Construct the geometric position feature as a first directed graph;

Input the first directed graph into the self-semantic extractor, and extract semantic features through the multi-head attention mechanism as appearance semantic features;

constructing the content features as a second directed graph;

The second directed graph is input into the self-semantic extractor, and semantic features are extracted through the multi-head attention mechanism as content semantic features.

Optionally, the collaboration block generation module is also used to:

Load a cross-context synthesizer, which is equipped with multiple multi-head attention mechanisms;

Input the visual features into the cross-context synthesizer, and learn to represent the first cooperative block between rows and columns through multiple multi-head attention mechanisms in parallel;

The appearance semantic features and the content semantic features are input into the cross-context synthesizer, and multiple multi-head attention mechanisms are used in parallel to learn to represent the second cooperation block between the cell and the text box and between the text box and the text information. The third collaboration block.

Optionally, the structural information generation module is also used to:

loading a structure prediction network having a plurality of prediction blocks;

The collaborative graph encoding is sequentially input into multiple prediction blocks for processing to identify structural information in the table.

For example, the number of the prediction blocks is three, and each prediction block has a third fully connected layer, a fourth fully connected layer and an activation layer.

According to another aspect of the present invention, an electronic device is provided, the electronic device including:

at least one processor; and

a memory communicatively connected to the at least one processor; wherein,

The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor, so that the at least one processor can execute the method described in any embodiment of the present invention. Table parsing method.

According to another aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. The computer program is used to implement any embodiment of the present invention when executed by a processor. The parsing method of the table.

In this embodiment, image data containing tables is received; the geometric position features of text blocks, the appearance features of cells, and the content features of text information are extracted from the image data; and the rows and columns are separated from the appearance features. Visual features; extract semantic features from geometric position features and content features respectively as appearance semantic features and content semantic features; learn to represent the first collaboration block between rows and columns based on visual features, appearance semantic features, and content semantic features , representing the second collaboration block between the cell and the text box, and the third collaboration block representing the text box and the text information; integrating the first collaboration block, the second collaboration block and the third collaboration block into collaboration graph coding; Identify the structural information in the table based on the collaboration graph encoding. This embodiment can generate context for each modality of the table, thereby merging and modulating the inter-modal interaction information of different tables. By stacking contexts in different dimensions multiple times, intra-modal context generation and inter-modal collaboration can be The hierarchical approach is carried out alternately, which enables intra-modal interactions to be continuously generated from low-level to top-level, that is, multi-modal low-level context information and high-level context information can cooperate with each other in the entire network, so that sufficient shallow-level context can be learned The image representation information can be better combined with the semantic information of the content in the cell, which greatly improves the accuracy of identifying the structural information of the table.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the invention, nor is it intended to limit the scope of the invention. Other features of the present invention will become easily understood from the following description.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a flow chart of a table parsing method provided according to Embodiment 1 of the present invention;

Figure 2 is a schematic structural diagram of a neural synergy map machine provided according to Embodiment 1 of the present invention;

Figure 3 is a schematic structural diagram of a self-semantic extractor and context synthesizer provided according to Embodiment 1 of the present invention;

Figure 4 is a schematic structural diagram of a table parsing device provided according to Embodiment 2 of the present invention;

FIG. 5 is a schematic structural diagram of an electronic device provided in Embodiment 3 of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only These are some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall within the scope of protection of the present invention.

It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the invention described herein are capable of being practiced in sequences other than those illustrated or described herein. Furthermore, the terms "include" and "having" and any variations thereof are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus that encompasses a series of steps or units and need not be limited to those explicitly listed. Those steps or elements may instead include other steps or elements not expressly listed or inherent to the process, method, product or apparatus.

Embodiment 1

Figure 1 is a flow chart of a table parsing method provided in Embodiment 1 of the present invention. The method can be executed by a table parsing device. The table parsing device can be implemented in the form of hardware and/or software. The table parsing device can be implemented in the form of hardware and/or software. The analysis device can be configured in electronic equipment. As shown in Figure 1, the method includes:

Step 101: Receive image data whose content includes a table.

In this embodiment, a neural collaborative graph machine (NCGM) is proposed, as shown in Figure 2, to convert various files containing tables (such as partition tables) in the power grid into image data. Then, the image data can also be called table images. data(TableImage).

Among them, the table contains multiple cells, and the cells contain text information. In general, the multiple cells have no obvious regular distribution, and the text information also has no obvious regularity.

Preprocess the image data such as paging, and input the image data into the Neural Collaboration Graph Machine (NCGM) for processing.

Step 102: Extract the geometric position features of the text block, the appearance features of the cells, and the content features of the text information from the image data.

In this embodiment, as shown in Figure 2, three-way feature extraction (Featureextraction) can be performed on the image data. One of the feature extractions is to extract the geometric position features of the text block from the image data, and the other is to extract the geometric position features of the text block from the image data. The appearance features of cells are extracted from the data, and the next step of feature extraction is to extract the content features of text information from the image data.

Among them, the text block is an area where text information is aggregated.

In the specific implementation, you can load the d-dimensional first fully connected layer (FC), residual network (ResNet, such as ResNet18), d-dimensional second fully connected layer, and word vector model (such as wrod2vec) , Convolutional layer.

Perform optical character recognition (Optical Character Recognition, OCR) on the image data to obtain a text block, where the text block has position information.

The position information of the text block is input into the first fully connected layer for mapping to obtain the geometric position feature (Geometry).

The image data is input into the residual network to extract image features.

Perform a region of interest aggregation operation (ROIalign) on the image features to obtain the features representing the text box corresponding to the cell.

The features of the text box are input into the second fully connected layer for mapping to obtain the appearance features of the cell (Appearance fseg).

Enter the text information corresponding to the text box into the word vector model (such as wrod2vec) for encoding, and encode it into the distribution space to obtain the text vector.

Input the text vector into the convolution layer, use a convolution kernel of size 7×1×d to perform a convolution operation with a step size of 1, and obtain the content features (content) of the text information.

Step 103: Separate visual features in rows and columns from appearance features.

In this embodiment, high-level feature extraction can be performed on the appearance features, and visual features in rows and columns can be separated from the appearance features.

In the specific implementation, as shown in Figure 2, the split-aggregation module (Split-aggregationModule) and the long-short-term memory network (lstm module) can be loaded.

Input the appearance features into the separate aggregation module to extract the fused features on rows and columns, and use the correlation relationship to extract the fused features into the first separation feature on the row and the second separation feature on the column.

The first separation feature and the second separation feature are input into the long short-term memory network for feature enhancement to fuse into visual features in rows and columns.

Step 104: Extract semantic features from geometric position features and content features respectively as appearance semantic features and content semantic features.

In this embodiment, high-level feature extraction can be performed on geometric position features and content features. Semantic features are extracted from the geometric position features and are recorded as appearance semantic features. Semantic features are extracted from content features and are recorded as content semantic features. feature.

In the specific implementation, as shown in Figure 2, it is loaded from the semantic extractor (ECE (Ego Context Extractor) module).

Among them, as shown in Figure 3, the mode of each feature input to ECE is constructed as a separate directed graph G ^~ = {v, ε}∈{G ^G , G ^A , G ^C }, in the graph In each decoupled modality, consider _{the corresponding} embedding of each text segment bounding box as _a node Connect with each other. In the constructed directed graph, each node can be an anchor or one of the contexts of other nodes. For graph representation, CNN (Convolutional Neural Network) has a strong inductive bias and may not be the optimal choice. In order to solve this problem, the self-semantic extractor ECE proposed in this embodiment is equipped with a multi-head attention mechanism (Multi-head Attention, MHA) associated through residual connections. The multi-head attention mechanism aggregates the fully connected graph information of all three modes. , The multi-head attention mechanism makes few assumptions about the input, and can learn to combine local behavior and global information based on the input content, and can better handle structural parsing in large and complex tables.

Furthermore, MHA introduces a memory compression module. At this time, MHA is also called CMHA (Compressed Multi-head Attention, compressed multi-head attention mechanism):

MC(H)=Norm(Reshape(x,∈)W ^h )

CMHA can compress the amount of "memory" to the amount equal to query (query, Q), so that CMHA can reduce the number of image pixels and greatly reduce the computational complexity in the operation of the multi-head attention mechanism.

In addition, this embodiment is also equipped with residual connections for CMHA, so that query information can be passed on well, which can be defined as:

P=MHA(Q, MC(K), MC(V)),

Among them, "FFN(■)" is the feedforward layer, and "Add&Norm(■)" means element addition and layer normalization.

On the one hand, the geometric position feature can be constructed as a first directed graph, the first directed graph is input into the semantic extractor, and the semantic features are extracted through a multi-head attention mechanism as appearance semantic features.

On the other hand, the content features can be constructed as a second directed graph, the second directed graph is input into the semantic extractor, and the semantic features are extracted through a multi-head attention mechanism as content semantic features.

Step 105: Learn the first collaboration block between rows and columns, the second collaboration block between cells and text boxes, and the collaboration between text boxes and text information based on visual features, appearance semantic features, and content semantic features. The third collaboration block.

In this embodiment, it is possible to learn to represent the first collaboration block between rows and columns based on visual features, to learn to represent the second collaboration block between cells and text boxes based on appearance semantic features and content semantic features, and to represent the second collaboration block between text boxes and text boxes based on appearance semantic features and content semantic features. Third collaboration block between text messages.

In the specific implementation, as shown in Figure 2, the cross-context synthesizer (CCS (Cross ContextSynthesizer) Intermodality) can be loaded.

Among them, as shown in Figure 3, multiple multi-head attention mechanisms are set up in the cross-context synthesizer.

In each collaboration block (i.e., the first collaboration block, the second collaboration block, and the third collaboration block), the extracted feature embeddings are constructed as context graphs, which are applied individually by ECE to form a "modal flow". This embodiment will They blend together in a collaborative manner and learn collaboration patterns between different modalities. CCS has three parallel CMHAs. Each CMHA has three modes. One mode is used as the query, and the other two modes jointly serve as the key K and the value V. That is, the query mode explores useful information from the other two modes. , allowing CCS to selectively fuse individual context information of different modalities into modal interactions maintained in the "modal flow".

As shown in Figure 2, the visual features are input into the cross-context synthesizer, and the first collaborative block Ap between rows and columns is learned to represent the rows and columns through multiple multi-head attention mechanisms in parallel.

The appearance semantic features and content semantic features are input into the cross-context synthesizer, and multiple multi-head attention mechanisms are used to learn the second cooperation block Ge between the cell and the text box and the third collaboration block Ge between the text box and the text information in parallel. Collaboration Block Co.

Step 106: Fusion of the first collaboration block, the second collaboration block and the third collaboration block into collaborative graph coding.

In this embodiment, as shown in Figure 2, the first collaboration block Ap, the second collaboration block Ge and the third collaboration block Co can be integrated into collaborative graph embeddings (collaborative graph embeddings) through functions such as concat, as shown Based on this, the i-th element and j-th element are spliced together along the channel direction as a pair to form a vector/>

Step 107: Identify the structural information in the table based on the collaborative graph encoding.

In this embodiment, as shown in Figure 2, structure prediction (Structure prediction) can be performed based on collaborative graph coding to obtain the structural information in the table, for example, the position information of each cell Cell, the row sequence Row of each cell , column number Col, etc., to complete the structure analysis of the table.

In a specific implementation, a structure prediction network can be loaded, which has multiple prediction blocks.

The prediction block is an encapsulation of the deep learning structure containing prediction. The structures of each prediction block may be the same or different, and this embodiment is not limited to this.

For example, the number of prediction blocks is three. Each prediction block has a third fully connected layer, a fourth fully connected layer and an activation layer (such as a softmax layer). The processing in each prediction block is as follows: The third fully connected layer provides fully connected operations, the fourth fully connected layer provides fully connected operations, and the activation layer provides activation operations.

The collaborative graph encoding is input into multiple prediction blocks in turn for processing to identify the structural information in the table.

In this embodiment, image data containing tables is received; the geometric position features of text blocks, the appearance features of cells, and the content features of text information are extracted from the image data; and the rows and columns are separated from the appearance features. Visual features; extract semantic features from geometric position features and content features respectively as appearance semantic features and content semantic features; learn to represent the first collaboration block between rows and columns based on visual features, appearance semantic features, and content semantic features , representing the second collaboration block between the cell and the text box, and the third collaboration block representing the text box and the text information; integrating the first collaboration block, the second collaboration block and the third collaboration block into collaboration graph coding; Identify the structural information in the table based on the collaboration graph encoding. This embodiment can generate context for each modality of the table, thereby merging and modulating the inter-modal interaction information of different tables. By stacking contexts in different dimensions multiple times, intra-modal context generation and inter-modal collaboration can be The hierarchical approach is carried out alternately, which enables intra-modal interactions to be continuously generated from low-level to top-level, that is, multi-modal low-level context information and high-level context information can cooperate with each other in the entire network, so that sufficient shallow-level context can be learned The image representation information can be better combined with the semantic information of the content in the cell, which greatly improves the accuracy of identifying the structural information of the table.

Embodiment 2

FIG. 4 is a schematic structural diagram of a table parsing device provided in Embodiment 2 of the present invention. As shown in Figure 3, the device includes:

Image data receiving module 401, used to receive image data containing tables;

Feature extraction module 402 is used to extract geometric position features of text blocks, appearance features of cells, and content features of text information from the image data;

The visual feature separation module 403 is used to separate visual features in rows and columns from the appearance features;

The semantic feature identification module 404 is used to extract semantic features from the geometric position features and the content features respectively as appearance semantic features and content semantic features;

The collaboration block generation module 405 is configured to learn a first collaboration block between rows and columns, a second collaboration block between cells and text boxes based on the visual features, appearance semantic features and content semantic features. Represents the third collaboration block between the text box and text information;

Collaboration graph coding generation module 406, configured to fuse the first collaboration block, the second collaboration block and the third collaboration block into collaboration graph coding;

The structural information generation module 407 is used to identify the structural information in the table according to the collaborative graph encoding.

In one embodiment of the present invention, the feature extraction module 402 is also used to:

Load the first fully connected layer, residual network, second fully connected layer, word vector model, and convolutional layer;

Perform optical character recognition on the image data to obtain text blocks, where the text blocks have position information;

Input the position information of the text block into the first fully connected layer for mapping to obtain geometric position features;

Input the image data into the residual network to extract image features;

Perform an aggregation operation on the region of interest on the image features to obtain features representing the text box corresponding to the cell;

Input the characteristics of the text box into the second fully connected layer for mapping to obtain the appearance characteristics of the cell;

Enter the text information corresponding to the text box into the word vector model for encoding to obtain a text vector;

The text vector is input into the convolution layer to perform a convolution operation to obtain the content features of the text information.

In one embodiment of the present invention, the visual feature separation module 403 is also used to:

Load the separate aggregation module and long short-term memory network;

Input the appearance features into the separate aggregation module to extract fusion features on rows and columns;

Extract the first separation feature on the row and the second separation feature on the column from the fusion feature;

The first separation feature and the second separation feature are input into the long short-term memory network and fused into visual features in rows and columns.

In one embodiment of the present invention, the semantic feature recognition module 404 is also used to:

Load the self-semantic extractor, which is provided with a multi-head attention mechanism associated through residual connections;

Construct the geometric position feature as a first directed graph;

Input the first directed graph into the self-semantic extractor, and extract semantic features through the multi-head attention mechanism as appearance semantic features;

constructing the content features as a second directed graph;

The second directed graph is input into the self-semantic extractor, and semantic features are extracted through the multi-head attention mechanism as content semantic features.

In one embodiment of the present invention, the cooperation block generation module 405 is also used to:

Load a cross-context synthesizer, which is equipped with multiple multi-head attention mechanisms;

Input the visual features into the cross-context synthesizer, and learn to represent the first cooperative block between rows and columns through multiple multi-head attention mechanisms in parallel;

The appearance semantic features and the content semantic features are input into the cross-context synthesizer, and multiple multi-head attention mechanisms are used in parallel to learn to represent the second cooperation block between the cell and the text box and between the text box and the text information. The third collaboration block.

In one embodiment of the present invention, the structural information generation module 407 is also used to:

loading a structure prediction network having a plurality of prediction blocks;

The collaborative graph encoding is sequentially input into multiple prediction blocks for processing to identify structural information in the table.

For example, the number of the prediction blocks is three, and each prediction block has a third fully connected layer, a fourth fully connected layer and an activation layer.

The table parsing device provided by the embodiment of the present invention can execute the table parsing method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the table parsing method.

Embodiment 3

FIG. 5 shows a schematic structural diagram of an electronic device 10 that can be used to implement embodiments of the present invention. Electronic devices are intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (eg, helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are examples only and are not intended to limit the implementation of the invention described and/or claimed herein.

As shown in Figure 5, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores There is a computer program that can be executed by at least one processor. The processor 11 can perform the operation according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. Perform various appropriate actions and processing. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via the bus 14. An input/output (I/O) interface 15 is also connected to bus 14 .

Multiple components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc. etc.; and communication unit 19, such as network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices through computer networks such as the Internet and/or various telecommunications networks.

Processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processing processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes various methods and processes described above, such as table parsing methods.

In some embodiments, the table parsing method may be implemented as a computer program, which is tangibly included in a computer-readable storage medium, such as the storage unit 18 . In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19 . When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the table parsing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the parsing method of the table in any other suitable manner (eg, by means of firmware).

Various implementations of the systems and techniques described above may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip implemented in a system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or a combination thereof. These various embodiments may include implementation in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor The processor, which may be a special purpose or general purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device. An output device.

Computer programs for implementing the methods of the invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that the computer program, when executed by the processor, causes the functions/operations specified in the flowcharts and/or block diagrams to be implemented. A computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display)) for displaying information to the user monitor); and a keyboard and pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and may be provided in any form, including Acoustic input, voice input or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., A user's computer having a graphical user interface or web browser through which the user can interact with implementations of the systems and technologies described herein), or including such backend components, middleware components, or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communications network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

Computing systems may include clients and servers. Clients and servers are generally remote from each other and typically interact over a communications network. The relationship of client and server is created by computer programs running on corresponding computers and having a client-server relationship with each other. The server can be a cloud server, also known as cloud computing server or cloud host. It is a host product in the cloud computing service system to solve the problems of difficult management and weak business scalability in traditional physical hosts and VPS services. defect.

Embodiment 4

An embodiment of the present invention also provides a computer program product. The computer program product includes a computer program. When executed by a processor, the computer program implements the table parsing method provided by any embodiment of the present invention.

During the implementation of the computer program product, computer program code for performing the operations of the present invention can be written in one or more programming languages or a combination thereof. Programming languages include object-oriented programming languages, such as Java, Smalltalk , C++, and also includes conventional procedural programming languages, such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In situations involving remote computers, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider through Internet connection).

It should be understood that various forms of the process shown above may be used, with steps reordered, added or deleted. For example, each step described in the present invention can be executed in parallel, sequentially, or in different orders. As long as the desired results of the technical solution of the present invention can be achieved, there is no limitation here.

The above-mentioned specific embodiments do not constitute a limitation on the scope of the present invention. It will be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions are possible depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims (10)

1. A table parsing method, characterized by including: Receive image data containing tables; Extract geometric position features of text blocks, appearance features of cells, and content features of text information from the image data respectively; Separating visual features in rows and columns from said appearance features; Extract semantic features from the geometric position features and the content features respectively as appearance semantic features and content semantic features; According to the visual features, the appearance semantic features and the content semantic features, learn to represent the first collaboration block between rows and columns, represent the second collaboration block between cells and text boxes, and represent between text boxes and text information. The third collaboration block; Fusion of the first collaboration block, the second collaboration block and the third collaboration block into collaborative graph coding; Structural information in the table is identified based on the collaboration graph encoding. 2. The method according to claim 1, wherein the step of extracting geometric position features of text blocks, appearance features of cells, and content features of text information from the image data includes: Load the first fully connected layer, residual network, second fully connected layer, word vector model, and convolutional layer; Perform optical character recognition on the image data to obtain text blocks, where the text blocks have position information; Input the position information of the text block into the first fully connected layer for mapping to obtain geometric position features; Input the image data into the residual network to extract image features; Perform an aggregation operation on the region of interest on the image features to obtain features representing the text box corresponding to the cell; Input the characteristics of the text box into the second fully connected layer for mapping to obtain the appearance characteristics of the cell; Enter the text information corresponding to the text box into the word vector model for encoding to obtain a text vector; The text vector is input into the convolution layer to perform a convolution operation to obtain the content features of the text information. 3. The method according to claim 1, characterized in that said isolating visual features in rows and columns from said appearance features includes: Load the separate aggregation module and long short-term memory network; Input the appearance features into the separate aggregation module to extract fusion features on rows and columns; Extract the first separation feature on the row and the second separation feature on the column from the fusion feature; The first separation feature and the second separation feature are input into the long short-term memory network and fused into visual features in rows and columns. 4. The method according to claim 1, wherein the semantic features extracted from the geometric position features and the content features respectively as appearance semantic features and content semantic features include: Load the self-semantic extractor, which is provided with a multi-head attention mechanism associated through residual connections; Construct the geometric position feature as a first directed graph; Input the first directed graph into the self-semantic extractor, and extract semantic features through the multi-head attention mechanism as appearance semantic features; constructing the content features as a second directed graph; The second directed graph is input into the self-semantic extractor, and semantic features are extracted through the multi-head attention mechanism as content semantic features. 5. The method of claim 1, wherein the first collaboration block between rows and columns, cells and text are learned based on the visual features, appearance semantic features and content semantic features. The second collaboration block between boxes and the third collaboration block representing text boxes and text information include: Load a cross-context synthesizer, which is equipped with multiple multi-head attention mechanisms; Input the visual features into the cross-context synthesizer, and learn to represent the first cooperative block between rows and columns through multiple multi-head attention mechanisms in parallel; The appearance semantic features and the content semantic features are input into the cross-context synthesizer, and multiple multi-head attention mechanisms are used in parallel to learn to represent the second cooperation block between the cell and the text box and between the text box and the text information. The third collaboration block. 6. The method according to any one of claims 1-5, characterized in that identifying the structural information in the table based on the collaborative graph encoding includes: loading a structure prediction network having a plurality of prediction blocks; The collaborative graph encoding is sequentially input into multiple prediction blocks for processing to identify structural information in the table. 7. The method according to claim 6, characterized in that the number of the prediction blocks is three, and each prediction block has a third fully connected layer, a fourth fully connected layer and an activation layer. 8. A table parsing device, characterized in that it includes: The image data receiving module is used to receive image data containing tables; A feature extraction module, configured to extract geometric position features of text blocks, appearance features of cells, and content features of text information from the image data; A visual feature separation module, used to separate visual features in rows and columns from the appearance features; A semantic feature recognition module, configured to extract semantic features from the geometric position features and the content features respectively as appearance semantic features and content semantic features; A collaboration block generation module, configured to learn a first collaboration block between rows and columns, a second collaboration block between cells and text boxes, and a representation based on the visual features, appearance semantic features, and content semantic features. The third collaboration block between text boxes and text information; A collaborative graph coding generation module, configured to fuse the first collaboration block, the second collaboration block and the third collaboration block into collaborative graph coding; A structural information generation module, configured to identify the structural information in the table based on the collaborative graph encoding. 9. An electronic device, characterized in that the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor, so that the at least one processor can execute any one of claims 1-7 The parsing method of the table. 10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program is used to implement the method described in any one of claims 1-7 when executed by a processor. Table parsing method.