RU2768544C1 - Method for recognition of text in images of documents - Google Patents

Abstract

FIELD: computing technology.

SUBSTANCE: computer-implemented method for automatic recognition of text in an image of a document, the method is executed on a computing apparatus containing a processor and a memory storing instructions executed by the processor and containing stages of: obtaining an image of the document; isolating an area of the document in the image; based on the isolated area of the document in the image, classifying a type of the document using a convolutional neural network; binarising the image of the determined type of document in order to separate the text of the document from the background; segmenting the image of the document, with the determined type of document, in order to determine text fields, by determining the coordinates of the bounding rectangles of fields; determining the type of found text fields using structural document templates; conducting text recognition on the identified text fields; validating the recognised text on the identified text fields based on a set of rules related to the data contained in the fields and the expected presentation formats of said data.

EFFECT: increase in the accuracy of extracting information from images of documents.

16 cl, 2 dwg

Description

FIELD OF TECHNOLOGY

The present technical solution relates to computer technology, and in particular to methods for recognizing text on document images using machine learning methods.

BACKGROUND OF THE INVENTION

Known from the prior art is the source of information US 8,897,563 B1, published on November 25, 2014, which discloses a method and system for automatic document processing. In this solution, the document analysis system receives and processes jobs from multiple users, where each job may contain multiple electronic documents, to extract data from them, through a method of automatically pre-processing each received electronic document using multiple images. The solution provides transformation algorithms to improve the subsequent extraction of data from the specified document. The specified method includes electronic splitting of each received page of the electronic document into parts; automatic processing of each page fragment of the received electronic document using each of the multiple image pre-processing algorithms to create a plurality of image options for each fragment, as well as analyzing the results of post-processing and data extraction for each of the image parts options to determine which result is best.

The source of information RU 2691214 C1, published on 06/11/2019, is also known from the prior art, revealing a text recognition method using artificial intelligence. The method includes obtaining an image of text, wherein the text in the image contains one or more words in one or more sentences; obtaining a text image as the first input data for a set of trained machine learning models that stores information about word compatibility and the frequency of their joint use in real sentences; obtaining one or more final outputs from a set of trained machine learning models; and extracting from one or more final outputs one or more suggested sentences from the text in the image. Each of the one or more proposed sentences contains probable word sequences.

The proposed solution differs from the solutions known from the prior art in that after highlighting the document area in the image, the document type is classified using machine learning algorithms, and the recognized text is validated according to a set of rules regarding the data contained in the fields and the expected presentation formats of these data.

SUMMARY OF THE INVENTION

The technical problem to be solved by the claimed solution is the development of a method for extracting information from images of documents using machine learning methods, described in an independent claim. Additional embodiments of the present invention are presented in dependent claims.

The technical result consists in increasing the accuracy of extracting information from document images. An additional technical result is to increase the speed of text recognition on document images. An additional technical result is an increase in the performance of the computing system when solving the task (i.e., it allows processing to obtain a result (product) in less time), thereby reducing the load on the central processor of the computing device, by reducing the number of processed requests.

The claimed technical result is achieved through the operation of a computer-implemented method for automatic text recognition on a document image, the method is performed on a computing device that contains a processor and a memory that stores instructions executable by the processor and contains the steps in which:

an image of the document is received on the computing device;

by means of the document localization module, the selection of the document area on the image is carried out;

by means of a document type classification module, based on the selected area of the document in the image, the document type is classified using a convolutional neural network;

by means of the binarization module, perform binarization of the image of a certain type of document to separate the text of the document from the background;

through the segmentation module, segmenting the image of the document, with a certain type of document at the stage of classification, to determine the text fields, by determining the coordinates of the bounding boxes of the fields;

by means of the mapping module, determine the type of text fields found, at the previous stage, using structural document templates;

by means of a text recognition module, perform text recognition on the identified text fields;

by means of the validation module, the recognized text is validated on the identified text fields according to a set of rules regarding the data contained in the fields and the expected formats for presenting this data.

In a private embodiment of the proposed method, the image is obtained by scanning.

In another particular embodiment of the proposed method, the image is obtained by photographing.

In another particular embodiment of the proposed method, the selection of the document area on at least one image is carried out using the algorithm for finding the contour of the largest area or the selection of the area is carried out by segmentation using a convolutional neural network.

In another particular embodiment of the proposed method, the selection of a document area in an image using the largest area contour search algorithm includes the following steps: compressing the size of the resulting document image to a target height depending on the type of document, converting a multi-channel image into a single-channel image, applying to the converted single-channel image median blur filter, select image edges, search for contours in the image, select the contour with the largest area, select the bounding box for the selected contour with the largest area, and flatten the document area using perspective projection with matrix.

In another particular embodiment of the proposed method, the selection of a document area in an image using segmentation using a convolutional neural network includes the following steps: a multi-channel document image is fed to the input of the trained convolutional neural network, and a binary mask of the input image size is obtained at the output, where 1 is the area document, 0 is the background.

In another particular implementation of the proposed method, the convolutional neural network is trained using the stochastic gradient descent algorithm with minimization of the loss function.

In another particular embodiment of the proposed method, the document type is understood as a document containing personal data.

In another private embodiment of the proposed method, a document containing personal data is: passport, SNILS, TIN, insurance policy, driver's license, birth certificate, death certificate, marriage certificate, divorce certificate, education document , military ID, work book.

In another particular embodiment of the proposed method, binarization is performed by analyzing the geometric characteristics of the associated components.

In another particular embodiment of the proposed method, binarization is performed based on color information using the HSV color model.

In another particular embodiment of the proposed method, additionally, after the binarization step, by means of the alignment module, at least one document image is aligned by recognizing lines of text strings.

In another particular embodiment of the proposed method, line recognition for alignment is carried out using the probabilistic Hough transform algorithm.

In another particular embodiment of the proposed method, to determine the text fields, by determining the coordinates of the bounding rectangles of the fields, the analysis of profile projection histograms is used.

In another particular implementation of the proposed method, the type of text fields is determined based on the grouping by lines, the maximum and minimum width and height of the text fields in the template.

In another particular embodiment of the proposed method, text recognition on the identified text fields is carried out using the TesseractOCR system.

DESCRIPTION OF THE DRAWINGS

The implementation of the invention will be described hereinafter in accordance with the accompanying drawings, which are presented to explain the essence of the invention and in no way limit the scope of the invention. The following drawings are attached to the application:

1 illustrates a block diagram of the operation of the proposed computer-implemented automatic text recognition method on at least one document image.

Fig. 2 illustrates an example of a computing device operation scheme.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the implementation of the invention, numerous implementation details are provided to provide a clear understanding of the present invention. However, one skilled in the art will appreciate how the present invention can be used, both with and without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to unnecessarily obscure the features of the present invention.

Furthermore, it will be clear from the foregoing that the invention is not limited to the present implementation. Numerous possible modifications, changes, variations and substitutions that retain the spirit and form of the present invention will be apparent to those skilled in the subject area.

The following terms and definitions apply in this application, unless otherwise expressly stated. References to techniques used in the description of this invention refer to well-known methods, including modifications of these methods and replacing them with equivalent methods known to those skilled in the art.

The term "document type" means the type of document containing personal data. The type of document can be, but not limited to: passport (citizen's passport or foreign passport), SNILS, TIN, insurance policy, driver's license, birth certificate, death certificate, marriage certificate, divorce certificate, education document, military ID, work book.

On FIG. 1 illustrates the scheme of operation of a computer-implemented method for automatic text recognition on a document image.

On the computing device, by means of network interaction, receive an image of the document. The image of the document can be obtained by any known method, for example, by scanning, photographing, and the image of the created electronic document can also be used.

At step 102, by means of the document localization module, the selection of the document area in the image is performed using the largest area contour search algorithm or the area selection is performed by segmentation using a convolutional neural network.

The selection of the document area on the image using the algorithm for finding the contour of the largest area includes the following steps.

1. Carry out the scaling of the image, i.e. compressing the image size to a certain target height, which depends on the document type. The operation allows you to speed up subsequent processing and exclude image details that are not required for document localization (for example, the background).

2. Convert the RGB color image of the document into a single-channel grayscale image.

3. Apply a median blur filter to smooth out document image details such as text. Each pixel of the image is replaced by the median value of the pixel intensities from the surrounding area of a square shape with a size equal to the characteristic height of the document text in the image.

4. Edge detection using the Canny algorithm described in J. F. Canny, A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 8, no. 6, pp. 679-698, November 1986.

5. Search for contours using the method described in the article by Satoshi Suzuki and others. Topological structural analysis of digitized binary images by border following. Computer Vision, Graphics, and Image Processing, 30(1):32-46, 1985.

6. Select the contour of the largest area, which is taken beyond the boundaries of the document in the image.

7. Select the bounding box for the selected maximum area contour.

8. Align the document area on the image using a perspective projection with a matrix, which is determined using the RANSAC algorithm to four points - the corners of the selected rectangle.

In another implementation, image segmentation using a U-Net convolutional neural network, described in Ronneberger, Olaf, is used to highlight a document area in an image; Fischer, Philipp; Brox, Thomas. U-Net: Convolutional Networks for Biomedical Image Segmentation. 2015.

The input of the convolutional neural network is a color scanned or photographed image, or an image of a created electronic document, the output is a binary mask of the size of the input image with elements 1 for the document area and 0 for the surrounding background. The neural network is trained using the stochastic gradient descent algorithm with minimization of the loss function on a set of labeled examples, which are pairs: input - original image, output - binary mask of the document area.

At step 104, by means of the document type classification module, based on the selected area of the document, a document type classification is performed on the document image using the ResNet convolutional neural network described in the article by He et al. Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

A color or black-and-white image of a document is fed to the input of a convolutional neural network, and a probability distribution vector for possible types of documents is obtained at the output.

1. The resulting image is converted to a size of 224×224 pixels.

2. Normalize the converted image.

3. The sequence of operations of the ResNet convolutional neural network is applied to the normalized image, namely the operations of convolution, pooling, softmax.

4. At the output of the neural network, a vector of probability distribution by types of documents is obtained, for example, [0.8, 0.1, 0.1].

5. The document type with the highest probability is taken as the result. The most probable document type is determined by the index of the largest element of the previously obtained vector, expressed by the value of the signal at the output of the neural network. The neural network model contains outputs for each type of document, the output with the largest signal value is selected. The signals are calculated by the neural network based on the input image through a sequence of matrix multiplication operations with parameters selected when training the model on a set of examples, as well as non-linear transformation operations (zeroing negative values) and pooling (selecting the maximum element from the region). For clarity, the output signals are converted to a probability distribution (a vector of positive elements of the same size that sum to a value of 1) using the softmax operation.

For example, the probability distribution vector is represented as [0.8, 0.1, 0.1]. It is necessary to determine which type of document this vector belongs to from three possible types: passport, SNILS, TIN, and the vector refers to the document type "passport", if the first element of the vector has the largest value [0.8, 0.1, 0.1], the vector refers to the document type "SNILS" if the second element of the vector has the largest value [0.1, 0.8, 0.1], the vector belongs to the document type "TIN" if the third element of the vector has the largest value [0.1, 0.1, 0.8]. Thus, the neural network will attribute the previously obtained probability distribution vector [0.8, 0.1, 0.1] to the “passport” document type.

The parameters of the neural network model are selected during the learning process using the stochastic gradient descent algorithm on a set of labeled examples. Examples for training are pairs: input - document image, output - document type.

At step 106, by means of the binarization module, the image of a certain type of document is binarized to separate the text of the document from the background. The value of each pixel in the original image is replaced by 1 if the pixel belongs to the text field of the document, or 0 otherwise.

In one implementation, binarization is performed based on color information, which is applicable for types of documents in which text differs from background in color. For example, the text is black and the background is white. In this case, binarization can be performed using the HSV color model, when it is assumed that the text pixels have values in certain, experimentally selected by analyzing a set of examples, ranges of the Hue and Value channels.

In another embodiment, the binarization of the document image area or individual subareas of the document image is performed by analyzing the geometric characteristics of the associated components according to the algorithm:

1. The color image of the document is converted to shades of gray according to the formula:

y=0.299⋅R+0.587⋅G+0.114⋅B

where R, G, B are the intensities of the red, green and blue channels, respectively,

y is the resulting grayscale value of the pixel intensity.

2. The image is binarized in shades of gray with the choice of a threshold according to the Father algorithm described in the article by Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE Trans. Sys. man. Cyber. 9(1): 62-66. doi:10.1109/TSMC. 1979.4310076, 1979. For pixels whose intensity is higher than the selected threshold value, the value is set to 1, for the rest - 0. In the resulting binary mask, text fields and separate, relatively bright, background elements are highlighted.

3. Gluing individual text characters is performed using a morphological closing operation, which is the result of sequential application of dilation and erosion operations with a rectangular kernel of size W × 1, where W is the parameter of the largest distance between characters in the field, which is selected experimentally on a set of examples. In the dilation operation, each pixel is replaced by the maximum value from the surrounding area of the nucleus, in the erosion operation - by the minimum value.

4. Related components are selected - groups of pixels with a value of 1, which have at least one pixel with a value of 1 among the surrounding 8 pixels. The algorithm described in the article by Kesheng Wu, Ekow Otoo, and Kenji Suzuki is used to select related components. Optimizing two-pass connected-component labeling algorithms. Pattern Analysis and Applications, 12(2): 117-135, Jun 2009. The result is an image mask where each pixel has a value equal to the index of the component it belongs to.

5. Select related components with features specific to text fields of a certain type of document. For example, the characteristic height of a text field in a document is used as a criterion. For example, when recognizing a passport, height is used as a selection criterion for components corresponding to recognizable text fields, since the text of the fields is larger than the rest of the document text in the considered visual zones. Pixels of components with a height less than the threshold value belong to the background (set to 0). This separates the text fields from the background of the document. The following geometric characteristics can also be used: width, area, center coordinates, moments of inertia.

Additionally, at step 108, by means of the alignment module, the alignment of the document image can be performed by recognizing the lines of text lines, if this alignment was not performed at step 102, according to the perspective projection method.

In one implementation, the alignment is performed according to the method of recognizing the lines of text strings using the probabilistic Hough transform algorithm described in the article by Jiri Matas, Charles Galambos, and Josef Kittler. Robust detection of lines using the progressive probabilistic hough transform. Computer Vision and Image Understanding, 78(1):119-137, 2000. The alignment algorithm includes the following steps:

1. A color image is converted to grayscale.

2. Binarization is performed with the choice of the threshold according to the Father's algorithm.

3. Apply the morphological closure operation to glue the characters of the text fields.

4. Select the edges using the Canny algorithm.

5. Perform line recognition using a probabilistic Hough transform.

6. The lines are clustered according to the angle of inclination to the horizontal axis.

7. The corner with the most lines representing it is considered the document's rotation angle.

8. Alignment of the document is carried out by turning to the found angle.

In step 110, the segmentation module performs the segmentation of the document image, with a certain type of document in the classification step, to determine the text fields, by determining the coordinates of the bounding boxes of the fields. The segmentation is carried out using a profile projection histogram analysis method, which includes the following steps.

1. A morphological closure operation is applied to the binary mask of the field text.

2. Build histograms of the vertical projection of the profiles: for each mark on the vertical axis, the number of pixels with a value of 1 is counted.

3. By analyzing the outliers on the histogram of the vertical projection of the profiles, the vertical coordinates of the boundaries of the lines of text on the image of the document are determined. The values on the histogram are compared with the threshold experimentally selected on a set of examples. Intervals on the histogram with values exceeding the threshold are considered to correspond to lines of text.

4. For each line selected at step 3, the horizontal coordinates of the boundaries of individual fields are determined using the histogram of the horizontal projection of the profiles (for each mark along the horizontal axis, the number of pixels with a value of 1 is calculated). Values on the histogram are compared with an experimentally selected threshold, values exceeding the threshold are considered to be related to the fields of the document.

5. Expansion of the found bounding rectangles of the fields is performed with some scale factor experimentally selected on a set of examples.

At step 112, by means of the mapping module, the type of text fields found in the previous step is determined using structural document templates.

The structural template of the document is a list of lines ordered by vertical coordinates, where each line consists of fields ordered by horizontal coordinates.

For example, a passport, according to the structural template of the document, contains the fields of the last name, first name, patronymic in different lines, gender and date of birth in one line, 1-3 lines of the place of birth. For the field structure obtained by analyzing the profile projection histograms in the form of division by rows, you can match the field types in the template.

Each field in the template is characterized by a type, maximum and minimum width and height. According to the method of automatic enumeration, field types are assigned in the best way corresponding to the document template. For example, when recognizing the basic information zone of a passport, 6 lines were obtained as a result of segmentation. In accordance with the template, line 1 contains the field type "Last name", line 2 - "First name", line 3 - "Patronymic", line 4 - the fields "Gender" and "Date of birth", lines 5, 6 - "Place of birth" . The number of fields segmented in rows is checked, as well as the size of the fields to satisfy the constraints specified in the template. If any inconsistencies with the template rules are found, the particular string and related fields are considered not recognized. In the described case, a simple enumeration of the template strings is performed.

In step 114, the text recognition module performs text recognition on the identified text fields using the TesseractOCR system described in R. Smith An Overview of the Tesseract OCR Engine. Ninth International Conference on Document Analysis and Recognition (ICDAR 2007). 2007.

At step 116, the recognized text in the identified text fields is validated by the validator module against a set of rules regarding the data contained in the fields and the expected formats for presenting this data.

For example, for the "document number" field, it is checked that all recognized characters are digits and the number of characters is equal to the number of positions of the number in the document.

On FIG. 2, the general scheme of the computing device (200) will be presented below, providing the data processing necessary for the implementation of the claimed solution.

In general, the device (200) contains such components as: one or more processors (201), at least one memory (202), a data storage medium (203), input/output interfaces (204), an I/O means ( 205), networking tools (206).

The processor (201) of the device performs the basic computing operations necessary for the operation of the device (200) or the functionality of one or more of its components. The processor (201) executes the necessary machine-readable instructions contained in the main memory (202).

The memory (202) is typically in the form of RAM and contains the necessary software logic to provide the desired functionality.

The data storage means (203) can be in the form of HDD, SSD disks, raid array, network storage, flash memory, optical information storage devices (CD, DVD, MD, Blue-Ray disks), etc. The means (203) allows long-term storage of various types of information, for example, the above-mentioned files with user data sets, a database containing records of time intervals measured for each user, user identifiers, etc.

Interfaces (204) are standard means for connecting and working with the server part, for example, USB, RS232, RJ45, LPT, COM, HDMI, PS/2, Lightning, FireWire, etc.

The choice of interfaces (204) depends on the specific implementation of the device (200), which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, and the like.

As means of I/O data (205) in any embodiment of the system that implements the described method, a keyboard can be used. The keyboard hardware can be any known: it can be either a built-in keyboard used on a laptop or netbook, or a separate device connected to a desktop computer, server, or other computer device. In this case, the connection can be either wired, in which the keyboard connection cable is connected to the PS / 2 or USB port located on the system unit of the desktop computer, or wireless, in which the keyboard exchanges data via a wireless communication channel, for example, a radio channel, with base station, which, in turn, is directly connected to the system unit, for example, to one of the USB ports. In addition to the keyboard, the following I/O devices can also be used: joystick, display (touchscreen), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

Means of network interaction (206) are selected from a device that provides network data reception and transmission, for example, an Ethernet card, WLAN/Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. With the help of means (205) the organization of data exchange over a wired or wireless data transmission channel, for example, WAN, PAN, LAN (LAN), Intranet, Internet, WLAN, WMAN or GSM, is provided.

The components of the device (200) are connected via a common data bus (210).

In these application materials, a preferred disclosure of the implementation of the claimed technical solution was presented, which should not be used as limiting other, private embodiments of its implementation, which do not go beyond the scope of the requested legal protection and are obvious to specialists in the relevant field of technology.

Claims (24)

1. A computer-implemented method for automatic text recognition on a document image, the method is performed on a computing device containing a processor and a memory that stores instructions executed by the processor and contains the steps of: an image of the document is received on the computing device; by means of the document localization module, the selection of the document area on the image is carried out; by means of a document type classification module, based on the selected area of the document in the image, the document type is classified using a convolutional neural network; by means of the binarization module, perform binarization of the image of a certain type of document to separate the text of the document from the background; through the segmentation module, segmenting the image of the document, with a certain type of document at the stage of classification, to determine the text fields, by determining the coordinates of the bounding boxes of the fields; by means of the mapping module, determine the type of text fields found, at the previous stage, using structural document templates; by means of a text recognition module, perform text recognition on the identified text fields; by means of the validation module, the recognized text is validated on the identified text fields according to a set of rules regarding the data contained in the fields and the expected formats for presenting this data. 2. The method according to p. 1, characterized in that the image is obtained by scanning. 3. The method according to p. 1, characterized in that the image is obtained by photographing. 4. The method according to claim 1, characterized in that the selection of the document area in the image is carried out using the algorithm for finding the contour of the largest area or the area is selected by segmentation using a convolutional neural network. 5. The method according to claim 4, characterized in that the selection of a document area in the image using the largest area contour search algorithm includes the following steps: compressing the size of the resulting document image to a target height depending on the type of document, converting a multi-channel image into a single-channel image, applying to the transformed single-channel image of the median blur filter, select the edges of the image, search for contours in the image, select the contour with the largest area, select the bounding box for the selected contour with the largest area, and flatten the document area using perspective projection with matrix. 6. The method according to claim 4, characterized in that the selection of the document area in the image using segmentation using a convolutional neural network includes the following steps: a multi-channel image of the document is fed to the input of the trained convolutional neural network, the output is a binary mask of the size of the input image, where 1 is the document area, 0 is the background. 7. The method according to claim 6, characterized in that the convolutional neural network is trained using the stochastic gradient descent algorithm with minimization of the loss function. 8. The method according to p. 1, characterized in that the document type is understood as a document containing personal data. 9. The method according to claim 8, characterized in that the document containing personal data is: passport, SNILS, TIN, insurance policy, driver's license, birth certificate, death certificate, marriage certificate, divorce certificate , document on education, military ID, work book. 10. The method according to p. 1, characterized in that the binarization is performed by analyzing the geometric characteristics of the associated components. 11. The method of claim. 1, characterized in that the binarization is performed based on color information using the HSV color model. 12. The method according to claim 1, characterized in that additionally, after the binarization step, by means of the alignment module, the document image is aligned by recognizing lines of text strings. 13. The method according to claim 12, characterized in that the line recognition for alignment is carried out using a probabilistic Hough transform algorithm. 14. The method according to claim 1, characterized in that to determine the text fields, by determining the coordinates of the bounding rectangles of the fields, the analysis of histograms of the profile projection is used. 15. The method according to claim 1, characterized in that the type of text fields is determined based on the grouping by lines, the maximum and minimum width and height of the text fields in the template. 16. The method according to claim 1, characterized in that text recognition on the identified text fields is carried out using the TesseractOCR system.

Images