KR20170136970A - Method and apparatus for scanned documents classification - Google Patents

Abstract

This disclosure relates to the field of document processing and management and, more particularly, to a method and apparatus for classifying document hardcopy images, and automatically classifying them using image descriptors of hardcopy documents. A plurality of trained classifiers are used to estimate the class probability of the document hard copy image based on the extracted image descriptor and the most probable class of document hard copy images is determined by the meta classifier trained using the estimated class probability And the hardcopy image of the document and the hardcopy image of the most probable class are input to the allocation means. Finally, the most probable class determined by the trained meta classifier is assigned to the document hardcopy image by the classification means and classified A hard copy image of the document is obtained.

Description

[0001] METHOD AND APPARATUS FOR SCANNED DOCUMENTS CLASSIFICATION [0002]

This disclosure relates to the field of document processing and management, and more particularly to a method, apparatus, and system for classifying document hardcopy.

Despite the breakthrough advances in electronic document management, business documents are still available in hard copy. General office documents that exist in hard copy relate to accounting, legal documents, personal information, and official mail. Typically, tables, stamps, logos, facsimiles, or signatures may be attached to such documents. A large number of hardcopy documents are delivered to the organization through mail, fax, and courier delivery services. In most cases, the classification of documents is done manually.

Therefore, an automatic document classification method which analyzes a document image more efficiently is required.

One embodiment of the present disclosure provides a method and apparatus for analyzing a hard copy image of a document to determine a class and automatically classifying it into an output tray.

The object of the present disclosure is to eliminate or mitigate the above-mentioned disadvantages of the known solutions from the prior art.

More specifically, the purpose of this disclosure is to provide a tool for classifying document hardcopy images based on classified document hardcopy images, classifying document hardcodes, and further processing scenarios for each document hardcopy . One exemplary scenario is that invoices, invoices, and financial reports must be sent to the account department, while mail must be delivered to the recipient by the recipient.

As described above, the technical result achieved by using this disclosure is to enable automatic classification of hard copy images of documents.

According to a first aspect of the present disclosure, a method of classifying a hard copy image of a document is provided. This method begins by providing a hardcopy image of the document. Hardcopy images have an image feature. The document hard copy image is then input to the first set of image descriptor extractors. The first set of image descriptor extractors extract image descriptors from the document hard copy images. Each image descriptor describes the image function of a hard copy image of a document. A plurality of trained classifiers are then used to estimate the class probability of the document hard copy image based on the extracted image descriptor. Next, the most likely class of the hardcopy image of the document is determined by the meta-classifier trained using the estimated class probability. The most probable class of document hardcopy images and document hardcopy are further input into the allocation means. Finally, the most probable class determined by the trained meta-classifier is assigned to the document hardcopy image by the classification means to obtain the classified hardcopy image of the document.

The plurality of trained classifiers and the trained meta classifiers are accomplished by the following steps.

Storing a training set including a training image and a class label in a memory, wherein the class label is associated with the training image, each training image comprising a training image feature; Inputting a training image of the training set stored in the memory to a second set of image descriptor extractors; Extracting a training image descriptor by a second set of image descriptor extractor sets describing training image features of each training image; Obtaining the trained plurality of classifiers by the classifier training means using the training image descriptor extracted by the image descriptor extractor of the second set and the class label associated with the training image; Estimating a class probability of a training image by a plurality of trained classifiers; estimating a class probability estimated by a plurality of trained classifiers and a meta classifier trained by a classifier training means based on a class label associated with the training image; Acquiring a classifier.

In one embodiment, the number of the plurality of trained classifiers equals the number in the second set of image descriptor extractors of the image descriptor extractor, and each of the plurality of trained classifiers is associated with one of the second set of image descriptor extractors.

In some embodiments, the image descriptor extractor in the first and second sets of image descriptor extractors includes a spatial local binary pattern (SLBP) extractor, a grayscale runlength histogram (GRLH) extractor, and a BMMFV (Bernoulli Mixture Model Fisher) extractor.

Each of the image descriptor and the training image descriptor may be a numeric vector of integer, real, or binary number. The document hardcopy image and the class of the training image may be integer or text labels. The image features and training image features may each be associated with the document hard copy image and the shape, texture and / or color of the training image.

In one embodiment, providing the document hard copy image comprises obtaining a document hard copy image via a scanner, a facsimile machine, a photocamera, a video camera, a reader, or a wireless or wired communication network.

In one embodiment, the first and second set image descriptor extractors are the same set of image descriptor extractors.

In one embodiment, the trained multiple classifier is a support vector machine (SVM).

Estimating the class probability of a hard copy image of a document may include obtaining a vector of real numbers that represents the probability of a hard copy image belonging to a particular class.

Determining the most probable class by the trained meta-classifier comprises concatenating a plurality of probabilistic vectors estimated by the plurality of trained classifiers into a single vector; Estimating a class probability of the document hard copy image using the SVM and a plurality of connected probability vectors; Select the most likely class as the most likely class.

Wherein storing the training set comprises: receiving the training set; Selecting a random subset of the training image and a class label from the training set; And storing the random subset in the memory.

The step of extracting the image descriptor or the training-image descriptor by the SLBP extractor comprises recursively subdividing the document hard copy image or each training image into a plurality of horizontal and vertical stripes; Downsampling each stripe to the same size; Extracting a local binary pattern (LBP) for each pixel of each downsampled stripe; Calculating a binary pattern histogram for each downsampled stripe; Coupling the computed binary pattern histograms to the image descriptor or the training-image descriptor; Normalizing the image descriptor or the training-image descriptor.

The step of extracting the image descriptor or the training-image descriptor by the GRLH extractor comprises: downsampling the document hard copy image or each training image; Recursively subdividing the downsampled hardcopy image or training image into a plurality of horizontal and vertical stripes; Extracting run lengths of similar luminance for each line of the stripe in the horizontal, vertical, diagonal, and diagonal directions; Run length histogram calculation for each stripe, luminance value and length; Linking the calculated run length histogram to the image descriptor or the training-image descriptor; Normalizing the image descriptor or the training-image descriptor.

The step of extracting the image descriptor or the training-image descriptor by the BMMFV extractor comprises: downsampling the document hard copy image or each training image; Recursively subdividing the downsampled hardcopy image or training image into a plurality of horizontal and vertical stripes; Extracting a binary local descriptor for each stripe; Reducing the dimension of the extracted binary local descriptor using a principal component analysis algorithm (PCA); Computing a Bernoulli mixture model for the dimensionally reduced local descriptor; Calculating a Fisher vector based on the calculated Bernoulli mixture model; Power normalization and L2 normalization of the calculated Fischer vector; And linking the normalized fisher vector for each stripe to an image descriptor or a training-image descriptor. The binary local descriptor may be one of the BRISK or ORB descriptors.

According to a second aspect of the present disclosure, an apparatus is provided for classifying a hard copy image of a document. This apparatus is used to perform the method according to the first aspect of the present disclosure. To this end, the device comprises a classification module and a training module.

The classification module is used to classify the document hard copy images and includes a first set of image descriptor extractors, a plurality of trained classifiers, a trained meta-classifier, and assignment means. The first set of image descriptor extractors are configured to extract an image descriptor for the document hard copy image input to the first set of image descriptor extractors. Document hardcopy images have an image function, and each image descriptor describes the image function. The plurality of trained classifiers are configured to estimate a class probability of a document hard copy image using an image descriptor extracted by a first set of image descriptor extractors. The trained meta-classifier is configured to determine the most probable class of the document hard copy image using the class probabilities estimated by the plurality of trained classifiers. The assignment means is configured to assign the most probable class determined by the trained meta-classifier to the document hardcopy image to obtain a classified document hardcopy image.

The training module is used to obtain the plurality of trained classifiers and the trained meta-classifiers, and includes a memory, a second set of image descriptor extractors, classifier training means and meta-classifier training means. The memory is configured to store a training set comprising a training image and a class label. The class label is associated with the training image, and each training image has a training image feature. The second set of image descriptor extractors are configured to receive the training images of the training set stored in the memory and to extract the training image descriptors. Each training image descriptor describes the training image features of each training image. The classifier training means is configured to acquire a training multiple classifier using the training image descriptor extracted by the second set of image descriptor extractors and the class label associated with the training image. The plurality of trained classifiers estimate the class probability of the training image. The meta-classifier training means is configured to obtain a classifier trained by the plurality of trained classifiers and a classifier trained using a class label associated with the training image.

An embodiment of the device according to the second aspect of the present disclosure is similar to the embodiment of the method according to the first aspect of the present disclosure.

According to a third aspect of the present disclosure, a system for classifying document hard copies is provided. The system includes a hard copy classification apparatus and an image processing system.

The hard copy classification apparatus comprises an input tray for hard copying of the document, image capturing means configured to capture a hard copy image of the document; A notification indicator configured to display an assigned class of each document hard copy image; One or more output separation cartridges for sorted document hardcopy.

The image processing system includes a device according to the second aspect of the present disclosure configured to assign a class to a hardcopy image of each document and a routing module configured to assign an output tray as assigned to a hardcopy image of each document . In particular, the routing module receives from the hard copy sorting device a plurality of output trays and information about non-empty output trays and a hard copy image having a class assigned from the device from the device according to the second aspect of the present disclosure, Assigns the number of output trays to each document hard copy according to the assigned class of each document hard copy image and the number of non-empty output trays, and sets the hard copy of the document from the input tray to the designated output tray.

The imaging means may be one of a scanner, a facsimile, a photocamera, a video camera, a reader for reading an image file from a storage medium, and an input unit for receiving an image file via the Internet. The alert indicator may be one of an LCD display or an LED indicator.

In one embodiment, the notification indicator is further configured to indicate the need to empty the output tray.

Other features and advantages of the present disclosure will become apparent from the following detailed description and the accompanying drawings.

An automatic document classification method in which a document image is efficiently analyzed based on analysis of an image and a descriptor according to the present disclosure is possible.

Hereinafter, the gist of the present disclosure will be described with reference to the accompanying drawings.
FIG. 1 illustrates a system for classifying scanned document hard copies in accordance with one embodiment of the present disclosure.
Figure 2 shows a document hard copy classification apparatus according to an embodiment of the present disclosure.
3 is a flow diagram of a training module in accordance with one embodiment of the present disclosure;
4 is a flow diagram of a prediction module in accordance with one embodiment of the present disclosure;
5 is a flow diagram of a hard copy sorting module in accordance with one embodiment of the present disclosure;
Figure 6 illustrates an image space pyramid subdivision process according to one embodiment of the present disclosure.
Figure 7 illustrates an LBP extraction process in accordance with one embodiment of the present disclosure.
Figure 8 illustrates a GRLH extraction process in accordance with one embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein. In order that the present disclosure may be more fully understood, the same reference numbers are used throughout the specification to refer to the same or like parts.

The terms used in the specification may be used to describe various components, but the components should not be limited by terms. Terms are used only for the purpose of distinguishing one component from another.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . The fact that a part is "connected" to another part includes the case where a part is in a state where it can perform data communication with another part through signal transmission / reception.

In addition, the expression that a part "includes" some element is interpreted as covering a non-exclusive inclusion, and each step or process describing the method does not include only the steps described, But it is not explicitly included or may include other steps that are inherent in the step or processor. Similarly, one or more devices or sub-systems or components or structures or components proceeding to "comprising " may include other devices, other sub-systems, other components, other structures, Sub-systems or additional components.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The electronic devices, methods and embodiments provided herein are for illustrative purposes only and are not intended to limit the scope of the rights.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail with reference to the accompanying drawings. The word "exemplary" is used herein to mean "used as an example or example. &Quot; Any embodiment described herein as "exemplary " is by no means necessarily to be interpreted as having advantages over other embodiments.

In document classification, hard copies of documents can be performed using a variety of methods based on image (visual) functionality and text (content-meaningful) functionality. For example, N. Chen, D. Blostein, International Journal of Document Analysis and Recognition, vol. 1, there are many approaches such as "document image classification search: problem description, classifier structure and performance evaluation". Document analysis and recognition International Journal, Volume 10, iss. 1, pp. 1-16, June 2007 - focuses on textual information, which is essential for document classification. It is true, however, that there are many situations for business documents, such as containing only handwritten text, where the amount of text in the document is relatively small, missing, or hard to recognize. Accordingly, some of the prior art solutions, such as those disclosed in, for example, U.S. Patent No. 8,831,361 and U.S. Patent No. 8,462,394, suggest using textual information with visual information to improve classification accuracy. However, this solution requires a lot of computation because it needs to perform the optical character recognition step for text analysis.

Another approach requires document layout extraction and document layout is provided in the form of a binary (or more complex) tree. One example is disclosed in U.S. Patent No. 8,744,183. The main drawback of this approach is the difficulty and sometimes the lack of robustness in layout extraction in complex documents. For example, headings, footers, and content blocks can easily be determined for well-printed documents on white paper. However, it is difficult or almost impossible to make such a decision if the document contains mixed tables or background graphics.

The image feature-based method provides a more robust method for all document layout and background work. More specifically, these methods include three main pipelines for document image descriptor extraction: a spatial local binary pattern, a gray scale runlength histogram, and a Fisher vector.

Local Binary Patterns (LBPs) are widely used for face recognition in recent years, but they are rarely applied to document classification. Conventional methods mainly focus on extracting LBPs for the entire image or a specific portion of the document in the scanned document (T. Ojala, M. Pietikainen, M. Maenpaa, "Multi Resolution Grayscale and Revolutionary Invariant Texture Classification Binary Pattern & , PAMI, 2010).

Document image descriptors based on run-length histograms are discussed in U.S. Patent No. 8,249,343, and document image descriptors are specifically designed for binary documents. However, an obvious disadvantage of the document image descriptor disclosed in U.S. Patent No. 8,249,343 is that it can only be applied to simple monochrome business documents, i.e. not used as a grayscale image descriptor.

As described above, the Fischer vector is also used to extract the document image descriptor. The image classification based on the Fischer vector can be found in J. Sanchez, F. Perronnin, T. Mensink, J. Verbeek, International Journal of Computer Vision pp.105. An example of a document image descriptor extracted with a Fischer vector is described in US 8,532,399.

In addition, some examples of each means for document classification based on the approaches and methods described above are set forth in U.S. Pat. No. 5,435,544, U.S. Pat. No. 5,525,031 and U.S. Pat. No. 5,602,973. A typical mailbox of a printer has a plurality of output bins and can classify printed hard copies as described in U.S. Patent No. 5,295,181 according to predefined similarities of users or groups of users. The main disadvantage of such an arrangement is that the relevance is pre-set and all documents sent to the printer have. The preference is determined using the barcode that must be printed on all documents. Therefore, documents without barcodes are not recognized. In addition, the number of output separation boxes limits the number of possible document classes.

FIG. 1 illustrates a system 100 for classifying document hard copies in accordance with an embodiment of the present disclosure.

As shown, the system 100 includes a scanning device 101, an image processing system 102, a hard copy sorting device 107, a printing device 108 and a storage device 109. In general, the scanning device 101 may be accompanied or replaced by other means suitable for image acquisition, such as a photocamera, a media storage device with an image file, and the like. The image processing system 102 includes an apparatus 103 and a routing module 106 for classifying document hard copy images. An apparatus 103 for classifying a hard copy image of a document comprises a training module 104 and a classification module 105.

The training module 104 includes classifier training means for acquiring a plurality of trained classifiers and training the meta classifier to obtain a trained meta classifier. The training module 104 will be described in more detail with reference to FIG.

The classification module 105 is used to classify hard copy images of documents and will be described in more detail below with reference to FIG. The routing module 106 allows the output tray to be assigned to each document hard copy according to the assigned class for each document hard copy image.

The class assigned to each document hard copy image is sent to hard copy sorting device 107 to perform hard copy sorting (sorting) according to each assigned class. The hard copy alignment includes separating a plurality of document hard copies into sets related to the appropriate classes defined by the training set. A training set consists of a training image and an assigned class label. Depending on user preferences or selections, the electronic copy of the document may be printed by the printing device 108, stored by the storage device 109, or shared by other means, such as the Internet, a local network, have.

Any module of the system 100 may be combined in a single off-line device, separated separately, connected by a wired network or wireless system for data exchange, or combined with other systems in a subsystem. For example, the scanning device 101 may include an image processing system 102 or a printing device 108, and may be embedded in a multifunction peripheral such as a multifunction peripheral (MFP). The image processing system 102 may be implemented with a processor and memory having instructions for image descriptor extraction, classifier training, and hard copy classification. In another embodiment, the modules of image processing system 102 may be implemented in hardware using a system-on-chip (SoC). As will be appreciated by those skilled in the art, the system 100 may be reconfigured or modified in accordance with user preferences and scenarios for each predefined class. For example, the printing device 108 may represent a network of printing devices distributed between different users or departments. Each class of document hardcopy image is accompanied by a scenario with a predefined scenario or list of necessary actions. For example, after hardcopy sorting a document using a predicted class invoice, the scanned copy is printed to the accounting department manager via e-mail to the printer located in the accounting department. Implementing the system 100 without the scanning device 101 and the hard copy classifier 107 by inputting a document image to the system via a local or remote repository, a photo or other appropriate device such as a video camera or other image capture device It is possible.

Also, the terms "image descriptor" or "visual descriptor" are used herein in the generic sense known to those skilled in the art and relate to the description of the entire image or the basic characteristics of each image area. Such an image descriptor may be provided in any form depending on the specific application. For example, the image descriptor may be an integer, real or binary numeric vector, but is not limited thereto.

FIG. 2 illustrates a hard copy classification apparatus 200 according to one embodiment of the present disclosure. The hard copy sorting apparatus 200 includes at least one input tray 201 and an output tray 203 for performing at least the functions of the scanning and hard copy sorting apparatus, and hard copying of documents. The display and control panel 204 is optional and can be replaced by a notification indicator.

When a user places a plurality of document hard copies in the input tray 201 and selects a specific alignment mode through the panel 204 or uses the default alignment mode, the user receives an aligned document hard copy at the output tray 203. The document hardcopy image is captured by an image capture means, which may be a scanner, a facsimile, a photocamera, a video camera, a reader for reading the image file from the storage medium, and an input unit device for receiving the image file via the Internet.

The classification mode is a set of classification parameters including a list of output trays assigned for each class of the hardcopy image (here also abbreviated as "hard copy class") of the desired classifier or classifier for the current classification process or And determines the sort category specified for the group of such classes. Thus, the classification process is applied by placing a hard copy on an appropriate output bin assigned for that class based on the hardcopy class prediction.

A class can be, for example, a kind in which a hard copy image is classified. For example, the first class based on hard copy image classification may be assigned an 'evaluation form', a second class 'contract', and a third class 'invoice'.

In the case where the hard copy sorting apparatus 200 has only one output tray 203, two embodiments are possible. The first embodiment only assumes the classification of a hard copy to output all hard copies to a single output bin. The second embodiment is to output the hard copy to the output tray only while the input hard code has the same class. If the current hard copy and the previous hard copy class are different, the hard copy classifier 200 empties the output tray 203 to the user (e.g., via panel 204) and places all hard copies Manually prompts to put in the corresponding hard copy bundle. The alignment process then proceeds. Of course, the hard copy sorting apparatus 200 with only one tray can be any scanner or MFP with an automatic paper feeder. Hard copy classification device 200 may include other system modules each including image processing system 102, printing or storage devices 108 and 109,

The training module 300 and the classification module 400 will be described with reference to Figs. 3 and 4. Fig.

The training module 300 is optionally used prior to the classification module 400 to adapt the system 100 to a user-defined training data set. The training module 300 is used again when a new hard copy class or class is added to the system 100. [

Training module 300 operation is based on classifier training according to a user-labeled hardcopy image. Here, the label may be a number or text, such as "1", "2", "3", "4" or "bill", "email", "report" The user pre-labels each image.

The training module 300 receives a training set consisting of a training image and an assigned label, selects a random subset of the training set, and stores the random subset in memory 301. Classifier training is applied separately for a plurality of classifiers (one classifier per descriptor extractor) and a meta classifier. In one embodiment of the present disclosure, three image descriptor extractors, namely a spatial local binary pattern (SLBP) 305, a grayscale run length histogram (GRLH) 306 and a Bernoulli mixed model Fisher Vector (BMMFV) Descriptor is used. The trained classifier is used in the classification module 105. It is possible to train a set of classifiers for different classification scenarios by changing the training set and the amount of classes desired.

The operation of the training module according to one embodiment of the present disclosure will be described with reference to FIG.

1. First, the training set is stored in the memory 301. The storage is associated with receiving the training set, comprising selecting a random subset from the training set and storing the random subset in the memory 301. [

2. Thereafter, three image descriptor extractors, namely a spatial local binary pattern extractor (SLBP) 305, a grayscale run length histogram extractor (GRLH) 306 and a Bernoulli mixed model Fisher Vector (BMMFV) The training image 303 is received from the memory 301 by the image descriptor extractor represented by the base-based extractor 307.

3. Training image descriptors 308, 309, and 310 for the training image 303 are extracted using the image descriptor extractors 305, 306, and 307.

The plurality of trained classifiers 312,31,3 and 314 may then be trained using training image descriptors 308,309 and 310 extracted from the training training set 302 and one or more classifications using class labels 304. [ Is trained using the self-training means 311. In one embodiment of the present disclosure, the classifier training is implemented via SVM (Support Vector Machines).

5. The class probabilities 315, 316, and 317 of the image 303 are then estimated using the trained classifiers 312, 313, and 314. The class probability is a real number vector representing the probability that a hardcopy image belongs to a particular class.

6. Finally, the meta-classifier 319 includes meta-classifier training means 318 that uses class probabilities 315, 316, and 317 and class labels 304 from the training set 302 for training ). ≪ / RTI > In one embodiment of the present disclosure, the meta-classifier is implemented via SVM. Class probabilities are linked by a single vector.

The trained classifiers 312, 313 and 314 are used in the classification module 105. The user may modify or retrain the classifier 312, 313, 314 during system operation by updating the training module 104 or supplementing the new training set.

The operation of the classification module 400 according to one embodiment of the present disclosure will now be described with reference to FIG.

1. First, a document hard copy image 401 is provided to the image descriptor extractors 402, 403 and 404.

2. The image descriptor extractors 402, 403, and 404 extract the image descriptors 405, 406, and 407, respectively. In one embodiment of the present disclosure, the image descriptor extractors are SLBP, GRLH, and BMMFV. The image descriptor extractors 405, 406, 407, 305, 306, 307 may be the same.

3. The plurality of trained classifiers 408, 409, and 410 then estimate class probabilities 411, 412, and 413, respectively. In one embodiment of the present disclosure, multiple classifiers are implemented via Support Vector Machines (SVM). It should be noted that the trained classifiers 408, 409, 410 and 312, 313, 314 are the same.

4. The trained meta classifier 414 then determines the most probable class 415 using the class probabilities 411, 412, 413 associated with a single vector. In one embodiment of the present disclosure, the trained meta-classifier is implemented via an SVM. It should be noted that the trained meta classifiers 414 and 319 are the same.

5. Finally, the most probable class 415 is assigned to the hard copy image 401 using the allocation means, which yields the classified hard copy 417.

FIG. 5 illustrates a classification process 500 for a device 200 in accordance with one embodiment of the present disclosure.

The sorting process continues while the hard copy is inserted into the input tray 201, and the input tray is blanked (step 501). Each input hard copy from the input tray 201 is scanned and sorted (step 502). If the device 200 has one output tray 203 determined in step 503, the device 200 receives and scans the input hard copies one by one through steps 502 and 506 until the input document has the same class. If the current document is different from the class of the previous document and the result of step 504 is No, the device 200 manually puts all the documents in the output tray 203 into the corresponding class bundle and empties the output tray 203 (Steps 507 and 510). If the amount of output trays 203 is less than the predefined classes or alignment category according to at least two output trays 203 determined at step 503 and the result of step 505, One class (N is the number of output trays) is converted to a separate tray and all the remaining classes are converted to the last tray (steps 502, 505, 508 and 509). If the amount of output trays 203 is equal to or greater than the predefined classes or sort categories, each hard copy should be output to the appropriate output tray 203 designated for the current document class.

According to one embodiment of the training 300 and classification 400 modules, three different classification methods may be applied to process the classification operation. All of these are divided into two steps: extraction of the image descriptors 405, 406, 407 and classification of the image descriptor using the corresponding classifiers 408, 409, 410. All three algorithms use an SVM classifier with different settings. The image descriptor extractor is a comprehensive local descriptor based on the Spatial Local Binary Pattern 402 (SLBP), Grayscale Run Length Histogram 403 (GRLH), and Bernoulli Mixture Model Fisher Vector 404 (BMMFV).

Figure 6 illustrates extraction of an image descriptor based on a spatial local binary pattern (SLBP) according to one embodiment of the present disclosure. SLBP is extracted as follows. First, the input image of the scanned document 601 is converted to a grayscale image, divided into a plurality of sub-images 602 and 603 using an image subdivision process, and each image is divided into a plurality of vertical and horizontal stripes Sub-divided. All images 601, 602, and 603 (a total of 13 images or stripes) are then downscaled to a pixel size of 100x100 if necessary, and each pixel of these images is transformed into a local binary pattern LBP). Scaling is performed using a bilinear transform.

(수학식 1)

(1)

Where g _i is the intensity of the ith neighboring pixel and s (g ₀ , g _i ) is a function that compares pixel intensities according to equation (2) and returns a binary code.

(수학식 2)

(2)

Figure 7 illustrates an LBP extraction process in accordance with one embodiment of the present disclosure. As shown in FIG. 7, the LBP is converted to a numeric binary sequence by extraction of the pattern elements starting from the upper left pixel of the counterclockwise direction 701 with a 3 pixel margin between the center pixel and its neighbors. The extracted 1's and 0's form an 8-bit pattern 702 that is interpreted as unsigned byte 703. These bytes are combined into an 8-bin histogram and [0; 1]. The length of the final descriptor is (13 sub-images) x (8-bin histogram) = 104.

Since the pixel size of each subimage is 100x100, the histogram value is divided by 10 ⁴ in the integer to [0; 1] can be converted to a real number. The purpose of scaling is to provide various levels of detail for each histogram band. This not only preserves details with small stripes, but also removes noise.

Figure 8 illustrates a GRLH extraction process in accordance with one embodiment of the present disclosure.

The GRLH extractor applies a spatially localized gray scale run length. This is calculated as follows.

First, the input image of the scanned document is converted to a grayscale image and downsampled to a predefined size. In one embodiment of the present disclosure, the predetermined size is equal to ⁵ x 10 pixels. Then, it is divided into a plurality of sub-images using the spatial pyramid described in Fig. Each image is processed in grayscale run length encoding. Since a run is a pixel sequence with a brightness change of an adjacent pixel that is less than a predefined value, all runs 802, labeled R1-R4 on the image line 801, repeat the pixel as seen in Figure 8, Can be found by looking for.

The GRLH extractor uses a threshold of 30 to distinguish neighboring runs. The run length is quantized using a logarithmic scale and stored in two 9 bin histograms according to the difference between the average pixel value of the current run and the previous run. If the current run is lighter than the previous run, it is stored in the first histogram or the second histogram. The GRLH extractor calculates line histograms for the horizontal, vertical, diagonal, and anti-diagonal directions. The histograms from all sub-images, directions and brightness are linked to a single feature vector of size 1512 bytes. The final feature vector is [0; 1].

Using a grayscale image instead of a binary avoids a binarization step that can completely destroy the image content if the binarization algorithm incorrectly estimates the brightness separation threshold. As an example, this is very useful when the contrast of the document image is low or there are a lot of overlapping images. A common example is a magazine. Unlike binary image-based descriptors, the present invention does not cause the aforementioned problems because the present disclosure performs some kind of local binarization along each direction, as local binarization exhibits fairly sensitive performance for small details.

Based on the Bernoulli Mixture Model (BMMFV), a local technician gathered as a Fisher vector (P. Viola, M. Jones, CVPR 2001, 2001 IEEE Computer Society, "Fast Object Detection Using Boosted Cascade of Simple Features," 2001 , Uchida, Yusuke and Shigeyuki Sakazawa, "Image Retrieval Using Fisher Vector of Binary Characteristics", ACPR (Pattern Recognition), 2013 2nd IAPR Asia Conference IEEE) is obtained by the following method.

Once the local descriptors X {x ₁ , ..., x _t , ..., x _T } are extracted from the input image and projected into a T / 2 dimensional space using PCA (Principal Component Analysis). There is no specific requirement for the local descriptor type, but the proposed implementation uses BRISK (Binary Robust Invariant Scalable Keypoints) or ORB (E. Rublee, V. Rabaud, K. Konolige, G. Bradski, "ORB: Efficient Alternatives to SIFT or SURF", Computer Vision (ICCV), 2011 IEEE International Conference, IEEE, 2011). Then the Bernoulli mixture model (BMM) λ = {w _i , μ _id , i = 1..N, d = 1..D}, where N is the number of components of the BMM model, Training. Using this model, the module calculates the Fisher score for each local descriptor as: < EMI ID = 3.0 >

(수학식 3)

(3)

Here, T is the number of binary features extracted from the image and γ _t (i) = p (i | x _s , λ). The module then obtains the Fisher matrix as shown in equation (4).

(수학식 4)

(4)

는 정규화된 피셔 점수 연결

(i = 1..N, d = 1..D) 로 얻어진다.Finally,

Normalized Fischer score connection

(i = 1..N, d = 1..D).

가 주어지면, 파워-정규화 된 벡터 f (z)는

(α= 0.5)와 같이 계산된다. The Fischer vector is further normalized by power normalization (taking the square root of each vector value) and L ₂ normalization (dividing each vector value by the vector Euclidean standard). Fisher vector

Given the power-normalized vector f (z), the power-normalized vector f

(? = 0.5).

After power normalization, L ₂ normalization is performed on f (z) to become the final Fisher vector descriptor of the binary feature set. The BMMFV extractor fetches one Fisher vector for each sub-image 601 and 602 (a total of five sub-images) and concatenates them all into a single feature vector descriptor.

The reason for using BMM instead of the well-known Gauss Mixing Model (GMM) is that the GMM is designed to operate at the top of the normal distribution, which is a distribution of arbitrary real numbers. A large number of descriptors are also real numbers, and in this case it is normal to use GMMs. However, binary descriptors such as BRISK and ORB are not consistent with the GMM and are consistent with the Bernoulli distribution.

Once the feature vectors are obtained, they are classified using SVM (Support Vector Machine). In the case of SLBP and GRLH, SVMs 408 and 409 use the following crossover kernel.

(수학식 5)

(5)

And the BMMFV SVM 410 uses the following linear kernel (Equation 6).

(수학식 6)

(6)

In both cases the SVM parameter C (cost) is set to 10.

커널을 갖는 SVM 인 메타 분류자를 사용하여 이미지 클래스의 예측에 사용된다.Each SVM provides a vector of class probabilities corresponding to the number of image classes. These vectors are connected as a single vector and treated as a new feature vector. This feature vector

It is used to predict the image class using a meta classifier, which is an SVM with a kernel.

(수학식 7)

(7)

Where K is the number of classifiers and N is the number of possible classes. The output of the meta-classifier is a probability vector of size N.

In summary, the present disclosure introduces a set of modules for hard copy image descriptor extraction and classification using 1) a plurality of trained classifiers, 2) a meta-classifier, and 3) a hard copy classifier. The image descriptor extractor includes a spatial local binary pattern extractor, a grayscale run length histogram extractor, and a Bernoulli mixed model fisher vector extractor. All extractors use binary, integer, or real-valued image features, which is useful when it is difficult to extract document layouts that are essential to layout-based methods. Thus, the present disclosure is applicable even if there are any limitations on the document contents.

Major improvements to the Spatial Local Binary Pattern (SLBP) compared to existing approaches based on LBP (eg T. Ojala, M. Pietikainen, M. Maenpaa, "Multi-resolution grayscale using local binaries of multiple binary solutions and Rotation Invariant Texture Classification Pattern ", PAMI, 2010 and Hongming Zhang, Wen Gao, Xilin Chen, Debin Zhao," Object Detection Using Spatial Histogram Features ", Image and Vision Computing, Volume 24, April 1, 2006 , Pp. 327-341, ISSN 0262-8856) suggests that the proposed implementation can avoid the histogram normalization by applying a recursive block-based spatial subdivision of the input image combined with block rescaling to a predefined size, And processed at a magnification.

Although the concept of the proposed gray scale run length histogram is close to the conventional approach, the present disclosure does not include a binarization step, which allows the system 100 to be used in simple black and white documents as well as in complex document images. This makes it possible to avoid binarization errors that are completely impossible to classify.

As for the Bernoulli mixed model fisher vector (BMMFV), the concept is described as follows, for example. J. Sanchez, F. Perronnin, T. Mensink, J. Verbeek, "Image Classification as a Fisher Vector: Theory and Practice", International Journal of Computer Vision 105, pp. 222, The present disclosure estimates the concept itself as a hardcopy classification problem. Instead of a known approach, the present disclosure combines a Fisher vector with a Bernoulli mixture model and spatial stripe-based subdivision to significantly improve classification accuracy and classifier robustness for various document transformations such as rotation, layout changes, and the like. Using the Bernoulli blend model is more accurate when using binary image descriptors such as ORB or BRISK, rather than using the Gaussian blend model, which is the most common choice of existing models.

The SLBP (Grayscale Run Length Histograms) and BMMFV (Bernoulli Mixture Model Fisher Vectors) are very different image descriptor extractors and tend to extract other image features. Therefore, using the meta classifier at the top of the three extractors combining these classifiers can improve classification accuracy.

This disclosure seeks to propose a flexible classifier utilization approach. All three classifiers or one classifier may be used to provide the end user with a different number of classification profiles. For example, GRLH can be applied to fast classification of fixed layout documents, BMMFV is suitable for flexible layout classification, and meta classification is the perfect solution for complex cases where the image can not be classified with sufficient accuracy.

The present disclosure can be efficiently applied to scanned document classification or hard copy alignment problems. The proposed system 100 can classify high precision or fixed layout documents with high accuracy, even if the document distortion such as distortion, rotation, low contrast and others is provided. The proposed system 100 may be embedded in any one of an MFP, smartphone, server-based solution, or may be applied as a standalone application. The high classification speed and multi-core based processing enable the proposed system 100 for multiple documents. The proposed hardcopy sourcing apparatus 200 can automatically classify a document bundle at a high classification speed, without user interaction.

Other aspects of the disclosure will become apparent from consideration of the drawings and the following description of one embodiment of the present disclosure. Those skilled in the art will recognize that other embodiments of the disclosure are possible and that the details of the disclosure may be modified in various ways without departing from the concept of the disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive. In the appended claims, reference to an element in the singular form does not exclude the presence of a plurality of such elements, unless expressly stated otherwise.

100; system
101; Storage Device
102; Image processing system
103; Device for classifying hardcopies
104; Training Module
105; Classification Module
106; Routing Module
107; Hardcopy sorting apparatus
108; Printing Device
109; 저장 장치
110; Internet / LAN / Fax
200; Hard copy sourcing device

Claims (19)

Providing a hardcopy image of a document comprising image features;
Extracting an image descriptor describing an image feature of the document hard copy image by a first set of image descriptor extractors;
Estimating a class probability of the document hard copy image by a plurality of trained classifiers based on the image descriptor;
Determining a most probable class of the document hardcopy image by a meta classifier trained based on the estimated class probability;
Inputting the document hard copy image and the document hard copy image of the most probable class into the allocation means; And
Assigning, by the assigning means, the most probable class determined by the trained meta classifier to the document hardcopy image to obtain a classified document hardcopy image. ≪ Desc / How to. The method according to claim 1,
Wherein the plurality of trained classifiers and the trained meta classifiers comprise:
CLAIMS What is claimed is: 1. A method comprising: storing a training set in a memory, the training set including a training image and a class label including a training image feature, wherein the class label is associated with the training image,
Extracting a training image descriptor describing a training image feature of the training image by a second set of image descriptor extractors;
Obtaining, by classifier training means, the plurality of trained classifiers using the training image descriptor and a class label associated with the training image;
Estimating a class probability of the training image by the plurality of trained classifiers; And
By the meta classifier training means, obtaining the trained meta classifier based on the estimated class probability and the class label associated with the training image. 3. The method of claim 2, wherein the number of the plurality of trained classifiers is equal to the number of image descriptor extractors in the second set of image descriptor extractors, Lt; RTI ID = 0.0 > 1, < / RTI > 3. The apparatus of claim 2, wherein each image descriptor extractor in the first and second set of image descriptor extractors comprises a spatial local binary pattern (SLBP) extractor, a grayscale runlength histogram and a Bernoulli mixed model fisher Bernoulli Mixture Model Fisher) vector (BMMFV) extractor. ≪ Desc / Clms Page number 21 > 5. The method of claim 4, wherein extracting the image descriptor or training image descriptor by the spatial local binary pattern (SLBP)
Recursively subdividing the document hard copy image or each training image into a plurality of horizontal and vertical stripes;
Downsampling each of the stripes to the same size;
Extracting a local binary pattern (LBP) for each pixel of each downsampled stripe;
Calculating a binary pattern histogram for each downsampled stripe;
Linking the computed binary pattern histogram to the image descriptor or the training image descriptor;
And normalizing the image descriptor or the training image descriptor. ≪ Desc / Clms Page number 19 > 5. The method of claim 4, wherein extracting the image descriptor or the training image descriptor by the GRLH extractor comprises:
Down sampling the document hard copy image or each training image;
Recursively subdividing the down-sampled document hardcopy image or training image into a plurality of horizontal and vertical stripes;
Extracting run lengths of similar brightness for each line of the stripe in the horizontal, vertical, diagonal, and anti-diagonal directions;
Calculating a run length histogram for each of the stripes, the brightness, and the length;
Linking the calculated run length histogram to the image descriptor or the training image descriptor; And
And normalizing the image descriptor or the training image descriptor. ≪ Desc / Clms Page number 21 > 5. The method of claim 4, wherein extracting the image descriptor or training image descriptor by the BMMFV extractor comprises:
Down sampling the document hard copy image or each training image;
Recursively subdividing the down-sampled document hardcopy image or training image into a plurality of horizontal and vertical stripes;
Extracting a binary local descriptor for each stripe;
Reducing the dimension of the extracted binary local descriptor using a principal component analysis algorithm (PCA);
Computing a Bernoulli mixture model for the dimensionally reduced local descriptor;
Calculating a Fisher vector based on the calculated Bernoulli mixture model;
Performing power normalization and L2 normalization of the calculated Fischer vector;
And concatenating the normalized Fischer vector for each stripe with the image descriptor or the training image descriptor. 3. The method of claim 2, wherein each of the image descriptor and the training image descriptor is a numeric vector of integer, real, or binary numbers. 3. The method of claim 2, wherein the image feature is associated with at least one of a shape, texture and color of the document hard copy image and wherein the training image feature is associated with at least one of a shape, How to categorize a hardcopy image of a document. 3. The method of claim 2, wherein the first and second sets of image descriptor extractors are the same set of image descriptor extractors. 3. The method of claim 2, wherein the document hard copy image and the class of the training image are integer or text labels. 3. The method of claim 2, wherein storing the training set comprises:
Receiving the training set;
Selecting a random subset of the training image and the class label from the training set; And
Storing the random subset in a memory. ≪ Desc / Clms Page number 21 > 2. The method of claim 1, wherein the plurality of trained classifiers is a support vector machine (SVM). 14. The method of claim 13, wherein determining the most probable class by the trained meta classifier comprises:
Concatenating a plurality of probabilistic vectors estimated by the plurality of trained classifiers into a single vector;
Estimating a class probability of the document hard copy image using the SVM and the single vector; And
Selecting a class with the highest probability as the most probable class based on the estimated probability. ≪ Desc / Clms Page number 19 > 2. The method of claim 1, wherein estimating the class probability of the document hard copy image comprises obtaining a real vector representing a probability of a document hard copy image belonging to a specific class. Way. A classification module for classifying a document hardcopy image comprising image features describing an image feature of the document hardcopy image, the classification module comprising: an image processor for generating an image for the document hardcopy image input to the first set of image descriptor extractors, A first set of image descriptor extractors configured to extract descriptors,
A plurality of trained classifiers configured to estimate a class probability of the document hard copy image using the image descriptor;
A trained meta classifier for determining the most probable class of the hardcopy image of the document using the estimated class probability;
And assigning means for assigning the most probable class to the document hardcopy image to obtain a classified document hardcopy image. 17. The method of claim 16,
Further comprising a training module for obtaining the plurality of trained classifiers and the trained meta classifier,
A memory configured to store a training set comprising a training image and a class label associated with the training image, wherein the training image comprises a training image feature,
A second set of image descriptor extractors configured to receive the training images of the training set stored in the memory and to extract the training image descriptors describing the training image features of each training image;
And classifier training means configured to obtain a plurality of classifiers trained using the training image descriptor extracted by the image descriptor extractor of the second set and the class label associated with the training image, Wherein the training image is configured to estimate a class probability of the training image, and
Classifier training means configured to obtain a trained meta classifier using a class probability estimated by the plurality of trained classifiers and a class label associated with the training image, . A system for classifying document hard copies,
A hard copy classification apparatus, wherein the hard copy classification apparatus comprises:
An input tray for loading document hard copies,
Image capture means configured to capture an image of the document hard copy,
A notification indicator configured to display an assigned class for each of the document hard copy images,
And at least one output tray for the hard copy of the classified document,
An image processing system, comprising:
An apparatus for classifying a document hardcopy image configured to assign the class to each document hardcopy image,
And a routing module configured to assign the output tray to each of the document hardcopy according to a class assigned to each of the document hardcopy images,
Wherein the routing module is adapted to receive information about the number of output trays and the number of non-empty output trays from the hard copy classification apparatus and the document hard copy image, wherein the document hard copy image has the assigned class from the apparatus for classifying the document hard copy image Receiving a hard copy image of the document,
Assigning the number of output trays to each document hard copy according to the assigned class of each document hard copy image and the number of non-empty output trays,
And redirecting a hard copy of the document from the input tray to the assigned output tray. 19. The apparatus according to claim 18, wherein the image capturing means is any one of a scanner, a facsimile, a photo camera, a video camera, a reader for reading an image file from a storage medium, and an input unit for receiving the image file via the Internet A system for sorting hard copy documents.

Images