CN116645675A - Character recognition method, device, equipment and medium - Google Patents

Abstract

The present disclosure provides a character recognition method, which relates to the field of artificial intelligence. The method comprises the following steps: acquiring an image to be recognized, wherein the image to be recognized comprises N characters to be recognized which are irregularly arranged, the irregular arrangement comprises that the characters are not arranged according to a straight line, and N is an integer greater than or equal to 2; inputting the image to be recognized into a character recognition model; obtaining recognition results of the N characters to be recognized, which are output by the character recognition model; wherein the character recognition model comprises an encoder and a decoder, the character recognition model being configured to be pre-trained by: a first loss function value is obtained based on an encoding result of the encoder, a second loss function value is obtained based on a decoding result of the decoder, and the encoder and the decoder are updated based on the first loss function value and the second loss function value. The present disclosure also provides a character recognition apparatus, device, medium, and program product.

Description

Character recognition method, device, equipment and medium

technical field

The present disclosure relates to the field of artificial intelligence, and more specifically, to a character recognition method, device, equipment, medium and program product.

Background technique

The traditional OCR (optical character recognition, optical character recognition) recognition model is mainly effective for recognizing characters arranged in straight lines such as horizontal or vertical, etc., and for irregularly arranged characters such as curved, multi-line text or symbols, such as seals, formulas, etc. identification is not very effective.

In related technologies, for the recognition schemes of irregularly arranged characters, some of them need to transform the specially arranged characters into horizontal arrangement and then use the traditional OCR recognition model for recognition. The steps are cumbersome, and the accuracy and robustness are not good. Ideal question.

Contents of the invention

In view of the above problems, the present disclosure provides a character recognition method, device, equipment, medium and program product.

An aspect of an embodiment of the present disclosure provides a method for character recognition, including: acquiring an image to be recognized, wherein the image to be recognized includes N characters to be recognized that are irregularly arranged, and the irregular arrangement includes not arranged in a straight line , N is an integer greater than or equal to 2; the image to be recognized is input into the character recognition model; the recognition results of the N characters to be recognized outputted by the character recognition model are obtained; wherein the character recognition model includes an encoder and a decoder, the character recognition model is configured to be pre-trained through the following operations: obtain a first loss function value based on the encoding result of the encoder, obtain a second loss function value based on the decoding result of the decoder, and obtain a second loss function value based on The first loss function value and the second loss function value update the encoder and the decoder.

According to an embodiment of the present disclosure, the acquiring the image to be recognized includes: determining a target image; using a target detection model to determine a first target area from the target image, the first target area including the N characters to be recognized; Segmenting the first target area from the target image to obtain the image to be recognized.

According to an embodiment of the present disclosure, the obtaining the recognition results of the N characters to be recognized output by the character recognition model includes: using the encoder to process the image to be recognized, and obtaining the first Image features; using a classification function to process the first image features to obtain intermediate recognition results of the N characters to be recognized, the classification function is used to classify the first image features; the first image features and the first sequence of text are input into the decoder to obtain the recognition results of the N characters to be recognized output by the decoder, and the first sequence of text is obtained by shifting the recognition results of each character in the intermediate recognition results to the right M character positions are obtained, and M is an integer greater than or equal to 1.

According to an embodiment of the present disclosure, the decoder includes a cross-attention layer and a classification layer, the first image feature and the first sequence of text are input into the decoder, and the N outputted by the decoder is obtained. The recognition result of a character to be recognized includes: using the cross-attention layer to process the first image feature and the first sequence of text to obtain the first attention feature, wherein the cross-attention layer is configured to be based on The data is processed by a cross-attention mechanism; the classification layer is used to process the first attention feature to obtain the recognition results of the N characters to be recognized, and the classification layer is used to classify the first attention feature.

According to an embodiment of the present disclosure, the obtaining the recognition results of the N characters to be recognized output by the character recognition model includes: using the encoder to process the image to be recognized, and obtaining the first Image features: input the first image features into the decoder, and obtain the recognition results of the N characters to be recognized outputted by the decoder.

According to an embodiment of the present disclosure, the encoding result includes a second image feature, and the obtaining the first loss function value based on the encoding result of the encoder includes: inputting a training image into the encoder, and obtaining an output of the encoder The second image features, wherein, the training image includes irregularly arranged N characters to be recognized; based on the second image features, a first prediction tensor is obtained, and the first prediction tensor includes the N characters Character prediction information of the character to be recognized; obtaining a first loss function value between the first prediction tensor and annotated text, where the annotated text includes a label of each character to be recognized.

According to an embodiment of the present disclosure, the decoding result includes a second prediction tensor, and the obtaining the second loss function value based on the decoding result of the decoder includes: inputting the second image feature into the decoder to obtain The second prediction tensor output by the decoder, the second prediction tensor including character prediction information for the N characters to be recognized; obtaining the relationship between the second prediction tensor and the marked text The second loss function value of .

According to an embodiment of the present disclosure, the updating the encoder and the decoder based on the first loss function value and the second loss function value includes: according to the first loss function value and its first weight , to obtain a first weighted value; according to the second loss function value and its second weight, obtain a second weighted value; obtain a comprehensive loss function value according to the first weighted value and the second weighted value; according to the The integrated loss function value updates the encoder and the decoder to obtain a trained character recognition model.

According to an embodiment of the present disclosure, the decoder includes a cross-attention layer and a classification layer, the inputting the second image feature into the decoder, and obtaining the second prediction tensor output by the decoder includes : Input the second image feature and the second sequence of text into the decoder, the second sequence of text is obtained by moving the label of each character to be recognized in the label text to the right by M character positions, and M is greater than or an integer equal to 1; use the cross-attention layer to process the second image feature and the second sequence text to obtain a second attention feature, wherein the cross-attention layer is configured to be based on cross-attention The data is processed by a mechanism; the classification layer is used to process the second attention feature to obtain the second prediction tensor.

According to an embodiment of the present disclosure, the obtaining the first prediction tensor based on the second image feature includes: performing dimension-up processing on the second image feature to obtain a third image feature; inputting the third image feature To the classification function, get the first prediction tensor.

Another aspect of the embodiments of the present disclosure provides a character recognition device, including: an image acquisition module, configured to acquire an image to be recognized, wherein the image to be recognized includes N characters to be recognized irregularly arranged, the Irregular arrangements include not arranged in a straight line, N is an integer greater than or equal to 2; the third input module is used to input the image to be recognized into the character recognition model; the recognition result module is used to obtain the output of the character recognition model The recognition results of the N characters to be recognized; wherein, the character recognition model includes an encoder and a decoder, and the character recognition model is configured to be pre-trained through the following operations:

Obtaining a first loss function value based on an encoding result of the encoder, obtaining a second loss function value based on a decoding result of the decoder, updating the encoding based on the first loss function value and the second loss function value device and the decoder.

Another aspect of the embodiments of the present disclosure provides an electronic device, including: one or more processors; a storage device for storing one or more programs, wherein, when the one or more programs are executed by the one When executed by one or more processors, one or more processors execute the method as described above.

Another aspect of the embodiments of the present disclosure also provides a computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor executes the above method.

Another aspect of the embodiments of the present disclosure further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the above method is implemented.

The above-mentioned one or more embodiments have the following beneficial effects: in the pre-training process, the encoding result of the encoder is not only input to the decoder, but also the first loss function value related to the encoding result is obtained, and the decoding result based on the decoder The obtained second loss function values are jointly used in the training, so as to obtain more information for learning, speed up the training speed of the character recognition model, and improve the recognition effect. In the character recognition scene, the image to be recognized can be directly recognized based on the trained character recognition model. In the actual application scene, it has the characteristics of high recognition efficiency and accuracy, good robustness, simple and effective.

Description of drawings

Through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, the above content and other objects, features and advantages of the present disclosure will be more clear, in the accompanying drawings:

FIG. 1 schematically shows an application scenario diagram of a character recognition model training method and a character recognition method according to an embodiment of the present disclosure;

Fig. 2 schematically shows a flow chart of a method for training a character recognition model according to an embodiment of the present disclosure;

3A-3B schematically illustrate examples of training images according to embodiments of the present disclosure;

FIG. 4 schematically shows a training architecture diagram of a character recognition model according to an embodiment of the present disclosure;

Fig. 5 schematically shows a flow chart of obtaining a training image according to an embodiment of the present disclosure;

Fig. 6 schematically shows an example of a sample image according to an embodiment of the present disclosure;

Fig. 7 schematically shows a flow chart of obtaining a second prediction tensor according to an embodiment of the present disclosure;

Fig. 8 schematically shows a flow chart of obtaining a comprehensive loss function value according to an embodiment of the present disclosure;

FIG. 9 schematically shows a flowchart of a character recognition method according to an embodiment of the present disclosure;

Fig. 10 schematically shows a flow chart of obtaining an image to be recognized according to an embodiment of the present disclosure;

Fig. 11 schematically shows a flow chart of obtaining recognition results of N characters to be recognized according to an embodiment of the present disclosure;

Fig. 12 schematically shows a flow chart of obtaining recognition results of N characters to be recognized according to another embodiment of the present disclosure;

Fig. 13 schematically shows a flow chart of an end-to-end special character recognition method based on YOLO and TrOCR according to an embodiment of the present disclosure;

Fig. 14 schematically shows a structural block diagram of a training device for a character recognition model according to an embodiment of the present disclosure;

FIG. 15 schematically shows a structural block diagram of a character recognition device according to an embodiment of the present disclosure; and

16 schematically shows a block diagram of an electronic device suitable for implementing a character recognition model training method or a character recognition method according to an embodiment of the present disclosure;

It should be noted that, for the sake of clarity, in the drawings used to describe the embodiments of the present disclosure, the overall/partial structure or the size of the whole/partial area may be enlarged or reduced, that is, these drawings are not drawn according to the actual scale .

Detailed ways

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. It should be understood, however, that these descriptions are exemplary only, and are not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Also, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concept of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting of the present disclosure. The terms "comprising", "comprising", etc. used herein indicate the presence of stated features, steps, operations and/or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

All terms (including technical and scientific terms) used herein have the meaning commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted to have a meaning consistent with the context of this specification, and not be interpreted in an idealized or overly rigid manner.

Where expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted as those skilled in the art would normally understand the expression (for example, "having A, B, and C A system of at least one of "shall include, but not be limited to, systems with A alone, B alone, C alone, A and B, A and C, B and C, and/or A, B, C, etc. ).

In order to facilitate the understanding of the technical solutions of the present disclosure, the terms involved in some embodiments of the present disclosure are defined as follows:

Yolo: A one-stage target detection algorithm with fast speed and high accuracy.

TrOCR: An OCR recognition method based on Vision Tranformer and Transformer Decoder, which has the characteristics of high accuracy and wide application scenarios.

Tensor: refers to a multidimensional array, which is based on the promotion of vectors and matrices.

Cross Attention Mechanism: Also known as Cross Attention, it is a variant of the multi-head attention mechanism that can be used in a sequence-to-sequence model, allowing the model to focus on different parts of the input while processing the input.

Classification function: a function used to map input data to different classification spaces to obtain classification results, such as softmax function, sigmod function or CTC (Connectionist Temporal Classification, connection time series classification function) function, etc.

Fig. 1 schematically shows an application scene diagram of a character recognition model training method and a character recognition method according to an embodiment of the present disclosure. It should be noted that, what is shown in FIG. 1 is only an example of the system architecture to which the embodiments of the present disclosure can be applied, so as to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other device, system, environment or scenario.

As shown in FIG. 1 , an application scenario 100 according to this embodiment may include terminal devices 101 , 102 , and 103 , a network 104 and a server 105 . The network 104 is used as a medium for providing communication links between the terminal devices 101 , 102 , 103 and the server 105 . Network 104 may include various connection types, such as wires, wireless communication links, or fiber optic cables, among others.

Users can use terminal devices 101 , 102 , 103 to interact with server 105 via network 104 to receive or send messages and the like. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (just for example).

The terminal devices 101, 102, 103 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers and the like.

The server 105 may be a server that provides various services, such as a background management server that provides support for websites browsed by users using the terminal devices 101 , 102 , 103 (just an example). The background management server can analyze and process received data such as user requests, and feed back processing results (such as webpages, information, or data obtained or generated according to user requests) to the terminal device.

The server 105 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud computing, network services, and middleware services.

In some embodiments, the training set can be prepared on the terminal devices 101, 102, 103, and the server 105 can be run to implement model training. The terminal devices 101, 102, 103 can take pictures with characters to be recognized by the camera, or obtain the pictures with characters to be recognized by downloading from the network, and send the recognition request to the server 105, and obtain the response from the server 105 to run the character recognition model. recognition results. In some embodiments, the character recognition model can be deployed on the terminal devices 101, 102, 103, and the training and recognition processes are performed locally on the terminal devices 101, 102, 103.

It should be understood that the numbers of terminal devices, networks and servers in Fig. 1 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

Based on the scene described in FIG. 1 , the training method of the character recognition model in the embodiment of the present disclosure will be described in detail through FIGS. 2 to 8 .

Fig. 2 schematically shows a flowchart of a method for training a character recognition model according to an embodiment of the present disclosure. 3A-3B schematically illustrate examples of training images according to an embodiment of the present disclosure. Fig. 4 schematically shows a training architecture diagram of a character recognition model according to an embodiment of the present disclosure. A character recognition model includes an encoder and a decoder.

As shown in Figure 2, the training method of the character recognition model of this embodiment comprises:

In operation S210, the training image is input into the encoder of the character recognition model, and the second image feature output by the encoder is obtained. The encoding result of the encoder includes the second image feature, wherein the training image includes N characters to be recognized that are arranged irregularly , the irregular arrangement includes not being arranged in a straight line, and N is an integer greater than or equal to 2.

In the embodiment of the present disclosure, before training the model, a large number of training images need to be collected, and the large number of training images may include different types of text images. As shown in FIG. 3A to FIG. 3B , the training images may include seal images and formula images, and may also include other irregularly arranged characters or symbols. Straight line arrangement includes writing from left to right or from top to bottom, all characters are roughly on a straight line and this line almost coincides with the horizontal/vertical direction.

Referring to Figure 3A, various seal shapes will affect the arrangement direction of the stamp characters, or there may be some stamp characters that are complete texts. Curved text means that the characters are almost not on the same straight line, and the center points of the characters are roughly connected. in a curved line. In addition, due to the usage scenarios of the stamp, overlapping texts may appear, that is, the irregularly arranged text shown at the bottom of FIG. 3A overlaps the regularly arranged text.

Continuing to refer to FIG. 3A , each stamp image includes at least one stamp, and the stamps can be various types of stamps such as a special seal for contracts and a special seal for invoices. The seal can be a regular shape such as a circular seal, an oval seal, a polygonal seal (for example, a rectangular seal), or an irregular shape.

Referring to FIG. 3B , the upper side shows a handwritten formula image, and the lower side shows label text in LaTex language. The training image may also be a print formula image. As shown in FIG. 3B , an image that conveys a group of information to the reader through multi-line arrangement is also within the scope of the present disclosure.

The training image may be a text image uploaded by a user or downloaded from the network, or a text image directly collected by an electronic device through a camera, and the source of the training image is not limited here.

In operation S210, the encoder is used to extract the second image features of the training image. Referring to the architecture of the TrOCR model shown in FIG. 4 , the encoder can be a Vision Transformer model. The Vision Transformer model is a classification model that combines computer vision (CV for short) and NLP to block the original image, flatten it into a sequence, input the coding part, and then process the coded output to complete the classification task. In particular, the present disclosure does not intend to limit the encoder in operation S210 to Vision Transformer, and it can be flexibly replaced with other image feature extraction models, such as Swin Transformer and CSwinTransformer.

In operation S220, a first prediction tensor is obtained based on the second image feature, and the first prediction tensor includes character prediction information for the N characters to be recognized.

Exemplarily, the prediction can be made directly based on the second image feature, for example, the first prediction tensor can be obtained by using a classification function, a classification model based on a neural network, or a classification model based on other machine learning algorithms, and the character prediction information can be obtained from The information in the dictionary to obtain the predicted results.

Specifically, the number of training images is recorded as Batch_size, and Batch_size can take an integer greater than or equal to 1. The number of training images equal to Batch_size is input to the encoder, and the first image feature is represented by Batch_size×L×C, where L represents the length of the character to be recognized, and C represents the dimension of the tensor.

In some embodiments, dimension-up processing is performed on the second image feature to obtain the third image feature. The third image feature is input to the classification function to obtain the first prediction tensor. For example, the first image feature of Batch_size×L×C is linearly transformed (just an example) to realize dimension-up processing to obtain Batch_size×L×D, where D is the dimension of the dictionary. Then input the second image feature of Batch_size×L×D to the classification function to obtain the first prediction tensor. After the dimension of the second image feature is increased, the dimension of the third image feature is the same as that of the dictionary, which is convenient for classification processing.

In operation S230, the second image feature is input into the decoder to obtain the second prediction tensor output by the decoder, the decoding result of the decoder includes the second prediction tensor, and the second prediction tensor includes characters for N characters to be recognized Forecast information.

Referring to FIG. 4 , the decoder in operation S230 may be a Transformer Decoder, which is a decoder in a natural language processing model for performing natural language processing tasks. It can make a prediction according to the first image feature to obtain the second prediction tensor, and the character prediction information also represents the information that can obtain the prediction result from the dictionary.

In operation S240, obtain the first loss function value between the first prediction tensor and the labeled text, and the second loss function value between the second predicted tensor and the labeled text, the labeled text includes the label of each character to be recognized .

The label text is the label text of the character to be recognized in the training image, such as the seal image shown in Figure 3A, the label text corresponding to the training image is "XX Co., Ltd", the ideal of the first prediction tensor or the second prediction tensor The result is that the token "XX Co., Ltd." is identified. Among them, the dictionary includes mappable words during translation, representing all translatable words when the model performs text translation. Based on the dictionary, each character can be mapped to an integer number (token), for example, "XX Co., Ltd." is converted into a sequence of token integers: "[1919, 25, 103, 429, 17, 151]". When calculating the first loss function value or the second loss function value, the numerical calculation of the loss value is performed using the token sequence of the text.

In operation S250, a comprehensive loss function value is obtained according to the first loss function value and the second loss function value.

In operation S260, the encoder and the decoder are updated according to the integrated loss function value to obtain a trained character recognition model.

According to the comprehensive loss function value, optimizers such as Adam or AdamW can be used for model training until the training image input is completed, or the comprehensive loss function value reaches the target convergence condition.

By obtaining the first prediction tensor and the second prediction tensor through operation S220 and operation S230 above, the character prediction result can be output directly. Since the model can directly extract the features of the text without rotating the text in the stamp image, it can avoid the problem of random rotation of the stamp and the inaccurate recognition results caused by the change of the arrangement trend of the text in the stamp on the paper .

According to the embodiments of the present disclosure, more information can be obtained from the output of the encoder and decoder for learning during the training process, and the training speed and recognition effect of the character recognition model can be accelerated. It can be directly recognized based on the extracted image features, and the trained character recognition model has the characteristics of high recognition efficiency and accuracy, good robustness, simplicity and effectiveness in practical application scenarios.

It should be noted that some steps of the above methods may be executed individually or in combination, and may be executed in parallel or sequentially, and are not limited to the specific operation sequence shown in the figure. For example, after the first prediction tensor is calculated in operation S220, the first loss function value can be calculated in operation S240 instead of calculating the first loss function value after operation S230.

In related technologies, there are one-stage end-to-end scene text recognition methods such as ABCNet and PGNet. This type of algorithm includes text detection, transformation, and recognition functions at the same time, but the recognition accuracy is not very high, and the labeling cost is relatively high, and the position of the text needs to be marked word by word. If there are 20 characters on a seal on a picture, about 40 coordinate points need to be marked, which is quite expensive. The present disclosure proposes a two-stage approach, one of which is to obtain training images as shown in FIG. 5 . Another stage is model training as shown in FIG. 2 .

Fig. 5 schematically shows a flow chart of obtaining training images according to an embodiment of the present disclosure. As shown in Figure 5, this embodiment obtains the training image and includes:

In operation S510, a sample image is determined.

For example, the sample image may be any image including a seal or a formula, such as an invoice image, a receipt image, or a document image, and the present disclosure is not limited thereto.

Fig. 6 schematically shows an example of a sample image according to an embodiment of the present disclosure. Some of the information in the illustration in Figure 6 is covered with texture images to avoid adverse effects. As shown in FIG. 6 , the receipt includes a circular seal, and the circular seal is a special seal for receipts, including relevant text information of the seal.

For example, the sample image may be an original image directly collected by the image acquisition device, or an image obtained after preprocessing the original image. For example, in order to avoid the influence of the data quality and data imbalance of the input image on the stamp image recognition, before processing the sample image, it may also include the operation of preprocessing the input image. Preprocessing can eliminate irrelevant information or noise information in the input image, so as to process the input image better. The preprocessing may include, for example, scaling, cropping, gamma (Gamma) correction, image enhancement, or noise reduction filtering on the input image.

In operation S520, a second target area is determined from the sample image by using the target detection model, the second target area includes N characters to be recognized.

In operation S530, the second target area is segmented from the sample image to obtain a training image.

According to the embodiment of the present disclosure, a two-stage end-to-end OCR recognition method is proposed, which has a good effect on any arrangement of text symbols. This method only needs to label the text on the picture, the overall coordinates and character representation of the symbol, and does not need to label the coordinates of each text symbol in a special arrangement, which is simple and effective.

In addition, referring to FIG. 6 , the sample image may include other redundant characters, such as amount, customer signature, or receipt title, etc., and there may also be multiple stamps. By determining the second target area, accurate data can be input to the character recognition model, thereby improving the character recognition accuracy.

In some embodiments, the classification function includes a joint time series classification function, and obtaining the first loss function value between the first prediction tensor and the labeled text in operation S240 includes: inputting the first predicted tensor and the labeled text into the first loss function to obtain the value of the first loss function, wherein the first loss function is constructed based on the connection time series classification function, and is used to compare the difference between the first prediction tensor and the labeled text. Referring to FIG. 4, the first loss function may be CTCloss.

According to an embodiment of the present disclosure, the obtained first loss function value represents the difference between the first prediction tensor and the labeled text, and cooperates with the difference between the second predicted tensor obtained by the decoder and the labeled text to participate in During the training process of the character recognition model, more image information and character information can be extracted for learning, and the model convergence speed and recognition accuracy can be improved.

Fig. 7 schematically shows a flow chart of obtaining a second prediction tensor according to an embodiment of the present disclosure. As shown in FIG. 7, this embodiment is one of the embodiments of operation S230, and obtaining the second prediction tensor includes:

In operation S710, the second image feature and the second sequence of text are input into the decoder of the character recognition model, and the second sequence of text is obtained by moving the label of each character to be recognized in the label text to the right by M character positions, where M is greater than or An integer equal to 1. The decoder includes a cross-attention layer and a classification layer. In some embodiments, M may be equal to 1.

In operation S720, the second image feature and the second sequence of text are processed by using a cross-attention layer to obtain a second attention feature, wherein the cross-attention layer is configured to process data based on a cross-attention mechanism.

In operation S730, the classification layer is used to process the second attention feature to obtain a second prediction tensor.

According to an embodiment of the present disclosure, the second prediction tensor is obtained by the decoder through the cross-attention layer and the classification layer according to the second image feature and the second sequence of text, because the second sequence of text passes each label text to be recognized The label of the character is obtained by shifting M character positions to the right, which can enable the decoder to fully obtain context information during data processing, and the output recognition result is more accurate.

In some embodiments, obtaining the second loss function value between the second prediction tensor and the labeled text in operation S240 includes: inputting the second predicted tensor and the labeled text into the second loss function to obtain the second loss function value , where the second loss function is used to compare the difference between the second prediction tensor and the labeled text.

Referring to FIG. 4 , the second loss function may be a classification loss function, specifically, a loss function such as CrossEntropy (cross-entropy loss function), KLDivLoss (divergence loss function), and the like.

Fig. 8 schematically shows a flow chart of obtaining a comprehensive loss function value according to an embodiment of the present disclosure. As shown in FIG. 8, this embodiment is one of the embodiments of operation S250, and obtaining the comprehensive loss function value includes:

In operation S810, a first weighted value is obtained according to the first loss function value and the first weighted value thereof.

In operation S820, a second weighted value is obtained according to the second loss function value and its second weight.

In operation S830, a comprehensive loss function value is obtained according to the first weighted value and the second weighted value.

Referring to FIG. 4 , the third loss function may be a weighted mixed loss function Loss, as shown in Equation 1.

Loss=αCTCLoss(y,y′ ₁ )+(1-α)ClassfyLoss(y,y′ ₂ ) Formula 1

Wherein, α is the first weight, and its value is between 0-1. 1-α is the second weight. CTCLoss(y, y′ ₁ ) is the first loss function value, ClassfyLoss(y, y′ ₂ ) is the second loss function value, ClassfyLoss is the second loss function, y is the sequence vector of the labeled text, and y′ ₁ is the first One prediction tensor, _y′2 is the second prediction tensor.

The first loss function value and the second loss function value are used to reflect different optimization situations of the encoder and decoder, and the respective weights are used to balance the importance between different encoders and decoders. The value of α can be assigned a certain value manually, or an initial value can be assigned first, and then dynamically updated during the training process.

For example, use the gradient descent algorithm to reversely update the encoder parameters, decoder parameters, and α values. The greater the weight of a loss function value, the more important the task (encoder or decoder) is, and the greater the contribution to the gradient, and vice versa. . The ability to automatically adjust the weights during training can further affect the update of parameters in the encoder and decoder. For a task with a larger weight, the gradient of the corresponding loss function is larger, which will make the model parameters move to the optimal solution of the task faster. For tasks with smaller weights, the gradient of the corresponding loss function is smaller, which will make the model parameters move more slowly towards the optimal solution of the task.

According to the embodiments of the present disclosure, in the training phase, multiple factors that affect the recognition accuracy, such as the second image feature, the first prediction tensor, and the second prediction tensor, can be considered, and the recognition accuracy can be optimized during actual recognition. Effect.

After the training phase is over, deploy the trained character recognition model to the actual application scenario. On the basis of the foregoing FIGS. 2 to 8 , the character recognition method will be further described below in conjunction with FIGS. 9 to 12 .

Fig. 9 schematically shows a flowchart of a character recognition method according to an embodiment of the present disclosure. As shown in Figure 9, the character recognition method of this embodiment comprises:

In operation S910, after the image to be recognized is obtained, the image to be recognized is input into the character recognition model, wherein the image to be recognized includes N characters to be recognized that are irregularly arranged, and the irregular arrangement includes not being arranged in a straight line, and N is greater than or equal to Integer of 2.

Exemplarily, the image to be recognized may be as shown in FIG. 3A to FIG. 3B . The image to be recognized may be a text image uploaded by a user or downloaded from the network, or a text image directly collected by an electronic device through a camera, and the source of the image to be recognized is not limited here.

The character recognition model is configured to be pre-trained by the following operations: obtain the first loss function value based on the encoding result of the encoder, obtain the second loss function value based on the decoding result of the decoder, and obtain the second loss function value based on the first loss function value and the second loss function The value updates the encoder and decoder. Specifically, you can refer to the training method described in any one of the embodiments described in FIGS. 2 to 8 .

In operation S920, the recognition results of N characters to be recognized outputted by the character recognition model are obtained.

If the image to be recognized is a seal image, the recognition result may include the text of the seal, as shown in FIG. 3A . If the image to be recognized is a formula image, the recognition result can be formula symbols or Latex language, as shown in the lower side of FIG. 3B .

According to the embodiments of the present disclosure, the features of the text can be directly extracted without performing operations such as rotation on the text in the image, which is equivalent to directly "translating" the recognition result from the image to be recognized. It can efficiently identify pictures of special characters such as various seals, formulas, and multi-line texts. It has the characteristics of high accuracy, good robustness, simplicity and effectiveness.

Fig. 10 schematically shows a flow chart of obtaining an image to be recognized according to an embodiment of the present disclosure. As shown in Figure 10, this embodiment obtains training image and comprises:

In operation S1010, a target image is determined. The target image may be as shown in FIG. 6 , but the disclosure is not limited thereto, and may also be a document image, an invoice image, and the like.

In operation S1020, a first target area is determined from the target image by using the target detection model, the first target area includes N characters to be recognized.

In operation S1030, the first target area is segmented from the target image to obtain an image to be recognized.

According to the embodiment of the present disclosure, a two-stage end-to-end OCR recognition method is used, by determining the first target area, accurate data can be input to the character recognition model, and the accuracy of character recognition can be improved. Good results.

Fig. 11 schematically shows a flow chart of obtaining recognition results of N characters to be recognized according to an embodiment of the present disclosure. As shown in Figure 11, this embodiment is one of the embodiments of operation S910, obtaining N characters to be recognized includes:

In operation S1110, the image to be recognized is processed by the encoder of the character recognition model, and a first image feature output by the encoder is obtained.

In operation S1120, the first image feature is processed by using a classification function to obtain intermediate recognition results of N characters to be recognized, and the classification function is used to classify the first image feature.

In operation S1130, the first image feature and the first sequence of text are input into the decoder of the character recognition model, and the recognition results of N characters to be recognized output by the decoder are obtained, and the first sequence of text is recognized by each character in the intermediate recognition result The result is obtained by shifting M characters to the right, where M is an integer greater than or equal to 1.

Referring to Fig. 4, the intermediate recognition result may correspond to the first prediction tensor, which is different from the right-shifted text obtained from the label text in the training stage, and the intermediate recognition result is used as the right-shifted text, which is input to the decoder together with the first image feature . The decoder is configured to reorder the intermediate recognition results based on the first image feature, and output the final recognition result, so as to improve the recognition accuracy.

In some embodiments, the first image feature and the first sequence of text are processed by using a cross-attention layer to obtain the first attention feature, wherein the cross-attention layer is configured to process data based on a cross-attention mechanism. The classification layer is used to process the first attention feature, and the recognition results of N characters to be recognized are obtained.

Fig. 12 schematically shows a flow chart of obtaining recognition results of N characters to be recognized according to another embodiment of the present disclosure. As shown in FIG. 12, this embodiment is one of the embodiments of operation S910, obtaining N characters to be recognized includes:

In operation S1210, the image to be recognized is processed by the encoder of the character recognition model, and a first image feature output by the encoder is obtained.

In operation S1220, the first image feature is input into the decoder of the character recognition model, and the recognition results of N characters to be recognized outputted by the decoder are obtained.

Different from the embodiment shown in FIG. 11 , the embodiment shown in FIG. 12 does not obtain an intermediate recognition result, but only inputs the first image feature to the decoder, and the recognition speed is fast and has sufficient accuracy. [CLS] can be input to the decoder as the text shifted to the right in Fig. 4 along with the first image feature. [CLS] A special encoding marker denoted as the start of a text sequence. It can be understood that [CLS] is only an example, and the flags can be set flexibly.

Referring to the two different ways of obtaining recognition results in FIG. 11 and FIG. 12 , the method in FIG. 11 or FIG. 12 can be considered according to the first weight "α" and the second weight "1-α" in the above formula 1. If α is large and the accuracy of model prediction is pursued, the CTC function can be used to realize the CTC beam search first, and then the intermediate recognition result is input to the decoder for decoding in the way of AttentionRescore, that is, the way in Figure 12. If α is small and the speed of model prediction is pursued, only the input to the decoder can be used to decode by Attention beam search, that is, the method in Figure 11.

Fig. 13 schematically shows a flowchart of an end-to-end special character recognition method based on YOLO and TrOCR according to an embodiment of the present disclosure. With joint reference to Figures 1 to 12 and any of the above-mentioned embodiments, the end-to-end special character recognition method shown in Figure 13 includes:

In operation S1310, use YOLO-V7 to train the target detection model, realize the detection of the character structure, detect and cut out the picture containing only the entire character structure. The character structure is a target area including characters to be recognized, which can be represented as a target area box.

In operation S1320, build a character recognition model based on TrOCR, its Encoder part can be any Vision Transformer model or image feature extraction model, such as Swin Transformer, CSwin Transformer etc., its Decoder part can be Transformer Decoder algorithm.

In operation S1330, a weighted hybrid loss function based on the CTC loss function and the classification loss function is constructed as a third loss function.

In operation S1340, perform model training on the constructed TrOCR character recognition model. Input the character structure picture, the character sequence id and label data (character sequence id) shifted one bit to the right into the model, and use the weighted mixed loss function constructed by S1330, and use Adam or AdamW and other optimizers for model training. During the model training process, a series of data enhancement processes can be performed on the picture data, such as Gaussian blur, random mask, panning pictures, adjusting picture color, saturation and brightness, etc.

In the training phase, in order to speed up the convergence speed of the model, the corresponding pre-trained model parameters for image classification can be loaded in the encoder part of the model, and the corresponding pre-trained model parameters for natural language processing can be loaded in the decoder part of the model , such as ERNIE3.0, BERT, etc. In addition, in order to obtain better model generalization and save the cost of manual labeling, computer programs can also be used to synthesize a large number of simulated labeling samples and use a small amount of manually labeled real data for tuning.

In operation S1350, the trained TrOCR model is input with a picture of an unknown character sequence, decoded using an Attention beam search method, and the character sequence on the picture is predicted for verification.

In operation S1360, the object detection model and the character recognition model are connected in series to form an end-to-end recognition model. In the end-to-end application, input the entire document picture to the end recognition model, use the target detection model to obtain the slice of the character structure to obtain the image to be recognized, and then decode the slice character recognition model to predict the character sequence on the slice.

According to the embodiments of the present disclosure, an end-to-end character recognition scheme is proposed, which can effectively recognize irregularly arranged characters, symbols, etc., and has the characteristics of high accuracy and good robustness. It can be effectively used for header recognition, formula recognition, and multi-line text recognition of various seals such as round stamps, oval stamps, and square stamps.

It can be understood that the target detection model used in this embodiment may be any target model. The Encoder used in the TrOCR algorithm in the present invention can be any image feature extraction model, and the Decoder part can be any natural language model based on Transformer Encoder/Decoder. The classification loss function in the mixed loss function used in the present invention can also be any classification loss function.

Based on the above character recognition model training method and character recognition method, the present disclosure also provides a character recognition model training device and a character recognition device. The device will be described in detail below with reference to FIGS. 14 and 15 .

Fig. 14 schematically shows a structural block diagram of a training device for a character recognition model according to an embodiment of the present disclosure.

As shown in Figure 14, the training device 1400 of the character recognition model of this embodiment includes a first input module 1410, a first prediction module 1420, a second input module 1430, a loss calculation module 1440, a comprehensive loss module 1450 and a model update module 1460 .

The first input module 1410 can perform operation S210, which is used to input the training image into the encoder of the character recognition model to obtain the second image features output by the encoder, wherein the training image includes N characters to be recognized that are arranged irregularly, irregular Arrangement includes non-linear arrangement, and N is an integer greater than or equal to 2.

The first prediction module 1420 may perform operation S220 for obtaining a first prediction tensor based on the second image feature, the first prediction tensor including character prediction information for the N characters to be recognized.

The second input module 1430 can perform operation S230, which is used to input the second image feature into the decoder of the character recognition model, and obtain the second prediction tensor output by the decoder, the second prediction tensor includes characters for N characters to be recognized Forecast information.

In some embodiments, the second input module 1430 may also perform operation S710 to operation S730, which will not be repeated here.

The loss calculation module 1440 may perform operation S240 for obtaining a first loss function value between the first predicted tensor and the labeled text, and a second loss function value between the second predicted tensor and the labeled text, the labeled text includes A label for each character to be recognized.

The comprehensive loss module 1450 may perform operation S250 for obtaining a comprehensive loss function value according to the first loss function value and the second loss function value.

In some embodiments, the second input module 1430 may also perform operation S810 to operation S830, which will not be repeated here.

The model update module 1460 may perform operation S260 for updating the encoder and the decoder according to the integrated loss function value to obtain a trained character recognition model.

In some embodiments, the training device 1400 may further include a first object detection module, which is configured to perform operation S510 to operation S530, which will not be repeated here.

In particular, the training device 1400 includes modules for executing the steps of any one of the embodiments described above in FIG. 2 to FIG. 8 .

Fig. 15 schematically shows a structural block diagram of a character recognition device according to an embodiment of the present disclosure.

As shown in FIG. 15 , the character recognition device 1500 of this embodiment includes a third input module 1510 and a recognition result module 1520 . In some embodiments, the character recognition apparatus 1500 may further include an image acquisition module, which is used to acquire an image to be recognized, which is not shown in FIG. 15 .

The third input module 1510 can perform operation S910, which is used to input the image to be recognized into the character recognition model, wherein the character recognition model is obtained according to the training method described in any one of the above embodiments, and the image to be recognized includes N to-be-recognized images arranged irregularly. Recognize characters, irregular arrangement includes not arranged in a straight line, N is an integer greater than or equal to 2.

The recognition result module 1520 may perform operation S930 for obtaining recognition results of N characters to be recognized outputted by the character recognition model.

The character recognition model is configured to be pre-trained by the following operations: obtain the first loss function value based on the encoding result of the encoder, obtain the second loss function value based on the decoding result of the decoder, and obtain the second loss function value based on the first loss function value and the second loss function The value updates the encoder and decoder. Specifically, reference may be made to the training method described in any one of the embodiments illustrated in FIGS. 2 to 8 .

In some embodiments, the recognition result module 1520 may also perform operation S1110 to operation S1130, or operation S1210 to operation S1220, which will not be repeated here.

In some embodiments, the character recognition apparatus 1500 may further include a second object detection module, which is configured to perform operation S1010 to operation S1030, which will not be repeated here.

In particular, the character recognition device 1500 includes modules for executing the steps of any one of the embodiments described above in FIG. 9 to FIG. 12 .

It should be noted that the implementations of modules/units/subunits, etc., the technical problems solved, the functions realized, and the technical effects achieved in the embodiments of the device part are respectively the same as those of the corresponding steps in the embodiments of the method part. , the technical problems solved, the functions realized, and the technical effects achieved are the same or similar, and will not be repeated here.

According to an embodiment of the present disclosure, any number of modules in the training device 1400 or the character recognition device 1500 can be implemented in one module, or any one module can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules may be combined with at least part of the functions of other modules and implemented in one module.

According to an embodiment of the present disclosure, at least one of the training device 1400 or the character recognition device 1500 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, A system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuits, such as hardware or firmware, or in software, hardware, and firmware Any one of them or an appropriate combination of any of them. Alternatively, at least one of the training device 1400 or the character recognition device 1500 may be at least partially implemented as a computer program module, and when the computer program module is executed, corresponding functions may be performed.

Fig. 16 schematically shows a block diagram of an electronic device suitable for implementing a character recognition model training method or a character recognition method according to an embodiment of the present disclosure.

As shown in FIG. 16, an electronic device 1600 according to an embodiment of the present disclosure includes a processor 1601, which can be loaded into a random access memory (RAM) 1603 according to a program stored in a read-only memory (ROM) 1602 or from a storage section 1608. Various appropriate actions and processing are performed by the program. The processor 1601 may include, for example, a general-purpose microprocessor (eg, a CPU), an instruction set processor and/or related chipsets, and/or a special-purpose microprocessor (eg, an application-specific integrated circuit (ASIC)), and the like. Processor 1601 may also include on-board memory for caching purposes. The processor 1601 may include a single processing unit or a plurality of processing units for executing different actions of the method flow according to the embodiments of the present disclosure.

In the RAM 1603, various programs and data necessary for the operation of the electronic device 1600 are stored. The processor 1601 , ROM 1602 , and RAM 1603 are connected to each other through a bus 1604 . The processor 1601 executes various operations according to the method flow of the embodiment of the present disclosure by executing programs in the ROM 1602 and/or RAM 1603 . It is to be noted that the programs may also be stored in one or more memories other than the ROM 1602 and the RAM 1603 . The processor 1601 may also perform various operations according to the method flow of the embodiments of the present disclosure by executing programs stored in one or more memories.

According to an embodiment of the present disclosure, the electronic device 1600 may further include an input/output (I/O) interface 1605 which is also connected to the bus 1604 . The electronic device 1600 may also include one or more of the following components connected to the I/O interface 1605: an input part 1606 including a keyboard, a mouse, and the like. An output section 1607 such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., a speaker, etc. is included. A storage section 1608 including a hard disk and the like. And a communication section 1609 including a network interface card such as a LAN card, a modem, and the like. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to the I/O interface 1605 as needed. A removable medium 1611, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is mounted on the drive 1610 as necessary so that a computer program read therefrom is installed into the storage section 1608 as necessary.

The present disclosure also provides a computer-readable storage medium, which may be included in the device/apparatus/system described in the above embodiments. It can also exist independently without being assembled into the equipment/device/system. The above-mentioned computer-readable storage medium carries one or more programs, and when the above-mentioned one or more programs are executed, the method according to the embodiment of the present disclosure is realized.

According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as may include but not limited to: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM) , erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include one or more memories other than the ROM 1602 and/or the RAM 1603 and/or the ROM 1602 and the RAM 1603 described above.

Embodiments of the present disclosure also include a computer program product, which includes a computer program including program codes for executing the methods shown in the flowcharts. When the computer program product runs in the computer system, the program code is used to make the computer system realize the method provided by the embodiments of the present disclosure.

When the computer program is executed by the processor 1601, the above-mentioned functions defined in the system/apparatus of the embodiment of the present disclosure are performed. According to the embodiments of the present disclosure, the above-described systems, devices, modules, units, etc. may be implemented by computer program modules.

In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program can also be transmitted and distributed in the form of a signal on network media, downloaded and installed through the communication part 1609, and/or installed from the removable media 1611. The program code contained in the computer program can be transmitted by any appropriate network medium, including but not limited to: wireless, wired, etc., or any appropriate combination of the above.

In such an embodiment, the computer program may be downloaded and installed from a network via communication portion 1609 and/or installed from removable media 1611 . When the computer program is executed by the processor 1601, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to the embodiments of the present disclosure, the above-described systems, devices, devices, modules, units, etc. may be implemented by computer program modules.

According to the embodiments of the present disclosure, the program codes for executing the computer programs provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, specifically, high-level procedural and/or object-oriented programming language, and/or assembly/machine language to implement these computing programs. Programming languages include, but are not limited to, programming languages such as Java, C++, python, "C" or similar programming languages. The program code can execute entirely on the user computing device, partly on the user device, partly on the remote computing device, or entirely on the remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (for example, using an Internet service provider). business to connect via the Internet).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that includes one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block in the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a A combination of dedicated hardware and computer instructions.

Those skilled in the art can understand that various combinations and/or combinations of the features described in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the various embodiments of the present disclosure and/or the features described in the claims can be combined and/or combined in various ways. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the various embodiments have been described separately above, this does not mean that the measures in the various embodiments cannot be advantageously used in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the present disclosure, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims (14)

1. A character recognition method, comprising: Acquiring an image to be recognized, wherein the image to be recognized includes N characters to be recognized that are irregularly arranged, the irregular arrangement includes not being arranged in a straight line, and N is an integer greater than or equal to 2; Inputting the image to be recognized into the character recognition model; obtaining the recognition results of the N characters to be recognized output by the character recognition model; Wherein, the character recognition model includes an encoder and a decoder, and the character recognition model is configured to be pre-trained through the following operations: Obtaining a first loss function value based on an encoding result of the encoder, obtaining a second loss function value based on a decoding result of the decoder, updating the encoding based on the first loss function value and the second loss function value device and the decoder. 2. The method according to claim 1, wherein said obtaining the image to be recognized comprises: Determine the target image; Using a target detection model to determine a first target area from the target image, the first target area includes the N characters to be recognized; Segmenting the first target area from the target image to obtain the image to be recognized. 3. The method according to claim 1, wherein said obtaining the recognition results of said N characters to be recognized outputted by said character recognition model comprises: using the encoder to process the image to be recognized to obtain a first image feature output by the encoder; Processing the first image feature by using a classification function to obtain an intermediate recognition result of the N characters to be recognized, the classification function being used to classify the first image feature; The first image feature and the first sequence of text are input into the decoder to obtain the recognition results of the N characters to be recognized output by the decoder, and the first sequence of text is obtained by adding the intermediate recognition results to Each character recognition result is obtained by shifting M character positions to the right, where M is an integer greater than or equal to 1. 4. The method according to claim 3, wherein the decoder comprises a cross-attention layer and a classification layer, the first image feature and the first sequence of text are input into the decoder to obtain the decoded The recognition results of the N characters to be recognized that the device outputs include: Using the cross-attention layer to process the first image feature and the first sequence of text to obtain a first attention feature, wherein the cross-attention layer is configured to process data based on a cross-attention mechanism; The classification layer is used to process the first attention feature to obtain recognition results of the N characters to be recognized, and the classification layer is used to classify the first attention feature. 5. The method according to claim 1, wherein said obtaining the recognition results of said N characters to be recognized outputted by said character recognition model comprises: using the encoder to process the image to be recognized to obtain a first image feature output by the encoder; The first image feature is input into the decoder, and the recognition results of the N characters to be recognized outputted by the decoder are obtained. 6. The method according to claim 1, wherein the encoding result comprises a second image feature, and the obtaining the first loss function value based on the encoding result of the encoder comprises: Inputting a training image into the encoder to obtain a second image feature output by the encoder, wherein the training image includes irregularly arranged N characters to be recognized; Obtaining a first prediction tensor based on the second image feature, the first prediction tensor including character prediction information for the N characters to be recognized; Obtain a first loss function value between the first prediction tensor and annotated text, where the annotated text includes a label of each character to be recognized. 7. The method according to claim 6, wherein the decoding result comprises a second prediction tensor, and the obtaining a second loss function value based on the decoding result of the decoder comprises: Inputting the second image feature into the decoder to obtain the second prediction tensor output by the decoder, the second prediction tensor including character prediction information for the N characters to be recognized; A second loss function value between the second prediction tensor and the labeled text is obtained. 8. The method of claim 7, wherein said updating said encoder and said decoder based on said first loss function value and said second loss function value comprises: Obtaining a first weighted value according to the first loss function value and its first weight; Obtaining a second weighted value according to the second loss function value and its second weight; obtaining a comprehensive loss function value according to the first weighted value and the second weighted value; updating the encoder and the decoder according to the comprehensive loss function value to obtain a trained character recognition model. 9. The method according to claim 7, wherein said decoder comprises a cross-attention layer and a classification layer, said inputting said second image feature into said decoder, obtaining said decoder outputted The second prediction tensor consists of: The second image feature and the second sequence of text are input into the decoder, the second sequence of text is obtained by moving the label of each character to be recognized in the label text to the right by M character positions, and M is greater than or an integer equal to 1; Using the cross-attention layer to process the second image feature and the second sequence of text to obtain a second attention feature, wherein the cross-attention layer is configured to process data based on a cross-attention mechanism; The classification layer is used to process the second attention feature to obtain the second prediction tensor. 10. The method according to claim 6, wherein said obtaining a first prediction tensor based on said second image feature comprises: performing dimension-up processing on the second image feature to obtain a third image feature; The third image feature is input to a classification function to obtain the first prediction tensor. 11. A character recognition device, comprising: An image acquisition module, configured to acquire an image to be recognized, wherein the image to be recognized includes N characters to be recognized that are irregularly arranged, the irregular arrangement includes not being arranged in a straight line, and N is an integer greater than or equal to 2; The third input module is used to input the image to be recognized into the character recognition model; A recognition result module, configured to obtain the recognition results of the N characters to be recognized output by the character recognition model; Wherein, the character recognition model includes an encoder and a decoder, and the character recognition model is configured to be pre-trained through the following operations: Obtaining a first loss function value based on an encoding result of the encoder, obtaining a second loss function value based on a decoding result of the decoder, updating the encoding based on the first loss function value and the second loss function value device and the decoder. 12. An electronic device comprising: one or more processors; storage means for storing one or more programs, Wherein, when the one or more programs are executed by the one or more processors, the one or more processors are made to execute the method according to any one of claims 1-10. 13. A computer-readable storage medium, on which executable instructions are stored, and the instructions, when executed by a processor, cause the processor to perform the method according to any one of claims 1-10. 14. A computer program product comprising a computer program, the computer program implementing the method according to any one of claims 1-10 when executed by a processor.