CN116012570A - A method, device and system for recognizing text information in an image - Google Patents

Abstract

The present application discloses a method, device and system for recognizing text information in an image, which relates to the field of computer vision (computer version, CV), and can quickly and accurately recognize information in an image. In this application, the scene analysis of the image is carried out by using the image semantic segmentation algorithm, and the corresponding special optical character recognition (OCR) capability is invoked according to the analysis result, such as a special model or a special algorithm for information recognition by scene, and finally integrated The information recognition results of multiple special models or special algorithms are obtained to obtain the information recognition results of the final image. The method can obtain more accurate and reliable information identification results, and provide users with a more convenient, efficient, and reliable intelligent information identification experience.

Description

A method, device and system for recognizing text information in an image

technical field

The embodiments of the present application relate to the field of computer vision (computer version, CV), and in particular to a method, device and system for recognizing text information in an image.

Background technique

With the popularity of smart terminals, terminals can recognize text information in images through optical character recognition (OCR) technology, such as recognizing text in documents, recognizing license plate numbers, and so on.

Different types of text information are presented in different forms, for example, the layout and content of table text and card text are quite different. Currently, the text in the card text image can be recognized by a card-related model or algorithm, and the text in the form text image can be recognized by a form-related model or algorithm. However, for some images with a slightly more complex situation, such as the content in the image includes both cards and forms, the recognition accuracy of the text in the image needs to be improved urgently.

Contents of the invention

The present application provides a method, device and system for recognizing text information in an image, which can quickly and accurately recognize the information in the image.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical solutions:

In a first aspect, a method for identifying text information in an image is provided, the method comprising: first, acquiring an image including multiple content areas; then, using multiple models to respectively identify the text information in the above multiple content areas; finally, Integrating the recognition results of the above-mentioned multiple models on the text information in the multiple content areas to obtain the above-mentioned text recognition result of the image. As an example, the above method can be applied to an information identification system.

The solution provided by the first aspect above can identify multiple content areas in the image through multiple special models or special algorithms, and finally integrate the information recognition results of multiple special models or special algorithms to obtain the information recognition result of the final image . The method can obtain accurate and reliable information recognition results, and provide users with a more convenient, efficient and reliable intelligent information recognition experience.

In a possible implementation manner, the above-mentioned multiple content areas correspond to multiple scenes; the above-mentioned method further includes: classifying the multiple content areas of the image, and determining the multiple scenes corresponding to the multiple content areas; Identifying the text information in the multiple content areas respectively includes: respectively identifying the text information in the multiple content areas through multiple models corresponding to the multiple scenes. As an example, the solution provided by this application can determine multiple scenes corresponding to multiple content areas by performing scene analysis on the image, so that the corresponding special model or special algorithm can be called for information recognition based on the specific scene and scene. , in order to obtain accurate and reliable information identification results.

In a possible implementation manner, the above-mentioned classifying multiple content areas and determining multiple scenes corresponding to the multiple content areas includes: performing image semantic segmentation on multiple content areas based on an image semantic segmentation model or algorithm, and determining Multiple scenes corresponding to multiple content areas. By using image semantic segmentation models or algorithms to determine multiple scenes corresponding to multiple content areas, accurate and fast analysis of complex image scenes can be achieved, and accurate and reliable information recognition results can be obtained.

In a possible implementation manner, the aforementioned identification of the text information in the multiple content areas through the multiple models corresponding to the multiple scenarios includes: according to the multiple scenarios corresponding to the multiple content areas, converting the multiple content areas cropping to obtain a plurality of image regions; respectively distributing the above-mentioned plurality of image regions to corresponding models, so as to recognize text information in the plurality of image regions through the plurality of models. In order to call the corresponding special model or special algorithm for information recognition by scene, as an example, the original image can be cropped according to the scene and then distributed to the corresponding special model or special algorithm to reduce the input to the special model or special algorithm At the same time, accurate and reliable information identification results can be obtained.

In a possible implementation manner, the above-mentioned multiple models corresponding to multiple scenes respectively identify the text information in multiple content areas, including: according to multiple scenes corresponding to multiple content areas, determine the multiple scenes in the image Position information of the corresponding multiple image areas; sending the images to multiple models, and respectively distributing the position information of the multiple image areas to the corresponding models, so as to recognize the text information in the multiple image areas through the multiple models. In order to call the corresponding special model or special algorithm for information recognition by scene, as an example, the position information of the image area corresponding to the scene can be notified to the special model or special algorithm, so as to obtain accurate and reliable information recognition results.

In a possible implementation, the above method further includes: calculating the saliency value of the pixel in the image based on a visual saliency detection (VSD) algorithm; determining the focus area of the image according to the saliency value of the pixel in the image ; Wherein, the saliency value of the pixel in the focus area of the image is greater than a preset threshold. In order to improve the speed and accuracy of information recognition, the visual salience analysis can also be performed on the image to identify the image area where the pixels with high user attention in the image are located, that is, the focus area.

In a possible implementation manner, the above method further includes: determining regional saliency values of multiple content regions according to the saliency values of pixels in multiple content regions and the area of the focus area respectively; The text information in the multiple content areas includes: using multiple models to respectively identify the text information in the content areas whose regional salience values are greater than a preset threshold. In order to improve the speed and accuracy of information recognition, regional saliency analysis can also be performed on the image to identify the image area with high user attention in the image, that is, the focus area.

其中， A_i＝{p_j}，p_j是内容区域A_i中的第j个像素，内容区域A_i中包括N个像素，1≤j≤N，N正整数，且N＞1；α是自定义参数；S(p_j)是像素p_j的视觉显著性值，H(p_j)是像素p_j的掩码值。作为一种示例，可以基于上述公式1计算得到多个内容区域的区域显著性值。In a possible implementation manner, the determination of regional saliency values of multiple content regions includes: calculating and obtaining the saliency OcrSaliency(A _i ) of the content region A _i based on the following formula 1, wherein the saliency of the content region A _i The property is used to characterize the importance of the object represented by the content area A _i ;

Among them, A _i ={p _j }, p _j is the jth pixel in the content area A _i , and the content area A _i includes N pixels, 1≤j≤N, N is a positive integer, and N>1; α is a custom parameter; S(p _j ) is the visual saliency value of pixel p _j , and H(p _j ) is the mask value of pixel p _j . As an example, the region salience values of multiple content regions can be calculated based on the above formula 1.

In a possible implementation manner, the above method further includes: performing text line analysis on the image, and determining that the image includes the first information, wherein the first part of the first information belongs to the first scene, and the second part of the first information belongs to the first scene. Two scenes; Merge the image regions corresponding to the first scene and the second scene. By combining the original isolated image scene analysis and text line analysis for image semantic segmentation, the results of the two can be complementary, avoiding wrong segmentation problems caused by shadows and other reasons, improving the fault tolerance of image semantic segmentation, and ensuring image semantic segmentation. reliability and accuracy.

In a possible implementation, the above method further includes: superimposing and displaying the text recognition result of the image on the image; or displaying the text recognition result of the image floating on the image; or displaying the text recognition result of the image in the typesetting format of the image The text recognition result of the image; or, rearrange the typesetting according to preset rules to display the text recognition result of the image, and the preset rules include: preset position order, preset significance order. This application does not limit the display form of the recognition result, which depends on specific requirements and device functions.

In a possible implementation manner, the multiple scenes corresponding to the above multiple content areas include any of the following: card scene, license plate scene, photo scene, signboard scene, poster scene, street sign scene, road sign scene, document Scenes, Table Scenes, Book Scenes, Environmental Scenes. The solution provided by this application can recognize text information in complex images under any combination of scenes, and has strong versatility and stable performance.

In a second aspect, a method for identifying text information in an image is provided, the method is applied to a first device, and the method includes: the first device acquires an image including a plurality of content areas; the first device classifies the plurality of content areas , determine multiple scenes corresponding to multiple content areas; the first device sends multiple information recognition tasks corresponding to the above multiple scenes to the second device; the first device receives the image information recognition result from the second device.

In the solution provided by the second aspect above, the first setting is to determine multiple scenes corresponding to multiple content areas by performing scene analysis on the image, so as to instruct the second device to call the corresponding special model or special algorithm analysis based on the specific scene. Scenarios for information identification to obtain accurate and reliable information identification results.

In a possible implementation manner, the above-mentioned first device classifies the multiple content areas, and determines the multiple scenes corresponding to the multiple content areas, including: the first device classifies the multiple content areas based on the image semantic segmentation model or algorithm Perform image semantic segmentation to determine multiple scenes corresponding to multiple content areas. By using image semantic segmentation models or algorithms to determine multiple scenes corresponding to multiple content areas, accurate and fast analysis of complex image scenes can be achieved, and accurate and reliable information recognition results can be obtained.

In a possible implementation manner, the above method further includes: the first device crops the above multiple content areas according to the multiple scenes corresponding to the above multiple content areas to obtain multiple sub-images; Sending multiple information recognition tasks corresponding to multiple scenes includes: the first device distributes multiple sub-images to corresponding models in the second device, so as to respectively recognize text information in the multiple sub-images through multiple models. In order to call the corresponding special model or special algorithm for information recognition by scene, as an example, the original image can be cropped according to the scene and then distributed to the corresponding special model or special algorithm to reduce the input to the special model or special algorithm At the same time, accurate and reliable information identification results can be obtained.

In a possible implementation manner, the above method further includes: the first device determines the position information of the multiple image areas corresponding to the multiple scenes in the image according to the multiple scenes corresponding to the multiple content areas; sending to the second device, and distributing the position information of the plurality of image regions to corresponding models in the second device, so as to respectively recognize the text information in the plurality of image regions through the plurality of models. In order to call the corresponding special model or special algorithm for information recognition by scene, as an example, the position information of the image area corresponding to the scene can be notified to the special model or special algorithm, so as to obtain accurate and reliable information recognition results.

In a possible implementation, the above method further includes: the first device calculates the saliency value of the pixel in the image based on the visual saliency detection VSD algorithm; the first device determines the focus of the image according to the saliency value of the pixel in the image region; where the saliency value of the pixels in the focus region of the image is greater than a preset threshold. In order to improve the speed and accuracy of information recognition, the visual salience analysis can also be performed on the image to identify the image area where the pixels with high user attention in the image are located, that is, the focus area.

In a possible implementation manner, the above method further includes: the first device determines the area saliency values of multiple content areas according to the saliency values of pixels in the multiple content areas and the area of the focus area respectively; the above-mentioned first device Sending multiple information identification tasks corresponding to multiple scenarios to the second device includes: the first device sends multiple information identification tasks corresponding to content areas whose area salience value is greater than a preset threshold among multiple content areas to the second device . In order to improve the speed and accuracy of information recognition, regional saliency analysis can also be performed on the image to identify the image area with high user attention in the image, that is, the focus area.

其中，A_i＝{p_j}，p_j是内容区域A_i中的第j个像素，内容区域A_i中包括N个像素，1≤j≤N，N正整数，且N＞1；α是自定义参数；S(p_j) 是像素p_j的视觉显著性值，H(p_j)是像素p_j的掩码值。作为一种示例，第一设备可以基于上述公式1计算得到多个内容区域的区域显著性值。In a possible implementation manner, the first device determining regional saliency values of multiple content regions includes: the first device calculates the saliency OcrSaliency(A _i ) of the content region A _i based on the following formula 1, where, The salience of the content area A _i is used to characterize the importance of the object represented by the content area A _i ;

Among them, A _i ={p _j }, p _j is the jth pixel in the content area A _i , and the content area A _i includes N pixels, 1≤j≤N, N is a positive integer, and N>1; α is a custom parameter; S(p _j ) is the visual saliency value of pixel p _j , and H(p _j ) is the mask value of pixel p _j . As an example, the first device may calculate area salience values of multiple content areas based on the foregoing formula 1.

In a possible implementation manner, the above method further includes: the first device performs text line analysis on the image, and determines that the image includes the first information, wherein the first part of the first information belongs to the first scene, and the second part of the first information The part belongs to the second scene; the first device merges image regions corresponding to the first scene and the second scene. By combining the original isolated image scene analysis and text line analysis for image semantic segmentation, the results of the two can be complementary, avoiding wrong segmentation problems caused by shadows and other reasons, improving the fault tolerance of image semantic segmentation, and ensuring image semantic segmentation. reliability and accuracy.

In a possible implementation, the above method further includes: the first device superimposes and displays the text recognition result of the image on the image; or, the first device hovers on the image to display the text recognition result of the image; or, the first The device displays the text recognition result of the image in the typesetting format of the image; or, the first device rearranges the typesetting according to preset rules to display the text recognition result of the image. The preset rules include: preset position order, preset significance order . This application does not limit the display form of the recognition result, which depends on specific requirements and device functions.

In a possible implementation manner, the multiple scenes corresponding to the above multiple content areas include any of the following: card scene, license plate scene, photo scene, signboard scene, poster scene, street sign scene, road sign scene, document Scenes, Table Scenes, Book Scenes, Environmental Scenes. The solution provided by this application can recognize text information in complex images under any combination of scenes, and has strong versatility and stable performance.

In a third aspect, a method for recognizing text information in an image is provided, the method is applied to a second device, and the method includes: the second device receives a plurality of information recognition tasks from an image of the first device, and the plurality of information recognition tasks Corresponding to multiple scenarios; the second device respectively performs multiple information recognition tasks through multiple models to obtain multiple information recognition results; the second device integrates multiple information recognition results obtained from multiple models to obtain text recognition results for images.

In the solution provided by the third aspect above, the second device can identify multiple content areas in the image through multiple special models or special algorithms, and finally integrate the information recognition results of multiple special models or special algorithms to obtain the final image. information identification results. The method can obtain accurate and reliable information recognition results, and provide users with a more convenient, efficient and reliable intelligent information recognition experience.

In a possible implementation manner, the above method further includes: the second device sends the text recognition result of the image to the first device, so as to facilitate the user to view the text information in the recognized image.

In a possible implementation, the above-mentioned multiple information recognition tasks respectively correspond to multiple sub-images of the image, and the above-mentioned multiple sub-images correspond to multiple scenes; the second device respectively performs multiple information recognition tasks through multiple models, and obtains multiple The information recognition result includes: the second device distributes the multiple sub-images to the models corresponding to the corresponding scenes, so as to respectively recognize the text information in the multiple sub-images through the multiple models. By performing information recognition on the sub-images corresponding to multiple scenes in the original image according to the scene, the amount of data input to the special model or special algorithm can be reduced, and accurate and reliable information recognition results can be obtained at the same time.

In a possible implementation, the above multiple information recognition tasks are respectively used to indicate the position information of the image and multiple image areas in the image, and the multiple image areas correspond to multiple scenes; the second device executes the tasks respectively through multiple models Multiple information recognition tasks to obtain multiple information recognition results, including: the second device distributes the position information of the image and multiple image regions to the model corresponding to the corresponding scene, so as to identify the information in the multiple image regions through multiple models text information. Accurate and reliable information recognition results can be obtained by performing information recognition on image regions corresponding to multiple scenes in the original image according to the scene and position information.

In a possible implementation manner, the second device integrates multiple information recognition results obtained from multiple models to obtain a text recognition result for an image, including: the second device integrates multiple information recognition results in an image typesetting format, Obtaining the text recognition result of the image; or, the second device integrates multiple information recognition results according to preset rules to obtain the text recognition result of the image; the preset rules include: preset position order and preset significance order. This application does not limit the integration form of the recognition results, which depends on specific requirements and device functions.

In a fourth aspect, a first device is provided, and the first device includes: a processing unit, configured to acquire an image including multiple content areas; and classify the multiple content areas, and determine multiple content areas corresponding to the multiple content areas. scenes. The transceiver unit is configured to send the multiple information recognition tasks corresponding to the above multiple scenarios to the second device; and receive the information recognition result of the image from the second device.

In the solution provided by the fourth aspect above, the first setting is to determine multiple scenes corresponding to multiple content areas by performing scene analysis on the image, so as to instruct the second device to call the corresponding special model or special algorithm analysis based on the specific scene. Scenarios for information identification to obtain accurate and reliable information identification results.

In a possible implementation manner, the above-mentioned processing unit classifies multiple content areas, and determines multiple scenes corresponding to the multiple content areas, including: the processing unit performs image segmentation on the multiple content areas based on an image semantic segmentation model or algorithm. Semantic segmentation, to determine multiple scenes corresponding to multiple content areas. By using image semantic segmentation models or algorithms to determine multiple scenes corresponding to multiple content areas, accurate and fast analysis of complex image scenes can be achieved, and accurate and reliable information recognition results can be obtained.

In a possible implementation manner, the processing unit is further configured to: crop the multiple content regions to obtain multiple sub-images according to the multiple scenes corresponding to the multiple content regions; the transceiver unit is specifically configured to: The sub-images are distributed to the corresponding models in the second device, so that the text information in the multiple sub-images can be recognized by the multiple models. In order to call the corresponding special model or special algorithm for information recognition by scene, as an example, the original image can be cropped according to the scene and then distributed to the corresponding special model or special algorithm to reduce the input to the special model or special algorithm At the same time, accurate and reliable information identification results can be obtained.

In a possible implementation manner, the above-mentioned processing unit is further configured to: determine the position information of multiple image areas corresponding to multiple scenes in the image according to multiple scenes corresponding to multiple content areas; the above-mentioned sending and receiving unit is specifically used to : sending the image to the second device, and distributing the location information of the plurality of image regions to corresponding models in the second device, so as to respectively recognize the text information in the plurality of image regions by the plurality of models. In order to call the corresponding special model or special algorithm for information recognition by scene, as an example, the position information of the image area corresponding to the scene can be notified to the special model or special algorithm, so as to obtain accurate and reliable information recognition results.

In a possible implementation manner, the above processing unit is further configured to: calculate the saliency value of the pixel in the image based on the visual saliency detection VSD algorithm; determine the focus area of the image according to the saliency value of the pixel in the image; wherein, The saliency value of pixels in the focus area of the image is greater than a preset threshold. In order to improve the speed and accuracy of information recognition, the visual salience analysis can also be performed on the image to identify the image area where the pixels with high user attention in the image are located, that is, the focus area.

In a possible implementation manner, the above processing unit is further configured to: determine the area saliency values of multiple content areas according to the saliency values of pixels in the multiple content areas and the area of the focus area respectively; the above sending and receiving unit specifically uses In: sending to the second device a plurality of information identification tasks corresponding to content regions whose region salience values are greater than a preset threshold. In order to improve the speed and accuracy of information recognition, regional saliency analysis can also be performed on the image to identify the image area with high user attention in the image, that is, the focus area.

其中，A_i＝{p_j}，p_j是内容区域A_i中的第j个像素，内容区域A_i中包括N个像素，1≤j≤N，N正整数，且N＞1；α是自定义参数；S(p_j)是像素p_j的视觉显著性值，H(p_j)是像素p_j的掩码值。作为一种示例，第一设备可以基于上述公式1计算得到多个内容区域的区域显著性值。In a possible implementation manner, the above processing unit is specifically configured to: calculate and obtain the saliency OcrSaliency(A _i ) of the content area A _i based on the following formula 1, wherein the saliency of the content area A _i is used to characterize the content area A the importance of the object represented by _i ;

Among them, A _i ={p _j }, p _j is the jth pixel in the content area A _i , and the content area A _i includes N pixels, 1≤j≤N, N is a positive integer, and N>1; α is a custom parameter; S(p _j ) is the visual saliency value of pixel p _j , and H(p _j ) is the mask value of pixel p _j . As an example, the first device may calculate area salience values of multiple content areas based on the foregoing formula 1.

In a possible implementation manner, the above processing unit is further configured to: perform text line analysis on the image to determine that the image includes the first information, wherein the first part of the first information belongs to the first scene, and the second part of the first information belonging to the second scene; and merging image regions corresponding to the first scene and the second scene. By combining the original isolated image scene analysis and text line analysis for image semantic segmentation, the results of the two can be complementary, avoiding wrong segmentation problems caused by shadows and other reasons, improving the fault tolerance of image semantic segmentation, and ensuring image semantic segmentation. reliability and accuracy.

In a possible implementation manner, the above-mentioned first device further includes: a display unit, configured to superimpose and display the text recognition result of the image on the image; or display the text recognition result of the image by floating on the image; or, use The typesetting format of the image shows the text recognition result of the image; or, retypesetting according to preset rules to show the text recognition result of the image. The preset rules include: preset position order, preset significance order. This application does not limit the display form of the recognition result, which depends on specific requirements and device functions.

In a possible implementation manner, the multiple scenes corresponding to the above multiple content areas include any of the following: card scene, license plate scene, photo scene, signboard scene, poster scene, street sign scene, road sign scene, document Scenes, Table Scenes, Book Scenes, Environmental Scenes. The solution provided by this application can recognize text information in complex images under any combination of scenes, and has strong versatility and stable performance.

In a fifth aspect, there is provided a second device, the second device comprising: a transceiver unit configured to receive multiple information recognition tasks of images from the first device, the multiple information recognition tasks corresponding to multiple scenarios; a processing unit, It is used to perform multiple information recognition tasks through multiple models to obtain multiple information recognition results; and integrate multiple information recognition results obtained from multiple models to obtain text recognition results for images.

In the solution provided in the fifth aspect above, the second device can use multiple special models or special algorithms to perform information identification on multiple content areas in the image, and finally integrate the information recognition results of multiple special models or special algorithms to obtain the final image. information identification results. The method can obtain accurate and reliable information recognition results, and provide users with a more convenient, efficient and reliable intelligent information recognition experience.

In a possible implementation manner, the above-mentioned transceiving unit is further configured to: send the text recognition result of the image to the first device, so as to facilitate the user to view the text information in the recognized image.

In a possible implementation manner, the above-mentioned multiple information recognition tasks correspond to multiple sub-images of the image, and the above-mentioned multiple sub-images correspond to multiple scenes; the above-mentioned processing unit is specifically used to: distribute the multiple sub-images to the corresponding scene A model to identify text information in multiple sub-images through multiple models. By performing information recognition on the sub-images corresponding to multiple scenes in the original image according to the scene, the amount of data input to the special model or special algorithm can be reduced, and accurate and reliable information recognition results can be obtained at the same time.

In a possible implementation manner, the above multiple information recognition tasks are respectively used to indicate the position information of the image and multiple image areas in the image, and the multiple image areas correspond to multiple scenes; the above processing unit is specifically used to: and the position information of multiple image areas are distributed to the models corresponding to the corresponding scenes, so that the text information in the multiple image areas can be respectively recognized by the multiple models. Accurate and reliable information recognition results can be obtained by performing information recognition on image regions corresponding to multiple scenes in the original image according to the scene and position information.

In a possible implementation, the above processing unit is specifically configured to: integrate multiple information recognition results in an image typesetting format to obtain a text recognition result for an image; or, integrate multiple information recognition results according to preset rules to obtain The text recognition result of the image; preset rules include: preset position order, preset significance order. This application does not limit the integration form of the recognition results, which depends on specific requirements and device functions.

According to a sixth aspect, a first device is provided, and the first device includes: a memory for storing a computer program; a transceiver for receiving or sending a radio signal; a processor for executing the computer program, so that the first The device executes the method in any possible implementation manner of the second aspect.

In a seventh aspect, a second device is provided, and the second device includes: a memory for storing a computer program; a transceiver for receiving or sending a radio signal; a processor for executing the computer program, so that the second The device executes the method in any possible implementation manner of the third aspect.

The eighth aspect provides an information identification system, the information identification system includes the first device in any possible implementation manner of the fourth aspect or the sixth aspect, and any possible implementation manner of the fifth aspect or the seventh aspect In an implementation, the second device. The communication system is used to implement the method in any possible implementation manner of the first aspect.

According to a ninth aspect, a computer-readable storage medium is provided. Computer program code is stored on the computer-readable storage medium. When the computer program code is executed by a processor, the processor implements any one of the second aspect or the third aspect. method in one possible implementation.

In a tenth aspect, a chip system is provided, the chip system includes a processor and a memory, and computer program code is stored in the memory; when the computer program code is executed by the processor, the processor implements the second aspect or the first A method in any of the possible implementations of the three aspects. The system-on-a-chip may consist of chips, or may include chips and other discrete devices.

In an eleventh aspect, a computer program product comprising computer instructions is provided. When the computer instructions are run on the computer, the computer is made to implement the method in any possible implementation manner of the second aspect or the third aspect.

Description of drawings

FIG. 1 is an example diagram of an information recognition result;

FIG. 2 is a schematic diagram of a hardware structure of a first device provided in an embodiment of the present application;

FIG. 3 is a schematic diagram of a process for identifying information in an image based on a specific model provided by an embodiment of the present application;

FIG. 4 is an example diagram of an information identification result provided by an embodiment of the present application;

Fig. 5 is an example diagram 1 of a complex image provided by the embodiment of the present application;

Figure 6 is the second example of a complex image provided by the embodiment of the present application;

Figure 7 is the third example of a complex image provided by the embodiment of the present application;

Fig. 8 is an example diagram 1 of the process of recognizing text information in an image provided by the embodiment of the present application;

FIG. 9 is a second example of the process of identifying text information in an image provided by the embodiment of the present application;

FIG. 10 is an example diagram of obtaining a significance distribution diagram based on a spectral residual significance detection algorithm provided by an embodiment of the present application;

FIG. 11 is an example diagram of coordinate information of an image area provided by an embodiment of the present application;

Fig. 12 is a comparison diagram of two image semantic segmentation effects provided by the embodiment of the present application;

Fig. 13 is a third example of the process of identifying information in an image provided by the embodiment of the present application;

Figure 14 is an example of Figure 4 of the process of identifying information in an image provided by the embodiment of the present application;

FIG. 15 is an example diagram of physical cropping results of two images provided in the embodiment of the present application;

Fig. 16A is an example diagram showing an information identification result provided by the embodiment of the present application;

FIG. 16B is an example diagram of displaying another information identification result provided by the embodiment of the present application;

Fig. 17 is an interactive flow chart of a method for recognizing text information in an image provided by an embodiment of the present application;

FIG. 18 is a schematic diagram of an image semantic segmentation stage provided by an embodiment of the present application;

Figure 19 is a comparison chart 1 of the information recognition effect provided by the embodiment of the present application;

Figure 20 is the second comparison of information recognition effects provided by the embodiment of the present application;

Figure 21 is the third comparison of information recognition effects provided by the embodiment of the present application;

Fig. 22 is a structural block diagram of a first device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Among them, in the description of the embodiments of this application, unless otherwise specified, "/" means or means, for example, A/B can mean A or B; "and/or" in this article is only a description of associated objects The association relationship of , which means that there can be three kinds of relationships, for example, A and/or B, can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "plurality" refers to two or more than two.

Hereinafter, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

An embodiment of the present application provides a method for recognizing text information in an image, and the method is applied in the process of recognizing information in an image based on optical character recognition (OCR) technology. Wherein, the types of text information identified through information identification may include but not limited to one or more of the following: characters, numbers, letters, symbols, and the like.

In this embodiment of the application, an image may include but not limited to a picture, an image frame, and the like. An image is used to represent one or more objects. For example, the object represented by the image may include but not limited to one or more of cards (such as ID cards, passports, bank cards, business cards, etc.), license plates, photos, signboards, posters, street signs, road signs, documents, forms, books, etc. Multiple, not limited by this application. Taking the image shown in FIG. 1 as an example, the objects represented by the picture A shown in FIG. 1 include documents, forms and cards.

Among them, a method for identifying text information in an image provided by the embodiment of the present application can be completed by a device on the device (such as a smart phone), or by a device on the cloud (such as a server), or by a device on the device and a cloud The equipment is completed together, which is not limited in this application.

Exemplarily, the end-side device (hereinafter referred to as "the first device") may include but is not limited to a smart phone, a personal computer (personal computer, PC) (such as a notebook computer, a desktop computer, an ultra-mobile personal computer (ultra-mobile personal computer) , UMPC), etc.), tablet PCs, TVs, augmented reality (augmented reality, AR)/virtual reality (virtual reality, VR) equipment, wireless terminals in industrial control (industrial control), self-driving (self-driving) Wireless terminals in transportation safety, wireless terminals in smart city, sensor devices (such as monitoring terminals), Internet of things (IOT) devices (such as smart home equipment), etc. The present application does not limit the specific function and structure of the first device.

Please refer to FIG. 2 . FIG. 2 shows a schematic diagram of a hardware structure of a first device (that is, a device at the end side) provided by an embodiment of the present application, taking a smart phone as an example. As shown in Figure 2, the first device may include a processor 210, a memory (including an external memory interface 220 and an internal memory 221), a universal serial bus (universal serial bus, USB) interface 230, a charging management module 240, and a power management module 241, battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, speaker 270A, receiver 270B, microphone 270C, earphone jack 270D, sensor module 280, button 290, motor 291, indicator 292, a camera assembly 293, a display screen 294, a subscriber identity module (subscriber identity module, SIM) interface 295, etc. Wherein, the sensor module 280 may include a gyroscope sensor, an acceleration sensor, a magnetic sensor, a touch sensor, a fingerprint sensor, a pressure sensor, an air pressure sensor, a distance sensor, a proximity light sensor, a temperature sensor, an ambient light sensor, a bone conduction sensor, and the like.

It can be understood that, the structure shown in the embodiment of the present invention does not constitute a specific limitation on the first device. In some other embodiments of the present application, the first device may include more or fewer components than shown in the figure, or some components may be combined, or some components may be separated, or different component arrangements may be made. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

Processor 210 may include one or more processing units. For example: the processor 210 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a flight controller, a video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

A memory may also be provided in the processor 210 for storing instructions and data. In some embodiments, the memory in processor 210 is a cache memory. The memory may hold instructions or data that the processor 210 has just used or recycled. If the processor 210 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 210 is reduced, thereby improving the efficiency of the system.

In some embodiments, processor 210 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuitsound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver transmitter (universal asynchronous receiver) /transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, and/or universal serial bus (universal serial bus, USB) interface wait.

The charging management module 240 is configured to receive charging input from the charger. The power management module 241 is used for connecting the battery 242 , the charging management module 240 and the processor 210 . The power management module 241 receives the input from the battery 242 and/or the charging management module 240 to provide power for the processor 210 , the internal memory 221 , the display screen 294 , the camera component 293 , and the wireless communication module 260 .

The wireless communication function of the first device may be realized by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor and the baseband processor.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the first device can be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

In this embodiment of the present application, the first device may send an information identification task to the cloud-side device through antenna 1 and/or antenna 2, and receive an information identification result from the cloud-side device.

The mobile communication module 250 may provide wireless communication solutions including 2G/3G/4G/5G/6G applied on the first device. The mobile communication module 250 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 250 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modem processor, convert it into electromagnetic wave and radiate it through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 250 may be set in the processor 210 . In some embodiments, at least part of the functional modules of the mobile communication module 250 and at least part of the modules of the processor 210 may be set in the same device.

A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 270A, a receiver 270B, etc.), or displays an image or video through a display screen 294 . In some embodiments, the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent of the processor 210, and be set in the same device as the mobile communication module 250 or other functional modules.

The wireless communication module 260 can provide wireless local area networks (wireless local area networks, WLAN) (such as Wi-Fi network), Bluetooth BT, global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation) applied on the first device. , FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 260 may be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 210 . The wireless communication module 260 can also receive the signal to be sent from the processor 210, frequency-modulate it, amplify it, and convert it into electromagnetic wave radiation through the antenna 2.

In some embodiments, the antenna 1 of the first device is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, so that the first device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), wideband code wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), new radio (NR), BT, GNSS, WLAN, NFC, FM, and/or IR technology, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou satellite navigation system (beidou navigation satellite system, BDS), a quasi-zenith satellite system (quasi- zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The first device implements the display function through the GPU, the display screen 294, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 294 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 294 is used to display images, videos and the like. Display 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrixorganic light-emitting diode) , AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the first device may include 1 or N display screens 294, where N is a positive integer greater than 1.

The first device can realize the shooting function through the ISP, the camera component 293 , the video codec, the GPU, the display screen 294 and the application processor.

The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, a solid state drive, etc., to expand the storage capacity of the first device. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. Such as saving music, video, images and other files in external memory card.

The internal memory 221 may be used to store computer-executable program codes including instructions. The internal memory 221 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data created during the use of the first device (such as audio data, phonebook, etc.) and the like. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like. The processor 210 executes various functional applications and data processing of the first device by executing instructions stored in the internal memory 221 and/or instructions stored in a memory provided in the processor.

The first device may implement an audio function through an audio module 270 , a speaker 270A, a receiver 270B, a microphone 270C, an application processor, and the like. Such as music playback, recording, etc. About audio module 270, loudspeaker 270A, receiver 270B and microphone 270C's specific working principle and effect, can refer to the introduction in conventional technology.

The keys 290 include a power key, a volume key and the like. The key 290 may be a mechanical key. It can also be a touch button. The first device can receive key input and generate key signal input related to user settings and function control of the first device.

It should be noted that the hardware modules included in the first device shown in FIG. 2 are only described as examples, and do not limit the specific structure of the first device. For example, if the first device is a television, the first device may further include components such as a remote control component. If the first device is a PC, the first device may further include components such as a keyboard and a mouse.

微软

苹果

等操作系统。In this application, the operating system of the first device may include but not limited to

Microsoft

apple

and other operating systems.

系统的第一设备为例，第一设备的软件可以分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。例如，第一设备的软件结构从上至下可以分为三层：应用程序层(简称应用层)，应用程序框架层(简称框架层)，系统库，安卓运行时和内核层(也称为驱动层)。to include layered architecture

Take the first device of the system as an example, the software of the first device can be divided into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. For example, the software structure of the first device can be divided into three layers from top to bottom: application program layer (abbreviated as application layer), application program framework layer (referred to as framework layer), system library, Android runtime and kernel layer (also referred to as driver layer).

Wherein, in the embodiment of the present application, the first device can provide an information identification service. For example, the application layer of the first device may include a text-aware application program (application, APP) for identifying information such as text. For another example, the first device may be integrated with functional modules related to providing text and other information identification. In another example, the first device may provide an interface related to information identification such as text, such as a quick application or a small program. For another example, the application installed in the first device may be integrated with information identification services such as text. The embodiment of the present application does not limit the specific manner in which the first device provides the information identification service. For the specific implementation of the information identification function, reference may be made to conventional technologies, which will not be repeated here.

In order to perform information identification more accurately and comprehensively, in a possible implementation manner, the scene of the image may be analyzed, and then the information in the image may be identified based on a specific model or a specific algorithm corresponding to the specific scene.

As an example, the scene of an image is related to the object represented by the image. For example, if the object represented by the image is a card, the image belongs to the card scene. If the object represented by the image is a table, the image belongs to the table scene. If the object represented by the image is a document, the image belongs to the document scene. If the object represented by the image is a license plate, the image belongs to the license plate scene.

As another example, the scene of the image may also be determined by one or more of the image's color, brightness, or texture. For example, if an image expressing a natural scenery is dark, the image belongs to a night scene. If the image expressing the natural scenery is brighter, the image belongs to the daytime scene.

As an example, the process of identifying information in an image based on a specific model may be shown in FIG. 3 . As shown in FIG. 3 , if there are texts in the image, scene analysis can be further performed on the image. If it is determined that the image belongs to scene 1, use model 1 to recognize the image and obtain the recognition result. If it is determined that the image belongs to scene 2, model 2 is used to identify the image. If it is determined that the image belongs to scene 3, model 3 is used to identify the image. If it is determined that the image belongs to scene 4, model 4 is used to identify the image. If it is determined that the image belongs to scene 4, model 4 is used to identify the image. If it is determined that the image belongs to scene 5, then use model 5 to identify the image. Among them, model 1 is the special model of scene 1, model 2 is the special model of scene 2, model 3 is the special model of scene 3, model 4 is the special model of scene 4, and model 5 is the special model of scene 5.

However, the recognition method based on the information in the image based on the specific model shown in Figure 3 still has the problem of missed detection or false detection when processing some complex images. As shown in Figure 4, the objects represented by picture A include documents, forms and cards, but the method of identifying information in images based on specific models can usually only determine one scene of picture A. For example, the scene of the image is determined based on the proportions and positions of the image areas where the above-mentioned multiple objects are located in the image.

Exemplarily, if the scene analysis of the picture A shown in Figure 4 determines that the picture belongs to the document scene, the special model corresponding to the document scene cannot recognize all the information in the table area due to the inability to recognize some symbols or formulas, and Due to the inability to recognize some text with concave-convex or hollow-out characteristics, it is impossible to successfully recognize all the information in the card area. In the same way, if the scene analysis of picture A determines that the picture belongs to the table scene, the special model corresponding to the document scene cannot successfully identify all the information of the document and the card area. If the scene analysis of picture A determines that the picture belongs to the card scene, the special model corresponding to the document scene cannot successfully identify all the information in the document and table areas. Among them, in the recognition result shown in FIG. 4 , the successfully recognized text in picture A is marked in a box.

It should be noted that the example shown in FIG. 4 is only an example in which the recognition result is superimposed on the image (for example, marked in a box). The embodiment of the present application does not limit the specific display form of the recognition result. For example, the recognition result can also display the information recognition result in the form of a list of recognized text and other information, and the information recognition result can be displayed in the form of a document (for example, the information that has not been successfully recognized is displayed in the form of garbled characters), and it can also be displayed in the form of a mask Display the information recognition result in the form of code value.

In order to solve the problems existing in the above-mentioned conventional methods, the embodiment of the present application provides a method for recognizing text information in an image. The method performs scene analysis on the image, and when the objects represented by the image belong to different scenes, the image is segmented according to the scene. to get multiple image regions. Then, use the corresponding special model or special algorithm to carry out text information recognition on multiple image regions respectively. Finally, the recognition results of multiple special models or special algorithms for multiple image regions are integrated to obtain the final recognition result of the text information in the image.

Through a method for recognizing text information in an image provided by an embodiment of the present application, the text information recognition of complex images can be performed more accurately and comprehensively. Among them, in the embodiment of the present application, the complex image may include but not limited to one or more of the following characteristics: the object represented by the image belongs to multiple scenes, the image contains small and dense text, the image contains interference background, etc.

Wherein, the object represented by the image belongs to multiple scenes such as shown in FIG. 4 or 6, wherein the object document, form and card shown in picture A in FIG. 4 belong to the document scene, form scene and card scene respectively. The objects represented by the picture C shown in FIG. 5 include tables and photos, which belong to the table scene and the picture scene respectively. The image contains small and dense text, for example, as shown in Figure 5. The screen in the center of picture B shown in Figure 5 contains small and dense text. Based on conventional methods, the fine and dense text in picture B shown in Figure 5 is not easy to be successfully recognized. The image contains interference background, for example, as shown in Figure 7, where the background of picture D shown in Figure 7 includes interference text.

Taking picture A as an example, as shown in Figure 8, a method for recognizing text information in an image provided by the embodiment of the present application may include the following four stages:

Phase 1: Focus Area Analysis Phase.

Wherein, the focus area analysis stage is used to estimate the area of interest of the user or the area of concern of the user in the image.

It can be understood that according to the human visual mechanism, when a user sees an image, the first thing that the user pays attention to is one or more of the following features in the image, including but not limited to: text, numbers, letters, symbols, graphics, logos, colors, textures , light, and the specific location of the above features, etc. In the embodiment of this application, for the areas in the image that the user is interested in or concerned about, it is necessary to prioritize the priority and accuracy of information identification; lower level or can be ignored.

Based on the human visual mechanism, in the embodiment of the present application, the region of the user's interest or attention in the image may be determined by simulating the human visual mechanism.

For example, based on visual saliency detection (visual saliency detection, VSD) algorithm, such as based on phase spectrum saliency detection algorithm, spectral residual (spectral residual) saliency detection algorithm, four-element Fourier transform and other algorithms, determine the The saliency value of the pixel to determine the focus area of the image by analyzing the saliency value of the pixel in the image. For example, the saliency value of a pixel in a focus area of the image is greater than a preset threshold.

Among them, the principle of the VSD algorithm is to simulate the visual characteristics of human beings through the computer vision (computer version, CV) field algorithm, and extract the technology of the human interest area in the image. Exemplarily, the VSD algorithm can use the color, brightness, edge and other feature representations of the image to judge the difference between the target area and its surrounding pixels, and then calculate the difference between the pixels in the image to determine the visual salient features. Based on the VSD algorithm, the first device can acquire image features concerned by the human visual system, such as text, numbers, letters, symbols, graphics, logos, colors, textures, light, and the like.

Taking the spectral residual saliency detection algorithm as an example, the principle of the spectral residual saliency detection algorithm is: through the logarithmic spectrum analysis of the input image, the spectral residual of the image in the spectral domain is extracted, and the corresponding spectral residual is constructed in the spatial domain. Saliency map. In some embodiments, it is also possible to obtain optimal detection results by setting different parameters, comparing their effects on the saliency map, and discussing the advantages and limitations of the algorithm itself.

Please refer to FIG. 10 , which shows an example of obtaining a significance distribution map based on a spectral residual significance detection algorithm. Wherein, the brightness of each pixel in FIG. 10 is used to identify the feature value of the image feature of the pixel point, and the greater the brightness, the larger the feature value of the image feature of the pixel point. The feature value is used to indicate the degree to which the pixel is paid attention to by the user. As shown in Figure 10, the brighter area in the saliency distribution map corresponds to the central screen area in picture B, and this image area has a higher degree of user attention; the brighter and smaller area in the saliency distribution map corresponds to the venue background in picture B area, the image area is less concerned by the user.

Taking picture A shown in FIG. 8 as an example, through focus area analysis, it can be determined that the image area where the characters, numbers, letters and symbols in picture A are located is the area that the user is interested in or concerned about. The areas in the image that do not include text, numbers, letters, symbols, graphics, logos, textures, etc. are not areas that are of interest or concern to the user, or are less interested or concerned by the user.

Stage 2: Image semantic segmentation stage.

Among them, the image semantic segmentation stage is used to determine the multiple objects represented by the image, the scenes to which the multiple objects belong, and the corresponding positions of the multiple scenes in the image through image scene analysis.

Further, the image semantic segmentation stage is also used to segment the image into multiple image regions according to the scenes to which the above multiple objects belong.

For example, the scene to which the object belongs may include but not limited to one or more of the following: card scene, license plate scene, photo scene, signboard scene, poster scene, street sign scene, road sign scene, document scene, form scene, book scene , environmental scenes, etc.

In some embodiments, the object represented by the image may be further determined based on the feature distribution in the focal area of the image. Among them, the object represented by the image may include but not limited to one or more of the following: cards (such as ID cards, passports, bank cards, business cards, etc.), license plates, photos, signboards, posters, road signs, road signs, documents , forms, books.

For example, if there are preset characters (such as ID cards), preset numbers (such as 18), preset logos or graphics, and preset textures distributed in the focus area of the image, it can be determined that the object represented by the image is an identity certificate.

It should be noted that, in the embodiment of the present application, after image semantic segmentation, an image region corresponding to a scene is used to represent one or more objects. That is, a scene may correspond to one or more image regions in an image.

As shown in FIG. 8 , after image scene analysis is performed on the image, three image regions 8 a , 8 b and 8 c corresponding to the document scene, form scene and card scene can be obtained respectively. Wherein, the object represented by the image area 8a belongs to the document scene, the object represented by the image area 8b belongs to the form scene, and the object represented by the image area 8c belongs to the card scene.

As an example, the respective positions of the multiple scenes in the image may be represented by coordinate information of image regions corresponding to the multiple scenes.

As an example, the first device may use U-Net and DenseNet to determine the scenes to which multiple objects represented by the image belong and the respective positions of the multiple scenes in the image.

Among them, U-Net is a deep fully convolutional network. In the embodiment of this application, U-Net can be used for image semantic segmentation. For example, U-Net can be modeled using a symmetric U-shaped structure containing compressed paths and expanded paths to determine the scene to which multiple objects represented by an image belong and the positions of multiple scenes in the image. DenseNet is a deep neural network. In the embodiment of this application, DenseNet can be used for image detection and classification. For example, DenseNet can use dense layer-skip connections to achieve the purpose of reusing the features of each convolutional layer, thereby achieving better scene recognition accuracy.

For example, as shown in Figure 11, multiple object documents, forms and cards represented by picture A belong to the document scene, form scene and card scene respectively. After image semantic segmentation, it can be determined that the position of the document scene in the image is the coordinate (0, 0)-(x1, y1), the position of the table scene in the image is the rectangular area of coordinates (x1, y2)-(x2, y1), and the position of the card scene in the image is the coordinate ( The rectangular area of x1,0)-(x2,y2).

Wherein, the coordinate system shown in FIG. 11 is only used as an example of a coordinate system, and the application does not limit the specific positions of the origin, x-axis and y-axis of the coordinate system. In addition, in some embodiments, the coordinate system may also be a polar coordinate system or other coordinate systems, and image regions corresponding to multiple scenes may also be represented by curve expressions, which is not limited in this application.

In some embodiments, in order to ensure the reliability and accuracy of image semantic segmentation and improve the fault tolerance of image semantic segmentation, after image scene analysis, text line analysis can also be combined to determine the scene and location of the object represented by the image. The position of the scene in the image.

For example, assuming that the image includes the first information (such as the first text line), and after image scene analysis, it is determined that the scenes to which the above-mentioned multiple objects belong include the first scene and the second scene, then the text line analysis can be performed on the image, in When it is determined that the first part of the first information belongs to the first scene and the second part of the first information belongs to the second scene, image regions corresponding to the first scene and the second scene are merged.

Take the image shown in FIG. 12 with a large contrast between light and dark areas as an example, wherein the image shown in FIG. 12 represents only one object, that is, a document. However, when image semantic segmentation is performed, as shown in (a) in Figure 12, if only image scene analysis is performed, due to the shadow relationship, the document is segmented into two image regions corresponding to two scenes, one bright and one dark ( Area 1 and Area 2) are shown in (a) in Figure 12).

If the image scene analysis and the text line analysis are combined, as shown in (b) in FIG. 12 , it can be determined that the text line 1201 straddles Area 1 and Area 2 . According to the definition of the scene, we can know that the same text line can only appear in the corresponding area of a scene. Therefore, for this case, as shown in (b) in FIG. 12 , the two image regions corresponding to the two scenes can be merged.

By combining the original isolated image scene analysis and text line analysis for image semantic segmentation, the results of the two can be complementary, avoiding the wrong segmentation problem caused by shadows as shown in (a) in Figure 12, and improving the image semantic segmentation. Fault tolerance, to ensure the reliability and accuracy of image semantic segmentation.

In some embodiments, in order to avoid resource occupation of the information identification unit by unimportant areas. As shown in FIG. 9 , after the image semantic segmentation is completed, regional saliency calculation can also be performed (that is, the regional saliency calculation stage shown in FIG. 9 , referred to as stage 5).

Wherein, the regional saliency calculation stage is used for determining the importance of multiple objects respectively represented by image regions corresponding to multiple scenes through saliency analysis.

As an example, the degree of importance may be determined according to the brightness, texture, mask value of the focus area, and the like in image areas corresponding to multiple scenes.

As an example, after the multiple image areas are obtained, the importance of the objects represented by the multiple image areas may be determined according to the saliency values of the pixels in the multiple image areas and the area of the focus area respectively.

For example, the saliency of an image region (expressed in OcrSaliency) can be calculated based on the following formula 1:

Among them, the higher the saliency value, the more important the corresponding image region; the lower the saliency value, the less important the corresponding image region. A _i is the i-th image area of the image, i is a positive integer, and i>1. A _i ={p _j }, p _j is the jth pixel in the image area A _i , the image area A _i includes N pixels, 1≤j≤N, N is a positive integer, and N>1. α is a custom parameter. S(p _j ) is the visual saliency value of pixel p _j in image region A _i . Wherein, the visual saliency value of pixel p _j can be calculated based on VSD algorithm (such as phase spectrum, spectral residual, quaternary Fourier transform, etc.) in the above stage 1. H(p _j ) is the mask value of the pixel p _j in the image area A _i , for example, if the pixel p _j is located at the text position, then H(p _j )=1.

In the embodiment of the present application, the calculated regional saliency results can be used to minimize the processing load according to the actual situation during subsequent information identification. For example, when the GPU is limited, the corresponding image area is not sent to the information recognition unit for the lower salient area, and the information identification is performed on the higher salient image area preferentially.

As an example, in the embodiment of the present application, the saliency threshold Th can be set, and if the calculated regional saliency is smaller than the saliency threshold Th, such as OcrSaliency(A _i )<Th, the image region A _i will be abandoned identify.

Please refer to FIG. 13 . FIG. 13 shows an example diagram of an information recognition process by taking the information recognition of the picture B shown in FIG. 5 as an example. As shown in Figure 13, the image 1301 is divided into a screen area 13a in the center of the image and a background area 13b composed of chairs, curtains, ceilings, etc. through image semantic segmentation, wherein the screen area 13a belongs to the form scene, and the background area 13b belongs to the environment scene. After calculating the regional saliency of the screen area 13a and the background area 13b, assuming that the saliency threshold Th is 0.25, the saliency of the screen area 13a is 0.41, and the saliency of the background area 13b is 0.18, as shown in Figure 13, because The salience (0.18) of the background region 13b is less than the salience threshold Th(0.25), so no information identification is performed on the background region 13b. Wherein, for the situation shown in FIG. 13 , when integrating the results, the information recognition result of the image 1301 is obtained only based on the recognition result of the screen area 13 a.

Through the calculation of regional saliency, unimportant areas in the image can be effectively filtered out, so as to avoid the resource occupation of the information identification unit by the unimportant areas, resulting in a high load of the information identification unit.

Phase 3: Task distribution phase.

Wherein, the task distribution stage is used to distribute image regions corresponding to multiple scenes to information recognition units of different scenes, so as to perform information recognition in corresponding scenes. Wherein, the information identification unit is integrated with a special model or a special algorithm for identifying information in a corresponding scene.

In a possible implementation, the complete image (such as picture A) can be sent to the information recognition unit of different scenes respectively, and at the same time, the information recognition unit is notified of the position of the corresponding image area in the image, so that when the information recognition unit performs information recognition , accurately identify the information in the object represented by the corresponding image region.

As shown in Figure 14, when the picture A is sent to the information identification unit a, the information identification unit b and the information identification unit c respectively, the image area 8a of the information identification unit a, the information identification unit b and the information identification unit c can be indicated respectively , 8b and 8c in picture A, so that information recognition unit a, information recognition unit b and information recognition unit c recognize the information in the corresponding area in picture A according to the positions of image areas 8a, 8b and 8c in picture A respectively .

In another possible implementation, after determining the positions of the multiple scenes in the image, the image can also be physically cropped according to the image regions corresponding to the multiple scenes to obtain multiple independent sub-images, and then the Multiple sub-images are sent to information recognition units of different scenes for text information recognition in corresponding scenes.

As shown in Figure 8, after physical cropping, the image areas (i.e., sub-images) 8a, 8b and 8c are respectively sent to the information recognition unit a for recognizing document information, the information recognition unit b for recognizing form information and the information recognition unit b for recognizing form information. An information identification unit c for identifying card information.

As an implementation manner, when performing physical cropping on an image, the cropping may be performed with edges of image regions corresponding to multiple scenes as boundaries, as shown in (a) in FIG. 15 .

As another implementation, when physically cropping an image, the minimum enclosing rectangle outside the image area corresponding to multiple scenes can be calculated first, and then cropped with the edges of multiple minimum enclosing rectangles as the boundary, as shown in Figure 15 (b) shown.

In some embodiments, when performing task distribution, image regions corresponding to multiple scenes can be distributed to different special interfaces (such as vertical interface or OCR interface), so as to send them to different scenes through special interfaces. Information identification unit. Wherein, in some cases, the scene is also called vertical scene or OCR scene. For example, the image areas 8a, 8b and 8c of the picture A shown in Figure 8 can be distributed to the document type interface, the form type interface and the card type interface, so as to be sent to the corresponding information identification unit through the type interface for further processing. Document information identification, form information identification and card information identification.

Phase 4: Results integration phase.

Wherein, the result integration stage is used to obtain the information recognition result of the image by arranging and merging the recognition results of the image regions corresponding to the multiple scenes by the multiple information recognition units.

For example, the recognition results of image regions corresponding to multiple scenes may be integrated according to preset rules. The embodiment of the present application does not limit specific rule settings, nor does it limit the basis for combining and integrating.

In some embodiments, after the result integration, the information recognition result of the image can be displayed by superimposing and displaying a mark on a corresponding position of the image. For example, by marking a box outside the corresponding information in the image to display the result, as shown in FIG. 8 , FIG. 9 , FIG. 13 or FIG. 14 .

In other embodiments, after the results are integrated, the image information recognition results can also be displayed in the form of documents, as shown in FIG. 16A .

It should be noted that the embodiment of the present application does not specifically limit the specific display form of the information recognition result of the image. For example, the image information recognition results can also be displayed in the form of mask values. As shown in FIG. 16B, in response to the operation of the user touching the image area 8a with a finger, the first device pops up a floating frame, wherein the floating frame includes information such as text in the recognized image area 8a. In another example, the first device may display the recognition result of the image in a typesetting format consistent with the image.

As an implementation manner, the integration may be performed according to preset rules according to positions of image regions corresponding to multiple scenes in the image. For example, the preset rule may be a preset position sequence, such as integrating recognition results of image regions corresponding to multiple scenes in a sequence from top to bottom and from left to right, as shown in FIG. 16A .

As another implementation manner, the integration may be performed according to preset rules according to the saliency values of image regions corresponding to multiple scenes in the image. For example, the preset rule may be a preset saliency order, such as integrating recognition results of image regions corresponding to multiple scenes in descending order of saliency values.

It should be noted that, in this embodiment of the application, the tasks of the above-mentioned stages 1 to 5 can be completed by the first device (such as a smart phone), or by a cloud-side device (such as a server), or partly by the first The device is responsible, and part of it is the cloud-side device (hereinafter referred to as "the second device"), which is not limited in this application.

For example, in order to reduce the computing pressure of the cloud-side device (that is, the second device), the first device may perform the tasks of processing the above-mentioned stages 1, 2, 5 and 3, and the second device may perform the tasks of the above-mentioned stage 4.

For another example, in order to maintain and protect the details of the algorithm, the first device may perform the tasks of processing the above-mentioned phase 1-phase 2, and the second device may perform the tasks of the above-mentioned phase 3, phase 5 and phase 4.

The following will take the first device to perform the tasks of the above-mentioned stage 1, stage 2, stage 5 and stage 3, and the second device to perform the tasks of the above-mentioned stage 4 as an example. In combination with the accompanying drawings, a Chinese image recognition method provided by the embodiment of the present application will be described. The method of this information is described in detail.

As shown in Figure 17, a method for identifying text information in an image provided by the embodiment of the present application may include the following S1701-S1706:

S1701. The first device determines a focus area of an image. The focus area is the area in the image that the user is interested in or the area that the user pays attention to.

As an example, the focus area of the image includes, but is not limited to, one or more of the following features: text, numbers, letters, symbols, graphics, logos, colors, textures, light, and specific positions of the above features.

As shown in Figure 8, after focus area analysis, it can be determined that the image area where the characters, numbers, letters, and symbols in picture A are located is the area that the user is interested in or concerned about, so it is the image area that needs to be prioritized in the information recognition process. However, the areas in the image that do not include text, numbers, letters, symbols, graphics, logos, textures, etc. are not areas that are of interest or concern to users, or are less interested or concerned by users. lower priority or can be ignored.

As another example, the screen area in the center of picture B shown in FIG. 5 is usually the area that the user is interested in or pays attention to, while the chair and curtain area are less interested or concerned by the user. The printed text in picture D shown in FIG. 7 is usually the area that the user is interested in or pays attention to, while the interfering text in the background is less interested or concerned by the user.

S1702. The first device divides the image into multiple image regions. The objects represented by the plurality of image regions belong to different scenes.

Wherein, the scenes to which the objects represented by the multiple image areas may include but not limited to one or more of the following: card scene, license plate scene, photo scene, signboard scene, poster scene, road sign scene, road sign scene, document Scenes, Table Scenes, Book Scenes, Environmental Scenes, etc.

As an implementation manner, the first device may perform image scene analysis on the image to determine the scene to which multiple objects represented by the image belong (also called "vertical scene") and the respective positions of the multiple scenes in the image.

Wherein, the multiple objects represented by the image may be determined by the first device based on feature distribution in the focus area of the image. Among them, the object represented by the image may include but not limited to one or more of the following: cards (such as ID cards, passports, bank cards, business cards, etc.), license plates, photos, signboards, posters, road signs, road signs, documents , forms, books, etc.

As another implementation, in order to ensure the reliability and accuracy of image semantic segmentation and improve the fault tolerance of image semantic segmentation, the first device can determine the multiple The scene to which the object belongs and the respective positions of multiple scenes in the image.

For example, assuming that the image includes the first information (such as the first text line), and after image scene analysis, it is determined that the scenes to which the above-mentioned objects belong include the first scene and the second scene, then the first device can perform text line information on the image. For analysis, when it is determined that the first part of the first information belongs to the first scene and the second part of the first information belongs to the second scene, image regions corresponding to the first scene and the second scene are merged.

For example, if it is determined through text line analysis that a part of the same text line belongs to scene 1 and another part belongs to scene 2, image regions corresponding to scene 1 and scene 2 are merged. As shown in (b) of FIG. 12 , the text line 1201 spans regions corresponding to two scenes. In this case, the first device merges the two image regions corresponding to the two scenes.

As an implementation manner, a common-stem dual-branch analysis unit may be integrated in the first device to determine the scenes to which multiple objects represented by the image belong and the respective positions of the multiple scenes in the image. Among them, the double branch refers to the image scene analysis branch (R branch) and the text line analysis branch (H branch). The common backbone refers to the image semantic segmentation backbone. As shown in Figure 18, a branch scene analysis branch (R branch) of the semantic segmentation backbone analyzes the scene to which the object represented by picture A belongs, and obtains the image scene analysis result; another branch text line analysis branch (H branch) analyzes picture A The text line distribution in , get the text line analysis result. Exemplarily, in the embodiment of the present application, the backbone may adopt a U-Net structure and a DenseNet structure.

S1703. The first device determines the importance of objects represented by multiple image regions.

For example, the first device may perform a saliency analysis to determine the importance of objects represented by multiple image regions.

As an example, the first device may determine the importance of multiple image regions according to the brightness, texture, mask value of the focus region, and the like.

As an example, the first device may calculate the saliency values of multiple image regions based on the foregoing formula 1. Among them, the higher the saliency value, the more important the corresponding image region; the lower the saliency value, the less important the corresponding image region.

In the embodiment of the present application, the first device may perform tasks in the subsequent task distribution stage according to the importance of objects represented by the multiple image regions. Exemplarily, the first device may respectively identify information in one or more corresponding image regions among the plurality of image regions based on a specific scene, in which the represented object is of high importance.

For example, when the saliency value of a certain image region is less than the saliency threshold, the task distribution and subsequent text information recognition for the image region are abandoned. By comprehensively considering the regional saliency, unimportant areas in the image can be effectively filtered out, so as to avoid the resource occupation of the information identification unit by the unimportant areas, resulting in a high load of the information identification unit.

It should be noted that, in some embodiments, the first device may not perform the foregoing S1703, that is, the first device continues to perform the following S1704 after performing the foregoing S1702.

S1704. The first device sends the information identification tasks of the multiple image regions to the corresponding information identification unit of the second device.

As an example, a specific model or a specific algorithm for identifying information in a corresponding scenario may be integrated in the information identification unit.

For example, assume that the object represented by the image belongs to three scenes (such as the first scene, the second scene and the third scene), and the three scenes correspond to the first image area, the second image area and the third image area in the image respectively , then in S1704, the first device may send the first recognition task, the second recognition task and the third recognition task to the second device (such as the information recognition management unit of the second device). After the second device (such as the information recognition management unit of the second device) receives the above recognition tasks, it distributes the first recognition task, the second recognition task and the third recognition task to the first recognition unit and the second recognition task according to the scene. unit and the third identification unit. Wherein, the first identification unit is used for information identification of the image area (i.e. the first image area) belonging to the first scene, and the second identification unit is used for information identification of the image area (i.e. the second image area) belonging to the second scene. Identifying, the third identifying unit is used to identify the information of the image area (ie the third image area) belonging to the third scene.

As an example, the information recognition task may carry the image and the position of the corresponding image region in the image (such as the coordinate information of the corresponding image region in the image). For example, the first recognition task carries the image and the position of the first image region in the image, the second recognition task carries the image and the position of the second image region in the image, and the third recognition task carries the image and the position of the first The position of the three image regions in the image.

As another example, the information recognition task may carry corresponding image regions. For example, the first recognition task carries the first image region, the second recognition task carries the second image region, and the third recognition task carries the third image region.

S1705. The multiple information identification units of the second device respectively identify the information in the corresponding image area, and obtain multiple information identification results.

Exemplarily, the multiple information identification units of the second device can respectively use a specific model or a specific algorithm corresponding to the scene to perform information identification on one or more image areas in the multiple image areas, and obtain the information on one or more image areas information identification results.

The object represented by the image belongs to three scenes (such as the first scene, the second scene and the third scene), and the first recognition unit, the second recognition unit and the third recognition unit of the second device are respectively responsible for the correspondence between the above three scenes Taking the information recognition task of the image area of the image area as an example, in S1705, the first identification unit identifies the information in the first image area to obtain the first identification result; the second identification unit identifies the information in the second image area to obtain the second identification Result; the third recognition unit recognizes the information in the third image area to obtain a third recognition result.

As an example, the information identification result output by the information identification unit may be recorded in the form of coordinates, may be superimposed on the image or the corresponding position of the image area, and may also be in a document format, which is not limited in this embodiment of the application.

S1706. The second device obtains the information recognition result of the image according to the multiple information recognition results.

For example, the second device may obtain the information recognition result of the image by arranging and merging the information recognition results of one or more image regions by the information recognition unit.

The object represented by the image belongs to three scenes (such as the first scene, the second scene and the third scene), and the first recognition unit, the second recognition unit and the third recognition unit respectively recognize the first image area, the second image area and the information in the third image area, respectively obtaining the first recognition result, the second recognition result and the third recognition result as an example, S1706 may specifically include: the second device according to the first recognition result, the second recognition result and the third recognition result As a result, an information recognition result of the image is obtained.

For example, the second device may integrate information recognition results of one or more image regions according to preset rules. The embodiment of the present application does not limit specific rule settings, nor does it limit the basis for result integration. For example, the preset rule may be a preset position sequence, such as integrating the recognition results of multiple image regions in a sequence from top to bottom and from left to right. As another example, the preset rule may be a preset saliency order, such as integrating the recognition results of multiple image regions in descending order of saliency values.

Further, as an example, the second device may send the obtained information recognition result of the image to the first device, so that the first device can display it.

The method for identifying text information in an image provided by the embodiment of the present application comprehensively utilizes a focus area analysis algorithm (such as a visual saliency detection algorithm) and an image semantic segmentation algorithm to analyze the scene of the image, and calls the corresponding special OCR capability according to the analysis result, For example, the special model performs information recognition in different scenes, and finally integrates the information recognition results of multiple special models to obtain the information recognition result of the final image.

Compared with conventional information recognition methods, the method for recognizing text information in images provided by the embodiments of the present application can obtain more accurate and reliable information recognition results, and provide users with a more convenient, efficient and reliable intelligent information recognition experience.

For example, compared to the result of identifying the picture A information using the conventional information identification method shown in Figure 1 or Figure 4, as shown in Figure 8, Figure 9 or Figure 14, the method for identifying text information in the image provided by the embodiment of the present application Image A can be segmented into an image area belonging to a document scene, an image area belonging to a form scene, and an image area belonging to a card scene through image semantic segmentation. Furthermore, through the information recognition of the image area of the document scene, the image area of the table scene and the image area of the card scene, the text in the picture comment was successfully recognized, which greatly improved the speed, success rate and accuracy of information recognition. Improve user experience.

As another example, as shown in Figure 19, compared to the information recognition of picture B using conventional information recognition methods, due to the limited input size of the information recognition model or special algorithm, it is impossible to recognize the small and dense images on the central screen of picture B. As a result of the text, the method for identifying text information in an image provided by the embodiment of the present application can divide the picture B into an image area belonging to the table scene and an image area belonging to the environment scene through image semantic segmentation. Furthermore, by calculating the regional saliency of multiple image regions, the image regions with low salience are not sent to the information recognition unit, which indirectly expands the image ratio of the small and dense text on the central screen of picture B, greatly improving the information The speed, success rate and accuracy of recognition improve user experience.

As another example, as shown in Figure 20, compared with the conventional information recognition method based on a single special model for information recognition of picture C, due to the large contrast between text comments and picture comments, the text in picture comments cannot be recognized As a result of missed detection, the method for identifying text information in an image provided by the embodiment of the present application can divide the picture C into an image area belonging to the table scene and an image area belonging to the picture scene through image semantic segmentation. Further, by performing information recognition on the image area of the table scene and the image area of the picture scene, the text in the picture comment is successfully recognized, which greatly improves the speed, success rate and accuracy of information recognition, and improves user experience.

As another example, as shown in Figure 21, compared to the result of recognizing the interfering text in the background area of the picture D without considering the salience of the information in the picture D by using the conventional information recognition method, the recognition of the text in the image provided by the embodiment of the present application The information method can filter the disturbing text in the image background by calculating the regional saliency, avoid the interference caused by the recognition of the disturbing text to the user, save the computing power of the information recognition model or algorithm, and improve the user experience.

It should be understood that various schemes of the embodiments of the present application can be used in a reasonable combination, and the explanations or descriptions of various terms appearing in the embodiments can be referred to or interpreted in each embodiment, which is not limited.

It should also be understood that in various embodiments of the present application, the serial numbers of the above-mentioned processes do not mean the order of execution, and the order of execution of the processes should be determined by their functions and internal logic, and should not be implemented in this application. The implementation of the examples constitutes no limitation.

It can be understood that, in order to realize the functions of any one of the foregoing embodiments, the first device includes hardware structures and/or software modules corresponding to each function. Those skilled in the art should easily realize that the present application can be implemented in the form of hardware or a combination of hardware and computer software in combination with the units and algorithm steps of each example described in the embodiments disclosed herein. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

In this embodiment of the present application, the first device may be divided into functional modules. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. It should be noted that the division of modules in the embodiment of the present application is schematic, and is only a logical function division, and there may be other division methods in actual implementation.

For example, in the case of dividing each functional module in an integrated manner, as shown in FIG. 22 , it is a structural block diagram of a first device provided in the embodiment of the present application. As shown in FIG. 22 , the first device may include a processing unit 2210, a transceiver unit 2220, a storage unit 2230, and a display unit 2240.

Wherein, the processing unit 2210 is used to support the first device to determine the focus area of the image, divide the image into multiple image areas according to the scene, determine the importance of objects represented by the multiple image areas, and identify information in the multiple image areas, Integrating recognition results of information in multiple image regions, and/or other processes related to the embodiments of the present application. The transceiver unit 2220 is used to support the first device to send the information recognition tasks of multiple image regions to the corresponding information recognition unit of the second device, receive the image information recognition result from the second device, and/or be related to the embodiment of the present application other processes. The storage unit 2230 is used to support the first device to store computer programs and implement the processing data and/or processing results (such as information identification results) in the methods provided in the embodiments of the present application. The display unit 2240 is configured to support the first device to display images, display information recognition results of images, and/or other interfaces related to the embodiments of the present application.

As an example, the transceiver unit 2220 may include a radio frequency circuit. Specifically, the first device may receive and send wireless signals through a radio frequency circuit. Typically, radio frequency circuitry includes, but is not limited to, an antenna, at least one amplifier, transceiver, coupler, low noise amplifier, duplexer, and the like. In addition, radio frequency circuits can also communicate with other devices through wireless communication. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile Communications, General Packet Radio Service, Code Division Multiple Access, Wideband Code Division Multiple Access, Long Term Evolution, Email, Short Message Service, etc.

It should be understood that each module in the first device may be implemented in the form of software and/or hardware, which is not specifically limited. In other words, electronic equipment is presented in the form of functional modules. The "module" here may refer to an application-specific integrated circuit ASIC, a circuit, a processor and memory executing one or more software or firmware programs, an integrated logic circuit, and/or other devices that can provide the above-mentioned functions.

In an optional manner, when software is used to implement data transmission, it may be implemented in whole or in part in the form of computer program products. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions described in the embodiments of the present application are realized in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium, (such as a floppy disk, a hard disk, etc. , magnetic tape), optical media (such as digital video disk (DVD)), or semiconductor media (such as solid state disk (SSD)), etc.

The steps of the methods or algorithms described in conjunction with the embodiments of the present application may be implemented in hardware, or may be implemented in a manner in which a processor executes software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, mobile hard disk, CD-ROM or any other form of storage known in the art medium. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be a component of the processor. The processor and storage medium can be located in the ASIC. Alternatively, the ASIC may be located in the electronic device. Of course, the processor and the storage medium may also exist in the first device as discrete components.

Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated according to needs It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

Claims (30)

1. A method for recognizing text information in an image, characterized in that the method comprises: acquiring an image, the image including a plurality of content areas; identifying the text information in the multiple content areas respectively by using multiple models; Integrating the recognition results of the multiple models for the text information in the multiple content areas to obtain the text recognition results for the image. 2. The method according to claim 1, wherein the plurality of content areas correspond to a plurality of scenes; the method further comprises: classify the multiple content areas, and determine multiple scenes corresponding to the multiple content areas; The identifying the text information in the multiple content areas respectively through multiple models includes: The text information in the multiple content regions is respectively identified through the multiple models corresponding to the multiple scenes. 3. The method according to claim 2, wherein the classifying the multiple content areas and determining the multiple scenes corresponding to the multiple content areas comprises: Based on the image semantic segmentation model or algorithm, perform image semantic segmentation on the multiple content areas, and determine the multiple scenes corresponding to the multiple content areas. 4. The method according to claim 2 or 3, wherein the identifying the text information in the plurality of content areas through the plurality of models corresponding to the plurality of scenes includes: According to the multiple scenes corresponding to the multiple content areas, crop the multiple content areas to obtain multiple image areas; The plurality of image regions are respectively distributed to corresponding models, so as to respectively recognize text information in the plurality of image regions by the plurality of models. 5. The method according to claim 2 or 3, wherein the identifying the text information in the plurality of content areas through the plurality of models corresponding to the plurality of scenes includes: According to the multiple scenes corresponding to the multiple content areas, determine the position information of the multiple image areas corresponding to the multiple scenes in the image; The images are sent to the multiple models, and the location information of the multiple image areas are respectively distributed to the corresponding models, so as to respectively recognize the text information in the multiple image areas by the multiple models. 6. The method according to any one of claims 1-5, wherein the method further comprises: Based on the visual saliency detection VSD algorithm, calculating the saliency value of the pixel in the image; determining a focus area of the image based on the saliency values of the pixels in the image; Wherein, the saliency value of the pixels in the focus area of the image is greater than a preset threshold. 7. The method according to claim 6, further comprising: determining regional saliency values of the plurality of content regions according to the saliency values of pixels in the plurality of content regions and the area of the focus region, respectively; The identifying the text information in the multiple content areas respectively through multiple models includes: Among the plurality of content areas, the text information in the content areas whose area saliency value is greater than a preset threshold is identified respectively by using multiple models. 8. The method according to any one of claims 1-7, wherein the method further comprises: performing text line analysis on the image, and determining that the image includes first information, wherein a first part of the first information belongs to a first scene, and a second part of the first information belongs to a second scene; Merge image regions corresponding to the first scene and the second scene. 9. The method according to any one of claims 1-8, wherein the method further comprises: Overlaying and displaying the text recognition result of the image on the image; or, hovering over the image to display the text recognition result of the image; or, displaying the text recognition results of said image in the typesetting format of said image; or, The layout is rearranged according to preset rules to display the text recognition results of the image, and the preset rules include: preset position order and preset significance order. 10. The method according to any one of claims 2-9, wherein the multiple scenes corresponding to the multiple content areas include any of the following: card scene, license plate scene, photo Scene, Signboard Scene, Poster Scene, Street Sign Scene, Road Sign Scene, Document Scene, Table Scene, Book Scene, Environmental Scene. 11. A method for recognizing text information in an image, characterized in that the method comprises: the first device acquires an image, the image including a plurality of content areas; The first device classifies the multiple content areas, and determines multiple scenes corresponding to the multiple content areas; The first device sends multiple information identification tasks corresponding to the multiple scenarios to the second device; The first device receives an information recognition result of the image from the second device. 12. The method according to claim 11, wherein the first device classifies the multiple content areas and determines the multiple scenes corresponding to the multiple content areas, comprising: The first device performs image semantic segmentation on the plurality of content regions based on an image semantic segmentation model or algorithm, and determines the plurality of scenes corresponding to the plurality of content regions. 13. The method according to claim 11 or 12, characterized in that the method further comprises: The first device crops the multiple content areas according to the multiple scenes corresponding to the multiple content areas to obtain multiple sub-images; The first device sends multiple information identification tasks corresponding to the multiple scenarios to the second device, including: The first device distributes the plurality of sub-images to corresponding models in the second device, so as to respectively recognize text information in the plurality of sub-images through the plurality of models. 14. The method according to claim 11 or 12, characterized in that the method further comprises: The first device determines, according to the multiple scenes corresponding to the multiple content areas, the position information of the multiple image areas corresponding to the multiple scenes in the image; The first device sends the image to the second device, and distributes the position information of the plurality of image regions to corresponding models in the second device, so as to identify Text information in the plurality of image areas. 15. The method according to any one of claims 11-14, further comprising: The first device calculates a saliency value of a pixel in the image based on a visual saliency detection (VSD) algorithm; determining, by the first device, a focal area of the image according to a saliency value of a pixel in the image; Wherein, the saliency value of the pixels in the focus area of the image is greater than a preset threshold. 16. The method of claim 15, further comprising: The first device determines area saliency values of the multiple content areas according to the saliency values of the pixels in the multiple content areas and the area of the focus area, respectively; The first device sends multiple information identification tasks corresponding to the multiple scenarios to the second device, including: The first device sends to the second device a plurality of information identification tasks corresponding to content regions whose region significance values are greater than a preset threshold among the plurality of content regions. 17. The method according to any one of claims 11-16, further comprising: The first device performs text line analysis on the image, and determines that the image includes first information, wherein a first part of the first information belongs to a first scene, and a second part of the first information belongs to a second scene. Scenes; The first device merges image regions corresponding to the first scene and the second scene. 18. The method according to any one of claims 11-17, further comprising: The first device superimposes and displays a text recognition result on the image on the image; or, The first device hovers over the image to display the text recognition result of the image; or, The first device displays the text recognition result of the image in the typesetting format of the image; or, The first device rearranges the typesetting according to preset rules to display the text recognition results of the image, and the preset rules include: a preset position order and a preset significance order. 19. The method according to any one of claims 11-18, wherein the multiple scenes corresponding to the multiple content areas include any of the following: card scene, license plate scene, photo Scene, Signboard Scene, Poster Scene, Street Sign Scene, Road Sign Scene, Document Scene, Table Scene, Book Scene, Environmental Scene. 20. A method for recognizing text information in an image, characterized in that the method comprises: The second device receives multiple information recognition tasks of images from the first device, and the multiple information recognition tasks correspond to multiple scenes; The second device respectively executes the multiple information recognition tasks through multiple models, and obtains multiple information recognition results; The second device integrates the multiple information recognition results obtained by the multiple models to obtain a text recognition result for the image. 21. The method of claim 20, further comprising: The second device sends a text recognition result of the image to the first device. 22. The method according to claim 20 or 21, wherein the multiple information recognition tasks correspond to multiple sub-images of the image, and the multiple sub-images correspond to multiple scenes; The second device respectively executes the multiple information recognition tasks through multiple models, and obtains multiple information recognition results, including: The second device distributes the plurality of sub-images to models corresponding to corresponding scenes, so as to respectively recognize text information in the plurality of sub-images through the plurality of models. 23. The method according to claim 20 or 21, wherein the plurality of information identification tasks are respectively used to indicate the image and the location information of a plurality of image regions in the image, the plurality of images Regions correspond to multiple scenes; The second device respectively executes the multiple information recognition tasks through multiple models, and obtains multiple information recognition results, including: The second device distributes the image and position information of multiple image areas to models corresponding to corresponding scenes, so as to respectively recognize text information in the multiple image areas through the multiple models. 24. The method according to any one of claims 20-23, characterized in that the second device integrates the multiple information recognition results obtained by the multiple models to obtain the text recognition of the image Results, including: The second device integrates the multiple information recognition results in the typesetting format of the image to obtain a text recognition result for the image; or, The second device integrates the multiple information recognition results according to a preset rule to obtain a text recognition result of the image; the preset rule includes: a preset position order and a preset significance order. 25. A first device, characterized in that the first device comprises: memory for storing computer programs; Transceivers for receiving or transmitting radio signals; A processor, configured to execute the computer program, so that the first device implements the method according to any one of claims 11-19. 26. A second device, characterized in that the second device comprises: memory for storing computer programs; Transceivers for receiving or transmitting radio signals; A processor, configured to execute the computer program, so that the second device implements the method according to any one of claims 20-24. 27. An information identification system, characterized in that the information identification system is used to implement the method according to any one of claims 1-10; the information identification system comprises: A first device as claimed in claim 25; and, A second device as claimed in claim 26. 28. A computer-readable storage medium, characterized in that computer program codes are stored on the computer-readable storage medium, and when the computer program codes are executed by a processing circuit, the implementation of claims 1-10 or 11-19 is achieved. any one of the methods described. 29. A chip system, characterized in that the chip system includes a processing circuit and a storage medium, and computer program codes are stored in the storage medium; when the computer program codes are executed by the processing circuit, the implementation of claim 1 - the method of any one of 10 or 11-19. 30. A computer program product, characterized in that the computer program product is configured to run on a computer to implement the method according to any one of claims 1-10 or 11-19.

Images