WO2022041406A1 - Ocr and transfer learning-based app violation monitoring method - Google Patents

Abstract

Disclosed in the present invention is an OCR and transfer learning-based app violation monitoring method. Said method comprises: periodically updating an APK, and performing data acquisition on a corresponding app according to the updated APK, the data acquisition comprising the capture of data packets and the acquisition of page screenshots; performing text recognition and extraction on the screenshots on the basis of an OCR algorithm; for the recognized text content, constructing a sample set by means of keywords and regular expressions, and performing manual annotation on same; inputting the manually annotated sample set into a pre-trained deep learning model for model adjustment, and implementing violation determination of text in different scenarios by means of division of service scenarios; and collecting statistics of scores of different apps according to the determination results outputted by the deep learning model, so as to obtain violation scores of the apps. By acquiring and analyzing data of apps, the present invention effectively and quickly detects the violation usage situations of the apps.

Description

An App Violation Monitoring Method Based on OCR and Transfer Learning technical field

The invention relates to the technical field of data monitoring, in particular to an APP violation monitoring method based on OCR and transfer learning.

Background technique

Through real-time automatic public opinion collection, public opinion analysis, public opinion summary, and public opinion monitoring of massive network public opinion information, and identify key public opinion information, timely notify relevant personnel, so as to respond to emergency in the first time, for correct public opinion orientation and collection of netizens A set of information platforms that provide direct support for opinions. However, it only collects public opinion data, and cannot detect special content; and generally only detects website data, and does not detect mobile data.

According to the user-defined task configuration, the semi-structured and unstructured data in the Internet target pages can be extracted in batches and accurately, converted into structured records, and saved in the local database for internal use or external network publishing. Realize the acquisition of external information. However, generally only for network data collection, there is no way to collect mobile APP data; and websites have different complexities and anti-crawling measures, the success rate of data crawling cannot be guaranteed.

The short text classification model refers to text forms with no more than 200 words, such as microblogs, chat messages, news topics, opinion comments, question texts, mobile phone messages, literature summaries, etc. The purpose of the short text classification task is to automatically process the short text input by the user to obtain valuable classification output. However, the short text classification model is supervised learning, which often requires massive data as support and requires a lot of manual labeling workload.

That is to say, it is impossible to obtain target information quickly and effectively in the existing technology, for example, the data required for a certain information violation monitoring cannot be obtained; if the APP has certain anti-crawling measures, it will not be able to use crawlers for data crawling; target information promotion Contains a large number of pictures and cannot process data in picture format; there are insufficient data samples for certain information violation monitoring, even if network data is obtained, a large amount of manual annotation is required; supervised deep learning models require massive training data, if you want to obtain Good results also require a lot of machine resources for training; there is a lack of data viewing and comparison platforms for certain information violation monitoring.

Therefore, the existing technology still needs to be improved and developed.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to provide an APP violation monitoring method based on OCR and transfer learning, which aims to solve the problem that target information cannot be quickly and effectively obtained in the prior art.

In order to achieve the above object, the present invention provides an APP violation monitoring method based on OCR and transfer learning, and the APP violation monitoring method based on OCR and transfer learning includes the following steps:

Regularly update the APK, and perform data collection of the corresponding APP according to the updated APK, and the data collection includes data capture and page screenshots;

Text recognition and extraction of screenshots based on OCR algorithm;

For the recognized text content, a sample set is constructed through keywords and regular expressions, and manual annotation is performed;

Input the manually labeled sample set into the pre-trained deep learning model for model adjustment, and realize the text violation discrimination in different scenarios by dividing the business scenarios;

According to the discrimination results output by the deep learning model, the scores of different APPs are counted to obtain the violation scores of the APPs.

Optionally, the method for monitoring APP violations based on OCR and transfer learning, wherein, in the regularly updating APK, data collection of the corresponding APP is performed according to the updated APK, specifically including:

Based on Java and with the help of Jsoup library, the APK of each application is crawled, and the APK of the application store is updated regularly, and the data collection of the corresponding APP is carried out according to the updated APK.

Optionally, in the method for monitoring APP violations based on OCR and transfer learning, the data collection method specifically includes: using a crawler to directly capture packets of promotional data and using an Appium script to automatically take screenshots of pages.

Optionally, in the method for monitoring APP violations based on OCR and transfer learning, wherein the manually annotated sample set is input into a pre-trained deep learning model for model adjustment, and by dividing business scenarios, the text in different scenarios is realized. Violation judgment, which previously included:

A corpus is constructed for supervising the training of the deep learning model.

Optionally, in the method for monitoring APP violations based on OCR and transfer learning, the construction process of the corpus includes:

Obtain multiple keywords, and match the keywords;

A keyword-based training corpus is constructed and labeled manually for generating the corpus.

Optionally, the described APP violation monitoring method based on OCR and transfer learning, wherein, the scores of different APPs are counted to obtain the APP violation scores, specifically including:

The violation score of the APP is obtained by weighted average:

表示加权平均数，f1～fk为每个维度违规项配置的权重，x1～xk为实际每个维度违规项的质检结果异常数，n表示维度总个数，不同维度表示不同违规场景。 in,

Indicates the weighted average number, f1~fk are the weights configured for the violation items of each dimension, x1~xk are the actual number of abnormal quality inspection results of the violation items in each dimension, n is the total number of dimensions, and different dimensions represent different violation scenarios.

Optionally, in the method for monitoring APP violations based on OCR and transfer learning, wherein, according to the discrimination results output by the deep learning model, the scores of different APPs are counted to obtain the violation scores of the APPs, and then the include:

Set a scheduled start task for all tasks.

Optionally, in the APP violation monitoring method based on OCR and transfer learning, the tasks include: APP crawling timed task, APP screenshot timed task, and violation monitoring timed task.

In addition, in order to achieve the above object, the present invention also provides an intelligent terminal, wherein the intelligent terminal includes: a memory, a processor, and an OCR-based and transfer learning based OCR and transfer learning system stored in the memory and running on the processor. The APP violation monitoring program, when the APP violation monitoring program based on OCR and transfer learning is executed by the processor, implements the steps of the above-mentioned OCR and transfer learning-based APP violation monitoring method.

In addition, in order to achieve the above object, the present invention also provides a storage medium, wherein the storage medium stores an APP violation monitoring program based on OCR and transfer learning, and the APP violation monitoring program based on OCR and transfer learning is processed by a processor When executed, the steps of implementing the above-mentioned OCR and transfer learning-based APP violation monitoring method.

The present invention periodically updates the APK, and collects data corresponding to the APP according to the updated APK. The data collection includes data packet capture and page screenshots; text recognition and extraction are performed on the screenshots based on the OCR algorithm; Keywords and regular expressions are used to construct a sample set, and manually annotate; input the manually labeled sample set into a pre-trained deep learning model for model adjustment, and divide business scenarios to achieve text violation discrimination in different scenarios; Based on the discrimination results output by the deep learning model, the scores of different APPs are counted to obtain the APP's violation score. The present invention effectively and quickly detects the illegal use of the APP by collecting and analyzing the data of the APP.

Description of drawings

Fig. 1 is a schematic diagram of a cross-platform, multi-language mobile terminal automated testing framework based on Client/Server architecture;

Figure 2 is a schematic diagram of the PaddleHub architecture in the pre-training model management and transfer learning tool;

Fig. 3 is the flow chart of CTPN algorithm;

Fig. 4 is the framework schematic diagram of the monitoring system of the mobile terminal based on OCR and transfer learning;

Figure 5 is a schematic diagram of a microservice architecture;

6 is a flowchart of a preferred embodiment of the method for monitoring APP violations based on OCR and transfer learning of the present invention;

7 is a schematic diagram of the execution process of a preferred embodiment of the APP violation monitoring method based on OCR and transfer learning of the present invention;

8 is a schematic diagram of a configuration path table structure formed when a screenshot is taken in a preferred embodiment of the APP violation monitoring method based on OCR and transfer learning of the present invention;

9 is a schematic diagram of a monitoring function for real-time monitoring of APP publicity data in a preferred embodiment of the APP violation monitoring method based on OCR and transfer learning of the present invention;

FIG. 10 is a schematic diagram of an operating environment of a preferred embodiment of an intelligent terminal of the present invention.

detailed description

In order to make the objectives, technical solutions and advantages of the present invention clearer and clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

Appium is a cross-platform, multi-language mobile automated testing framework based on the Client/Server architecture. Appium can support both Android and iOS across platforms. As shown in Figure 1, through any client (such as Java, Python, Javascript, etc.) that implements the WebDriver JSONWriteProtocol protocol corresponding to the Appium client class library, the Appium server can perform operations such as screenshots and clicks on the mobile device through parsing. such as commands. In addition, Appium Destop can easily and accurately assist developers to obtain the coordinates of each APP control and xpath (language used to determine the location of a certain part of the XML document) path attribute information.

Transfer Learning is a sub-field of deep learning. The goal of this research field is to use the similarity between data, tasks, or models to transfer the knowledge learned in the old field and apply it to the new field. , you can usually perform Fine-tune (fine-tuning) on the pre-trained model to achieve model migration, so as to achieve the purpose of adapting the model to new field data, as shown in Figure 2, PaddleHub is an open source pre-trained model management and migration learning tool , Through PaddleHub, developers can use high-quality pre-trained models combined with the Fine-tune API to quickly complete the entire process from transfer learning to application deployment and complete model convergence in a shorter time, while enabling the model to have better generalization capabilities.

The OCR algorithm can generally be divided into two parts: text detection (detecting the area where the text is located) and character recognition (recognizing the text in the area). For example, the CTPN (Connectionist Text Proposal Network) model, which greatly simplifies the detection process, The seamless combination of RNN and CNN is used to improve the detection accuracy (CNN is used to extract deep features, and RNN is used for sequence feature recognition), so that the effect, speed and robustness of text detection have been qualitatively improved. A sequence composed of "characters, parts of characters, and multiple characters", so the detection target of text is not independent and closed, but related before and after. Therefore, RNN (Recurrent Neural Networks) is used in CTPN. ) to use the information of the context to predict the text position. The algorithm in the CTPN paper can be implemented through the Keras+TensorFlow framework: specifically, a series of proposals (pre-selection boxes) are generated by using the feature map (feature map) output by VGG16 convolution for detection, and the CTPN text recognition model is trained on the VOC2007_text_detection dataset. The text area of the image can be detected using the CTPN algorithm. This algorithm can detect text lines of multiple sizes and aspect ratios on a single-scale image by setting vertical anchors and fine-grained detection strategies. It also limits CTPN to only horizontal direction. The detected text effect is relatively good, but the detection effect in other directions is relatively poor. However, since most of the picture data collected by this system through Appium's automatic screenshots are in horizontal direction, there is less interference background, there is almost no edge overlap, and the picture tilt angle is not large, so the CTPN text detection algorithm has a high accuracy rate. Its algorithm steps are as follows As shown in Figure 3:

(1) Using VGG16 (a classic model of CNN convolutional neural network) as the extraction feature, a feature map of size W×H×C is obtained, and a sliding window of size 3×3 is set on the basis of the fifth layer. The windows all get a feature vector of length 3×3×C.

(2) The convolution feature obtained in step (1) is used as the input of 256-dimensional bidirectional LSTM (two 128-dimensional LSTM), and the output of length W × 256 is obtained. The introduction of LSTM is to solve the problem of RNN layer gradient disappearance and further expansion RNN layer.

(3) The output layer part contains three outputs, which are 2k vertical coordinates, 2k scores, and k side-refinements, using a standard non-maximum suppression algorithm (NMS) to filter out duplicate text boxes.

After getting the areas containing text in the image, the next task is to identify the text in each area. DenseNet is one of the character recognition algorithms. It uses Relu as the activation function and uses 3 Dense Block layers for calculation. The DenseNet network is formed by connecting each Dense Block through the Transition structure. Finally, it is obtained by training with CTC loss. In the data model, after the data is processed by the DenseBlock layer, the convolution operation is performed and then transmitted to the Transition structure for parameter integration specification. The parameters are reduced by pooling and then transmitted to the lower Dense Block structure to achieve higher accuracy. During the training process, When the accuracy rate and loss value reach the limit and fall into oscillation, the learning rate needs to be reduced exponentially, which can make the accuracy rate jump sharply immediately.

Further, as shown in Figure 4, the technology stack used by the system includes front-end, back-end, algorithm and operation and maintenance, wherein the front-end, back-end, algorithm and operation and maintenance can be classified as follows:

Front-end: Vue, ElementUI, Vuex, Axios;

Backend: Java, SpringBoot, SpringCloud, Nacos, SpringGateWay, SpringAdmin, Feign, XXL-JOB, Mybatis-Plus, Maven;

Algorithms: Flask, TensorFlow, Keras, Pytorch, CTPN, CRNN;

Operation and maintenance: Docker, Linux;

The system adopts the design idea of separating front and back ends. The front end is based on Vue, combined with ElementUI, Vuex and Axios and other technology stacks to build the management system interface. Realize the configuration center and registration center, uniformly access the backend interface through the SpringGateWay gateway component, and monitor the backend service uniformly through SpringAdmin. The timing task background implements distributed timing tasks based on Java combined with the open source framework XXL-JOB. The algorithm background is based on Flask combined with TensorFlow and Keras to implement the OCR text recognition algorithm, and fine-tuned on the basis of PaddleHub's pre-training model to generate a violation detection model.

In the background data persistence layer, MySQL is used, Mybatis Plus is used to add, delete, modify and search data, MongoDb is used to store HTTP packet capture data, and Redis is used to cache hot data and implement distributed locks.

According to functional requirements, the background is divided into three categories: web background, timed task background and algorithm background.

Web backend: Based on Java, it provides basic interfaces for adding, deleting, modifying and searching various data, including but not limited to data and model management. Timing task background: Based on Java and XXL-JOB, it is used to periodically execute timing tasks. Algorithm background: OCR algorithm, violation detection algorithm and semantic similarity algorithm are implemented based on Python, CTPN, CRNN and PaddleHub.

Drawing on the idea of microservices, the background can be abstractly subdivided into five categories: data query services, timed task services, data collection services, data violation detection services, and data analysis services. The basic functions are provided for the above two through data collection service, data violation detection service and data analysis service. The microservice architecture is shown in Figure 5.

Through the above-mentioned various technical frameworks, a mobile terminal violation monitoring system based on OCR and transfer learning is constructed, and each module of the system is integrated based on the technical architecture to ensure the maintainability and scalability of the system.

The method for monitoring APP violations based on OCR and transfer learning according to the preferred embodiment of the present invention is shown in FIG. 6 and FIG. 7 . The method for monitoring APP violations based on OCR and transfer learning includes the following steps:

Step S10, update the APK regularly, and perform data collection of the corresponding APP according to the updated APK, where the data collection includes data packet capture and page screenshots.

Specifically, based on Java, the APK of each application is crawled with the help of the Jsoup library, and the APK of the application store is updated regularly, and the data of the corresponding APP is collected according to the updated APK. The collected application stores include Xiaomi App Store, App Store, and Baidu App Assistant. , 360 Mobile Assistant, Pea Pod, PP Assistant, Sogou Mobile Assistant, etc.

The purpose of data collection is to monitor the relevant behavior of the APP, such as the fund overview, fund manager, fund announcement, fund promotion carousel, fund promotion activity introduction and other content in the APP.

The data collection method specifically includes: using a crawler to directly capture packets of publicity data and using an Appium script to automatically take screenshots of pages.

For example, when capturing packets through crawler data, some APPs use Fidder to capture packets, such as Tiantian Fund, China Asset Manager, Egg Roll Fund, GF Yitaojin, Flush Fund, Xingquan Fund, Cathay Fund, Haomai Funds, Profit Funds, etc. First, obtain the URL and parameters of the fund list interface through Fidder. According to the fund code, you can obtain the fund profile and fund announcement data; then construct the corresponding http request through Python, and store the result in the MongoDb database; finally through the Python requests Libraries to easily build requests and get corresponding data results.

For example, when taking automated screenshots, the difficulty of automated screenshots based on Appium is how to ensure the stability and comprehensiveness of automated click scripts and the accuracy of OCR recognition. The present invention mainly locates control elements through text, and simultaneously uses Appium Destop to obtain XPath path assistance to locate control elements.

For example, because the interface of the fund APP that needs to be monitored by screenshots has a common feature of low interface depth and few interface search buttons, Appium can configure the design of the path table to cover the automated screenshot requirements of most APPs. The finally formed configuration path table structure is shown in Figure 8.

In order to achieve the versatility of scripts, the Appium configuration path is abstracted into five fields, namely application, root path, sub-path, exception log, and enable status. The specific format definitions of the root path and subpath are as follows:

【Home-0|Current-0|com.hctforgf.gff:id/risk_warn_tv-5-0】;

Among them, the three paths (home-0, current-0 and com.hctforgf.gff:id/risk_warn_tv-5-0) separated by the | symbol represent the three element controls that are clicked in sequence, home-0 and current-0 The 0 in com.hctforgf.gff:id/risk_warn_tv-5-0 represents positioning by the specified method, and the 0 represents positioning the control by element ID. From this, it can be concluded that the general format of the path is text-positioning control numerical method-extra parameter|text-positioning control numerical method-extra parameter|text-positioning control numerical method-extra parameter. The specific numbers and their meanings are shown in the table below:

In the carousel map of the APP, by providing the positioning carousel map with text and coordinates, the essence of positioning the carousel map by text is still positioning the carousel map by coordinates, and the carousel map can be slid left and right with the help of coordinates. At the same time, the problem that needs to be considered at this time is the problem that the automatic sliding of the APP carousel image and the sliding of the script result in an abnormal number of carousel images. The additional parameters of the positioning carousel image method define the total number of carousel images. Image similarity prevents the same carousel from being processed until the total number of corresponding carousels is obtained before exiting.

Step S20, performing text recognition and extraction on the screenshot based on the OCR algorithm.

Specifically, after all the screenshots are obtained, the OCR algorithm is used to identify and extract the text to obtain the required information.

Step S30 , constructing a sample set by using keywords and regular expressions for the recognized text content, and performing manual annotation.

Specifically, for the recognized text content, through keywords and regular expressions (regular expressions are a kind of logical formula for string operations, that is, some pre-defined specific characters and combinations of these specific characters are used to form A "rule string", this "rule string" is used to express a filtering logic for strings) to construct a sample set and improve reliability through manual annotation.

Step S40: Input the manually labeled sample set into the pre-trained deep learning model to adjust the model, and realize the violation judgment of text in different scenarios by dividing the business scenarios.

Before the step S40, the method further includes: constructing a corpus for supervising the training of the deep learning model. Specifically: obtaining a plurality of keywords, and matching the keywords; constructing training corpus based on keywords, and manually labeling them for generating the corpus.

For example, the fund sales violation discrimination model is trained in a supervised manner, so a corpus needs to be constructed for the training of a supervised deep learning model. Since the richness of the corpus will directly affect the accuracy of the semantic labels, in order to ensure the quality of the semantic label samples, the corpus of the illegal propaganda of fund sales is collected manually from the network.

Through the analysis and summary of network violation cases, we sorted out keywords such as high yield, zero risk, cash red envelopes, quick purchases, guaranteed, guaranteed, etc. for the parade. Based on the above keywords, the keywords are matched by means of including, not including, greater than, less than, equal to, and regular expressions, to construct a training corpus based on keywords, and manually label them for model training. Work.

Further, since deep learning requires a high amount of data in the model training process, it is impossible to obtain an effective model by directly training the model from scratch. Therefore, the system uses the PaddleHub framework to build a violation monitoring model. On the basis of the training model Ernie model, the model fine-tuning work based on transfer learning (ie model retraining) is carried out. In the training process, separate training work is carried out according to different violation scenarios (such as exaggerating income scenarios, slandering other fund managers, etc.) to classify violation discrimination models in different scenarios; finally, the classification of judging whether the data is illegal in each scenario is obtained. Model.

That is to say, in order to obtain a classification model, a large amount of labeled data is put into the model for training, and the model adjusts the weight of its own neural network nodes through these data and labels, and finally obtains the specific weight. The model can be used for data classification.

Step S50, according to the discrimination result output by the deep learning model, count the scores of different APPs, and obtain the violation scores of the APPs.

Specifically, the violation score of the APP is obtained by the weighted average (that is, the weighted average represents the violation score of the APP):

表示加权平均数，f1～fk为每个维度违规项配置的权重，x1～xk为实际每个维度违规项的质检结果异常数，n表示维度总个数，不同维度表示不同违规场景(例如“缺少合理风险提示”、“承诺保本收益”、“夸大收益”等)；根据质检结果异常数查找指定范围内的分数即是维度得分，同理可以计算出5(即n＝5 时)个维度的加权平均分作为平均得分，例如以非法承诺收益、缺少合理风险提示、诋毁其他基金管理人、承诺保本保收益、夸大收益等五个维度计算违规得分。 in,

Indicates the weighted average number, f1~fk are the weights configured for the violation items of each dimension, x1~xk are the actual number of abnormal quality inspection results of the violation items in each dimension, n is the total number of dimensions, and different dimensions indicate different violation scenarios (for example, "Lack of reasonable risk warning", "promised capital-guaranteed return", "exaggerated return", etc.); according to the abnormal number of quality inspection results to find the score within the specified range is the dimension score, and similarly, 5 can be calculated (that is, when n=5) The weighted average score of each dimension is used as the average score, for example, the violation score is calculated in five dimensions, such as illegal promised income, lack of reasonable risk warning, slandering other fund managers, promise to guarantee capital and income, and exaggerated income.

Further, after implementing the above steps, set a timed start task for all tasks, as shown in Figure 9, including the APP crawling timed task, the APP screenshot timed task and the violation monitoring timed task, so as to meet the requirements of real-time monitoring of APP publicity data. monitoring function.

The invention realizes the automatic crawling of the mobile terminal APP data regularly; for the data that cannot be crawled by the mobile terminal, the Appium simulates the touch screen operation to take screenshots; the OCR technology is used to perform character monitoring and character recognition on the pictures obtained from the screenshots, which solves the problem of fundraising. The problem of character monitoring in publicity pictures; for the problem of violation monitoring of publicity texts, a corpus construction method based on keywords and regular expressions is proposed; for the problem that deep learning domain models require a large amount of training corpus, the transfer learning model is used, and the pre-training model is used. Fine-tune the labeled data on the basis of the data set; build a dataset to train the deep learning model, and obtain a model capable of judging the illegal text classification of the fund; Automated violation monitoring for fund APPs.

The invention can collect and monitor the network fund publicity data uniformly, and the data and model results that can be obtained subsequently can be made into a knowledge base, which can be used in big data analysis, knowledge map and other fields.

Further, as shown in FIG. 10 , based on the above-mentioned APP violation monitoring method based on OCR and transfer learning, the present invention also provides an intelligent terminal correspondingly, the intelligent terminal includes a processor 10 , a memory 20 and a display 30 . FIG. 10 only shows some components of the smart terminal, but it should be understood that it is not required to implement all the shown components, and more or less components may be implemented instead.

In some embodiments, the memory 20 may be an internal storage unit of the smart terminal, such as a hard disk or a memory of the smart terminal. In other embodiments, the memory 20 may also be an external storage device of the smart terminal, such as a plug-in hard disk equipped on the smart terminal, a smart memory card (Smart Media Card, SMC), a secure digital (Secure) Digital, SD) card, flash card (Flash Card), etc. Further, the memory 20 may also include both an internal storage unit of the smart terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the smart terminal, such as program codes for installing the smart terminal. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, an APP violation monitoring program 40 based on OCR and transfer learning is stored on the memory 20, and the APP violation monitoring program 40 based on OCR and transfer learning can be executed by the processor 10, so as to realize the OCR-based application in this application. and transfer learning method for APP violation detection.

In some embodiments, the processor 10 may be a central processing unit (Central Processing Unit, CPU), a microprocessor or other data processing chips, for running program codes or processing data stored in the memory 20, such as Execute the APP violation monitoring method based on OCR and transfer learning, etc.

In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like. The display 30 is used for displaying information on the smart terminal and for displaying a visual user interface. The components 10-30 of the intelligent terminal communicate with each other through the system bus.

In one embodiment, when the processor 10 executes the APP violation monitoring program 40 based on OCR and transfer learning in the memory 20, the following steps are implemented:

Regularly update the APK, and perform data collection of the corresponding APP according to the updated APK, and the data collection includes data capture and page screenshots;

Text recognition and extraction of screenshots based on OCR algorithm;

For the recognized text content, a sample set is constructed through keywords and regular expressions, and manual annotation is performed;

Input the manually labeled sample set into the pre-trained deep learning model for model adjustment, and realize the text violation discrimination in different scenarios by dividing the business scenarios;

According to the discrimination results output by the deep learning model, the scores of different APPs are counted to obtain the violation scores of the APPs.

Wherein, for the regularly updated APK, data collection of the corresponding APP is performed according to the updated APK, which specifically includes:

Based on Java and with the help of Jsoup library, the APK of each application is crawled, and the APK of the application store is updated regularly, and the data collection of the corresponding APP is carried out according to the updated APK.

The data collection method specifically includes: using a crawler to directly capture packets of publicity data and using an Appium script to automatically take screenshots of pages.

Wherein, the manual annotated sample set is input into a pre-trained deep learning model for model adjustment, and the text violation judgment in different scenarios is realized by dividing the business scenarios, which also includes:

A corpus is constructed for supervising the training of the deep learning model.

Wherein, the construction process of the corpus includes:

Obtain multiple keywords, and match the keywords;

A keyword-based training corpus is constructed and labeled manually for generating the corpus.

Wherein, according to the statistics of the scores of different APPs, the violation scores of the APPs are obtained, including:

The violation score of the APP is obtained by weighted average:

表示加权平均数，f1～fk为每个维度违规项配置的权重，x1～xk为实际每个维度违规项的质检结果异常数，n表示维度总个数，不同维度表示不同违规场景。 in,

Indicates the weighted average number, f1~fk are the weights configured for the violation items of each dimension, x1~xk are the actual number of abnormal quality inspection results of the violation items in each dimension, n is the total number of dimensions, and different dimensions represent different violation scenarios.

Wherein, according to the judgment result output by the deep learning model, the scores of different APPs are counted to obtain the violation scores of the APPs, and the following further includes:

Set a scheduled start task for all tasks.

The tasks include: APP crawling timed tasks, APP screenshot timed tasks, and violation monitoring timed tasks.

The present invention also provides a storage medium, wherein the storage medium stores an APP violation monitoring program based on OCR and transfer learning, and when the APP violation monitoring program based on OCR and transfer learning is executed by a processor, the above-mentioned Steps of an APP violation detection method based on OCR and transfer learning.

To sum up, the present invention provides an APP violation monitoring method based on OCR and transfer learning. The method includes: updating the APK regularly, and collecting data corresponding to the APP according to the updated APK, and the data collection includes data packet capture and page screenshots; text recognition and extraction based on the OCR algorithm; for the recognized text content, a sample set is constructed through keywords and regular expressions, and manually labeled; the manually labeled sample set is input into the pre-trained The deep learning model adjusts the model, and realizes the violation judgment of texts in different scenarios by dividing the business scenarios; according to the judgment results output by the deep learning model, the scores of different APPs are counted to obtain the APP violation scores. The present invention effectively and quickly detects the illegal use of the APP by collecting and analyzing the data of the APP.

Of course, those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware (such as processors, controllers, etc.) through a computer program, and the programs can be stored in a In a computer-readable storage medium, when the program is executed, it may include the processes of the foregoing method embodiments. The storage medium may be a memory, a magnetic disk, an optical disk, or the like.

It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above descriptions, and all these improvements and transformations should belong to the protection scope of the appended claims of the present invention.

Claims (10)

An APP violation monitoring method based on OCR and transfer learning, characterized in that, the APP violation monitoring method based on OCR and transfer learning includes: Regularly update the APK, and perform data collection of the corresponding APP according to the updated APK, and the data collection includes data capture and page screenshots; Text recognition and extraction of screenshots based on OCR algorithm; For the recognized text content, a sample set is constructed through keywords and regular expressions, and manual annotation is performed; Input the manually labeled sample set into the pre-trained deep learning model for model adjustment, and realize the text violation discrimination in different scenarios by dividing the business scenarios; According to the discrimination results output by the deep learning model, the scores of different APPs are counted to obtain the violation scores of the APPs. The method for monitoring APP violations based on OCR and transfer learning according to claim 1, characterized in that, in the regularly updating APK, data collection of the corresponding APP is performed according to the updated APK, specifically comprising: Based on Java and with the help of Jsoup library, the APK of each application is crawled, and the APK of the application store is updated regularly, and the data collection of the corresponding APP is carried out according to the updated APK. The method for monitoring APP violations based on OCR and transfer learning according to claim 1, wherein the data collection method specifically includes: using a crawler to directly capture packets of publicity data and using an Appium script to automatically take screenshots of pages. The method for monitoring APP violations based on OCR and transfer learning according to claim 1, wherein the manually labeled sample set is input into a pre-trained deep learning model for model adjustment, and the business scenarios are divided into different scenarios. Violation judgment of text, which previously included: A corpus is constructed for supervising the training of the deep learning model. The APP violation monitoring method based on OCR and transfer learning according to claim 4, wherein the construction process of the corpus comprises: Obtain multiple keywords, and match the keywords; A keyword-based training corpus is constructed and labeled manually for generating the corpus. The APP violation monitoring method based on OCR and transfer learning according to claim 1, is characterized in that, the described score of different APP is counted, the violation score of APP is obtained, specifically comprises: The violation score of the APP is obtained by weighted average:

in,

Indicates the weighted average number, f1~fk are the weights configured for the violation items of each dimension, x1~xk are the actual number of abnormal quality inspection results of the violation items in each dimension, n is the total number of dimensions, and different dimensions represent different violation scenarios. The method for monitoring APP violations based on OCR and transfer learning according to claim 1, wherein, according to the discrimination result output by the deep learning model, the scores of different APPs are counted to obtain the violation scores of the APPs, After that it also includes: Set a scheduled start task for all tasks. The method for monitoring APP violations based on OCR and transfer learning according to claim 7, wherein the tasks include: APP crawling timing tasks, APP screenshot timing tasks, and violation monitoring timing tasks. An intelligent terminal, characterized in that the intelligent terminal comprises: a memory, a processor, and an APP violation monitoring program based on OCR and transfer learning that is stored on the memory and can run on the processor, and the The OCR and transfer learning-based APP violation monitoring program is executed by the processor to implement the steps of the OCR and transfer learning-based APP violation monitoring method according to any one of claims 1-8. A storage medium, characterized in that the storage medium stores an APP violation monitoring program based on OCR and transfer learning, and the APP violation monitoring program based on OCR and transfer learning is executed by a processor. The steps of any one of the OCR and transfer learning-based APP violation monitoring methods.

Images