CN116758525A - Medicine box real-time identification method and system based on deep learning - Google Patents

Abstract

The invention discloses a real-time identification method and system for medicine boxes based on deep learning, which belongs to the technical field of medicine management. It includes setting up a camera above the medicine box to collect the appearance image information of the medicine box; using pre-trained medicine boxes based on yolov5 Box recognition model to obtain pill box edge and appearance-based pill box recognition results; perform text detection through the pre-trained DBNet-based text detection model to obtain the area of the pill box text; use pre-trained text based on DenseNet and CTC The recognition model performs text recognition and identifies the text information of the pill box; each word or phrase of the text recognition result is sequentially searched and matched with a low threshold value in the drug database. The invention can effectively solve the problem that pharmacies require manual verification when taking medicine, and this method greatly ensures the accuracy of identification and reduces the workload of pharmacy workers.

Description

A method and system for real-time identification of pill boxes based on deep learning

Technical field

The present invention relates to the technical field of drug management, specifically a real-time identification method and system for pill boxes based on deep learning.

Background technique

In recent years, with the rapid economic growth, the people's demand for medical and health care standards and services has increased rapidly, and various automated and intelligent medical equipment have developed rapidly. While the conditions of medical equipment and the quality of doctors have been greatly improved, the types and quantities of medicines have also grown rapidly. The traditional pharmacy service model mainly relies on manpower to deposit and withdraw medicines. In this model, the tasks of preparing and dispensing medicines are completed by professional pharmacists. Patients receive prescriptions through the window, while pharmacists are busy dispensing medicines and are in a passive position of dispensing medicines. This model mainly has two disadvantages: First, for pharmacists, due to the wide variety of drugs, strict confirmation is required for dispensing drugs. The workload of pharmacists is huge and the technical content cannot be improved, which dilutes the technical content of pharmacists' work. It wastes precious human resources and medical resources; at the same time, long-term repetitive work also poses safety risks of getting the wrong medicines; on the other hand, this service model hinders the communication between doctors and patients, because pharmacists are busy dispensing medicines. , unable to provide patients with more medication consultation services; moreover, patients will have to wait in line when taking medicine due to the low efficiency of manual medicine taking, which affects the patient's medical experience. Therefore, realizing pharmacy automation and informatization is a reform trend for outpatient pharmacies and is also a guarantee for further improving medical quality.

With the development of computer technology, deep learning technology has also been widely used. Image recognition technology can be used to accurately detect and identify pill boxes, thus enabling automated management of medicines. However, although the current recognition technology has been developed, due to the small difference in shape and color of the pill boxes and the wide variety, the current pill box image recognition algorithm has poor performance and is far from accurate enough for actual pharmacy use. Therefore, improving the performance of the pill box recognition algorithm is the key to the entire smart pharmacy verification process. The quality of the recognition effect will directly affect the patient's medication safety.

Contents of the invention

The technical task of the present invention is to address the above shortcomings and provide a real-time identification method and system for medicine boxes based on deep learning, which can accurately identify the type and quantity of medicines, increase the speed of drug dispensing in the pharmacy, and reduce labor costs.

The technical solutions adopted by the present invention to solve the technical problems are:

A method for real-time identification of pill boxes based on deep learning, including the following steps:

Set up a camera above the pill box to collect appearance image information of the pill box;

Use the pre-trained pill box recognition model based on yolov5 to detect and identify pill boxes, and obtain pill box edge and appearance-based pill box recognition results;

The original image is cropped according to the edge of the obtained pill box, and multiple images containing only a single pill box are obtained. For each cropped pill box image, text detection is performed through a pre-trained DBNet-based text detection model. Get the area of the pill box text;

For the recognized text area, use the pre-trained text recognition model based on DenseNet and CTC to perform text recognition and identify the text information of the pill box;

Each word or phrase of the text recognition result is sequentially subjected to low-threshold fuzzy search and matching in the drug database. The content of fuzzy search and matching is the drug name, brand, manufacturer, and dosage attributes of the drug, and one or more drugs are obtained. serial number;

Search the unique drug number based on appearance recognition among the drug numbers queried based on text recognition; if the search is successful, output the recognition result;

If the search fails, each word or phrase of the text recognition result will be sequentially searched and matched intelligently in the drug database, and finally the pill box recognition result will be obtained.

Further, the step of setting up a camera above the pill box to collect appearance image information of the pill box includes:

Before using the neural network model for pill box appearance recognition and text recognition, all models need to be trained, including the pill box appearance image recognition model based on yolov5, the text detection model based on DBnet, and the text recognition model based on DenseNet+CTC.

Set up a camera above the pill box to collect the appearance image information of the pill box. The steps include:

The medicine box is placed in the medicine box information collection box, and a camera is set up above to take a top-down picture of the medicine box. The inner wall of the medicine box information collection box is made of non-reflective black material and provides a stable light source;

Use the pre-trained pill box recognition model based on yolov5 to detect and identify the pill box, and obtain the edge of the pill box and the unique number of the medicine based on the appearance of the pill box;

Crop the original image according to the edge of the obtained pill box, and obtain multiple pictures in which each picture only contains a single pill box;

For each cropped pill box picture, text detection is performed through the pre-trained DBNet-based text detection model to obtain the area of the pill box text;

For the recognized text area, the pre-trained text recognition model based on DenseNet and CTC is used for text recognition to identify the text information of the pill box.

Further, the training of the pill box appearance image recognition model based on yolov5 includes the following steps:

The pill box is placed in the pill box information collection box, and a movable camera is set up above. The inner wall of the pill box information collection box is made of non-reflective black material and provides a stable light source;

The camera takes pictures of all the medicines to be collected from a bird's-eye view to obtain sufficient appearance pictures of the medicine boxes. To ensure the quality of training, the effective pictures collected for each medicine box should not be less than 20;

Use the labeling tool to label the obtained pill box images by region and category;

Load yolov5s pre-trained model;

During the training process, the appearance data of the pill box is data enhanced through mosaic data enhancement and other methods. After appropriate training rounds, the neural network model based on yolov5 reaches convergence.

Further, the steps for training the text detection model based on DBnet and the text recognition model based on DenseNet+CTC include:

Using a public data set, the data set contains a total of about 3.64 million images, divided into a training set and a verification set according to 99:1. The data uses the Chinese corpus (news + classical Chinese), through font, size, grayscale, blur, perspective, Stretching and other changes are randomly generated, including Chinese characters, English letters, numbers and punctuation, a total of 5990 characters. Each sample has a fixed 10 characters. The characters are randomly intercepted from sentences in the corpus, and the image resolution is unified to 280×3;

Use text synthesis programs to synthesize text images of commonly used drugs to increase the amount of training data;

Load pre-trained models trained on other large text recognition datasets;

Use the data set to train the text detection model of DBnet and the text recognition model based on DenseNet+CTC until the model converges.

Further, the pre-trained yolov5-based pill box recognition model is used to detect and identify pill boxes, and the yolov5 neural network model in the pill box edge and appearance-based pill box identification results is constructed using yolov5s, which is composed of the input end ( It consists of five parts: Input, Backbone, Neck, Head and Output;

The image size input at the input end is 640×640×3, and strategies such as Mosaic data enhancement, Anchor adaptive anchor frame calculation and image scaling are used to preprocess the image;

In yolov5, CSPDarknet53 is used as the backbone network of the model, including Focus module, Conv module, C3 module and SPP module. Its function is to extract rich semantic features from the input image; FPN and PAN are used in the neck to generate feature pyramids for enhancement. Detection of multi-scale targets; the head predicts the features passed from the neck and generates feature maps of three different scales;

The structure of the Conv module is Conv2d+BN+SiLU, followed by the convolution layer, normalization operation and activation function; the purpose of the Focus module is to reduce the calculation amount of the model and speed up the training speed of the network; first, the input size is 3×640 The image of After a convolution operation, the final feature map is 32×320×320;

The C3 module consists of two branches. In the first branch, the input feature map passes through 3 consecutive Conv modules and multiple stacked Bottleneck modules; in the second branch, the feature map only passes through one Conv module, and finally The two branches are spliced together by channel; among them, the Bottleneck module can better extract high-level features of the target, and mainly consists of two consecutive convolution operations and a residual operation;

The SPP module is a spatial pyramid pooling module, used to expand the receptive field of the network; in yolov5s, the input feature map size of the SPP module is 512×20×20, and the number of channels is halved after passing a Conv module; then the volume is used on the feature map The accumulation kernels are maximum pooling operations of 5×5, 9×9, and 13×13 respectively. The three feature maps and the input feature map are spliced by channel and then passed through a Conv module. The final output feature map size is 512× 20×20.

Furthermore, the text detection model of DBNet in text detection is performed through the pre-trained DBNet-based text detection model, which consists of the DBNet text detection network. DBNet mainly uses differentiable binarization as a text detector based on a simple segmentation network. The differentiable binarization formula is shown in Equation 1:

(1)

Among them, B represents the approximate binary image, T is the threshold feature map of network learning, and k represents the magnification factor; here, the empirical value of 50 is taken, so that the foreground and background can be better distinguished.

Furthermore, the text recognition model of DenseNet and CTC that uses pre-trained text recognition models based on DenseNet and CTC for text recognition consists of a DenseNet neural network composed of a CNN layer, an RNN layer, and a CTC transcription layer;

The functions of each layer of the text recognition model from bottom to top are as follows: the convolutional layer (CNN) to extract the features of the input image; the recurrent layer (RNN) to predict the distribution of the input feature sequence of the CNN layer; the transcription layer (CTC) to filter the sequence tags redundant data;

The DenseNet network uses Relu as the activation function and uses 3 Dense Block layers for calculation. The DenseNet network composed of each Dense Block connected together through the Transition structure is trained with CTC_loss to obtain the final data model;

The DenseNet network uses skip splicing to retain the original features and reduce the occurrence of gradient disappearance. However, as the network depth continues to deepen, the number of channels and parameters increase, making it difficult for the model to extract deep features, so DenseNet sets up a Transition conversion module ;After this module is used in Dense Block, it is mainly used to reduce the number of channels; at the same time, in order to reduce the number of channels, a bottleneck structure is added before each DenseBlock splicing to reduce the number of channels; parameters are reduced through pooling and then transmitted to the lower layer. Dense Block structure to achieve higher accuracy.

Further, each word or phrase of the text recognition result is sequentially subjected to low-threshold fuzzy search and matching in the drug database. The content of fuzzy search and matching is the name, manufacturer, dosage and other attributes of the drug, and one or more The steps for drug numbering are:

Considering that pill box text recognition may not recognize all the text, the matching threshold of fuzzy matching is low. The low threshold is 0.5, that is, if it contains half of the same characters, it is considered a successful match; text recognition can obtain multiple words or sentences. Perform low-threshold fuzzy search and matching in the drug database. The content of fuzzy search and matching is first the name of the drug. If the drug name is matched, the manufacturer, dosage and other attributes of the matched drugs will continue to be matched. If one of the If the attribute does not match any drug, it is considered a match failure; a match failure will not narrow the matching scope, and will directly skip the matching of the attribute; finally, one or more drug numbers can be obtained through fuzzy search and matching.

Further, the unique drug number based on appearance recognition is searched among the drug numbers queried based on text recognition. If the search is successful, the pill box recognition result is directly output; if the search fails, each word or phrase of the text recognition result is output. Intelligent matching is performed in the drug database in turn, and the pill box identification result is finally obtained:

Considering that no medicine can be found in the results obtained by text recognition due to errors in appearance-based recognition results; at this time, text recognition is completely relied on to confirm the information of the pill box; first, each word or phrase of the text recognition result is The intelligent matching algorithm is used to match the name attributes of the drugs in the drug database in turn; after obtaining the matching drugs, the intelligent matching algorithm is used to match the brand, manufacturer, dosage and other attributes of these drugs, and finally a drug number is obtained;

The intelligent matching algorithm is as follows:

The first thing to match is the drug name attribute. The initial matching threshold is 1, that is, if they are identical, the match is considered successful. If there is no match for any drug, it will be deemed that the matching failed. If the match fails, the matching threshold is lowered by 0.1 until the match is successful; after the first successful match with a threshold value other than 1, the matching threshold is increased by 0.01 until the match fails; after the match fails, the result of the successful match before the failed match is regarded as final matching result.

The present invention also claims a real-time identification system for pill boxes based on deep learning, including an image acquisition device and a data processing device,

The image acquisition device is used to collect appearance image information of the medicine box;

The data processing device is used to realize real-time recognition of the collected pill box appearance image information based on yolov5 and text recognition. The data processing device includes a pill box recognition module based on yolov5, a text detection module based on DBNet, and text based on DenseNet and CTC. Recognition module and pill box recognition module based on text recognition and appearance recognition;

This system can implement the above-mentioned real-time identification method of pill boxes based on deep learning.

The beneficial effects achieved by the present invention are:

The invention discloses a real-time identification method of medicine boxes based on deep learning. Obtain the appearance picture of the pill box through the camera; use the pre-trained pill box recognition model based on yolov5 to detect and identify the pill box, and obtain the pill box edge and appearance-based pill box recognition results; compare the original image according to the obtained pill box edge Crop to obtain multiple pictures each containing only a single pill box; for each clipped pill box picture, perform text detection through a pre-trained DBNet-based text detection model to obtain the area of the pill box text; for recognition In the text area found, the pre-trained text recognition model based on DenseNet and CTC is used for text recognition to identify the text information of the pill box; each word or phrase of the text recognition result is sequentially subjected to low-threshold fuzzy in the drug database. Search and match. The content of fuzzy search and matching is the drug name, brand, manufacturer, and dosage attributes of the drug, and one or more drug numbers are obtained; the unique number of the drug based on appearance recognition is added to the drug number queried based on text recognition. Search in; if the search is successful, the recognition result is output; if the search fails, each word or phrase of the text recognition result is sequentially searched and matched in the drug database through intelligent fuzzy search, and finally the pill box recognition result is obtained. The invention comprehensively applies image recognition and text recognition technology to greatly ensure the accuracy of recognition and ensure the safety of medication for patients. The present invention can effectively solve the problem that pharmacies require manual verification when taking medicine, and this method greatly ensures the accuracy of identification, reduces the workload of pharmacy workers, and at the same time ensures the safety of patients' medication.

Description of the drawings

Figure 1 is a schematic flowchart of a method for real-time identification of pill boxes based on deep learning provided by an embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and specific examples, so that those skilled in the art can better understand the present invention and implement it. However, the illustrated embodiments are not intended to limit the present invention. In the absence of conflict, Below, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

As shown in Figure 1, the example of the present invention proposes a real-time drug identification method based on deep learning, which includes the following steps:

S100. Set up a camera above the pill box to collect appearance image information of the pill box;

The pill box is placed in the pill box information collection box, and a camera is set up above. In order to reduce the interference of the platform's reflection on the recognition results, the inner wall of the pill box information collection box is made of non-reflective black material, and a stable industrial light source is provided inside.

S200: Use the pre-trained pill box recognition model based on yolov5 to detect and identify the pill box, and obtain the edge of the pill box and the unique number of the medicine based on the appearance of the pill box.

The yolov5 neural network model is constructed using yolov5s and consists of five parts: input, backbone, neck, head and output:

The image size input at the input end is 640×640×3, and strategies such as Mosaic data enhancement, Anchor adaptive anchor frame calculation and image scaling are used to preprocess the image;

CSPDarknet53 is used as the backbone network of the model in yolov5, including Focus module, Conv module, C3 module and SPP module, and FPN and PAN are used in the neck to generate feature pyramid;

The structure of the Conv module is Conv2d+BN+SiLU, followed by the convolution layer, normalization operation and activation function; the Focus module first divides the image with an input size of 3×640×640 into 4 slices, where each slice The size is 3×320×320; then use the splicing operation to splice the 4 slices through the channel dimension, and the resulting feature map scale is 12×320×320; after another convolution operation, the final feature is 32×320×320 picture;

The C3 module consists of two branches. In the first branch, the input feature map passes through 3 consecutive Conv modules and multiple stacked Bottleneck modules; in the second branch, the feature map only passes through one Conv module, and finally The two branches are spliced together by channel; among them, the Bottleneck module mainly consists of two consecutive convolution operations and a residual operation;

The SPP module is a spatial pyramid pooling module; the input feature map size of the SPP module is 512×20×20, and the number of channels is halved after passing through a Conv module; then the convolution kernels used for the feature maps are 5×5 and 9×9 respectively. , 13×13 maximum pooling operation, and the three feature maps and the input feature map are spliced by channel and then passed through a Conv module. The final output feature map size is 512×20×20.

The steps to train the pill box appearance image recognition model based on yolov5 are as follows:

The pill box is placed in the pill box information collection box, and a movable camera is set up above. The inner wall of the pill box information collection box is made of non-reflective black material and provides a stable light source;

Based on the consideration of imaging quality, the camera chooses a camera with better product performance. The camera takes pictures of all the drugs to be collected from a bird's-eye view to obtain sufficient appearance pictures of the pill boxes. To ensure the quality of training, the effective pictures collected for each pill box should not be less than 20; to ensure the effectiveness of pill box identification , the angle of the collected pill box picture should meet the angle that appeared when the photo was collected in step 100.

Use the labeling tool to label the regions and categories of the obtained pill box pictures; the labeling should be close to the edge of the pill box.

Load yolov5s pre-trained model;

During the training process, the appearance data of the pill box is data enhanced through mosaic data enhancement and other methods. After appropriate training rounds, until the neural network model based on yolov5 reaches convergence;

After completing the model training, in this step, use the trained yolov5-based pill box recognition model to detect and identify the collected pill box photos, and obtain the edge of the pill box and the unique drug number of the pill box based on appearance.

S300. Crop the original image according to the edge of the obtained pill box, and obtain multiple pictures in which each picture only contains a single pill box.

According to the edge of the pill box detected by yolov5, the original picture is cropped to obtain multiple pill box pictures, and each picture only contains a single pill box.

S400. For each cropped pill box picture, perform text detection through the pre-trained DBNet-based text detection model to obtain the area of the pill box text.

DBNet's text detection model consists of DBNet text detection network, which mainly uses differentiable binarization as a text detector based on a simple segmentation network. The differentiable binarization formula is as follows:

(1)

Among them, B represents the approximate binary image, T is the threshold feature map learned by the network, and k represents the magnification factor.

S500. For the recognized text area, use the pre-trained text recognition model based on DenseNet and CTC to perform text recognition and identify the text information of the pill box.

The text recognition models of DenseNet and CTC are composed of DenseNet neural network, which consists of CNN layer, RNN layer and CTC transcription layer;

The functions of each layer of the text recognition model from bottom to top are as follows: the convolutional layer (CNN) to extract the features of the input image; the recurrent layer (RNN) to predict the distribution of the input feature sequence of the CNN layer; the transcription layer (CTC) to filter the sequence tags redundant data;

The DenseNet network uses Relu as the activation function and uses 3 Dense Block layers for calculation. The DenseNet network composed of each Dense Block connected together through the Transition structure is trained with CTC_loss to obtain the final data model;

The DenseNet network uses skip splicing to retain the original features and reduce the occurrence of gradient disappearance. However, as the network depth continues to deepen, the number of channels and parameters increase, making it difficult for the model to extract deep features, so DenseNet sets up a Transition conversion module ;After this module is used in Dense Block, it is mainly used to reduce the number of channels; at the same time, in order to reduce the number of channels, a bottleneck structure is added before each DenseBlock splicing to reduce the number of channels; parameters are reduced through pooling and then transmitted to the lower layer. Dense Block structure to achieve higher accuracy.

The steps for training a text detection model based on DBnet and a text recognition model based on DenseNet+CTC include:

Using a public data set, the data set contains a total of about 3.64 million images, divided into a training set and a verification set according to 99:1. The data uses the Chinese corpus (news + classical Chinese), through font, size, grayscale, blur, perspective, Stretching and other changes are randomly generated, including Chinese characters, English letters, numbers and punctuation, a total of 5990 characters. Each sample has a fixed 10 characters. The characters are randomly intercepted from sentences in the corpus, and the image resolution is unified to 280×3;

Use text synthesis programs to synthesize text images of commonly used drugs to increase the amount of training data;

Load pre-trained models trained on other large text recognition datasets;

Use the data set to train the text detection model of DBnet and the text recognition model based on DenseNet+CTC until the model converges.

S600. Perform low-threshold fuzzy search and matching on each word or phrase of the text recognition result in the drug database in sequence. The contents of the fuzzy search and matching are the drug name, brand, manufacturer, and dosage attributes of the drug. One or more Drug number.

Considering that pill box text recognition may not recognize all the text, the matching threshold of fuzzy matching is low. The low threshold is 0.5, that is, if it contains half of the same characters, it is considered a successful match; text recognition can obtain multiple words or sentences. Perform low-threshold fuzzy search and matching in the drug database. The content of fuzzy search and matching is first the name of the drug. If the drug name is matched, the manufacturer, dosage and other attributes of the matched drugs will continue to be matched. If one of the If the attribute does not match any drug, it is considered a match failure; a match failure will not narrow the matching scope, and will directly skip the matching of the attribute; finally, one or more drug numbers can be obtained through fuzzy search and matching.

S700. Search the unique drug number based on appearance recognition in the drug number queried based on text recognition:

The unique drug number based on appearance recognition is searched among the drug numbers queried based on text recognition. If the search is successful, the pill box identification result is directly output.

S800. Search the unique drug number based on appearance recognition in the drug number queried based on text recognition. If the search fails, intelligently match each word or phrase of the text recognition result in the drug database in turn, and finally obtain the medicine box. Recognition results:

Considering that no medicine can be found in the results obtained by text recognition due to errors in appearance-based recognition results; at this time, text recognition is completely relied on to confirm the information of the pill box; first, each word or phrase of the text recognition result is The intelligent matching algorithm is used to match the name attributes of the drugs in the drug database in turn; after obtaining the matching drugs, the intelligent matching algorithm is used to match the brand, manufacturer, dosage and other attributes of these drugs, and finally a drug number is obtained;

The intelligent matching algorithm is as follows:

The first thing to match is the drug name attribute. The initial matching threshold is 1, that is, if they are identical, the match is considered successful. If there is no match for any drug, it will be deemed that the matching failed. If the match fails, the matching threshold is lowered by 0.1 until the match is successful; after the first successful match with a threshold value other than 1, the matching threshold is increased by 0.01 until the match fails; after the match fails, the result of the successful match before the failed match is regarded as final matching result.

Embodiments of the present invention also provide a real-time pill box recognition system based on deep learning, including an image acquisition device and a data processing device. The system can implement the deep learning-based real-time pill box identification method described in the above embodiment.

The image acquisition device is used to collect appearance image information of the medicine box.

The data processing device is used to realize real-time recognition of the collected pill box appearance image information based on yolov5 and text recognition. The data processing device includes a pill box recognition module based on yolov5, a text detection module based on DBNet, and text based on DenseNet and CTC. Recognition module and pill box recognition module based on text recognition and appearance recognition.

The steps of setting up a camera above the pill box to collect the appearance image information of the pill box include:

Before using the neural network model for pill box appearance recognition and text recognition, all models need to be trained, including the pill box appearance image recognition model based on yolov5, the text detection model based on DBnet, and the text recognition model based on DenseNet+CTC.

Set up a camera above the pill box to collect the appearance image information of the pill box. The steps include:

The medicine box is placed in the medicine box information collection box, and a camera is set up above to take a bird's-eye view of the medicine box. The inner wall of the medicine box information collection box is made of non-reflective black material and provides a stable light source.

Use the pre-trained pill box recognition model based on yolov5 to detect and identify the pill box, and obtain the edge of the pill box and the unique number of the medicine based on the appearance of the pill box;

Crop the original image according to the edge of the obtained pill box, and obtain multiple pictures in which each picture only contains a single pill box;

For each cropped pill box picture, text detection is performed through the pre-trained DBNet-based text detection model to obtain the area of the pill box text;

For the recognized text area, use the pre-trained text recognition model based on DenseNet and CTC to perform text recognition and identify the text information of the pill box;

Each word or phrase of the text recognition result is sequentially subjected to low-threshold fuzzy search and matching in the drug database. The content of fuzzy search and matching is the drug name, brand, manufacturer, and dosage attributes of the drug, and one or more drugs are obtained. serial number;

Search the unique drug number based on appearance recognition among the drug numbers queried based on text recognition; if the search is successful, output the recognition result;

If the search fails, each word or phrase of the text recognition result will be sequentially searched and matched intelligently in the drug database, and finally the pill box recognition result will be obtained.

The training of the pill box appearance image recognition model based on yolov5 includes the following steps:

The pill box is placed in the pill box information collection box, and a movable camera is set up above. The inner wall of the pill box information collection box is made of non-reflective black material and provides a stable light source;

The camera takes pictures of all the medicines to be collected from a bird's-eye view to obtain sufficient appearance pictures of the medicine boxes. To ensure the quality of training, the effective pictures collected for each medicine box should not be less than 20;

Use the labeling tool to label the obtained pill box images by region and category;

Load yolov5s pre-trained model;

During the training process, the appearance data of the pill box is data enhanced through mosaic data enhancement and other methods. After appropriate training rounds, the neural network model based on yolov5 reaches convergence.

The steps for training a text detection model based on DBnet and a text recognition model based on DenseNet+CTC include:

Use a public data set, which contains a total of about 3.64 million images, divided into a training set and a verification set according to 99:1. The data uses the Chinese corpus (news + classical Chinese), through font, size, grayscale, blur, perspective, Stretching and other changes are randomly generated, including Chinese characters, English letters, numbers and punctuation, a total of 5990 characters. Each sample has a fixed 10 characters. The characters are randomly intercepted from sentences in the corpus, and the image resolution is unified to 280×3;

Use text synthesis programs to synthesize text images of commonly used drugs to increase the amount of training data;

Load pre-trained models trained on other large text recognition datasets;

Use the data set to train the text detection model of DBnet and the text recognition model based on DenseNet+CTC until the model converges.

Use the pre-trained pill box recognition model based on yolov5 to detect and identify pill boxes, and obtain the edge of the pill box and the appearance-based pill box identification results. The yolov5 neural network model in the results uses yolov5s for model construction, which consists of the input terminal (Input), It consists of five parts: Backbone, Neck, Head and Output;

The image size input at the input end is 640×640×3, and strategies such as Mosaic data enhancement, Anchor adaptive anchor frame calculation and image scaling are used to preprocess the image;

In yolov5, CSPDarknet53 is used as the backbone network of the model, including Focus module, Conv module, C3 module and SPP module. Its function is to extract rich semantic features from the input image; FPN and PAN are used in the neck to generate feature pyramids for enhancement. Detection of multi-scale targets; the head predicts the features passed from the neck and generates feature maps of three different scales;

The structure of the Conv module is Conv2d+BN+SiLU, followed by the convolution layer, normalization operation and activation function; the purpose of the Focus module is to reduce the calculation amount of the model and speed up the training speed of the network; first, the input size is 3×640 The image of After a convolution operation, the final feature map is 32×320×320;

The C3 module consists of two branches. In the first branch, the input feature map passes through 3 consecutive Conv modules and multiple stacked Bottleneck modules; in the second branch, the feature map only passes through one Conv module, and finally The two branches are spliced together by channel; among them, the Bottleneck module can better extract high-level features of the target, and mainly consists of two consecutive convolution operations and a residual operation;

The SPP module is a spatial pyramid pooling module, used to expand the receptive field of the network; in yolov5s, the input feature map size of the SPP module is 512×20×20, and the number of channels is halved after passing a Conv module; then the volume is used on the feature map The accumulation kernels are maximum pooling operations of 5×5, 9×9, and 13×13 respectively. The three feature maps and the input feature map are spliced by channel and then passed through a Conv module. The final output feature map size is 512× 20×20.

The DBNet text detection model in text detection is performed through the pre-trained DBNet-based text detection model. The DBNet text detection model consists of the DBNet text detection network. DBNet mainly uses differentiable binarization as a text detector based on a simple segmentation network. The differentiable binarization formula is shown in Equation 1:

(1)

Among them, B represents the approximate binary image, T is the threshold feature map of network learning, and k represents the magnification factor; here, the empirical value of 50 is taken, so that the foreground and background can be better distinguished.

The text recognition model of DenseNet and CTC that uses pre-trained text recognition models based on DenseNet and CTC for text recognition consists of a DenseNet neural network composed of a CNN layer, an RNN layer, and a CTC transcription layer;

The functions of each layer of the text recognition model from bottom to top are as follows: the convolutional layer (CNN) to extract the features of the input image; the recurrent layer (RNN) to predict the distribution of the input feature sequence of the CNN layer; the transcription layer (CTC) to filter the sequence tags redundant data;

The DenseNet network uses Relu as the activation function and uses 3 Dense Block layers for calculation. The DenseNet network composed of each Dense Block connected together through the Transition structure is trained with CTC_loss to obtain the final data model;

The DenseNet network uses skip splicing to retain the original features and reduce the occurrence of gradient disappearance. However, as the network depth continues to deepen, the number of channels and parameters increase, making it difficult for the model to extract deep features, so DenseNet sets up a Transition conversion module ;After this module is used in Dense Block, it is mainly used to reduce the number of channels; at the same time, in order to reduce the number of channels, a bottleneck structure is added before each DenseBlock splicing to reduce the number of channels; parameters are reduced through pooling and then transmitted to the lower layer. Dense Block structure to achieve higher accuracy.

Each word or phrase of the text recognition result is sequentially subjected to low-threshold fuzzy search and matching in the drug database. The content of fuzzy search and matching is the name, manufacturer, dosage and other attributes of the drug, and one or more drug numbers are obtained. The steps are:

Considering that pill box text recognition may not recognize all the text, the matching threshold of fuzzy matching is low. The low threshold is 0.5, that is, if it contains half of the same characters, it is considered a successful match; text recognition can obtain multiple words or sentences. Perform low-threshold fuzzy search and matching in the drug database. The content of fuzzy search and matching is first the name of the drug. If the drug name is matched, the manufacturer, dosage and other attributes of the matched drugs will continue to be matched. If one of the If the attribute does not match any drug, it is considered a match failure; a match failure will not narrow the matching scope, and will directly skip the matching of the attribute; finally, one or more drug numbers can be obtained through fuzzy search and matching.

Search the unique number of the drug based on appearance recognition in the drug number queried based on text recognition. If the search is successful, the pill box recognition result will be output directly; if the search fails, each word or phrase of the text recognition result will be added to the drug in turn. Intelligent matching is performed in the database, and the pill box identification result is finally obtained:

Considering that no medicine can be found in the results obtained by text recognition due to errors in appearance-based recognition results; at this time, text recognition is completely relied on to confirm the information of the pill box; first, each word or phrase of the text recognition result is The intelligent matching algorithm is used to match the name attributes of the drugs in the drug database in turn; after obtaining the matching drugs, the intelligent matching algorithm is used to match the brand, manufacturer, dosage and other attributes of these drugs, and finally a drug number is obtained;

The intelligent matching algorithm is as follows:

The first thing to match is the drug name attribute. The initial matching threshold is 1, that is, if they are identical, the match is considered successful. If there is no match for any drug, it will be deemed that the matching failed. If the match fails, the matching threshold is lowered by 0.1 until the match is successful; after the first successful match with a threshold value other than 1, the matching threshold is increased by 0.01 until the match fails; after the match fails, the result of the successful match before the failed match is regarded as final matching result.

Although preferred examples of the invention have been described, those skilled in the art will be able to make additional changes and modifications to these examples once the basic inventive concepts are apparent. Therefore, it is intended that the appended claims be construed to include the preferred examples and all changes and modifications that fall within the scope of the invention. Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the invention. In this way, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims (10)

1. A real-time identification method for pill boxes based on deep learning, which is characterized by including the following steps: Set up a camera above the pill box to collect appearance image information of the pill box; Use the pre-trained pill box recognition model based on yolov5 to detect and identify pill boxes, and obtain pill box edge and appearance-based pill box recognition results; Crop the original image according to the edge of the obtained pill box to obtain multiple pictures each containing only a single pill box; for each cropped pill box picture, perform text detection through a pre-trained DBNet-based text detection model. Get the area of the pill box text; For the recognized text area, use the pre-trained text recognition model based on DenseNet and CTC to perform text recognition and identify the text information of the pill box; Each word or phrase of the text recognition result is sequentially subjected to low-threshold fuzzy search and matching in the drug database. The content of fuzzy search and matching includes the name, brand, manufacturer, and dosage attributes of the drug, and one or more drugs are obtained. serial number; Search the unique drug number based on appearance recognition among the drug numbers queried based on text recognition; if the search is successful, output the recognition result; If the search fails, each word or phrase of the text recognition result will be sequentially searched and matched intelligently in the drug database, and finally the pill box recognition result will be obtained. 2. A method for real-time recognition of pill boxes based on deep learning as claimed in claim 1, characterized in that the step of setting up a camera above the pill box and collecting appearance image information of the pill box includes: Before using the neural network model for pill box appearance recognition and text recognition, all models are trained, including the pill box appearance image recognition model based on yolov5, the text detection model based on DBnet, and the text recognition model based on DenseNet+CTC. 3. A method for real-time recognition of pill boxes based on deep learning as claimed in claim 2, characterized in that the training of the yolov5-based pill box appearance image recognition model includes the following steps: The pill box is placed in the pill box information collection box, and a movable camera is set up above. The inner wall of the pill box information collection box is made of non-reflective black material and provides a stable light source; The camera takes pictures of all the medicines to be collected from a bird's-eye view to obtain the required number of pictures of the appearance of the medicine boxes. To ensure the quality of training, the effective pictures collected for each medicine box should not be less than 20; Use the labeling tool to label the obtained pill box images by region and category; Load yolov5s pre-trained model; During the training process, data enhancement is performed on the appearance data of the pill box, and after appropriate training rounds, the neural network model based on yolov5 reaches convergence. 4. A method for real-time recognition of medicine boxes based on deep learning as claimed in claim 2, characterized in that the steps of training a text detection model based on DBnet and a text recognition model based on DenseNet+CTC include: Using a public data set, it is divided into a training set and a verification set according to 99:1. The data uses the Chinese corpus to randomly generate a total of 5990 characters including Chinese characters, English letters, numbers and punctuation through parameter changes. The parameters include font, size, Grayscale, blur, perspective, stretching; each sample has 10 fixed characters, characters are randomly intercepted from sentences in the Chinese corpus, and the image resolution is unified to 280×3; Use text synthesis programs to synthesize text images of commonly used drugs to increase the amount of training data; Load pre-trained models trained on other large text recognition datasets; Use the data set to train the text detection model of DBnet and the text recognition model based on DenseNet+CTC until the model converges. 5. A method for real-time identification of pill boxes based on deep learning as claimed in claim 1, characterized in that the pre-trained pill box identification model based on yolov5 is used to detect and identify pill boxes and obtain pill box edges. And the yolov5 neural network model in the appearance-based pill box recognition results is built using yolov5s, which consists of five parts: input end, backbone network, neck and head, and output end: The image size input at the input end is 640×640×3, and strategies such as Mosaic data enhancement, Anchor adaptive anchor frame calculation and image scaling are used to preprocess the image; CSPDarknet53 is used as the backbone network of the model in yolov5, including Focus module, Conv module, C3 module and SPP module; FPN and PAN are used in the neck to generate feature pyramid; The structure of the Conv module is Conv2d+BN+SiLU, followed by the convolution layer, normalization operation and activation function; the Focus module first divides the image with an input size of 3×640×640 into 4 slices, where each slice The size is 3×320×320; then use the splicing operation to splice the 4 slices through the channel dimension, and the resulting feature map scale is 12×320×320; after another convolution operation, the final feature is 32×320×320 picture; The C3 module consists of two branches. In the first branch, the input feature map passes through 3 consecutive Conv modules and multiple stacked Bottleneck modules; in the second branch, the feature map passes through only one Conv module; finally The two branches are spliced together by channel; among them, the Bottleneck module includes two consecutive convolution operations and a residual operation; The SPP module is a spatial pyramid pooling module; the input feature map size of the SPP module is 512×20×20, and the number of channels is halved after passing through a Conv module; then the convolution kernels used for the feature maps are 5×5 and 9×9 respectively. , 13×13 maximum pooling operation, and the three feature maps and the input feature map are spliced by channel and then passed through a Conv module. The final output feature map size is 512×20×20. 6. A method for real-time recognition of pill boxes based on deep learning as claimed in claim 1, characterized in that the text detection model of DBNet in the text detection is performed by the pre-trained DBNet-based text detection model. It consists of a text detection network that uses differentiable binarization as a text detector based on a simple segmentation network. The differentiable binarization formula is shown in Equation 1: (1) Among them, B represents the approximate binary image, T is the threshold feature map learned by the network, and k represents the magnification factor. 7. A method for real-time recognition of medicine boxes based on deep learning as claimed in claim 1, characterized in that the text recognition of DenseNet and CTC using pre-trained text recognition models based on DenseNet and CTC is used for text recognition. The model structure is as follows: The text recognition models of DenseNet and CTC are composed of CNN layer, RNN layer and CTC transcription layer; The DenseNet network uses Relu as the activation function and uses 3 Dense Block layers for calculation. Each DenseBlock is connected together through the Transition structure to form a DenseNet network. It is trained with CTC_loss and the final data model is obtained; DenseNet has a Transition conversion module, which is used after Dense Block. A bottleneck structure is added before each DenseBlock splicing, and the parameters are reduced through pooling and then transferred to the lower Dense Block structure. 8. A method for real-time identification of medicine boxes based on deep learning as claimed in claim 1, characterized in that each word or phrase of the text recognition result is sequentially subjected to low-threshold fuzzy search and matching in the drug database. , the content of fuzzy search and matching is the drug name, brand, manufacturer, and dosage attributes of the drug. The steps to obtain one or more drug numbers are: The low threshold is 0.5, that is, if it contains half of the same characters, it is considered a successful match; multiple words or sentences are obtained through text recognition, and low-threshold fuzzy search and matching are performed in the drug database. The content of fuzzy search and matching is firstly drugs. If the drug name is matched, the brand, manufacturer, and dosage attributes of the matched drugs will continue to be matched. If one of the attributes does not match any drug, the match will be deemed to have failed; the match will not be narrowed down if the match fails. range, the matching of this attribute will be skipped directly; finally, one or more drug numbers can be obtained through fuzzy search and matching. 9. A method for real-time recognition of medicine boxes based on deep learning as claimed in claim 1, characterized in that each word or phrase of the text recognition result is sequentially subjected to intelligent fuzzy search and matching in the medicine database, and finally the medicine is obtained. The steps for box identification results are: Each word or phrase of the text recognition result is sequentially used in the drug database to match the drug name attributes of the drug using an intelligent matching algorithm; after filtering out the matching drugs, under these drugs, the intelligent matching algorithm is used to match the brand, manufacturer, and dosage attributes, and finally obtain a drug number; The intelligent matching algorithm is as follows: The first thing to match is the drug name attribute. The initial matching threshold is 1, that is, if they are identical, the match is considered successful. If no drug is matched, the match is considered failed. If the match fails, the matching threshold is lowered to 0.1. , until the match is successful; after the first successful match with a threshold value other than 1, the matching threshold is increased by 0.01 in turn until the match fails; after the match fails, the result of the successful match before the failed match is regarded as the final match result. 10. A real-time pill box identification system based on deep learning, characterized by including an image acquisition device and a data processing device, The image acquisition device is used to collect appearance image information of the medicine box; The data processing device is used to realize real-time recognition of the collected pill box appearance image information based on yolov5 and text recognition. The data processing device includes a pill box recognition module based on yolov5, a text detection module based on DBNet, and text based on DenseNet and CTC. Recognition module and pill box recognition module based on text recognition and appearance recognition; The system can implement the deep learning-based real-time identification method of pill boxes described in any one of claims 1 to 9.