TWI684959B - Mouth and nose occluded detecting method and system thereof - Google Patents

Abstract

A mouth and nose occluded detecting method includes a detecting step and a warning step, wherein the detecting step includes capturing an image by an image capturing device, detecting a face portion image from the image by a face detecting method, using an image extracting method so as to extract a mouth portion image from the face portion image, and importing the mouth portion image to an occluding convolutional neural network so as to produce a judgment result, the judgment result is divided into an occluding state or a normal state. When the judgment result is the normal state, the processor performs the detecting step. When the judgment result is the occluding state, an alarming device provides an alarm. Therefore, the mouth and nose occluded detecting method can decrease a misjudgment caused by an environment factor of the image via entering the mouth portion image into the occluding convolutional neural network.

Description

Mouth and nose foreign body shielding detection method and system

The invention relates to a method and a system for masking detection of foreign body of mouth and nose, in particular to a method and system for masking detection of foreign body of mouth and nose by judging by convolutional neural network.

Since the caregiver cannot be on the side, in order to avoid the suffocation of the caregiver's mouth and nose due to the foreign body shielding, the caregiver often assists the care through the mouth and nose foreign body shielding detection system to reduce the burden, but the traditional The mouth and nose foreign body detection system will cause the mouth and nose foreign body shadow detection system to judge abnormally due to the ambient light or the color of the caregiver's clothing.

In view of this, it is very important to develop a method and system for shielding the mouth and nose due to foreign objects that will not be affected by ambient light or the color of the caregiver's clothing.

Therefore, the object of the present invention is to provide an oral and nose foreign body masking detection method and system that are not affected by environmental factors, which judges by inputting the mouth image to the foreign body masking convolutional neural network to reduce image data The environmental factors have caused misjudgment in the detection method and system of foreign body and nose masking.

According to one embodiment of the method aspect of the present invention, there is provided a masking detection method for foreign body of nose and nose, which includes a detection step and a warning step. The detection step includes a face detection step, an image extraction step, and a foreign object occlusion judgment step. The face detection step is to capture image data through an image capture device, and then perform face detection on the image data to obtain a face image. The image extraction step uses image extraction to extract the mouth area of the face image to form a mouth In the image, the foreign body shadowing judgment step is to input the mouth image to the foreign body shadowing convolutional neural network to generate a judgment result, and the judgment result is classified into a state without foreign matter or a state with foreign matter. The warning step is to issue a warning based on the judgment result. When the judgment result is in the state of no foreign objects, the detection step is executed, and when the judgment result is in the state of foreign objects, a warning is issued.

In this way, the mouth and nose foreign body shielding detection method of the present invention utilizes the input of the mouth image to the foreign body shielding convolutional neural network to reduce the detection misjudgment caused by environmental factors in the image data. Moreover, when the foreign body shielding convolutional neural network determines that the mouth and nose of the care recipient are in a state of foreign body, a warning can be issued to remind the care recipient to deal with it as soon as possible.

According to the masking detection method for foreign body of nose and nose as described in the previous paragraph, it further includes a modeling step. The modeling steps include database creation step, image processing step and data training step. The database creation step is to create a foreign object obscuration detection database. The foreign object obscuration detection database includes a plurality of foreign object obscuration images and a plurality of normal images. The image processing step performs image processing on each foreign object obscuration image and each normal image to form a processed foreign object obscuration Detect images and deposit foreign objects Occlusion detection database, the data training step is to train the foreign object masking convolutional neural network by using the foreign object masking image in the foreign object masking detection database, the normal image and the processed foreign object masking detection image.

According to the masking detection method for foreign body of nose and nose as described in the preceding paragraph, the image processing may be image inversion, histogram equalization, logarithmic conversion, gamma processing or Laplace processing.

According to the mouth and nose foreign body shielding detection method described in the preceding paragraph, the foreign body shielding convolutional neural network may include six convolutional layers, three pooling layers, a hidden layer, and an output layer.

According to the masking detection method for foreign body of nose and nose as described in the preceding paragraph, the convolutional layer includes complex convolution kernels, wherein the size of the convolution kernel is 3×3, and the step size of the convolution kernel is 1.

According to the masking detection method for foreign body of nose and nose as described in the previous paragraph, each convolution layer uses a filling method to adjust the size of the feature map of the complex feature map output by each convolution layer.

According to the masking detection method for foreign body of nose and nose as described in the previous paragraph, each pooling layer adopts the maximum pooling method, each pooling layer may include a pooling filter, the size of the pooling filter is 2×2, and the pooling filter The pace of the device is 2.

According to the masking detection method for foreign body of nose and nose as described in the preceding paragraph, the hidden layer includes a fully connected layer, and the number of neurons in the fully connected layer is 128.

According to the mouth and nose foreign body occlusion detection method described in the previous paragraph, the image extraction method is to perform nine-grid lattice on the face image to generate the facial nine-grid image, and extract the partial images of the lower three grids of the facial nine-grid image to generate the mouth image.

According to the mouth and nose foreign body masking detection method described in the previous paragraph, the face detection uses a multiplexed convolutional neural network to detect the face position of the image data.

According to the mouth and nose foreign body masking detection method described in the previous paragraph, the foreign body masking convolutional neural network includes a normalized index layer, the normalized index layer includes image state, mouth image, image state parameters, image state probability, and image state Probability set, where y ^{( i )} is the image state, k is the number of image states, x ^{( i )} is the mouth image, θ ₁ , θ ₂ ..., θ _K is the image state parameter, p ( y ^{( i )} = k | x ^{( i )} ; θ ) is the image state probability, h _θ ( x ^{( i )} ) is the image state probability set, T is the transpose matrix, and the normalized index layer conforms to the following formula:

According to one embodiment of the structural aspect of the present invention, a mouth and nose foreign body shielding detection system is provided, which includes an image capturing device, a processor, and a warning device. The image capturing device is used to capture image data. The processor includes a face detection module, an image extraction module and a foreign object occlusion judgment module. The face detection module is electrically connected to the image capture device. The face detection module captures image data through the image capture device, and performs face detection on the image data to obtain a facial image. The image extraction module is electrically connected to the face detection module, and the image extraction module uses the image extraction method to extract the mouth area of the face image to form a mouth image. The foreign object masking judgment module is electrically connected to the image extraction module, and the foreign object masking judgment module inputs the mouth image to the foreign object masking convolutional neural network to generate a judgment result. The signal of the warning device is connected to the processor, and the warning device sends When a warning is issued, the detection step is executed when the judgment result is in the state of no foreign objects, and when the judgment result is in the state of foreign objects, a warning is issued.

Therefore, the mouth and nose foreign body shielding detection system of the present invention uses the foreign body shielding judgment module to input the mouth image to the foreign body shielding convolutional neural network to reduce the detection misjudgment caused by environmental factors in the image data.

According to the mouth and nose foreign body shielding detection system described in the preceding paragraph, the image capturing device is a camera.

According to the mouth and nose foreign body shading detection system described in the preceding paragraph, the foreign body shading convolutional neural network includes six convolution layers, three pooling layers, a hidden layer and an output layer.

According to the mouth and nose foreign body occlusion detection system described in the preceding paragraph, the face detection uses a multiplexed convolutional neural network to detect the face position of the image data.

S100‧‧‧ Masking detection method of foreign body

s110‧‧‧detection steps

s111‧‧‧Face detection steps

s112‧‧‧Image extraction steps

s113‧‧‧judgment procedure

s120‧‧‧Warning steps

s130‧‧‧Modeling steps

s131‧‧‧Data creation steps

s132‧‧‧Image processing steps

s133‧‧‧Data training steps

200‧‧‧ Foreign matter shielding convolutional neural network architecture

cl1‧‧‧The first convolution layer

cl2‧‧‧second convolution layer

cl3‧‧‧third convolution layer

cl4‧‧‧ fourth convolution layer

cl5‧‧‧fifth convolution layer

cl6‧‧‧sixth convolution layer

pl1‧‧‧The first pooling layer

pl2‧‧‧Second pooling layer

pl3‧‧‧The third pooling layer

hl‧‧‧ hidden layer

op‧‧‧ output layer

sl‧‧‧Normalized index layer

300‧‧‧ facial image

310‧‧‧ mouth image

400‧‧‧Mouth and nose foreign body shielding detection system

410‧‧‧Image capture device

420‧‧‧ processor

421‧‧‧Face detection module

422‧‧‧Image extraction module

423‧‧‧Foreign object blocking judgment module

430‧‧‧Warning device

Fig. 1 shows a flow chart of the steps of the method for detecting the masking of foreign bodies of the nose and nose according to one embodiment of a method aspect of the invention; Fig. 2 shows the mouth and nose of another embodiment according to a method aspect of the invention Step flow chart of foreign object masking detection method; FIG. 3 shows a schematic diagram of a foreign object masking convolutional neural network architecture according to the embodiment of FIG. 1; FIG. 4 shows an embodiment according to FIG. 1 Schematic diagram of facial nine-grid image of the detection method of foreign body and nose masking; FIG. 5 is a schematic view of a mouth image of a mouth and nose foreign body shielding detection method according to the embodiment of FIG. 1; and FIG. 6 is a mouth and nose foreign body shielding detection system according to an embodiment of a structural aspect of the present invention. Block diagram.

Hereinafter, a plurality of embodiments of the present invention will be described with reference to the drawings. For clarity, many practical details will be explained in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are unnecessary. In addition, for the sake of simplifying the drawings, some conventionally used structures and elements will be shown in a simple schematic manner in the drawings; and repeated elements may be indicated by the same number.

FIG. 1 is a flow chart showing the steps of a method s100 for detecting a masking of foreign body of nose and nose according to an embodiment of a method aspect of the present invention. It can be seen from FIG. 1 that the method for detecting the masking of the foreign body of the nose and nose includes a detection step s110 and a warning step s120.

In detail, the detection step s110 may include a face detection step s111, an image extraction step s112, and a foreign object masking determination step s113. The face detection step s111 captures image data through the image capture device 410 (marked in Figure 6), and performs face detection on the image data to obtain a facial image; the image extraction step s112 uses the image extraction method to extract the face The mouth area of the partial image to form the mouth image 310 (marked in FIG. 5); the foreign body masking judgment step s113 inputs the mouth image 310 to the foreign body masking convolutional neural network to The judgment result is generated, and the judgment result can be divided into at least a state without foreign objects or a state with foreign objects. The warning step s120 is to issue a warning according to the judgment result. When the result of the determination is that there is no foreign object, the detection step s110 is performed to continuously monitor the state of the care recipient, and when the result of the determination is that there is a foreign object, a warning is issued to prompt the caregiver to process as soon as possible. In this way, by inputting the mouth image 310 to the foreign body shielding convolutional neural network to determine whether the mouth and nose of the care recipient are covered by foreign bodies, to reduce the foreign body shielding convolutional neural network from being affected by environmental factors in the image data Misjudgment occurs, and the environmental factor may be the ambient light or the color of the caregiver's clothing.

FIG. 2 is a flow chart showing the steps of a method s100 for detecting a masking of foreign body of nose and nose according to another embodiment of a method aspect of the present invention. The mouth and nose foreign body shadow detection method s100 includes a detection step s110, a warning step s120, and a modeling step s130.

Please refer to FIG. 1 together. In the embodiment of FIG. 2, the detection step s110 and the warning step s120 are the same as the corresponding steps in FIG. 1 and will not be repeated here. In particular, the method for detecting occluded foreign body occlusion in the embodiment shown in FIG. 2 may further include a modeling step s130. The modeling step s130 includes a data creation step s131, an image processing step s132, and a data training step s133, wherein the data creation step s131 creates a foreign object obscuration detection database. The foreign object obscuration detection database may include plural foreign object obscuration images and plural normal images, where The plural masked images may include quilt masked images, milk spit masked images, etc.; the image processing step s132 performs image processing on each foreign object masked image and each normal image to form a processed foreign object masked detection image and stores it in the foreign object masked detection database; data Training step s133 uses foreign object occlusion detection Foreign object obscured images, normal images, and processed foreign object obscured detection images in the database train the foreign object obscured convolutional neural network. In this way, by processing the complex foreign object masked image and the complex normal image separately to obtain processed foreign object masked detection images, the training samples of the foreign object masked convolutional neural network are increased, thereby improving the judgment of the foreign object masked convolutional neural network Accuracy.

In order to increase the training samples of the foreign matter masking convolutional neural network, the image processing method in the above image processing step s132 may be image inversion, histogram equalization, logarithmic conversion, gamma processing, or Laplace processing. The purpose of the image processing of the foreign object masking image is to simulate the illuminance and image contour in various situations to train the foreign object masking convolutional neural network, where the histogram equalization can redistribute the brightness in the foreign object masking image uniformly, so that the foreign object masking the image The darker part of the image becomes brighter, and the brighter part of the image obscured by foreign objects becomes darker. Logarithmic conversion (Log) can transform the low-brightness part of the image obscured by foreign objects through logarithmic conversion to increase the brightness. Gamma processing (Gamma) It is to adjust the gamma value to lighten the darker part of the image obscured by foreign objects or darken the lighter part of the image obscured by foreign objects. Laplace processing is obtained by using the principle of second-order partial differential Foreign objects obscure the outline, shape and distribution of the image. In simple terms, histogram equalization, logarithmic conversion, gamma processing, and Laplace processing are performed on the foreign object masked image, and the histogram equalization, logarithmic conversion, and gamma are performed after the image of the foreign object masked image is inverted. After Ma processing and Laplacian processing, nine images of foreign body masking detection after processing can be obtained to increase the number of training samples of foreign body masking convolutional neural network. Although the image processing method is disclosed as above, it is not based on the content of this disclosure. The disclosed embodiments are limited. In addition, please refer to Table 1, which shows the accuracy of the first embodiment and the first comparative example Degree comparison table, in which the foreign matter shielding convolutional neural network of the first embodiment has the same network architecture as the foreign matter shielding convolutional neural network of the first comparative example, the difference is that the foreign body shielding convolutional neural network of the first embodiment The number of training samples of the network is higher than the number of training samples of the foreign object masking convolutional neural network of the first comparative example, wherein the training samples of the foreign object masking convolutional neural network of the first embodiment are foreign object masking images, normal images and processing After the foreign object masking detection image, the training samples of the foreign object masking convolutional neural network of the first comparative example are the foreign object masking part image and the normal image, as can be seen from Table 1, the accuracy of the foreign object masking convolutional neural network of the first comparative example is The degree of accuracy is 84%, and the accuracy of the foreign object masking convolutional neural network of the first embodiment is 94%, that is, during the modeling step s130, using more training samples to train the foreign object masking image can improve foreign object masking The accuracy of convolutional neural networks.

FIG. 3 is a schematic diagram of a foreign object masking convolutional neural network architecture 200 according to the oral and nose foreign object masking detection method s100 according to the embodiment of FIG. 1, and Table 2 is a foreign object masking convolutional neural network architecture 200 of the embodiment of FIG. 3. List. Please refer to FIG. 3 and Table 2 together. The foreign object masking convolutional neural network architecture 200 may include six convolutional layers (not labeled), three pooling layers (not labeled), a hidden layer hl, and an output layer op. The convolution layer includes a first convolution layer cl1, a second convolution layer cl2, a third convolution layer cl3, a fourth convolution layer cl4, a fifth convolution layer cl5 and a sixth convolution layer cl6, wherein the first convolution layer cl1 and the second convolution layer cl2 can be conv3_16, namely the first convolution layer cl1 and the second convolution layer The size of the convolution kernel of cl2 is 3×3 and the number of feature maps output by the first convolution layer cl1 and the second convolution layer cl2 is 16, and the third convolution layer cl3 and the fourth convolution layer cl4 can be conv3_32, which is the third convolution The convolution kernel size of the convolution layer cl3 and the fourth convolution layer cl4 is 3×3 and the number of feature maps output by the third convolution layer cl3 and the fourth convolution layer cl4 is 32. The fifth convolution layer cl5 and the sixth convolution layer cl6 may be conv3_64, that is, the size of the convolution kernel of the fifth convolution layer cl5 and the sixth convolution layer cl6 is 3×3 and the number of feature maps output by the fifth convolution layer cl5 and the sixth convolution layer cl6 is 64. In addition, each convolutional layer can include complex convolution kernels to output a plurality of feature maps, the size of each convolution kernel is 3×3, the step size of each convolution kernel can be 1, and each convolution layer can be adjusted using a filling method The feature map size of the feature map output by the convolution layer. In detail, the size of the image data can be 50×50. Before performing the convolution operation of the convolution layer, the image data can be filled using the filling method to obtain the filled image. Image data, the size of the filled image data is 52×52, the filling method can be zero-filling method, and then the convolution operation is performed on the filled image data. The pooling layer includes the first pooling layer pl1, the second pooling layer pl2, and the third pooling layer pl3. Each pooling layer can use the maximum pooling method, and each pooling layer includes a pooling filter, pooling filter The size of the filter is 2×2 and the step size of the pooling filter is 2. The hidden layer hl may include a first fully connected layer, and the first fully connected layer may be FC_128, that is, the number of neurons in the first fully connected layer is 128. The output layer op may include a second fully connected layer, and the second fully connected layer may be FC_2, that is, the number of neurons in the second fully connected layer is 2.

藉此，異物遮蔽卷積神經網路可用以判斷嘴部影像310之各影像狀態的機率，其中影像狀態至少為有異物影像及無異物影像，當有異物影像的機率高於無異物影像的機率時，判斷結果為有異物狀態，當無異物影像的機率高於有異物影像的機率時，判斷結果為無異物狀態。 The foreign object masking convolutional neural network architecture 200 may further include a normalized index (softmax) layer sl. The normalized index layer sl may be used to calculate the probability of having a foreign object state and the probability of having no foreign object state to generate a judgment result. The normalized index layer sl contains the image state, the number of image states, the mouth image 310, the image state parameters, the image state probability, and the image state probability set, where y ^{( i )} is the image state, k is the number of image states, x ^{( i )} is the mouth image 310, θ is the set of image state parameters, θ ₁ , θ ₂ ..., θ _K is the image state parameter, p ( y ^{( i )} = k | x ^{( i )} ; θ ) is the image State probability, h _θ ( x ^{( i )} ) is the set of image state probabilities, T is the transpose matrix, and the normalized index layer sl conforms to the following formula:

In this way, the foreign object masking convolutional neural network can be used to determine the probability of each image state of the mouth image 310, wherein the image state is at least a foreign object image and a non-foreign object image. When the foreign object image probability is higher than the foreign object image probability At that time, the judgment result is in the state with foreign objects. When the probability of the image without foreign objects is higher than the probability of the images with foreign objects, the judgment result is in the state without foreign objects.

Please refer to Table 3 and Table 4 together. Table 3 shows the comparison of the foreign object shielding convolutional neural network architecture 200 and the second embodiment of the second embodiment of the present invention. Examples, third comparative examples, fourth comparative examples, fifth comparative examples, and sixth comparative examples of foreign body shielding convolutional neural network architecture comparison table, Table 4 shows the second embodiment of foreign body shielding convolutional neural network A comparison table of the accuracy of the road architecture 200 and the accuracy of the foreign matter masking convolutional neural network architecture of the second comparative example, third comparative example, fourth comparative example, fifth comparative example, and sixth comparative example. As can be seen from Tables 3 and 4, the accuracy of the foreign matter masking convolutional neural network architecture 200 of the second embodiment is higher than that of the second comparative example, third comparative example, fourth comparative example, fifth comparative example, and sixth comparison Example of foreign body shadowing convolutional neural network architecture.

In order to obtain facial images from image data, face detection can use multi-task Cascaded Convolutional Networks (MTCNN) to detect the facial position of the image data. Convolutional neural networks include Proposal Net (P-Net), Reduced Network (Refine-Net, R-Net) and Output Network (Output-Net, O-Net), where the recommended network is through It is recommended that the network convolutional neural network generate complex face frames, the reduced network is to exclude non-face frames by reducing the network convolutional neural network, and the output network is to output the face through the output network convolutional neural network feature. In this way, facial images can be obtained by inputting image data into a multiplexed convolutional neural network.

FIG. 4 is a schematic diagram of a facial jigsaw image 300 of an oral and nose foreign body shadow detection method s100 according to the embodiment of FIG. 1, and FIG. 5 is a mouth image of an oral and nose foreign body shadow detection method s100 according to the embodiment of FIG. 310 schematic. Please refer to Fig. 4 and Fig. 5, the image extraction method can be to perform nine-grid lattice on the face image to generate the facial nine-grid image 300, and extract the partial images of the lower three grids of the facial nine-grid image 300 to generate the mouth Image 310, and then input the mouth image 310 to the foreign object masking convolutional neural network to generate a judgment result, and use the mouth image 310 extracted from the image data as the input data of the foreign object masking convolutional neural network to reduce the environment The influence of factors can avoid misjudgment of the convolutional neural network shielded by foreign objects.

Due to the difference in the size of the human face, in order to avoid the difference in the size of the mouth image 310, which leads to the misjudgment of the muzzle detection method s100, the muzzle detection method s100 may further include the normalization of the mouth image 310 For processing, please refer to Table 5. Table 5 shows the accuracy of the foreign matter masking convolutional neural network of the third embodiment and the foreign matter of the seventh comparative example, the eighth comparative example, the ninth comparative example, and the tenth comparative example The accuracy comparison table of the masked convolutional neural network, wherein the size of the mouth image 310 of the third embodiment after normalization is 50×50, and the size of the mouth image of the seventh comparative example after normalization is 25×25, the first The size of the mouth image of the eight comparative examples after normalization is 75×75, the size of the mouth image of the ninth comparative example after normalization is 100×100, and the size of the mouth image of the tenth comparative example after normalization is 150×150. It can be seen from Table 5 that the foreign object masking convolutional neural network in the third embodiment has high accuracy. That is, when the size of the mouth image 310 is 50×50, the accuracy of the foreign object masking convolutional neural network is high.

FIG. 6 shows a block diagram of a mouth and nose foreign body shadow detection system 400 according to an embodiment of a structural aspect of the present invention. As can be seen from FIG. 6, the mouth and nose foreign body shadow detection system 400 includes an image capturing device 410 and processing器420 and warning device 430. The image capturing device 410 can be used to capture images Like information. The processor 420 is electrically connected to the image capture device 410 and includes a face detection module 421, an image extraction module 422, and a foreign object shading determination module 423. The face detection module 421 is electrically connected to the image capturing device 410, and the face detection module 421 captures image data through the image capturing device 410, and performs face detection on the image data to obtain a facial image. The image extraction module 422 is electrically connected to the face detection module 421, and the image extraction module 422 uses an image extraction method to extract the mouth area of the face image to form the mouth image 310. The foreign object masking determination module 423 is electrically connected to the image extraction module 422, and the foreign object masking determination module 423 inputs the mouth image 310 to the foreign object masking convolutional neural network to generate a determination result. The signal of the warning device 430 is connected to the processor 420, and the warning device 430 can issue a warning according to the judgment result. When the result of the determination is that there is no foreign object, the detection step s110 is executed, and when the result of the determination is that there is a foreign object, a warning is issued.

In detail, the image capturing device 410 can be used to capture the image of the care receiver to generate image data, wherein the image capturing device 410 can be a camera. The face detection module 421 of the processor 420 can perform face detection on the image data to obtain a face image, wherein the face detection can use a multiplexed convolutional neural network to detect the face of the image data部位置。 Department location. The image extraction module 422 of the processor 420 can use the image extraction method to extract the mouth area in the face image to form the mouth image 310. The image extraction method can be to nine-grid the facial image to generate the facial nine-grid image 300. And extract the lower three frames of the facial nine-grid image 300 to form the mouth image 310. Furthermore, the foreign object masking judgment module 423 of the processor 420 can input the mouth image 310 to the foreign object masking convolutional neural network to generate a judgment result, and the judgment result can be classified as having foreign objects State and no foreign body state. The processor 420 may be a microprocessor, a central processing unit (CPU), or other electronic operation processing unit. The warning device 430 issues a warning according to the judgment result. When the judgment result is in the state without foreign objects, the image capturing device 410 captures the image of the care recipient again to continuously monitor the state of the care recipient, and when the judgment result is in the state of foreign objects, the warning device 430 issues a warning to remind the caregiver Handle as soon as possible. The warning device 430 may be an image warning (such as a flashing light) or a sound warning (such as a buzzer). The mouth and nose foreign body shielding detection system 400 can be applied to a computer or a mobile phone.

In order to improve the accuracy of the mouth and nose foreign body masking detection system 400, the foreign body masking convolutional neural network architecture 200 may include six convolution layers, three pooling layers, hidden layers hl, and output layer op, and the foreign body masking convolutional neural network architecture 200 It is the same as Figure 3, Table 2, Table 3, and Table 4, and will not be repeated here.

In summary, the mouth and nose foreign body shielding detection method and mouth and nose foreign body shielding detection system of the present invention can provide the following advantages:

(1) Through image processing to increase the training samples of the foreign body shielding convolutional neural network, to improve the accuracy of the foreign body shielding convolutional neural network.

(2) By inputting the image of the mouth to the foreign object shielding convolutional neural network to avoid the environmental factors in the image data causing the foreign object shielding convolutional neural network to misjudge, thereby improving the accuracy of the foreign object shielding convolutional neural network.

(3) The foreign body shielding detection method of the oral and nose foreign body shielding detection system and the foreign body shielding convolutional neural network architecture with high accuracy.

Although the present invention has been disclosed as above in an embodiment, it is not intended to limit the present invention. Anyone who is familiar with this skill will not deviate from the essence of the present invention. Within the scope of the gods, various changes and modifications can be made, so the scope of protection of the present invention shall be subject to the scope defined in the appended patent application.

S100‧‧‧ Masking detection method of foreign body

s110‧‧‧detection steps

s111‧‧‧Face detection steps

s112‧‧‧Image extraction steps

s113‧‧‧judgment procedure

s120‧‧‧Warning steps

Claims (15)

An oral and nose foreign body masking detection method, including a detection step, including: a face detection step, which is to capture image data through an image capture device and perform a face detection on the image data to obtain a face Image; an image extraction step, which uses an image extraction method to extract a mouth area of the face image to form a mouth image; and a foreign body masking judgment step, which inputs the mouth image to a foreign body mask Convolutional neural network to generate a judgment result, the judgment result is divided into a state with foreign objects or a state without foreign objects; and a warning step, based on the judgment result to issue a warning, when the judgment result is the state without foreign objects At this time, the detection step is executed, and when the result of the judgment is that there is a foreign object state, a warning is issued. The mouth and nose foreign body shielding detection method as described in item 1 of the patent application scope further includes: a modeling step, including: a data creation step, which is to create a foreign body shielding detection database, and the foreign body shielding detection database contains plural numbers Foreign objects obscuring images and plural normal images; An image processing step is to perform an image processing on each of the foreign object obscured images and each of the normal images to form a processed foreign object obscured detection image and store it in the foreign object obscured detection database; and a data training step that utilizes The foreign object obscured images of the foreign object obscured detection database, the normal images, and the processed foreign object obscured detected images train the foreign object obscured convolutional neural network. The mouth and nose foreign body occlusion detection method as described in item 2 of the patent application scope, wherein the image processing is an image flip, histogram equalization, logarithmic conversion, a gamma processing or a Laplace processing. The mouth and nose foreign body shielding detection method as described in item 1 of the patent application scope, wherein the foreign body shielding convolutional neural network includes six convolutional layers, three pooling layers, a hidden layer and an output layer. The masking detection method for foreign body of nose and nose as described in item 4 of the patent application scope, wherein each of the convolutional layers includes a plurality of convolutional kernels, wherein the size of each of the convolutional kernels is 3×3, and the step size of each of the convolutional kernels Is 1. The masking detection method for foreign body of nose and nose as described in item 4 of the patent application scope, wherein each convolution layer uses a filling method to adjust the size of the feature map of the complex feature map output by each convolution layer. For the detection method of nose and mouth foreign body occlusion detection as described in item 4 of the patent scope, wherein each pooling layer adopts the maximum pooling method, each pooling layer includes a pooling filter, and the size of the pooling filter is 2×2, and the step size of the pooling filter is 2. The masking detection method for foreign body of nose and nose as described in item 4 of the patent application scope, wherein the hidden layer includes a fully connected layer, and the number of neurons in the fully connected layer is 128. The mouth and nose foreign body occlusion detection method as described in item 1 of the patent scope, wherein the image extraction method is to perform a nine-grid lattice on the face image to generate a face nine-grid image, and extract the lower three of the face nine-grid image Part of the image to generate the mouth image. The mouth and nose foreign body occlusion detection method as described in item 1 of the patent scope, wherein the face detection uses a multiplexed convolutional neural network to detect a face position of the image data. The mouth and nose foreign body masking detection method as described in item 1 of the patent scope, wherein the foreign body masking convolutional neural network includes a normalized index layer, the normalized index layer includes at least one image state, the mouth Image, at least one image state parameter, at least one image state probability, and a set of image state probability, where y ^{( i ) is} the at least one image state, k is the number of image states, x ^{( i ) is} the mouth image, θ ₁ , θ ₂ ..., θ _K are the image state parameters, p ( y ^{( i )} = k | x ^{( i )} ; θ ) is the probability of the image state, h _θ ( x ^{( i )} ) is the image state Probability set, T is the transposed matrix, the normalized index layer conforms to the following formula:

A mouth and nose foreign body shielding detection system, comprising: an image capturing device for capturing an image data; a processor, electrically connected to the image capturing device, and comprising: a face detection module, electrical Connected to the image capture device, the face detection module captures the image data through the image capture device, performs a face detection on the image data to obtain a face image; an image extraction module, electronic Is connected to the face detection module, the image extraction module uses an image extraction method to extract a mouth area of the face image to form a mouth image; and a foreign object occlusion judgment module is electrically connected to the An image extraction module, the foreign object masking judgment module inputs the mouth image to a foreign object masking convolutional neural network to generate a judgment result; and A warning device, a signal is connected to the processor, the warning device issues a warning according to the judgment result, when the judgment result is the foreign object-free state, a detection step is performed, and when the judgment result is the foreign object state, A warning is issued. The mouth and nose foreign body shielding detection system as described in item 12 of the patent application scope, wherein the image capturing device is a camera. The mouth and nose foreign body shielding detection system as described in item 12 of the patent application scope, wherein the foreign body shielding convolutional neural network includes six convolutional layers, three pooling layers, a hidden layer and an output layer. The mouth and nose foreign body occlusion detection system as described in item 12 of the patent scope, wherein the face detection uses a multiplexed convolutional neural network to detect a face position of the image data.

Images