CN117333703A - Tongue image quality evaluation method and system based on deep learning and feature fusion - Google Patents

Abstract

The invention discloses a tongue image quality assessment method based on deep learning and feature fusion, which comprises the following steps: constructing a tongue image dataset; training the evaluation model based on tongue image data to obtain a trained evaluation model; the evaluation model comprises a tongue segmentation module, a feature extraction module and a tongue image classification module; inputting the tongue image to be tested into the trained evaluation model; dividing the tongue image to be detected based on a tongue dividing module to obtain a target tongue image; respectively carrying out feature extraction processing on the target tongue body image based on a feature extraction module to obtain shallow features and deep semantic features; the shallow layer features at least comprise texture features, natural scene statistical features and color features; based on the tongue image classification module, shallow features and deep semantic features are fused in series, classification is carried out according to the fused feature vectors, a quality evaluation result is obtained, and the quality condition of the tongue image can be efficiently and accurately judged after the tongue image classification module is used.

Description

Tongue image quality assessment method and system based on deep learning and feature fusion

Technical field

The present invention relates to the technical field of traditional Chinese medicine image processing, and in particular to a tongue image quality assessment method and system based on deep learning and feature fusion.

Background technique

Tongue diagnosis, as an important part of traditional Chinese medicine inspection, has the advantages of being non-invasive, simple and easy to perform. In recent years, with the rapid development of science and technology, digital tongue diagnosis has begun to rise. By using modern scientific and technological means to objectively collect patient's tongue image data, combined with the theory of traditional Chinese medicine, information such as the deficiency and excess of yin and yang in the human body can be obtained. However, unqualified tongue images will not only affect subsequent data processing, but may even lead to misdiagnosis, seriously hindering the development and application of intelligent TCM diagnosis and TCM remote diagnosis and treatment.

Changes in the color of the tongue due to the influence of ambient lighting, blurring of the tongue image due to motion jitter, etc. will all cause certain image distortion, posing challenges to the intelligent development of tongue diagnosis in traditional Chinese medicine. In order to ensure the accuracy of digital intelligent tongue diagnosis, it is often necessary to manually determine whether the collected tongue images are qualified. The above method not only has high labor costs and low efficiency, but will also greatly restrict the intelligent development of traditional Chinese medicine tongue diagnosis.

Therefore, how to efficiently and accurately realize tongue image quality assessment is crucial to the intelligent and objective research of tongue diagnosis in traditional Chinese medicine.

Contents of the invention

The purpose of the present invention is to provide a tongue image quality assessment method and system based on deep learning and feature fusion to solve the existing problems of inaccurate and inefficient tongue image quality judgment.

In order to solve the above technical problems, the present invention provides two solutions. First, a tongue image quality assessment method based on deep learning and feature fusion is provided, which includes the following steps:

Construct tongue image dataset;

The evaluation model is trained based on the tongue image data to obtain a trained evaluation model;

Input the image of the tongue to be tested into the trained evaluation model;

The evaluation model includes a tongue segmentation module, a feature extraction module and a tongue image classification module;

Segment the tongue image to be measured based on the tongue segmentation module to obtain a target tongue image;

Based on the feature extraction module, perform feature extraction processing on the target tongue image respectively to obtain shallow features and deep semantic features;

Wherein, the shallow features include at least texture features, natural scene statistical features and color features;

The shallow features and the deep semantic features are normalized and fused in series based on the tongue image classification module, and image classification is performed according to the fused feature vectors to obtain tongue image quality evaluation results.

In some embodiments of the first aspect, the shallow feature extraction step specifically includes: converting the target tongue image into a gray level co-occurrence matrix, and obtaining the target tongue image by calculating the gray level co-occurrence matrix. multi-dimensional texture features; perform statistical feature extraction of natural images on the target tongue image to obtain multi-dimensional natural scene statistical features of the target tongue image; perform color space conversion on the target tongue image to obtain the target Multidimensional color features of tongue body images.

In some embodiments of the first aspect, the texture features include contrast, dissimilarity, uniformity, energy, correlation and angular second-order moment, which are specifically calculated as follows:

The formula for calculating the contrast:

The calculation formula for the dissimilarity is:

The calculation formula for the uniformity is:

The energy calculation formula is:

The calculation formula for the correlation is:

in,

The calculation formula of the second-order angular matrix:

Among them, i and j represent the gray value corresponding to the pixel (x, y) and the pixel (x+Δx, y+Δy) respectively; P(i, j, d, θ) represents the pixel (x, The probability that y) appears at the same time as the pixel (x+Δx, y+Δy) with gray level j; d is the separation distance between the two pixels, d takes 1 and 2; θ is the generation direction of the grayscale matrix, θ takes There are four directions: 0o, 45o, 90o, and 135o; n is the gray level of the pixel.

In some embodiments of the first aspect, performing statistical feature extraction of natural images on the target tongue image to obtain multi-dimensional natural scene statistical features of the target tongue image includes the following steps: The target tongue image is down-sampled to obtain the down-sampled tongue image; the GGD fitting distribution method is used to calculate the shape parameter α and scale parameter σ of the target tongue image and the down-sampled tongue image respectively. Feature extraction is used to describe the multi-dimensional natural scene distribution statistical characteristics of whether the tongue image is distorted; the AGGD fitting distribution method is used to perform horizontal adjacent, vertical adjacent and summation of the target tongue image and the down-sampled tongue image. Two pairs of diagonally adjacent four-direction fitting shape parameters η, mean parameter μ, and left scale parameter and right scale parameters/> Feature extraction to obtain multi-dimensional natural scene edge distribution statistical features used to describe whether the tongue image is distorted;

in,

The specific calculation formula of GGD is as follows:

In the above formula: mean parameter;

The specific calculation formula of AGGD is as follows:

In the above formula: x is the mean loss contrast normalization coefficient corresponding to the brightness value of a certain pixel block in the natural image; eta controls the shape of the distribution, and the left and right scale parameters The variances on the left and right sides of the distribution are controlled respectively, β _l and β _r are both calculation intermediate variables, and μ is the mean parameter.

In some embodiments of the first aspect, the color space of the target tongue image is converted into YIQ, YcbCr, HSV and CIELAB space images, and the grayscale components in the YIQ, YcbCr, HSV and CIELAB space images are extracted to obtain the The multi-dimensional color features of the target tongue image are described.

In some embodiments of the first aspect, the step of extracting deep semantic features specifically includes: adding a fully connected layer with multiple neurons to the ResNet-34 network model, and inputting the data set of the tongue body image into the ResNet-34 network model. The ResNet-34 network model outputs the neuron features of the fully connected layer as multi-dimensional deep semantic features of the tongue body image; wherein the number of neurons in the fully connected layer is the same as the number of dimensions of the shallow layer features.

In some embodiments of the first aspect, the tongue body segmentation module includes a YOLOv5s network model and a U ² -Net network model; the collected tongue images are input to the YOLOv5s network model, and targets containing the complete tongue body are extracted. region; input the target region into the U ² -Net network model, segment and extract the complete tongue body image, and obtain the target tongue body image.

In some embodiments of the first aspect, the shallow features and the deep semantic features are fused in series based on the tongue image classification module, and image classification is performed according to the fused feature vectors to obtain tongue image quality. Evaluating the results, this step specifically includes: the data set includes multiple categories; combining the data sets of all categories in pairs to form multiple two-classification tasks; for each of the two-classification tasks, based on the tongue The volume image classification module performs serial fusion of the shallow features and the deep semantic features; trains the corresponding two-class support vector machine SVM model according to the fused feature vectors; trains multiple two-class support vector machine SVM models based on the tongue image data set. Classify support vector machine models, and the number of SVM models obtained is equal to the number of pairwise combinations of categories in the data set; input the tongue image data to be tested into all the trained SVM models, and obtain multiple binary classification results; classify the results Use a relative majority voting strategy to make decisions and complete multi-classification evaluation tasks.

In some embodiments of the first aspect, before the multi-dimensional shallow features and the multi-dimensional deep semantic features are serially fused, the following steps are further included: merging the multi-dimensional shallow features and the multi-dimensional deep semantic features. All are subjected to linear normalization processing to obtain data scaled to [-1,1].

In the second aspect, this solution provides a tongue image quality assessment system based on deep learning and feature fusion that applies the first aspect, including: a data processing unit used to segment the tongue image to obtain a target tongue body image; A controller construction unit is used to construct a tongue image quality assessment model based on deep and shallow feature fusion; a calculation unit is used to input the target tongue body image into the tongue image quality assessment model for assessment, and obtain a tongue image quality assessment result.

The beneficial effects of the present invention are as follows:

The tongue body segmentation module based on deep learning inputs the collected tongue images into the YOLOv5s network model, extracts the target area containing the complete tongue body; inputs the target area into the U ² -Net network model, segments and Extract a complete tongue body image; compared with traditional tongue body segmentation, it can effectively reduce the computational complexity of tongue image processing on the basis of improving the accuracy of tongue body segmentation.

A tongue image quality assessment model based on the fusion of deep and shallow features, which fuses the deep semantic features and shallow features of tongue images in series, and uses the serially fused image feature vectors for model training to obtain a tongue image quality assessment model; Compared with the model trained with a single feature, it can obtain more complete tongue image features and objectively and comprehensively judge the tongue image quality. Its classification accuracy is higher, and it is suitable for tongue image analysis systems to help it screen high-quality tongue images.

Description of drawings

In order to explain the technical solution of the present invention more clearly, the drawings needed to be used in the implementation will be briefly introduced below. Obviously, the drawings in the following description are only some implementations of the present invention. For ordinary people in the art, For technical personnel, other drawings can also be obtained based on these drawings without exerting creative work.

Figure 1 is a flow chart of the present invention's TCM tongue image quality assessment based on deep learning and feature fusion.

Figure 2 is a schematic diagram of the YOLOv5 tag annotation process of the present invention.

Figure 3 is a schematic diagram of the U ² -Net labeling process of the present invention.

Figure 4 is a mask picture converted after U ² -Net annotation of the present invention.

Figure 5 is a YOLOv5s network structure diagram of the present invention.

Figure 6 is a U ² -Net network structure diagram of the present invention.

Figure 7 is the tongue target area after rough segmentation by YOLOv5s according to the present invention.

Figure 8 is the target tongue image after U ² -Net fine segmentation according to the present invention.

Figure 9 is a schematic diagram of the deep semantic feature extraction method based on ResNet-34 of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

In the existing TCM tongue image data collection, the color change of the tongue image and tongue due to the influence of environmental lighting, the blurring of the tongue body image due to motion jitter, etc. will cause certain image distortion, which is a key issue for TCM tongue diagnosis. Intelligent development brings challenges. In order to ensure the accuracy of digital tongue diagnosis, it is often necessary to manually judge whether the collected tongue images are qualified. The above methods not only have high labor costs, but also have low efficiency, which greatly restricts the intelligent development of tongue diagnosis in traditional Chinese medicine.

To this end, the present invention designs a two-stage tongue image quality assessment method of "segment first, then evaluate". As shown in Figure 1, the coarse segmentation and fine segmentation of the tongue image are completed in sequence, and then the deep and shallow features of the image are extracted. , and finally the tongue image quality classification is performed based on the TCM tongue image quality assessment model after feature fusion. Through the above work, on the one hand, it can reduce the calculation amount of the algorithm, improve the accuracy of tongue image classification, and solve the problem of inaccurate and inefficient judgment of existing tongue images; on the other hand, it can provide high-quality, complete tongue images for clinical researchers of traditional Chinese medicine. A standard tongue image data set of image information, which is of great significance for establishing a standard database of TCM tongue images and promoting the development of TCM intelligent tongue diagnosis technology.

The present invention provides a tongue image quality assessment method based on deep learning and feature fusion, which includes the following steps:

S1, construct a data set of tongue images;

Specifically, it refers to classifying the collected tongue image data sets according to tongue image quality, dividing the data set, hierarchically selecting 80% of the tongue image data as the training set, 10% of the data as the verification set, and the remaining 10% is used as the test set.

Among them, the hierarchical selection is based on the divided high-quality tongue image, blurred tongue image, high-brightness tongue image and low-brightness tongue image. From then on, hierarchical selection is performed separately among the four. For example, 80% of the tongues are selected from the high-quality tongue image. The image data is used as a training set, 10% of the data is used as a verification set, and the remaining 10% is used as a test set. The blurred tongue image, high-brightness tongue image, and low-brightness tongue image can be deduced in this way.

S2, train the tongue image quality assessment model based on the tongue image data, and obtain the trained assessment model;

The evaluation model includes a tongue segmentation module, a feature extraction module and a tongue image classification module.

For the segmentation module, the tongue segmentation part is shown in Figure 1. The tongue segmentation module is mainly used to segment the tongue image to be measured to obtain a complete tongue image.

Specifically, the segmentation module includes coarse segmentation based on the YOLOv5s network model (the output is the target area containing the complete tongue) and fine segmentation based on the U ² -Net network model (the output is the target tongue image). The YOLOv5s network model and U ² -Net network models all need to be pre-trained to meet segmentation requirements.

Before pre-training the segmentation model, first use the labelImg annotation tool to annotate the tongue images. Figure 2 is a schematic diagram of the annotation process. Select the save type as YOLO format. This format file contains the border coordinates of the target tongue area of each tongue image. information, and save the border coordinate information of the target area in a .txt file format; similarly, in order to train the U2-Net network, the collected tongue images are manually annotated using labelme software. Figure 3 is a schematic diagram of the annotation process, and Save in .json file format. Each .json file contains the label category of the image, the coordinate information of each point, the category shape of the annotated graphic, and image information. Convert the file to mask form and save it as .png The image format is saved, and Figure 4 is the annotated mask image.

Regarding the coarse segmentation network model, the YOLOv5s network (its network structure is shown in Figure 5), which has a small network depth but the fastest detection speed, is used to detect the target area containing the complete tongue body. The YOLOv5s model includes the input layer (Input), the backbone network layer (Backbone), the feature pyramid layer (Neck) and the prediction layer (Predict); after inputting the image to the input layer, the features are extracted through Backone and the feature map is continuously reduced, using Neck structure to achieve different levels of feature map fusion, and predict the target border based on the fused features. Based on this model, the preprocessed training set is used for model training. The tongue image size input during the training process is adjusted to 640*640. The batchsize (batch size) of each iteration is 10, and the training is 200 epochs (number of iterations). Finish. The verification set is then used for verification optimization, and the above process is repeated until the segmentation accuracy requirements are met, and the trained YOLOv5 network model is obtained.

Regarding the fine segmentation network model, the ^U2 -Net network is used to remove a small amount of interference information such as lips, and extract a complete target tongue image. Figure 6 is a U ² -Net network structure diagram. The overall framework of the structure is a large U-shaped structure composed of 11 stages. Each stage is filled with U-shaped residual blocks (Residual U-blocks, RSU). The network contains two The RSU of the structure is the encoder (Encoder, En) and the decoder (Decoder, De). The encoder is used to downsample the feature map, while the decoder is used to upsample the feature map. The bottom of the network is expanded. Convolution, its function is to keep the size of the feature map unchanged, and finally fuse the feature maps of each stage. After the convolution layer and upsampling restore the image size, it is spliced, and the final segmentation result is output through the sigmoid function. Based on this network, the preprocessed training set is used for model training, in which the batchsize (batch size) of each iteration is 10, the hyperparameter learning rate is set to 0.001, and the training is completed for 360 epoch (number of iterations), and then the Perform verification optimization on the above verification set, repeat the above process until the segmentation accuracy requirements are met, and the trained U ² -Net network model is obtained.

After completing the training of the YOLOv5s network model and U2 ^- Net network model, input the tongue image data to be tested into the trained YOLOv5s network model, perform rough segmentation, and detect and obtain the coordinates of the target area containing the complete tongue body in the tongue image. , and output the target area to complete the rough segmentation of the tongue image. Figure 7 shows the tongue target area after rough segmentation; then perform fine segmentation, that is, input the above tongue target area into the trained U ² -Net network model, Output the complete target tongue image, and set the interference information background to black to complete the tongue segmentation. Figure 8 shows the target tongue image after fine segmentation.

For the feature extraction module, the feature extraction module is used to perform feature extraction processing on the segmented tongue body images to obtain shallow features and deep semantic features.

Regarding shallow features, shallow features include texture features, natural scene statistical features and color features, with a total of 88-dimensional features.

In texture feature extraction, the gray-level co-occurrence matrix is mainly used. The gray-level co-occurrence matrix is a texture description method based on the probability of repeated gray-level structures in the image. It is defined as the pixel separation distance d along the θ direction. The gray value is the frequency of co-occurrence of pixel pairs i and j respectively, and the present invention obtains it by calculating the contrast, dissimilarity, uniformity, energy, correlation and angular second-order moment of the gray co-occurrence matrix of the tongue image data its texture characteristics.

Contrast calculation formula:

Dissimilarity calculation formula:

Calculation formula for uniformity:

Energy calculation formula:

Correlation calculation formula:

in,

The calculation formula of angular second-order matrix:

Among them, i and j represent the gray value corresponding to the pixel (x, y) and the pixel (x+Δx, y+Δy) respectively; P(i, j, d, θ) represents the pixel (x, y) The probability of appearing simultaneously with the pixel (x+Δx, y+Δy) with gray level j; d is the distance between the two pixels, here respectively taken as 1 and 2; θ is the generation direction of the gray matrix, usually Take four directions: 0°, 45°, 90°, and 135°; n is the gray level of the pixel. That is, the present invention extracts from four directions of 0°, 45°, 90°, and 135°, and takes the separation distance d as 1 and 2 respectively, thereby obtaining 8 gray-level co-occurrence matrices. Each gray-level co-occurrence matrix passes The formulas of contrast, dissimilarity, uniformity, energy, correlation and angular second moment each obtain 6 texture features, resulting in a total of 48-dimensional texture features.

In the natural scene statistical feature extraction, the present invention uses GGD (zero-mean generalized Gaussian distribution) fitting distribution for each tongue image to obtain the shape parameter α and scale parameter σ as the natural scene distribution statistical feature 1 and statistical feature 2 of the tongue image. , a total of 2-dimensional statistical characteristics of natural scenes.

We also use AGGD (zero-mean asymmetric generalized Gaussian model) for each tongue image from horizontal adjacent (Horizontal, H), vertical adjacent (Vertical, V), diagonal adjacent (Diagonal Direction, D ₁ , D ₂ ) Fit the statistical distribution of adjacent coefficients of natural images in four directions, and extract the statistical coefficient distributions of the above four adjacent natural scenes:

Shape parameters Mean parameter/>

left scale parameter and right scale parameters/>

A total of 16-dimensional statistical characteristics of natural scenes.

Therefore, by combining the extraction of each tongue body image with GGD and AGGD, a total of 18-dimensional natural scene statistical features are extracted. In order to obtain richer statistical features of natural scenes of tongue images and characterize changes in statistical characteristics of tongue image quality under natural scenes, this paper uses two different scale feature forms, including original images and downsampled images that downsample the image once.

Specifically, the tongue image is first down-sampled to obtain the down-sampled tongue image; then the GGD fitting distribution method is used to characterize the shape parameter α and scale parameter σ on the segmented tongue image and the down-sampled tongue image. Extract and obtain multi-dimensional natural scene distribution statistical features used to describe whether the tongue image is distorted.

Among them, the specific calculation of GGD is as follows:

in,

In the above formula: x is the mean subtracted contrast normalized (MSCN) coefficient corresponding to the brightness value of a certain pixel block in the natural image (because the normalized brightness value of the natural image approaches unity Gaussian characteristics, so the MSCN coefficient is often used to describe the normalized brightness value of the tongue image); α controls the shape of the distribution; σ is the standard deviation, which controls the variance; β is the calculation intermediate variable; t is the natural value of the gamma function Γ(·) Variable, the integration interval is [0,+∞).

After that, the AGGD fitting distribution method is used to classify the tongue image and the down-sampled tongue image from horizontal adjacent (Horizontal, H), vertical adjacent (Vertical, V), and diagonal adjacent (Diagonal Direction, respectively). D ₁ , D ₂ ) fit the statistical distribution of adjacent coefficients of natural images in four directions, and extract the shape parameter η, mean parameter μ, and left scale parameter of the above four adjacent natural scene statistical coefficient distributions. and right scale parameters/> The multi-dimensional natural scene edge distribution statistical characteristics used to describe whether the tongue image is distorted are obtained.

The specific calculation of AGGD is as follows:

in,

In the above formula: x is the mean subtracted contrast normalized (MSCN) coefficient corresponding to the brightness value of a certain pixel block in the natural image; eta controls the shape of the distribution, and the left and right scale parameters The variances on the left and right sides of the distribution are controlled respectively, β _l and β _r are both calculation intermediate variables, and μ is the mean parameter.

Then the fitted model parameters are extracted to form the following 16-dimensional natural scene edge distribution statistical characteristics:

Finally, for each tongue image, a total of natural scene statistical features in the 36-dimensional spatial domain are obtained.

In color feature extraction, in order to meet practical application requirements, the RGB color space of the tongue image needs to be converted into other color spaces. This invention uses the color function in the scikit-image library to perform color space transformation on the RGB tongue image, transforms the input RGB tongue image into YIQ, YcbCr, HSV, CIELAB color space images, and extracts the grayscale in each type of color space respectively. components to obtain 4-dimensional tongue image color features.

Among them, RGB represents the color of the three channels of red, green, and blue; YIQ color space is derived from the YUV color space and aims to take advantage of human color response characteristics. Among them, Y represents brightness (Brightness), indicating the brightness of each color. A brightness signal, I represents the in-phase color, the color ranges from orange to cyan, Q represents the quadrature-phase color, the color ranges from purple to yellow-green; YCbCr is a scaled and shifted version of the YUV color space , where Y is the brightness signal, Cb and Cr both refer to color, but they are expressed in different ways; HSV is a color space created based on the intuitive characteristics of color, HSV refers to hue (Hue), saturation (Saturation), Lightness (Value); the CIELAB color model is designed based on people's perception of color. Its numerical value describes all the colors that people with normal vision can see. It consists of three elements: brightness (L), color channels a and b, The colors a includes are from dark green (low brightness value) to gray (medium brightness value) to bright pink (high brightness value); b is from bright blue (low brightness value) to gray (medium brightness value) to yellow (high brightness value).

So far, for each input tongue image to be tested, a total of 88-dimensional shallow features have been extracted, namely 48-dimensional texture features based on gray-level co-occurrence matrix, natural scene statistical features in the 36-dimensional spatial domain, and 4-dimensional color features.

Regarding deep semantic features, as shown in Figure 9, the extraction of deep semantic features is to add a fully connected layer with multiple neurons to the ResNet-34 network model, input the tongue body image data set to the ResNet-34 network model, and output The neuron features of the fully connected layer are used as multi-dimensional deep semantic features of the tongue body image. The number of neurons in the fully connected layer is the same as the number of dimensions of the shallow features.

Specifically, the annotated tongue body image training set is input into the ResNet-34 network for model training. During the training process, the batch size of each iteration is 10, and the training is completed after 300 epochs. The verification set is then used for verification optimization, and repeated The above process is carried out until the classification accuracy requirements are met and the trained ResNet-34 network is obtained. For the tongue image data to be evaluated, the 88-dimensional feature values output by the fully connected layer are extracted as the deep semantic features of the tongue image.

For the tongue image quality classification module, the classification module is used to normalize and fuse shallow features and deep semantic features in series, and perform image classification based on the fused feature vectors to obtain tongue image quality assessment results.

First, in order to eliminate the dimensional influence between different types of feature values, the extracted feature data needs to be normalized, and the linear normalization method is used to scale the data to between [-1,1]. The formula is as follows:

In the formula, x represents the original data, x' represents the linearly normalized data, x _max represents the maximum value in the original data, x _min represents the minimum value, and x _mean represents the mean.

Then, a series fusion method is used to fuse the extracted 88-dimensional shallow features and 88-dimensional deep semantic features to obtain a new 176-dimensional tongue image feature vector;

Finally, the data sets of all categories are combined in pairs to form multiple two-classification tasks; for each of the two-classification tasks, the shallow feature summation is performed on the input tongue image data based on the tongue image quality classification module. Feature extraction, normalization processing and serial fusion of the deep semantic features; training the corresponding two-class support vector machine SVM model according to the fused feature vectors; training multiple two-class support vectors based on the tongue image data set machine model, and the number of SVM models obtained is equal to the number of pairwise combinations of categories in the data set; input the tongue image data to be tested into all the trained SVM models to obtain multiple binary classification results; and then classify the results Use a relative majority voting strategy to make decisions and complete multi-classification evaluation tasks.

Among them, the support vector machine model specifically uses Gaussian kernel function (RBF kernel function), the kernel function coefficient is 0.1, and the error term penalty factor C is 1.

The RBF kernel function formula is as follows:

In the formula, K( _xi ,x _j ) represents the kernel function, δ is the broadband of the Gaussian kernel, and its value is greater than zero.

At this point, the establishment of a TCM tongue image quality assessment model based on deep learning and feature fusion has been completed.

S3: Input the tongue image to be tested into the trained TCM tongue image quality assessment model.

Specifically, the tongue image to be tested is input into the above-mentioned TCM tongue image quality assessment model, the tongue image quality is assessed, and the final assessment result is obtained. If it is a high-quality tongue image, the image is directly saved for subsequent tongue image processing; Otherwise, it will prompt that there is a quality problem with the tongue image.

To sum up, in the above embodiments, the proposed TCM tongue image quality assessment based on deep learning and feature fusion is divided into two stages: TCM tongue body segmentation based on YOLOv5 and ^U2 -Net and deep and shallow tongue image-based Quality assessment of traditional Chinese medicine tongue images using feature fusion. In the first stage, the frame coordinates of the tongue target area are first obtained based on the YOLOv5 network, and the tongue target area containing the complete tongue is extracted; then the target area is input into the U ² -Net network, and the complete tongue is segmented from it Tongue body image is used as input data for subsequent tongue image quality assessment. In the second stage, a total of 88-dimensional shallow features are extracted from the tongue image based on the texture features based on the gray level co-occurrence matrix, features based on natural scene statistics, and color features, and 88 neurons are added in front of the ResNet-34 output layer. The fully connected layer is used to obtain the 88-dimensional deep semantic features of the tongue image. After normalizing the shallow features and deep semantic features and fusing the concatenated features, the support vector machine model is trained, and the quality assessment of the TCM tongue image is finally completed.

As can be seen from the above, the basic contents of the first aspect of the present invention are as follows. A system applying the method of the first aspect will be given below. This solution provides a tongue image quality assessment system based on deep learning and feature fusion.

Specifically, it includes a data processing unit for segmenting the tongue image to obtain a segmented target tongue image; a controller construction unit for constructing a tongue image quality assessment model based on deep and shallow feature fusion; and a calculation unit for converting all the tongue images into The tongue body image is input to the tongue image quality assessment model for evaluation, and a tongue image quality assessment result is obtained.

The above is the preferred embodiment of the present invention. It should be pointed out that for those of ordinary skill in the art, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also regarded as It is the protection scope of the present invention.

Claims (10)

1. The tongue image quality evaluation method based on deep learning and feature fusion is characterized by comprising the following steps of:

constructing a tongue image dataset;

training the evaluation model based on the tongue image data to obtain a trained evaluation model;

inputting the tongue image to be tested into the trained evaluation model;

the evaluation model comprises a tongue segmentation module, a feature extraction module and a tongue image classification module;

dividing the tongue image to be detected based on the tongue dividing module to obtain a target tongue image;

respectively carrying out feature extraction processing on the target tongue body image based on the feature extraction module to obtain shallow features and deep semantic features;

wherein the shallow features at least comprise texture features, natural scene statistical features and color features;

and carrying out normalization and series fusion on the shallow layer features and the deep semantic features based on the tongue image classification module, and carrying out image classification according to the fused feature vectors to obtain tongue image quality assessment results.

2. The tongue image quality evaluation method based on deep learning and feature fusion according to claim 1, wherein the shallow feature extraction step specifically comprises:

converting the target tongue image into a gray level co-occurrence matrix, and obtaining multidimensional texture features of the target tongue image by calculating the gray level co-occurrence matrix;

carrying out statistic feature extraction of natural images on the target tongue image to obtain multidimensional natural scene statistic features of the target tongue image;

and performing color space conversion on the target tongue image to obtain multi-dimensional color characteristics of the target tongue image.

3. The tongue image quality assessment method based on deep learning and feature fusion according to claim 2, wherein the texture features include contrast, dissimilarity, homogeneity, energy, correlation and angular second moment, which are specifically calculated as follows:

the calculation formula of the contrast ratio is as follows:

the calculation formula of the dissimilarity is as follows:

the calculation formula of the uniformity is as follows:

the energy calculation formula:

the calculation formula of the correlation is as follows:

wherein,

the calculation formula of the angular second-order array comprises the following steps:

wherein i and j represent the gray values corresponding to the pixel (x, y) and the pixel (x+Δx, y+Δy), respectively; p (i, j, d, θ) represents the probability that a pixel (x, y) with a gray level i and a pixel (x+Δx, y+Δy) with a gray level j are present at the same time; d is the separation distance between two pixels, d is taken as 1 and 2; θ is the generation direction of the gray matrix, and θ takes four directions of 0 °, 45 °, 90 ° and 135 °; n is the number of gray levels of the pixel.

4. The tongue image quality evaluation method based on deep learning and feature fusion according to claim 2, wherein the step of extracting the statistical features of the natural image from the target tongue image to obtain the multi-dimensional natural scene statistical features of the target tongue image comprises the steps of:

downsampling the target tongue image to obtain a downsampled tongue image;

the feature extraction of the shape parameter alpha and the scale parameter sigma is respectively carried out on the target tongue image and the downsampled tongue image by using a GGD fitting distribution method, so as to describe whether the tongue image is distorted or not;

performing four-direction fitting shape parameters eta, mean parameter mu and left scale parameter of horizontal adjacent, vertical adjacent and two pairs of diagonal adjacent on the target tongue image and the downsampled tongue image by using an AGGD fitting distribution methodAnd right scale parameter->The feature extraction of the tongue image is used for obtaining multi-dimensional natural scene edge distribution statistical features for describing whether the tongue image is distorted;

wherein,

the specific calculation formula of GGD is as follows:

in the formula: x is the average value loss contrast normalization coefficient corresponding to the brightness value of a certain pixel block in the natural image; the α controls the shape of the distribution; sigma is standard deviation, and the variance is controlled; beta is a calculated intermediate variable; t is the argument of gamma function Γ (), the integration interval is set to be 0, ++ infinity a) is provided;

the specific calculation formula of AGGD is as follows:

in the formula: x is the average value loss contrast normalization coefficient corresponding to the brightness value of a certain pixel block in the natural image; η controls the shape of the distribution, the left and right scale parametersControlling the variance of the left and right sides of the distribution, beta _l And beta _r All are calculated intermediate variables, and mu is a mean parameter.

5. The tongue image quality assessment method based on deep learning and feature fusion according to claim 2, wherein,

and converting the color space of the target tongue image into YIQ, ycbCr, HSV and CIELAB space images, and extracting gray components in YIQ, ycbCr, HSV and CIELAB space images to obtain the multi-dimensional color characteristics of the target tongue image.

6. The tongue image quality assessment method based on deep learning and feature fusion according to claim 1, wherein the deep semantic feature extraction step specifically comprises:

adding a full-connection layer with a plurality of neurons to a ResNet-34 network model, inputting a data set of the tongue image to the ResNet-34 network model, and outputting neuron characteristics of the full-connection layer as multidimensional deep semantic characteristics of the tongue image;

wherein the number of neurons of the fully connected layer is the same as the number of dimensions of the shallow features.

7. The tongue image quality assessment method based on deep learning and feature fusion according to claim 1, wherein the tongue segmentation module comprises a YOLOv5s network model and a U ² -Net network model;

inputting the collected tongue image into the YOLOv5s network model, and extracting a target area containing a complete tongue body;

inputting the target area into the U ² -Net network model, segmenting and extracting complete tongue image, obtaining target tongue image.

8. The tongue image quality evaluation method based on deep learning and feature fusion according to claim 1, wherein the step of performing series fusion on the shallow features and the deep semantic features based on the tongue image classification module and performing image classification according to the fused feature vectors to obtain a tongue image quality evaluation result specifically comprises the following steps:

the dataset includes a plurality of categories;

combining all types of data sets pairwise to form a plurality of classification tasks;

aiming at each classification task, carrying out series fusion on the shallow layer features and the deep semantic features based on the tongue image classification module;

training a corresponding bi-classification Support Vector Machine (SVM) model according to the fused feature vectors;

training a plurality of two-classification support vector machine models based on the tongue image data set, wherein the number of the obtained SVM models is equal to the number of the two-to-two combination of the data set types;

respectively inputting tongue image data to be detected into all trained SVM models to obtain a plurality of classification results;

and adopting a relative majority voting strategy to make a decision on the classification result, and completing the multi-classification evaluation task.

9. The tongue image quality assessment method based on deep learning and feature fusion according to claim 1, further comprising the steps of, before the multi-dimensional shallow features and the multi-dimensional deep semantic features are fused in series:

and carrying out linear normalization processing on the multidimensional shallow layer features and the multidimensional deep semantic features to obtain data scaled to [ -1,1 ].

10. A tongue image quality evaluation system based on deep learning and feature fusion, characterized in that the tongue image quality evaluation method based on deep learning and feature fusion according to any one of claims 1 to 9 is applied, comprising:

the data processing unit is used for carrying out segmentation processing on the tongue image to obtain a target tongue image;

the controller building unit is used for building a tongue image quality evaluation model based on depth feature fusion;

the computing unit is used for inputting the target tongue image into the tongue image quality evaluation model for evaluation, and obtaining a tongue image quality evaluation result.