CN117576405A - Tongue picture semantic segmentation method, device, equipment and medium - Google Patents

Abstract

The invention relates to a tongue picture semantic segmentation method, a tongue picture semantic segmentation device, tongue picture semantic segmentation equipment and a tongue picture semantic segmentation medium. The method comprises the steps of obtaining a target tongue picture and carrying out feature extraction treatment to obtain a shallow feature picture; carrying out multi-scale feature extraction processing on the shallow feature map to obtain a multi-scale feature map, and carrying out channel dimension reduction on the multi-scale feature map to obtain an integrated feature map; performing channel dimension reduction on the shallow feature map to obtain a first feature map, upsampling the integrated feature map to obtain a second feature map, and performing feature fusion on the first and second feature maps to obtain a deep feature map; and inputting the shallow and deep feature images into an image segmentation model to obtain a segmentation prediction result. The method can better identify and classify the objects, reduce the complexity of the model, improve the running speed and accuracy, better understand and identify the image content, improve the semantic segmentation of tongue picture, provide more accurate and objective diagnosis basis, and facilitate the diagnosis and health detection of traditional Chinese medicine.

Description

Tongue image semantic segmentation methods, devices, equipment and media

Technical field

The invention is applicable to the technical field of image processing, and in particular relates to a tongue image semantic segmentation method, device, equipment and medium.

Background technique

Semantic segmentation is a technology in the field of computer vision. Its main purpose is to segment an image into different regions and label these regions as different semantic categories. Compared with traditional image segmentation techniques, semantic segmentation can more accurately understand different regions in an image and match them with corresponding semantic categories. Therefore, semantic segmentation technology has broad application prospects in many visual scenarios.

In recent years, tongue image segmentation has become a research hotspot in the field of modernization of traditional Chinese medicine. Many scholars have conducted related research on tongue image segmentation and tried many segmentation methods. Tongue image segmentation has also achieved some research results. The existing DeepLabV3+ model performs semantic segmentation, but this model has a large number of parameters, low edge segmentation accuracy, and cannot achieve good segmentation results.

Therefore, how to improve existing models and improve the semantic segmentation effect of tongue images has become an urgent problem to be solved.

Contents of the invention

In view of this, embodiments of the present invention provide a tongue image semantic segmentation method, device, equipment and medium to solve the problem of how to improve the existing model and improve the tongue image semantic segmentation effect.

In a first aspect, a tongue image semantic segmentation method is provided. The tongue image semantic segmentation method includes:

Obtain the target tongue image, perform feature extraction preprocessing on the target tongue image, and obtain the shallow feature map of the target tongue image;

Perform multi-scale feature extraction processing on the shallow feature map to obtain a multi-scale feature map of the target tongue image, perform channel dimensionality reduction processing on the multi-scale feature map, and obtain an integrated feature map of the target tongue image;

Perform channel dimensionality reduction processing on the shallow feature map to obtain the first feature map of the target tongue image. Upsample the integrated feature map to obtain the second feature map of the target tongue image. Combine the first feature map and the second feature map. Feature fusion is performed on the images to obtain the deep feature map of the target tongue image;

Input the shallow feature map and deep feature map into the preset image segmentation model for semantic segmentation tasks to obtain the segmentation prediction results of the target tongue image.

In a second aspect, a tongue image semantic segmentation device is provided. The tongue image semantic segmentation device includes:

The shallow feature map acquisition module is used to obtain the target tongue image, perform feature extraction preprocessing on the target tongue image, and obtain the shallow feature map of the target tongue image;

The integrated feature map acquisition module is used to perform multi-scale feature extraction processing on the shallow feature map to obtain the multi-scale feature map of the target tongue image, and perform channel dimensionality reduction processing on the multi-scale feature map to obtain the integrated features of the target tongue image. picture;

The deep feature map acquisition module is used to perform channel dimensionality reduction processing on the shallow feature map to obtain the first feature map of the target tongue image, upsample the integrated feature map to obtain the second feature map of the target tongue image, and The first feature map and the second feature map are feature fused to obtain the deep feature map of the target tongue image;

The prediction result generation module is used to input the shallow feature map and the deep feature map into the preset image segmentation model for the semantic segmentation task, and obtain the segmentation prediction result of the target tongue image.

In a third aspect, embodiments of the present invention provide a computer device. The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. The processor executes the The computer program implements the tongue image semantic segmentation method described in the first aspect.

In a fourth aspect, embodiments of the present invention provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the tongue image semantic segmentation as described in the first aspect is implemented. method.

In a fifth aspect, embodiments of the present invention provide a computer program product. When the computer program product is run on a computer device, it causes the computer device to execute the tongue image semantic segmentation method described in the first aspect.

Compared with the existing technology, the beneficial effects of the present invention are: through the above steps, the present invention can fully analyze the characteristics of images at different scales, better identify and classify objects, reduce the complexity of the model, and improve the operation efficiency. With its speed and accuracy, it can better understand and identify image content, effectively improve the semantic segmentation of tongue images, accurately segment various parts of the tongue, provide more accurate and objective diagnostic basis, and facilitate traditional Chinese medicine diagnosis and health testing. , reducing the impact of human factors on diagnostic results.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings in the following description are only illustrative of the present invention. For some embodiments, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of an application environment of a tongue image semantic segmentation method provided by Embodiment 1 of the present invention;

Figure 2 is a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 2 of the present invention;

Figure 3 is a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 3 of the present invention;

Figure 4 is a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 4 of the present invention;

Figure 5 is a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 5 of the present invention;

Figure 6 is a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 6 of the present invention;

Figure 7 is a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 7 of the present invention;

Figure 8 is a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 8 of the present invention;

Figure 9 is a schematic diagram of the model architecture of a tongue image semantic segmentation method provided in Embodiment 9 of the present invention;

Figure 10 is a schematic structural diagram of a tongue image semantic segmentation device provided in Embodiment 10 of the present invention;

FIG. 11 is a schematic structural diagram of a computer device provided by Embodiment 11 of the present invention.

Detailed ways

In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the present invention in unnecessary detail.

It will be understood that, when used in the description of the present invention and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements and/or components but does not exclude one or more other The presence or addition of features, integers, steps, operations, elements, components and/or collections thereof.

It will also be understood that the term "and/or" as used in the specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

As used in the description of the present invention and the appended claims, the term "if" may be interpreted as "when" or "once" or "in response to determining" or "in response to detecting" depending on the context. ". Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, to mean "once determined" or "in response to a determination" or "once the [described condition or event] is detected ]" or "in response to detection of [the described condition or event]".

In addition, in the description of the present specification and appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description and shall not be understood as indicating or implying relative importance.

Reference in the specification of the invention to "one embodiment" or "some embodiments" or the like means that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the invention. Therefore, the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in other embodiments", etc. appearing in different places in this specification are not necessarily References are made to the same embodiment, but rather to "one or more but not all embodiments" unless specifically stated otherwise. The terms “including,” “includes,” “having,” and variations thereof all mean “including but not limited to,” unless otherwise specifically emphasized.

It should be understood that the sequence number of each step in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

In order to illustrate the technical solution of the present invention, specific examples will be described below.

The tongue image semantic segmentation method provided in Embodiment 1 of the present invention can be applied in the application environment as shown in Figure 1, in which the client communicates with the server. Among them, clients include but are not limited to handheld computers, desktop computers, notebook computers, ultra-mobile personal computers (UMPC), netbooks, cloud computer equipment, personal digital assistants (personal digital assistants, PDAs), etc. Computer equipment. The server can be implemented as an independent server or a server cluster composed of multiple servers.

Refer to Figure 2, which is a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 2 of the present invention. The above tongue image semantic segmentation method can be applied to the client in Figure 1. The user uses the client to perform a tongue body image Analysis, the above tongue image semantic segmentation method is executed in the client to output the result of tongue image semantic segmentation. As shown in Figure 2, the tongue image semantic segmentation method can include the following steps:

Step S201: Obtain a target tongue image, perform feature extraction preprocessing on the target tongue image, and obtain a shallow feature map of the target tongue image.

Among them, the target tongue image can be obtained from public databases or studies, or can be obtained directly by collecting images of the patient's tongue through medical equipment, or can be obtained by the user manually taking pictures and uploading them.

The shallow feature map is a feature map used for preliminary processing of the target tongue image. This shallow feature map can specifically show the detailed features of the original tongue image. The "shallow layer" is to define a name, rather than having an essence of the feature map. According to the above limitation, the shallow feature map is an image obtained by performing feature extraction preprocessing on the camera image.

Perform feature extraction preprocessing on the obtained target tongue image, which may include steps such as resizing, color normalization, and denoising, in order to perform feature extraction. The extracted features usually include: color, texture, shape and other information. After feature extraction preprocessing, an image mainly represented by some features can be obtained. Specifically, it can be an image expressed in the form of a matrix.

Step S202: Perform multi-scale feature extraction processing on the shallow feature map to obtain a multi-scale feature map of the target tongue image, and perform channel dimensionality reduction processing on the multi-scale feature map to obtain an integrated feature map of the target tongue image.

Among them, multi-scale feature extraction refers to processing the shallow feature map multiple times, using different scales or parameters for each processing, and finally merging the different processing results to obtain the multi-scale feature map of the target tongue image. When performing multi-scale feature extraction processing on shallow feature maps, the shallow feature maps can be reduced or enlarged to varying degrees to obtain image feature information at different scales. The network can also be adjusted by adjusting the structural parameters of the network. Depth, convolution kernel size, step size and other feature representations.

A large number of feature channels may be generated during the multi-scale feature extraction process, which will increase the complexity and computational burden of subsequent processing. Therefore, channel dimensionality reduction processing is required to reduce the number of feature channels while retaining key information. The integrated feature map contains important information about the tongue image at different scales and channels, which can provide a comprehensive feature representation for subsequent analysis.

Among them, channel dimension processing refers to the process of dimensionality reduction of data channels, and channels usually refer to the number of feature maps in the input data. The purpose of channel dimensionality reduction is to reduce the complexity and number of parameters of the model, thereby improving the training and inference efficiency of the model. At the same time, it can effectively prevent overfitting.

Step S203: Perform channel dimensionality reduction processing on the shallow feature map to obtain the first feature map of the target tongue image, upsample the integrated feature map to obtain the second feature map of the target tongue image, and combine the first feature map with The second feature map performs feature fusion to obtain the deep feature map of the target tongue image.

The first feature map refers to the image or graph expression obtained after channel dimensionality reduction processing of the shallow feature map. For example, the shallow feature map is a 3-channel feature map, and channel dimensionality reduction processing can reduce it to 2 channel or 1 channel to obtain the corresponding first feature map.

The second feature map refers to the image or graph representation obtained by upsampling the integrated feature map. The integrated feature map is the image or graph expression obtained after channel dimensionality reduction processing of the multi-scale feature map. It should be understood that the size of the integrated feature map is smaller than the first feature map, so it needs to be upsampled, and a second feature map that is consistent in size with the first feature map is obtained through interpolation or permutation convolution.

The fusion of two feature maps of the same size can combine the information of the first feature map and the second feature map to obtain a richer and multi-dimensional deep feature map. Among them, the fusion can use additive fusion, splicing fusion, and weighted fusion. Fusion and other methods are carried out, and these fusion methods can be based on.

The deep feature map not only contains the texture and color information of the original target tongue image, but may also contain semantic information after multi-level processing and fusion.

Step S204: Input the shallow feature map and the deep feature map into a preset image segmentation model for semantic segmentation tasks to obtain segmentation prediction results of the target tongue image.

The image segmentation model can be a pre-trained model, which includes a machine learning model, a neural network model, etc. During training, the collected tongue images are selected as a training set, so that the trained model can be used for tongue segmentation. . In this embodiment, it is specifically embodied in accurately segmenting different areas of the tongue image image, for example, segmenting the tongue body and background in the tongue image image, extracting tongue quality, etc.

By inputting the deep feature map into the image segmentation model, preliminary segmentation prediction results can be obtained. Since global features may not be considered in the image segmentation process of deep features, resulting in missing features, therefore, in order to avoid missing features, shallow layers are used. The segmentation prediction results generated by the feature map are trimmed to improve the segmentation results and improve the integrity of the tongue images.

The tongue image semantic segmentation method in this embodiment fully understands the characteristics of images at different scales through multi-scale extraction processing, and better identifies and classifies objects. The application of dimensionality reduction processing reduces the complexity of the model and improves operation efficiency. The speed and accuracy combine shallow and deep information to better understand and identify image content, effectively improve the semantic segmentation of tongue images, accurately segment various parts of the tongue, and provide a more accurate and objective diagnosis basis. It facilitates traditional Chinese medicine diagnosis and health testing, and automatically analyzes and processes target tongue images through machine learning and deep learning, reducing the impact of human factors on diagnostic results.

Refer to Figure 3, which is a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 3 of the present invention. As shown in Figure 3, in step S202, the shallow feature map is subjected to multi-scale feature extraction processing to obtain the target tongue image. Multi-scale feature maps of images, including:

Step S301: Pass the shallow feature map through a pooling layer and at least one convolution layer respectively.

Among them, the shallow feature map needs to go through a pooling layer and a convolution layer or multiple convolution layers. The pooling layer is usually used for downsampling to reduce the dimension of the feature map while retaining important information, reducing the amount of calculation and the risk of overfitting, and improving higher-level features; the detailed information in the image can be captured through the convolution layer. Multiple convolutional layers can be connected to further extract features, helping to better learn and understand complex features in tongue images.

Step S302, when the number of convolutional layers is greater than two layers, each convolutional layer is used to perform dilation convolution processing with different dilation rates on the shallow feature map to obtain each convolutional feature map of the target tongue image.

Among them, dilated convolution is a special convolution operation. By introducing the expansion rate in the convolution process, the step size of the convolution kernel expansion on the input feature map can be controlled. When there are multiple convolution layers, it can be It is necessary to set different expansion rates so that each convolutional layer can expand the receptive field in different ways, thereby extracting features from different angles and scales. The dilation convolution process for each dilation rate can generate a convolution feature map.

Step S303: The pooling layer is used to perform average pooling processing on the shallow feature map to obtain the pooled feature map of the target tongue image.

Global average pooling of shallow feature maps can help further reduce the dimensionality of feature maps, retain important information, and provide a more concise and efficient feature representation for subsequent image segmentation and classification tasks.

Step S304: Feature fusion is performed on each convolution feature map and the pooling feature map to obtain a multi-scale feature map.

Feature fusion is to combine feature maps from different levels to obtain multi-scale feature representation, fuse each processed feature map, and fuse the embodied features together to form a feature map with multiple scales. The specific fusion method can be to concatenate the feature maps of different layers in the channel dimension, or to make the size of the feature maps consistent through upsampling and downsampling.

For example, set up four convolutional layers and one pooling layer, the four convolutional layers are 1×1 convolutional layers, and the 3×3 dilated convolutional layers with expansion rates of 3, 6, and 9 respectively. Each convolutional or pooling layer generates a processed feature map.

The tongue image semantic segmentation method in this embodiment captures multi-scale features by using pooling and convolution with different expansion rates, enabling better understanding and analysis of tongue image images, reducing the dimensionality of the feature map, and reducing calculations. resources and time, improve processing efficiency, and have certain flexibility in the selection of convolutional layers.

Refer to Figure 4, which is a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 4 of the present invention. As shown in Figure 4, in step S302, dilation convolution processing with different expansion rates is performed on the shallow feature map to obtain the target tongue After each convolution feature map of the image, it also includes:

Step S401: For each convolution layer, calculate the channel dimension attention weight of the convolution feature map according to the preset channel attention module, and calculate the spatial dimension attention of the convolution feature map according to the preset spatial attention module. Weights.

Each convolution layer can output a convolution feature map. The output convolution feature map is input into the attention fusion module to calculate the attention weight in the channel dimension of the convolution feature map and the attention in the spatial dimension. Weights. Methods for calculating channel dimension attention weight include global average pooling, self-attention mechanism, etc. Methods for calculating spatial dimension attention weight include position embedding, spatial pyramid pooling, etc.

Step S402, use the channel dimension attention weight and the spatial dimension attention weight to process the convolution feature map to obtain a weighted feature map of the convolution feature map.

The calculated attention weights are multiplied with the convolutional feature map, and these weights are used to adjust the output of the convolutional layer to obtain a weighted feature map of the convolutional feature map. Specifically, the weight can be multiplied by the channel feature to obtain the weighted channel feature. Similarly, for each position in the feature map, the attention weight of the spatial dimension can be used to weight the features of the position.

The tongue image semantic segmentation method in this embodiment uses attention weights to perform weighted calculations on the convolution feature map, which can better capture the results and semantic information of the input data, help the model better understand and process the input data, and improve improve the performance and accuracy of the model.

Refer to Figure 5, which is a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 5 of the present invention. As shown in Figure 5, in step S402, the convolution feature map is processed using the channel dimension attention weight, including :

Step S501, perform an average pooling operation on the convolutional feature map to obtain a first pooling result.

Perform an average pooling operation on the convolution feature map, and perform an average operation on each local area in the feature map to obtain the average value in the area as the output. Each pixel in the feature map can be replaced with an average value. , thereby reducing the dimensionality of the feature map.

Step S502: Perform a maximum pooling operation on the convolutional feature map to obtain a second pooling result.

Maximum pooling selects the maximum value as the output within the area covered by the pooling kernel. Compared with average pooling, maximum pooling replaces each pixel with the maximum value in the area and pays more attention to extracting mutations in the feature map. information.

Step S503: Input the first pooling result and the second pooling result into the first multi-layer perceptron to obtain the first output feature map and the second output feature map respectively.

The first multi-layer perceptron (MLP) further processes and analyzes the first output feature map and the second output feature map respectively. The specific operation process is: perform layer-by-layer convolution on the input feature map. and pooling operations to extract deeper features; use activation functions in each layer to perform nonlinear transformation on the convolution results to increase the expressive ability of the model; in the last layer, flatten the feature map into a one-dimensional vector, and Perform a full connection operation and obtain the output result.

Step S504: Add the first output feature map and the second output feature map to obtain the channel dimension attention weight.

Merge the first output feature map and the second output feature map to obtain a richer feature fusion result, perform a weighted sum for each channel, and determine the attention weight of the channel dimension, thereby emphasizing important channels and keeping unimportant ones. aisle. The calculated channel dimension attention weight can be further used to guide the model's attention allocation to the feature map, improving the performance and generalization ability of the model.

Step S505: Multiply the channel dimension attention weight and the convolution feature map to obtain the channel attention mechanism fusion feature map.

The channel attention mechanism is a method that allows the model to focus on important channel information. By assigning different weights to each channel, the model can pay more attention to important features, thereby improving the performance of the model. Multiply the obtained channel dimension attention weight with the convolutional feature map, so that the model can better focus on important channels in the feature map and suppress irrelevant or redundant channels, thereby improving the performance and generalization of the model. ability.

The tongue image semantic segmentation method in this embodiment performs attention fusion in the channel dimension, which effectively improves the representation ability and classification accuracy of the model. The combination of average pooling and maximum pooling reduces the dimension of the feature map while retaining Important information in the image enables better segmentation of tongue images.

Referring to Figure 6, it is a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 6 of the present invention. As shown in Figure 6, after obtaining the channel attention mechanism fusion feature map in step S505, it also includes:

Step S601: Perform channel-based maximum pooling and average pooling operations on the channel attention mechanism fusion feature map to obtain maximum pooling results and average pooling results respectively.

Common pooling methods are channel-based max pooling and average pooling operations. Max pooling selects the maximum value within the pooling window as the output, while average pooling calculates the average value within the pooling window as the output. Reduce the size of feature maps and preserve important feature information. The specific operation is to divide the feature map along the channel direction so that each channel corresponds to a feature map, and then perform maximum pooling and average pooling on the feature map of each channel to obtain a new feature vector, that is, maximum pooling. pooling results and average pooling results.

Step S602: Combine the maximum pooling result and the average pooling result to obtain a spatial attention combined feature map.

Specifically, the maximum pooling result and the average pooling result are flattened into one-dimensional vectors respectively, and the two one-dimensional vectors are spliced together to form an information feature vector. This feature vector contains the channel attention mechanism fusion feature map in The maximum and average information on each channel are adjusted to this new feature vector to form a new spatial attention merged feature map.

Step S603: Perform convolution activation on the spatial attention combined feature map to obtain the spatial dimension attention weight.

To perform a convolution operation on the spatial attention merged feature map, you can use one convolution kernel for one convolution, or use multiple convolution kernels for multiple convolutions. The convolution operation can extract local features in the feature map and generate For the new feature map, after the convolution operation, the activation function is used to nonlinearly transform the feature map, and the sigmoid activation function is commonly used for processing. Under the action of the activation function, each pixel in the feature map will be assigned a value, which represents the attention weight of the pixel in the spatial dimension.

Step S604: Multiply the spatial dimension attention weight and the channel attention mechanism fusion feature map to obtain a weighted feature map.

The calculated spatial dimension attention weight should have the same size as the channel attention mechanism fusion feature map, and be multiplied element by element, that is, the feature map of each channel is weighted, so that the model can pay more attention to important spatial features. Element-wise multiplication is performed to obtain a weighted feature map, in which the value of each pixel is the product of the corresponding channel attention weight and the spatial dimension attention weight.

The tongue image semantic segmentation method in this embodiment improves the representation ability of the feature map. The combination of maximum pooling and average pooling reduces the calculation amount of subsequent convolution operations, improves the operating efficiency of the model, and can better process A variety of different image data improves the generalization ability of the model.

Referring to Figure 7, a schematic flow chart of a tongue image semantic segmentation method provided in Embodiment 7 of the present invention is shown. As shown in Figure 7, in step S204, the shallow feature map and the deep feature map are input into the preset for The image segmentation model for the semantic segmentation task obtains the segmentation prediction results of the target tongue image, including:

Step S701: Perform convolution feature extraction on the deep feature map to obtain a feature representation result of the deep feature map.

By further processing the deep feature map, the deep feature map after convolution feature extraction can obtain a more effective feature representation, and the local area of each feature map is convolved to extract more discriminative features.

Step S702: Perform point sampling around the pixel points of the feature representation result to obtain several preset sampling points.

Among them, the preset sampling points make the feature representation results relatively fuzzy and the points that cannot clearly determine the semantic information can be selected evenly around the pixel points. The specific sampling process can be designed according to actual needs to control the density and distribution of sampling points to ensure It can fully cover the important information in the feature representation results. It should be noted that the number and distribution of sampling points need to be selected and adjusted according to the characteristics of the specific task and data set to ensure the best results.

Step S703: Obtain the feature vector of each preset sampling point according to the feature representation result.

It should be understood that the feature vector of each preset sampling point can be extracted from the feature representation result, and the appropriate feature vector extraction method and parameter settings can be selected according to the actual situation to improve the performance and accuracy of the classification task.

In one implementation, as shown in Figure 8, obtaining the feature vector of each preset sampling point includes:

Step S801, obtain the point coordinates of each preset sampling point according to the feature representation result;

Step S802, sample the shallow feature map according to the point coordinates to obtain the low-level feature vector of the preset sampling point;

Step S803, sample the feature representation result according to the point coordinates to obtain the high-level feature vector of the preset sampling point;

Step S804: Merge the low-level feature vector and the high-level feature vector to obtain the feature vector of the preset sampling point.

Among them, the position of the preset sampling point (that is, the point coordinates of the preset sampling point) is determined from the feature representation result. Since the size of the shallow layer feature map is consistent with the size of the feature representation result, the shallow layer can be obtained based on the point coordinates. The feature vector at the corresponding position in the feature map is the low-level feature vector of the preset sampling point. According to the point coordinates, the feature vector at the corresponding position in the feature representation result can be obtained, which is the high-level feature vector. The low-level feature vector of the same preset sampling point The vector and the high-level feature vector are combined to obtain the feature vector of the preset sampling point.

Through the fusion of high-level feature vectors and low-level feature vectors, different levels of feature information can be obtained to better understand the essential characteristics of images or data, effectively improve the representation ability of features, and be able to handle different tasks and data sets more flexibly. Improve the comprehensiveness and accuracy of feature representation and enhance the generalization ability of the model.

Step S704: Calculate the feature vector through the second multi-layer perceptron to obtain a segmentation prediction result.

The multi-layer perceptron is composed of multiple neurons. Each neuron receives an input signal and outputs a value. Through training, the multi-layer perceptron can learn the mapping relationship from the input feature vector to the output segmentation result. The specific training process can be carried out using optimization algorithms such as the back propagation algorithm. By continuously adjusting the weight parameters of the second multi-layer perceptron, the error between the prediction results and the actual segmentation results can be minimized. It should be noted that the structure and parameter settings of the second multi-layer perceptron need to be selected and adjusted according to the characteristics of the specific task and data set. After calculation, the segmentation prediction result is finally obtained with the same size as the original image.

In one embodiment, the obtained segmentation prediction results can also be subjected to iterative repair processing to continuously optimize the segmentation results to achieve more accurate results. By continuously updating the segmentation results and gradually approaching the optimal solution, the weight parameters of the second multi-layer perceptron can be adjusted according to the difference between the current segmentation prediction results and the actual segmentation results to obtain more accurate segmentation results. Then, the more accurate segmentation results can be obtained. The segmentation result is used as a new input feature vector, and the calculation of the second multi-layer perceptron is performed again to obtain further optimized segmentation prediction results. In this way, the error between the segmentation prediction results and the actual segmentation results is gradually reduced, and the accuracy and stability of the segmentation are improved. In addition, the number of iterative repair processes and parameter settings need to be selected and adjusted according to the characteristics of the specific task and data set to achieve the best segmentation effect.

As shown in Figure 9, it is a schematic diagram of the model architecture of the tongue image semantic segmentation method. In Figure 9, a lightweight network MobileNet V2 is input to obtain a shallow feature map, and the shallow feature map is input into a 1×1 convolution. layer (Conv), a 3×3 convolutional layer with 3x expansion rate, a 3×3 convolutional layer with 6x expansion rate, a 3x3 convolutional layer with 9x expansion rate, and an average The pooling layer processes and outputs feature maps separately, and fuses these feature maps to obtain multi-scale feature maps. After the shallow feature map passes through the convolution layer, CBAM (Convolutional Block Attention Module, convolutional attention mechanism module) can be used to process the output feature map to incorporate channel attention and spatial attention. The feature map of the shallow feature map that passes through the pooling layer does not need to be processed by CBAM. Perform channel dimensionality reduction on the multi-scale feature map to obtain an integrated feature map. Merge the upsampled integrated feature map (i.e. the second feature map) and the shallow feature map after channel dimensionality reduction (i.e. the first feature map). (Concat), get the deep feature map. After the deep feature map and shallow feature map of the convolution layer are input to the point rendering (Pointrend) module, the output is sorted and the segmentation prediction results of the target tongue image are obtained.

The tongue image semantic segmentation method in this embodiment selects some preset sampling points for feature vector calculation, and the obtained segmentation prediction results are more accurate, which improves the accuracy and stability of segmentation and facilitates tongue image analysis in traditional Chinese medicine.

Referring to Figure 10, which is a schematic structural diagram of a tongue image semantic segmentation device provided in Embodiment 10 of the present application. Based on the above tongue image semantic segmentation method, the tongue image semantic segmentation device in Embodiment 10 includes: shallow feature map acquisition Module 101 is used to obtain the target tongue image, perform feature extraction processing on the target tongue image, and obtain the shallow feature map of the target tongue image;

The integrated feature map acquisition module 102 is used to perform multi-scale feature extraction processing on the shallow feature map to obtain a multi-scale feature map of the target tongue image, and perform channel dimensionality reduction processing on the multi-scale feature map to obtain an integration of the target tongue image image. feature map;

The deep feature map acquisition module 103 is used to perform channel dimensionality reduction processing on the shallow feature map to obtain the first feature map of the target tongue image, and upsample the integrated feature map to obtain the second feature map of the target tongue image. Perform feature fusion on the first feature map and the second feature map to obtain the deep feature map of the target tongue image;

The prediction result generation module 104 is used to input shallow feature maps and deep feature maps into a preset image segmentation model for semantic segmentation tasks to obtain segmentation prediction results of the target tongue image.

Optionally, the above integrated feature map acquisition module 102 includes:

The feature map processing submodule is used to pass the shallow feature map through a pooling layer and at least one convolution layer respectively;

When the number of convolutional layers is greater than two, each convolutional layer is used to perform dilation convolution processing with different expansion rates on the shallow feature map to obtain each convolutional feature map of the target tongue image;

The pooling layer is used to average pool the shallow feature map to obtain the pooled feature map of the target tongue image;

The feature fusion submodule is used to perform feature fusion on each convolution feature map and pooling feature map to obtain a multi-scale feature map.

Optionally, the above feature map processing sub-module includes:

The weight calculation unit is used to calculate the channel dimension attention weight of the convolution feature map according to the preset channel attention module for each convolution layer, and calculate the space of the convolution feature map according to the preset spatial attention module. Dimension attention weight;

The weighted feature map acquisition unit is used to process the convolution feature map using the channel dimension attention weight and the spatial dimension attention weight to obtain a weighted feature map of the convolution feature map.

Optionally, the above weighted feature map acquisition unit includes:

The average pooling subunit is used to perform an average pooling operation on the convolutional feature map to obtain the first pooling result;

The maximum pooling subunit is used to perform maximum pooling operations on the convolution feature map to obtain the second pooling result;

The feature map output subunit is used to input the first pooling result and the second pooling result into the first multi-layer perceptron to obtain the first output feature map and the second output feature map respectively;

The channel attention weight calculation subunit is used to add the first output feature map and the second output feature map to obtain the channel dimension attention weight;

The channel attention fusion subunit is used to multiply the channel dimension attention weight and the convolution feature map to obtain the channel attention mechanism fusion feature map.

Optionally, the above channel attention fusion subunit also includes:

The channel pooling subunit is used to perform channel-based maximum pooling and average pooling operations on the channel attention mechanism fusion feature map to obtain the maximum pooling result and the average pooling result respectively;

The pooling merging subunit is used to merge the maximum pooling results and the average pooling results to obtain the spatial attention merging feature map;

The spatial attention weight acquisition subunit is used to perform convolution activation on the spatial attention combined feature map to obtain the spatial dimension attention weight;

The weighted feature map acquisition subunit is used to multiply the spatial dimension attention weight and the channel attention mechanism fusion feature map to obtain a weighted feature map.

Optionally, the above prediction result generation module 104 includes:

The feature extraction submodule is used to extract convolutional features from deep feature maps to obtain the feature representation results of deep feature maps;

The point sampling submodule is used to perform point sampling around the pixels of the feature representation results to obtain several preset sampling points;

The feature vector acquisition submodule is used to obtain the feature vector of each preset sampling point based on the feature representation results;

The prediction result acquisition submodule is used to calculate the feature vector through the second multi-layer perceptron to obtain the segmentation prediction result.

Optionally, the above feature vector acquisition sub-module includes:

The point coordinate acquisition unit is used to obtain the point coordinates of each preset sampling point based on the feature representation results;

The low-level feature vector acquisition unit is used to sample the shallow feature map according to the point coordinates to obtain the low-level feature vector of the preset sampling point;

The high-level feature vector acquisition unit is used to sample the feature representation results according to the point coordinates to obtain the high-level feature vector of the preset sampling point;

The feature vector acquisition unit is used to merge low-level feature vectors and high-level feature vectors to obtain feature vectors of preset sampling points.

It should be noted that the information interaction, execution process, etc. between the above modules are based on the same concept as the method embodiments of the present invention. For details of their specific functions and technical effects, please refer to the method embodiments section, which will not be discussed here. Again.

FIG. 11 is a schematic structural diagram of a computer device provided by Embodiment 11 of the present invention. As shown in Figure 10, the computer device of this embodiment includes: at least one processor (only one is shown in Figure 10), a memory, and a computer program stored in the memory and executable on at least one processor. The processor executes The computer program implements the steps in any of the above embodiments of the tongue image semantic segmentation method.

The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that FIG. 10 is only an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or less components than shown in the figure, or may combine certain components, or use different components. , for example, it may also include a network interface, a display screen, an input device, etc.

The so-called processor can be a CPU, which can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field- Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

The memory includes readable storage media, internal memory, etc., wherein the internal memory can be the memory of the computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of the computer device, or in other embodiments may be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SMC), Secure Digital (SD) card, Flash Card, etc. Further, the memory may also include both internal storage units of the computer device and external storage devices. The memory is used to store the operating system, application programs, boot loader (Boot Loader), data and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or is to be output.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional units and modules is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs. Module completion means dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be hardware-based. It can also be implemented in the form of software functional units. In addition, the specific names of each functional unit and module are only for the convenience of distinguishing each other and are not used to limit the scope of the present invention.

For the specific working processes of the units and modules in the above device, reference can be made to the corresponding processes in the foregoing method embodiments, which will not be described again here. Integrated units may be stored in a computer-readable storage medium if they are implemented in the form of software functional units and sold or used as independent products. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can be processed after being processed. When the processor is executed, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. Computer-readable media may at least include: any entity or device capable of carrying computer program code, recording media, computer memory, read-only memory (ROM), random access memory (Random Access Memory, RAM), electronic Carrier signals, telecommunications signals, and software distribution media. For example, U disk, mobile hard disk, magnetic disk or CD, etc. In some jurisdictions, subject to legislation and patent practice, computer-readable media may not be electrical carrier signals and telecommunications signals.

The present invention can implement all or part of the processes in the above method embodiments, and can also be completed through a computer program product. When the computer program product is run on a computer device, the computer device can implement the steps in the above method embodiments when executed. step.

In the above embodiments, each embodiment is described with its own emphasis. For parts that are not detailed or documented in a certain embodiment, please refer to the relevant descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered to be beyond the scope of the present invention.

In the embodiments provided by the present invention, it should be understood that the disclosed apparatus/computer equipment and methods can be implemented in other ways. For example, the apparatus/computer equipment embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components. can be combined or can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

A unit described as a separate component may or may not be physically separate. A component shown as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

The above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions of the foregoing embodiments. Modifications are made to the recorded technical solutions, or equivalent substitutions are made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of the present invention, and should all be included in the present invention. within the scope of protection.

Claims (10)

1. The tongue picture semantic segmentation method is characterized by comprising the following steps of:

obtaining a target tongue picture, and carrying out feature extraction pretreatment on the target tongue picture to obtain a shallow feature picture of the target tongue picture;

Performing multi-scale feature extraction processing on the shallow feature map to obtain a multi-scale feature map of the target tongue picture, and performing channel dimension reduction processing on the multi-scale feature map to obtain an integrated feature map of the target tongue picture;

performing channel dimension reduction processing on the shallow feature map to obtain a first feature map of the target tongue picture, up-sampling the integrated feature map to obtain a second feature map of the target tongue picture, and performing feature fusion on the first feature map and the second feature map to obtain a deep feature map of the target tongue picture;

and inputting the shallow feature map and the deep feature map into a preset image segmentation model for semantic segmentation tasks to obtain a segmentation prediction result of the target tongue picture.

2. The tongue picture semantic segmentation method according to claim 1, wherein the performing multi-scale feature extraction processing on the shallow feature map to obtain a multi-scale feature map of the target tongue picture comprises:

respectively passing the shallow feature map through a pooling layer and at least one convolution layer;

when the number of the convolution layers is greater than two, each convolution layer is used for performing expansion convolution processing of different expansion rates on the shallow feature images to obtain each convolution feature image of the target tongue image picture;

The pooling layer is used for carrying out average pooling treatment on the shallow feature map to obtain a pooled feature map of the target tongue picture;

and carrying out feature fusion on each convolution feature map and the pooled feature map to obtain the multi-scale feature map.

3. The tongue picture semantic segmentation method according to claim 2, wherein after performing expansion convolution processing of different expansion rates on the shallow feature map to obtain each convolution feature map of the target tongue picture, the method further comprises:

for each convolution layer, calculating the channel dimension attention weight of the convolution feature map according to a preset channel attention module, and calculating the space dimension attention weight of the convolution feature map according to a preset space attention module;

and processing the convolution feature map by using the channel dimension attention weight and the space dimension attention weight to obtain a weighted feature map of the convolution feature map.

4. A tongue semantic segmentation method according to claim 3, wherein processing the convolution feature map with the channel dimension attention weights comprises:

carrying out average pooling operation on the convolution feature map to obtain a first pooling result;

Performing maximum pooling operation on the convolution feature map to obtain a second pooling result;

inputting the first pooling result and the second pooling result into a first multi-layer perceptron to obtain a first output characteristic diagram and a second output characteristic diagram respectively;

adding the first output feature map and the second output feature map to obtain the channel dimension attention weight;

multiplying the channel dimension attention weight with the convolution feature map to obtain a channel attention mechanism fusion feature map.

5. The tongue semantic segmentation method according to claim 4, further comprising, after the obtaining the channel attention mechanism fusion feature map:

carrying out maximum pooling and average pooling operation based on the channel attention mechanism fusion feature map to respectively obtain a maximum pooling result and an average pooling result;

combining the maximum pooling result and the average pooling result to obtain a spatial attention combination feature map;

performing convolution activation on the spatial attention merging feature map to obtain the spatial dimension attention weight;

multiplying the spatial dimension attention weight by the channel attention mechanism fusion feature map to obtain the weighted feature map.

6. The tongue picture semantic segmentation method according to claim 1, wherein the step of inputting the shallow feature map and the deep feature map to a preset image segmentation model for a semantic segmentation task to obtain a segmentation prediction result of the target tongue picture comprises:

performing convolution feature extraction on the deep feature map to obtain a feature representation result of the deep feature map;

performing point sampling around the pixel points of the characteristic representation result to obtain a plurality of preset sampling points;

according to the feature representation result, obtaining a feature vector of each preset sampling point;

and calculating the feature vector through a second multi-layer perceptron to obtain the segmentation prediction result.

7. The tongue semantic segmentation method according to claim 6, wherein the obtaining the feature vector of each preset sampling point comprises:

acquiring point coordinates of each preset sampling point according to the characteristic representation result;

sampling the shallow feature map according to the point coordinates to obtain a low-level feature vector of the preset sampling point;

sampling the feature representation result according to the point coordinates to obtain a high-level feature vector of the preset sampling point;

And combining the low-level characteristic vector and the high-level characteristic vector to obtain the characteristic vector of the preset sampling point.

8. The tongue picture semantic segmentation device is characterized by comprising:

the shallow feature map acquisition module is used for acquiring a target tongue picture, and carrying out feature extraction pretreatment on the target tongue picture to obtain a shallow feature map of the target tongue picture;

the integrated feature map acquisition module is used for carrying out multi-scale feature extraction processing on the shallow feature map to obtain a multi-scale feature map of the target tongue picture, and carrying out channel dimension reduction processing on the multi-scale feature map to obtain an integrated feature map of the target tongue picture;

the deep feature map acquisition module is used for carrying out channel dimension reduction processing on the shallow feature map to obtain a first feature map of the target tongue picture, up-sampling the integrated feature map to obtain a second feature map of the target tongue picture, and carrying out feature fusion on the first feature map and the second feature map to obtain a deep feature map of the target tongue picture;

the prediction result generation module is used for inputting the shallow feature image and the deep feature image into a preset image segmentation model for a semantic segmentation task to obtain a segmentation prediction result of the target tongue picture.

9. A computer device comprising a processor, a memory and a computer program stored in the memory and executable on the processor, the processor implementing the tongue semantic segmentation method according to any one of claims 1 to 7 when the computer program is executed by the processor.

10. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the tongue semantic segmentation method according to any one of claims 1 to 7.