CN110503636A - Parameter adjustment method, lesion prediction method, parameter adjustment device and electronic equipment - Google Patents

Abstract

The disclosure provides a method for adjusting parameters of a single-mode detection network, a method for predicting lesions in eye images, a device for adjusting parameters of a single-mode detection network, and electronic equipment; it relates to the technical field of artificial intelligence. The method includes: performing feature encoding on the first image feature to obtain a first distribution function, and performing feature encoding on the second image feature to obtain a second distribution function; determining the first loss function according to the first distribution function and the second distribution function value; according to the fusion of the sampling result of the second distribution function and the first image, the features of the image to be compared are determined; the value of the second loss function is determined according to the comparison between the feature of the image to be compared and the label to be compared, and according to the first loss The function value and the second loss function value adjust the network parameters of the single mode detection network until the loss function value converges. The method in the present disclosure can overcome the problem of poor parameter adjustment effect to a certain extent, and then improve the processing effect of the network model on the input image.

Description

Parameter adjustment method, lesion prediction method, parameter adjustment device and electronic equipment

technical field

The present disclosure relates to the field of artificial intelligence technology, and to machine learning technology, specifically, to a method for adjusting parameters of a single-mode detection network, a method for predicting lesions in eye images, a device for adjusting parameters of a single-mode detection network, and Electronic equipment.

Background technique

With the continuous development of artificial intelligence, more and more network models for image processing have emerged to perform feature extraction, recognition, and classification operations on images.

Before the image is processed by the image processing network model, the network model needs to be trained, and the usual training method is supervised training. For example, input the sample image to the network model, and determine the corresponding loss function according to the comparison between the classification result output by the network model and the target classification result, so as to adjust the network parameters of the network model according to the loss function, and then complete the training of the network model .

However, in the above-mentioned training method, the input sample image is usually a single sample image, which will lead to poor parameter tuning effect, which in turn will affect the processing effect of the network model on the input image.

It should be noted that the information disclosed in the above background section is only for enhancing the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to those of ordinary skill in the art.

Contents of the invention

The purpose of the present disclosure is to provide a method for adjusting parameters of a single-modal detection network, a method for predicting lesions in eye images, a device for adjusting parameters of a single-modal detection network, and electronic equipment, so as to overcome the poor effect of parameter adjustment to a certain extent problem, and then improve the processing effect of the network model on the input image.

Other features and advantages of the present disclosure will become apparent from the following detailed description, or in part, be learned by practice of the present disclosure.

According to the first aspect of the present disclosure, a method for adjusting parameters of a single-mode detection network is provided, including:

performing feature extraction on the first image, and performing feature encoding on the extracted first image features to obtain a first distribution function, and performing feature encoding on the extracted second image features to obtain a second distribution function; wherein , the first image feature corresponds to the first image, and the second image feature is obtained by feature fusion of the first image and the second image;

determining a first loss function value according to the first distribution function and the second distribution function;

Determining the features of the image to be compared according to the fusion of the sampling result of the second distribution function and the first image;

Determine the second loss function value according to the comparison between the image features to be compared and the label to be compared, and adjust the network parameters of the single-modal detection network according to the first loss function value and the second loss function value until the loss function value Convergence; wherein, the loss function value includes a first loss function value and a second loss function value.

In an exemplary embodiment of the present disclosure, feature encoding is performed on the extracted first image features to obtain a first distribution function, including:

Extracting first image features corresponding to the first image through a single-mode detection network;

Perform feature encoding on the features of the first image to determine a first distribution function corresponding to the first image.

In an exemplary embodiment of the present disclosure, feature encoding is performed on the extracted second image features to obtain a second distribution function, including:

Fusing the first image and the second image through a multimodal detection network, and generating second image features according to the fusion result;

Feature coding is performed on the features of the second image to determine a second distribution function corresponding to the first image and the second image.

In an exemplary embodiment of the present disclosure, the characteristics of the image to be compared are determined according to the fusion of the sampling result of the second distribution function and the first image, including:

Fusing the sampling result of the second distribution function with the first image feature corresponding to the first image to obtain a third image feature, and fusing the sampling result of the second distribution function with the second image feature to obtain a fourth image feature;

determining a third distribution function corresponding to the third image feature, and determining a fourth distribution function corresponding to the fourth image feature;

determining a third loss function value according to the third distribution function and the fourth distribution function;

Fusing the sampling result of the fourth distribution function with the third image feature to obtain the fifth image feature, and fusing the sampling result of the fourth distribution function with the fourth image feature to obtain the sixth image feature;

determining a fifth distribution function corresponding to the fifth image feature, and determining a sixth distribution function corresponding to the sixth image feature;

determining a fourth loss function value according to the fifth distribution function and the sixth distribution function;

The sampling result of the sixth distribution function is fused with the fifth image feature to obtain the image feature to be compared.

In an exemplary embodiment of the present disclosure, determining the second loss function value according to the comparison between the image feature to be compared and the label to be compared includes:

Perform feature processing on the features of the image to be compared, compare the feature-processed image feature to be compared with the label to be compared, and determine the second loss function value according to the comparison result; wherein, the feature processing includes convolution processing, pooling processing and non-linear activation processing.

In an exemplary embodiment of the present disclosure, the parameter adjustment method of the single-mode detection network further includes:

The sum of the first loss function value, the second loss function value, the third loss function value and the fourth loss function value is determined as the loss function value.

In an exemplary embodiment of the present disclosure, the network parameters of the single-mode detection network are adjusted according to the first loss function value and the second loss function value until the loss function value converges, including:

The network parameters of the unimodal detection network are adjusted according to the loss function value until the loss function value converges.

According to a second aspect of the present disclosure, a method for predicting lesions in eye images is provided, including:

Input the eye image into the single-mode detection network, and fit the first distribution function corresponding to the eye image according to the single-mode detection network;

Fusing the sampling result of the first distribution function with the first image feature corresponding to the eye image to obtain the second image feature;

Fitting a second distribution function corresponding to the second image feature according to the single-mode detection network, and fusing the sampling result of the second distribution function with the second image feature to obtain a third image feature;

Fit the third distribution function corresponding to the third image feature according to the single-modal detection network, and fuse the sampling result of the third distribution function with the third image feature to obtain the fourth image feature and predict the eye according to the fourth image feature lesions in the image;

Wherein, the single-mode detection network is adjusted according to a parameter adjustment method of the single-mode detection network provided in the first aspect.

According to a third aspect of the present disclosure, a parameter adjustment device for a single-mode detection network is provided, including a distribution function determination unit, a loss function value determination unit, a feature fusion unit, and a parameter adjustment unit, wherein:

A distribution function determination unit, configured to perform feature extraction on the first image, and perform feature encoding on the extracted first image features to obtain a first distribution function, and perform feature encoding on the extracted second image features to obtain A second distribution function is obtained; wherein, the first image feature corresponds to the first image, and the second image feature is obtained by feature fusion of the first image and the second image;

A loss function value determining unit, configured to determine a first loss function value according to the first distribution function and the second distribution function;

A feature fusion unit is used to determine the features of the image to be compared according to the fusion of the sampling result of the second distribution function and the first image;

The parameter adjustment unit is used to determine the second loss function value according to the comparison between the image features to be compared and the label to be compared, and perform network parameters of the single-modal detection network according to the first loss function value and the second loss function value Adjust until the loss function value converges; wherein, the loss function value includes the first loss function value and the second loss function value.

In an exemplary embodiment of the present disclosure, the distribution function determination unit performs feature encoding on the extracted first image features to obtain the first distribution function as follows:

The distribution function determination unit extracts the first image feature corresponding to the first image through a single-mode detection network;

The distribution function determination unit performs feature encoding on the features of the first image to determine the first distribution function corresponding to the first image.

In an exemplary embodiment of the present disclosure, the distribution function determination unit performs feature encoding on the extracted second image features to obtain the second distribution function as follows:

The distribution function determination unit fuses the first image and the second image through the multimodal detection network, and generates the second image features according to the fusion result;

The distribution function determining unit performs feature encoding on the features of the second image to determine a second distribution function corresponding to the first image and the second image.

In an exemplary embodiment of the present disclosure, the manner in which the feature fusion unit determines the feature of the image to be compared according to the fusion of the sampling result of the second distribution function and the first image is as follows:

The feature fusion unit fuses the sampling result of the second distribution function with the first image feature corresponding to the first image to obtain the third image feature, and fuses the sampling result of the second distribution function with the second image feature to obtain the fourth image feature. image features;

The feature fusion unit determines a third distribution function corresponding to the third image feature, and determines a fourth distribution function corresponding to the fourth image feature;

The feature fusion unit determines the third loss function value according to the third distribution function and the fourth distribution function;

The feature fusion unit fuses the sampling result of the fourth distribution function with the third image feature to obtain the fifth image feature, and fuses the sampling result of the fourth distribution function with the fourth image feature to obtain the sixth image feature;

The feature fusion unit determines a fifth distribution function corresponding to the fifth image feature, and determines a sixth distribution function corresponding to the sixth image feature;

The feature fusion unit determines a fourth loss function value according to the fifth distribution function and the sixth distribution function;

The feature fusion unit fuses the sampling result of the sixth distribution function with the fifth image feature to obtain the image feature to be compared.

In an exemplary embodiment of the present disclosure, the method for the parameter adjustment unit to determine the second loss function value according to the comparison between the features of the image to be compared and the label to be compared is specifically:

The parameter adjustment unit performs feature processing on the features of the image to be compared, and compares the features of the image to be compared with the label to be compared, and determines the second loss function value according to the comparison result; wherein, the feature processing includes convolution processing, pooling, and non-linear activation processing.

In an exemplary embodiment of the present disclosure, the loss function value determination unit is further configured to determine the sum of the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value as the loss function value.

In an exemplary embodiment of the present disclosure, the loss function value determination unit adjusts the network parameters of the single-mode detection network according to the first loss function value and the second loss function value until the loss function value converges, specifically as follows :

The loss function value determination unit adjusts the network parameters of the single-mode detection network according to the loss function value until the loss function value converges.

According to a fourth aspect of the present disclosure, there is provided an apparatus for predicting lesions in eye images, including a function fitting unit, an image feature fusion unit, and an image feature acquisition unit, wherein:

The function fitting unit is used to input the eye image into the single-mode detection network, and fits the first distribution function corresponding to the eye image according to the single-mode detection network;

The image feature fusion unit is used to fuse the sampling result of the first distribution function with the first image feature corresponding to the eye image to obtain the second image feature;

The image feature acquisition unit is used to fit the second distribution function corresponding to the second image feature according to the single-mode detection network, and fuse the sampling result of the second distribution function with the second image feature to obtain the third image feature;

The image feature acquisition unit is also used to fit the third distribution function corresponding to the third image feature according to the single-mode detection network, and fuse the sampling result of the third distribution function with the third image feature to obtain the fourth image feature and Predicting a lesion in the eye image according to the fourth image feature;

Wherein, the single-mode detection network is adjusted according to a parameter adjustment method of the single-mode detection network provided in the first aspect.

According to a fifth aspect of the present disclosure, there is provided an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to Perform any of the methods described above.

According to a sixth aspect of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

Exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

In the parameter adjustment method of a single-mode detection network provided in an exemplary embodiment of the present disclosure, feature extraction may be performed on a first image (such as an OCT image), and feature encoding is performed on the extracted first image features, To obtain the first distribution function, and perform feature encoding on the extracted second image feature to obtain the second distribution function; wherein, the first image feature corresponds to the first image, and the second image feature is composed of the first image and the second image feature Two images (such as fundus images) are obtained by feature fusion; the first loss function value is determined according to the first distribution function and the second distribution function; furthermore, the fusion of the sampling result of the second distribution function and the first image can be used to determine the value to be obtained. Compare the image features; furthermore, the second loss function value can be determined according to the comparison between the image features to be compared and the label to be compared, and the network of the single-modal detection network can be calculated according to the first loss function value and the second loss function value The parameters are adjusted until the loss function value converges; wherein, the loss function value includes a first loss function value and a second loss function value. According to the description of the above solution, on the one hand, the present disclosure can overcome the problem of poor parameter adjustment effect to a certain extent, and then improve the processing effect of the network model on the input image; Adjust the network parameters of the unimodal detection network (for example, unimodal lesion detection network) in order to realize the training of the unimodal detection network, improve the network training effect, and then improve the classification effect of the unimodal detection network for the input image and identification effect, when the embodiments of the present disclosure are applied to lesion identification, the accuracy of lesion identification for fundus images can be improved.

It should be noted that the single-mode detection network trained by the embodiments of the present disclosure can be applied to an eye disease recognition system. In the traditional eye disease identification system, the patient's fundus image and OCT image need to be input into the system in pairs, and then the system can determine the lesion by combining the characteristics of the fundus image and OCT image; the eye disease identification system is equivalent to many Modality detection network, modality can be understood as an image. However, the unimodal detection network obtained through the training of the embodiments of the present disclosure can only determine the lesion through the fundus image or the OCT image, specifically, when the fundus image (or OCT image) is input to the unimodal detection network, the unimodal The image features extracted by the detection network can not only contain the features of the fundus image (or OCT image), but also conceal the features of the OCT image (or fundus image), which can reduce the accuracy of the input image while ensuring the recognition accuracy. Requirements, in the past, it was necessary to input pairs of images to perform lesion identification, but using the single-modal detection network in the embodiment of the present disclosure only needs to input a single image to perform lesion identification, which improves the convenience of lesion identification to a certain extent and The efficiency of lesion identification.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure. Apparently, the drawings in the following description are only some embodiments of the present disclosure, and those skilled in the art can obtain other drawings according to these drawings without creative efforts.

FIG. 1 shows a schematic diagram of an exemplary system architecture of a method for adjusting parameters of a single-mode detection network and a device for adjusting parameters of a single-mode detection network that can be applied to embodiments of the present disclosure;

FIG. 2 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present disclosure;

FIG. 3 schematically shows a flowchart of a method for adjusting parameters of a single-mode detection network according to an embodiment of the present disclosure;

Fig. 4 schematically shows a flow chart of a method for predicting a lesion in an eye image according to an embodiment of the present disclosure;

Fig. 5 schematically shows a schematic diagram of an eye image according to an embodiment of the present disclosure;

FIG. 6 schematically shows an architecture diagram of a parameter adjustment method of a single-mode detection network according to an embodiment of the present disclosure;

Fig. 7 schematically shows the architecture diagram of a method for predicting a lesion in an eye image according to an embodiment of the present disclosure;

FIG. 8 schematically shows a structural block diagram of a parameter adjustment device of a single-mode detection network according to an embodiment of the present disclosure;

Fig. 9 schematically shows a structural block diagram of an apparatus for predicting a lesion in an eye image according to an embodiment of the present disclosure.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details being omitted, or other methods, components, devices, steps, etc. may be adopted. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and/or processor means and/or microcontroller means.

Fig. 1 shows a schematic diagram of a system architecture of an exemplary application environment of a method for adjusting parameters of a single-mode detection network and an apparatus for adjusting parameters of a single-mode detection network according to embodiments of the present disclosure.

As shown in FIG. 1 , the system architecture 100 may include one or more of terminal devices 101 , 102 , 103 , a network 104 and a server 105 . The network 104 is used as a medium for providing communication links between the terminal devices 101 , 102 , 103 and the server 105 . Network 104 may include various connection types, such as wires, wireless communication links, or fiber optic cables, among others. The terminal devices 101, 102, and 103 may be various electronic devices with display screens, including but not limited to desktop computers, portable computers, smart phones, and tablet computers. It should be understood that the numbers of terminal devices, networks and servers in Fig. 1 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

The method for adjusting parameters of a single-mode detection network and the method for predicting lesions in eye images provided by the embodiments of the present disclosure are generally executed by the server 105. The prediction device is generally set in the server 105 . However, those skilled in the art can easily understand that the parameter adjustment method of the single-modal detection network and the lesion prediction method in the eye image provided by the embodiments of the present disclosure can also be executed by the terminal devices 101, 102, and 103. Correspondingly, The device for adjusting parameters of the single-mode detection network and the device for predicting lesions in eye images may also be set in the terminal devices 101, 102, 103, which are not specifically limited in this exemplary embodiment. For example, in an exemplary embodiment, the server 105 may perform feature extraction on the first image, and perform feature encoding on the extracted first image features to obtain a first distribution function, and perform feature encoding on the extracted first image features. The second image feature is subjected to feature encoding to obtain the second distribution function, and then, the first loss function value is determined according to the first distribution function and the second distribution function, and the value to be determined is determined according to the fusion of the sampling result of the second distribution function and the first image. Compare the image features, and then determine the second loss function value according to the comparison between the image features to be compared and the label to be compared, and calculate the network parameters of the single-mode detection network according to the first loss function value and the second loss function value Adjustments are made until the loss function value converges. The server 105 can also input the eye image into the single-mode detection network, fit the first distribution function corresponding to the eye image according to the single-mode detection network, and compare the sampling result of the first distribution function with the first distribution function corresponding to the eye image. The image features are fused to obtain the second image feature, and the second distribution function corresponding to the second image feature is fitted according to the single-mode detection network, and the sampling result of the second distribution function is fused with the second image feature to obtain the second image feature The three image features, and then, according to the single-mode detection network, the third distribution function corresponding to the third image feature is fitted, and the sampling result of the third distribution function is fused with the third image feature to obtain the fourth image feature and according to the first Four-image features predict lesions in eye images.

FIG. 2 shows a schematic structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present disclosure.

It should be noted that the computer system 200 of the electronic device shown in FIG. 2 is only an example, and should not limit the functions and application scope of the embodiments of the present disclosure.

As shown in FIG. 2 , a computer system 200 includes a central processing unit (CPU) 201 that can be programmed according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage section 208 into a random-access memory (RAM) 203 Instead, various appropriate actions and processes are performed. In the RAM 203, various programs and data necessary for system operation are also stored. The CPU 201 , ROM 202 , and RAM 203 are connected to each other via a bus 204 . An input/output (I/O) interface 205 is also connected to the bus 204 .

The following components are connected to the I/O interface 205: an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 208 including a hard disk, etc. and a communication section 209 including a network interface card such as a LAN card, a modem, or the like. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I/O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 210 as necessary so that a computer program read therefrom is installed into the storage section 208 as necessary.

In particular, according to an embodiment of the present disclosure, the processes described below with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, where the computer program includes program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via communication portion 209 and/or installed from removable media 211 . When the computer program is executed by a central processing unit (CPU) 201, various functions defined in the method and apparatus of the present application are performed. In some embodiments, the computer system 200 may further include an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is configured to process computing operations related to machine learning.

In some embodiments, the computer system 200 may further include an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is used to process computing operations related to machine learning.

Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the nature of intelligence and produce a new kind of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Artificial intelligence technology is a comprehensive subject that involves a wide range of fields, including both hardware-level technology and software-level technology. Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning/deep learning.

Computer Vision Technology (Computer Vision, CV) Computer vision is a science that studies how to make machines "see". More specifically, it refers to machine vision that uses cameras and computers instead of human eyes to identify, track and measure targets. And further graphics processing, so that the computer processing becomes an image that is more suitable for human observation or sent to the instrument for detection. As a scientific discipline, computer vision studies related theories and technologies, trying to build artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and maps It also includes common face recognition, fingerprint recognition and other biometric recognition technologies.

Machine learning (Machine Learning, ML) is a multi-field interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. Specializes in the study of how computers simulate or implement human learning behaviors to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its application pervades all fields of artificial intelligence. Machine learning and deep learning usually include techniques such as artificial neural network, belief network, reinforcement learning, transfer learning, inductive learning, and teaching learning.

In the early days of traditional machine learning, people needed to carefully design network parameters to shorten the gap between the results predicted by the neural network and the real results. In the current era of machine learning, people can make the neural network automatically optimize the network parameters according to the multi-layer loss function. In many scenarios, it is no longer necessary to carefully design the network parameters.

The technical solutions of the embodiments of the present disclosure are described in detail below:

With the continuous growth of China's population and the aggravation of population aging, the form of eye health is becoming more and more serious. According to statistics, more than 50% of people have not received routine eye examinations, and more than 90% of people receive treatment after the onset of the disease. For example, there are about 110 million diabetic patients in China, of which more than 40 million suffer from retinopathy, which can easily lead to blindness if not treated early. The risk of blindness can be reduced by 94.4% if regular eye examinations are carried out in the early stage of the disease.

Optical coherence tomography (OCT) is a new imaging technology that can image various aspects of biological tissues, such as structural information, blood flow, and elastic parameters. The clarity of OCT in observing the fundus structure is usually higher than that of other inspection methods. When observing the fundus through OCT, the retinal nerve fibers, inner and outer nerves, nuclear layer, cone rod cell layer, pigment epithelium and other ocular tissues can be clearly distinguished. Therefore, Diagnosis of macular hole, central serous chorioretinopathy, cystoid edema and other eye diseases by OCT can usually achieve better results. In addition, since OCT equipment can simultaneously obtain fundus images and OCT images during imaging, it can perform lesion diagnosis based on the two modalities of fundus images and OCT images at the same time, so as to greatly reduce the risk of missing lesions.

At present, the recognition of eye diseases based on eye images is usually achieved by training a network model capable of classifying natural images. The traditional network has only one fixed branch, and it is difficult to deal with the problem of inconsistent target size (for example, a dog may occupy a large area of the photo in one photo, but only a small part in another photo). Typically, fixed-size convolution kernels cannot handle this information asymmetry. InceptionV4 can use the method of widening the network width to increase the sampling information of the network for targets of different sizes. In addition, DenseNet can improve the performance of the network by increasing the depth of the network. Due to the problem of gradient disappearance in the training of traditional networks, the gradient of the network is 0 when calculating the backpropagation, which makes it impossible to perform backpropagation, which leads to the failure of training. Therefore, DenseNet proposes to use dense connections between all previous layers and subsequent layers to strengthen the backpropagation of gradients during training.

In general, InceptionV4 and DenseNet have achieved performance improvements in different dimensions, but both of the above only consider one modality (ie, an image) as input, and a single-structure network cannot be effective at the same time. The two modalities of OCT images and fundus images are processed, and the lack of information of a certain modality may cause poor parameter adjustment effects during training and incorrect classification of some data by the network during testing.

Based on one or more of the above problems, this example embodiment provides a method for adjusting parameters of a single-mode detection network. The method for adjusting parameters of the single-mode detection network may be applied to the above-mentioned server 105, and may also be applied to one or more of the above-mentioned terminal devices 101, 102, 103, which is not specifically limited in this exemplary embodiment. Referring to Fig. 3, the parameter adjustment method of the single-mode detection network may include the following steps S310 to S340:

Step S310: Perform feature extraction on the first image, and perform feature encoding on the extracted first image features to obtain a first distribution function, and perform feature encoding on the extracted second image features to obtain a second distribution function function; wherein, the first image feature corresponds to the first image, and the second image feature is obtained by feature fusion of the first image and the second image.

Step S320: Determine a first loss function value according to the first distribution function and the second distribution function.

Step S330: Determine the features of the image to be compared according to the fusion of the sampling result of the second distribution function and the first image.

Step S340: Determine the second loss function value according to the comparison between the image features to be compared and the label to be compared, and adjust the network parameters of the single-modal detection network according to the first loss function value and the second loss function value until The loss function value converges; wherein, the loss function value includes a first loss function value and a second loss function value.

In the parameter adjustment method of a single-mode detection network provided in an exemplary embodiment of the present disclosure, feature extraction may be performed on a first image (such as an OCT image), and feature encoding is performed on the extracted first image features, To obtain the first distribution function, and perform feature encoding on the extracted second image features to obtain the second distribution function; wherein, the first image feature corresponds to the first image, and the second image feature is composed of the first image and the second image feature Two images (such as fundus images) are obtained by feature fusion; the first loss function value is determined according to the first distribution function and the second distribution function; furthermore, the fusion of the sampling result of the second distribution function and the first image can be used to determine the value to be obtained. Compare the image features; furthermore, the second loss function value can be determined according to the comparison between the image features to be compared and the label to be compared, and the network of the single-modal detection network can be calculated according to the first loss function value and the second loss function value The parameters are adjusted until the loss function value converges; wherein, the loss function value includes a first loss function value and a second loss function value. According to the description of the above solution, on the one hand, the present disclosure can overcome the problem of poor parameter adjustment effect to a certain extent, and then improve the processing effect of the network model on the input image; The network parameters of the single-mode detection network are adjusted in a fusion manner to realize the training of the single-mode detection network, improve the network training effect, and then improve the classification effect and recognition effect of the single-mode detection network for the input image, which is implemented in this disclosure For example, when applied to focus recognition, it can improve the accuracy of focus recognition for fundus images.

In addition, the embodiments of the present disclosure can be applied to OCT and fundus disease detection algorithms, by inputting paired OCT images and fundus images of patients, and then combining OCT images and fundus images to diagnose eye diseases. Due to its unsupervised nature in the testing phase, it can be embedded in the OCT or fundus screening system at will, providing another modality of information as a supplement to the system to simultaneously improve the diagnostic capabilities of the fundus and the OCT network.

It should be noted that the fundus image mentioned in the embodiments of the present disclosure may be obtained by taking a fundus camera, and the OCT image may be obtained by taking an OCT imaging device. Optionally, the fundus image and the OCT image may also be acquired simultaneously by an OCT imaging device, which is not limited in this embodiment of the present disclosure. Wherein, the OCT image is a coherence tomography image of a specific position in the fundus image.

In addition, the above-mentioned fundus camera is a kind of medical camera used to acquire retinal images of human eyes (ie, fundus images). In addition, OCT imaging equipment is used to detect the back reflection or scattering signals of incident weak coherent light at different depths of biological tissue through the basic principle of weak coherent light interferometer, and obtain two-dimensional or three-dimensional structural images of biological tissue through scanning (ie , OCT image).

Next, the above-mentioned steps of this exemplary embodiment will be described in more detail.

In step S310, perform feature extraction on the first image, and perform feature encoding on the extracted first image features to obtain the first distribution function, and perform feature encoding on the extracted second image features to obtain the first Two distribution functions; wherein, the first image feature corresponds to the first image, and the second image feature is obtained by feature fusion of the first image and the second image.

In this exemplary embodiment, both the first image and the second image may be eye images, wherein the eye image includes a fundus image and an OCT image, and if the first image is a fundus image, then the second image is an OCT image; if The first image is an OCT image, and the second image is a fundus image. Wherein, the data size of the fundus image may be 496×496, the data size of the OCT image may be 496×X, and X may be 496, 768 or 1024.

In this exemplary embodiment, if the data size of the fundus image can be 496×496, and the data size of the OCT image can be 496×496, before step S310, the parameter adjustment method of the single-modal detection network can also include the following steps:

calculating the ratio of the image mean value corresponding to the first image to the image variance and the ratio of the image mean value corresponding to the second image to the image variance, so as to realize the standardization of the first image and the second image;

Perform operations such as random rotation, random horizontal flip, random elastic deformation, or adding random speckle noise on the standardized first image and second image to obtain multiple sets of images of different shapes, each set of images includes the first image and the second image Two images are used in step S310.

In this example embodiment, optionally, the extracted first image feature is subjected to feature encoding to obtain the first distribution function as follows:

Extracting first image features corresponding to the first image through a single-mode detection network;

Perform feature encoding on the features of the first image to determine a first distribution function corresponding to the first image.

In this example implementation, the first image feature is a subset of the first distribution function, and the expression form of the first distribution function may be a set (of features).

In this exemplary embodiment, the method of extracting the first image feature corresponding to the first image through the single-modal detection network is specifically: performing N times of convolution processing on the first image through the single-modal detection network, and performing non-convolutional processing on the convolution result. Linear activation processing (that is, relu layer processing), and determining the nonlinear activation processing result as the first image feature corresponding to the first image; wherein, N is a positive integer.

In this example embodiment, the method of performing feature encoding on the first image feature to determine the first distribution function corresponding to the first image is specifically: input the first image feature into the encoder so that the encoder performs feature encoding on it, Then determine the first distribution function corresponding to the first image; wherein, the encoder is also called Encoder, which is composed of a series of convolution, activation layers, and batch normalization layers.

It can be seen that, implementing this optional embodiment, the encoder can perform feature encoding on the first image to determine the first distribution function, which can then be used to approximate its similarity with the second distribution function and improve single-mode detection Image feature generation effects for networks.

In this example embodiment, optionally, the extracted second image features are subjected to feature encoding to obtain the second distribution function as follows:

Fusing the first image and the second image through a multimodal detection network, and generating second image features according to the fusion result;

Feature coding is performed on the features of the second image to determine a second distribution function corresponding to the first image and the second image.

In this example embodiment, the second image feature is a subset of the second distribution function, and the expression form of the second distribution function may be a set (of features).

In this exemplary embodiment, the method of fusing the first image and the second image through the multi-modal detection network, and generating the features of the second image according to the fusion result is as follows: combining the first image and the second image through the multi-modal detection network The images are fused into an image to be processed, and then a second image feature corresponding to the image to be processed is extracted; wherein, the second image feature includes an image feature corresponding to the first image and an image feature corresponding to the second image.

Further, the method of extracting the second image feature corresponding to the image to be processed is specifically: performing N times of convolution processing on the image to be processed through a multimodal detection network, performing nonlinear activation processing on the convolution result, and performing nonlinear activation processing The result is determined as the second image feature corresponding to the image to be processed; wherein, N is a positive integer.

In this example embodiment, the method of performing feature encoding on the second image features to determine the second distribution function corresponding to the first image and the second image is specifically: input the second image features into the encoder so that the encoder can It performs feature encoding, and then determines the second distribution function corresponding to the first image and the second image.

It can be seen that, implementing this optional embodiment, the encoder can perform feature encoding on the second image to determine the second distribution function, which can then be used to approximate its similarity with the first distribution function and improve single-mode detection Image feature generation effects for networks.

In step S320, a first loss function value is determined according to the first distribution function and the second distribution function.

In this example implementation, the first loss function value is used to characterize the difference between the first distribution function and the second distribution function.

In step S330, the features of the image to be compared are determined according to the fusion of the sampling result of the second distribution function and the first image.

In this example embodiment, the method of determining the sampling result of the second distribution function may be: assign a value to the unknown in the second distribution function, so as to determine the value of the second distribution function according to the expression after the assignment, as the second distribution function The sampling result; wherein, the sampling result may include at least one second distribution function value.

In this exemplary embodiment, optionally, the method of determining the features of the image to be compared according to the fusion of the sampling result of the second distribution function and the first image is specifically as follows:

Fusing the sampling result of the second distribution function with the first image feature corresponding to the first image to obtain a third image feature, and fusing the sampling result of the second distribution function with the second image feature to obtain a fourth image feature;

determining a third distribution function corresponding to the third image feature, and determining a fourth distribution function corresponding to the fourth image feature;

determining a third loss function value according to the third distribution function and the fourth distribution function;

Fusing the sampling result of the fourth distribution function with the third image feature to obtain the fifth image feature, and fusing the sampling result of the fourth distribution function with the fourth image feature to obtain the sixth image feature;

determining a fifth distribution function corresponding to the fifth image feature, and determining a sixth distribution function corresponding to the sixth image feature;

determining a fourth loss function value according to the fifth distribution function and the sixth distribution function;

The sampling result of the sixth distribution function is fused with the fifth image feature to obtain the image feature to be compared.

In this exemplary embodiment, the method of obtaining the third image feature by fusing the sampling result of the second distribution function with the first image feature corresponding to the first image may specifically be: performing N times of convolution processing on the first image feature; Among them, N is a positive integer; the hidden variables collected in the Gaussian distribution fitted by the posterior encoder are fused with the first image features after convolution processing, and then the fusion results are subjected to maximum value pooling processing to obtain the third image features. Similarly, the method of obtaining the fourth image feature, the fifth image feature and the sixth image feature is the same as the above-mentioned method of obtaining the third image feature, and will not be repeated here.

In this exemplary embodiment, the sampling result of the second distribution function may be understood as: a latent variable collected in the Gaussian distribution fitted by the first posterior encoder. The sampling result of the fourth distribution function can be understood as: the latent variables collected in the Gaussian distribution fitted by the second posterior encoder. The sampling result of the sixth distribution function can be understood as: the latent variables collected in the Gaussian distribution fitted by the third posterior encoder.

In addition, it should be noted that the single-mode detection network may include three prior networks, and the multi-mode detection network may include three posterior networks, and each network includes a corresponding encoder; The number of a priori networks included in the modal detection network may be at least one, and the number of a posteriori networks included in the multimodal detection network may also be at least one, which is not limited in the embodiments of the present disclosure.

Wherein, the first a posteriori network includes a first a posteriori encoder, the second a posteriori network includes a second a posteriori encoder, and the third a posteriori network includes a third a posteriori encoder. Similarly, the first prior network includes a first prior encoder, the second prior network includes a second prior encoder, and the third prior network includes a third prior encoder. Both the prior encoder and the posterior encoder are used to feature encode the image features to determine the corresponding distribution function.

In this exemplary embodiment, the first distribution function is determined through the first prior network, the second distribution function is determined through the first posterior network; the third distribution function is determined through the second prior network , the fourth distribution function is determined through the second posterior network; the fifth distribution function is determined through the third prior network, and the sixth distribution function is determined through the third posterior network.

It can be seen that by implementing this optional implementation, three loss function values can be determined to adjust the network parameters of the single-modal detection network according to the three loss function values, so as to improve the classification effect and recognition of the input image by the single-modal detection network. Effect.

In step S340, the second loss function value is determined according to the comparison between the image features to be compared and the label to be compared, and the network parameters of the single-modal detection network are adjusted according to the first loss function value and the second loss function value , until the loss function value converges; wherein, the loss function value includes the first loss function value and the second loss function value.

In this example implementation, the network parameters of the unimodal detection network are used to extract image features. The label to be compared can be the preset label corresponding to the input image. If it can be determined that it matches the label to be compared by classifying the features of the image to be compared, it indicates that the classification effect of the single-modal detection network is better, and the input The label corresponding to the image. In addition, the network parameters in the single-mode detection network can use the parameters pre-trained on the ImageNet dataset of the Dense121 network (that is, a deep learning network), where the newly added convolutional layer/fusion layer/encoder-decoder sub-network/ The fully connected layer can be initialized with a Gaussian distribution with a variance of 0.01 and a mean of 0, and then perform corresponding convolution processing, fusion processing, feature encoding processing, or classification processing on image features.

Wherein, it should be noted that the parameters and bias parameters of the above convolutional layer can be obtained based on Adam's gradient descent method. The most basic optimization algorithm for neural networks is the backpropagation algorithm plus gradient descent. Through the gradient descent method, the network parameters can be continuously converged to the global (or local) minimum value. However, due to the large number of neural network layers, it is usually necessary to propagate the error layer by layer from the output to the input through the backpropagation algorithm. The network parameters are updated layer by layer. Since the gradient direction is the fastest direction in which the function value becomes larger, the negative gradient direction is the fastest direction in which the function value becomes smaller. By iterating step by step along the negative gradient direction, it can quickly converge to the minimum value of the function. In addition, Adam is an optimizer that combines the advantages of two optimization algorithms, AdaGrad and RMSProp. Variance) is considered comprehensively to calculate the update step size.

In this example embodiment, optionally, the method of determining the second loss function value according to the comparison between the image feature to be compared and the label to be compared is specifically:

Perform feature processing on the features of the image to be compared, compare the feature-processed image feature to be compared with the label to be compared, and determine the second loss function value according to the comparison result; wherein, the feature processing includes convolution processing, pooling processing and non-linear activation processing.

In this exemplary embodiment, the image features to be compared after feature processing are compared with the labels to be compared, and the method of determining the second loss function value according to the comparison result is as follows: the image features to be compared after feature processing Input the fully connected layer to determine the category to which the image feature to be compared belongs, and determine the second loss function value according to the comparison between the label corresponding to the category to which it belongs and the label to be compared.

It can be seen that implementing this optional implementation mode can determine a loss function for adjusting network parameters through comparison with the tags to be compared, and then improve the image recognition effect of the network model through the loss function.

In this example embodiment, optionally, the parameter adjustment method of the single-mode detection network may also include the following steps:

The sum of the first loss function value, the second loss function value, the third loss function value and the fourth loss function value is determined as the loss function value.

Further, the network parameters of the single-mode detection network are adjusted according to the first loss function value and the second loss function value until the loss function value converges as follows:

The network parameters of the unimodal detection network are adjusted according to the loss function value until the loss function value converges.

In this example implementation, the first loss function value, the third loss function value, and the fourth loss function value can all correspond to the KL loss function, and the KL loss function is used to narrow the distribution between the three groups of prior networks and posterior networks . In addition, the second loss function corresponding to the second loss function value may be a cross-entropy loss function, and the cross-entropy loss function is used to guide the prior network to learn the lesion information existing in the fundus image or the OCT image or to classify and judge the lesion; Wherein, the lesion information includes at least the area of the lesion area and the central coordinates of the lesion area, etc., which are not limited in the embodiments of the present disclosure.

It can be seen that, implementing this optional implementation manner, the network parameters can be adjusted through the above-mentioned loss function value, which can improve the network training effect, and further improve the recognition effect of the single-modal detection network for the input image.

It can be seen that the implementation of the parameter adjustment method of the single-modal detection network shown in Figure 3 can overcome the problem of poor parameter adjustment effect to a certain extent, and then improve the processing effect of the network model on the input image; and, through multi-modal The network parameters of the single-mode detection network are adjusted in a fusion manner to realize the training of the single-mode detection network, improve the network training effect, and then improve the classification effect and recognition effect of the single-mode detection network for the input image, which is implemented in this disclosure For example, when applied to focus recognition, it can improve the accuracy of focus recognition for fundus images.

In addition, this exemplary embodiment also provides a method for predicting a lesion in an eye image. The method for predicting lesions in eye images may be applied to the above server 105, and may also be applied to one or more of the above terminal devices 101, 102, 103, which is not specifically limited in this exemplary embodiment. Referring to FIG. 4, the method for predicting lesions in eye images may include the following steps S410 to S440:

Step S410: Input the eye image into the single-modal detection network, and fit the first distribution function corresponding to the eye image according to the single-modal detection network.

Step S420: Fusion the sampling result of the first distribution function with the first image feature corresponding to the eye image to obtain the second image feature.

Step S430: Fit the second distribution function corresponding to the second image feature according to the single-modal detection network, and fuse the sampling result of the second distribution function with the second image feature to obtain the third image feature.

Step S440: Fitting the third distribution function corresponding to the third image feature according to the single-modal detection network, and fusing the sampling result of the third distribution function with the third image feature to obtain the fourth image feature and according to the fourth image feature Predicting lesions in eye images.

Wherein, the single-mode detection network is adjusted according to a parameter adjustment method of the single-mode detection network provided by the embodiments of the present disclosure.

Next, the above-mentioned steps of this exemplary embodiment will be described in more detail.

In step S410, the eye image is input into the single-modal detection network, and the first distribution function corresponding to the eye image is fitted according to the single-modal detection network.

In this exemplary embodiment, the above-mentioned eye image may be an OCT image or a fundus image, which is not limited in this embodiment of the present disclosure.

In this exemplary embodiment, the method of fitting the first distribution function corresponding to the eye image according to the single-modal detection network is specifically: performing N times of convolution processing on the eye image through the multi-modal detection network, and performing convolutional processing on the convolution result Non-linear activation processing, and determining the non-linear activation processing result as the image feature corresponding to the eye image; wherein, N is a positive integer; inputting the image feature corresponding to the eye image into the encoder so that the encoder performs feature encoding , and then determine the first distribution function corresponding to the image feature corresponding to the eye image.

In step S420, the sampling result of the first distribution function is fused with the first image feature corresponding to the eye image to obtain the second image feature.

In this exemplary embodiment, the method of obtaining the second image feature in step S420 is the same as the method of obtaining the third image feature in FIG. 3 . Please refer to the embodiment shown in FIG. 3 , and details will not be repeated here.

In step S430, the second distribution function corresponding to the second image feature is fitted according to the single-modal detection network, and the sampling result of the second distribution function is fused with the second image feature to obtain a third image feature.

In this exemplary embodiment, the manner of obtaining the third image feature in step S430 is the same as the manner of obtaining the third image feature in FIG. 3 . Please refer to the embodiment shown in FIG. 3 , and details will not be repeated here.

In step S440, the third distribution function corresponding to the third image feature is fitted according to the single-modal detection network, and the sampling result of the third distribution function is fused with the third image feature to obtain the fourth image feature and according to the fourth Image features predict lesions in ocular images.

In this example embodiment, if the eye image is an OCT image, then the first image feature, the second image feature, the third image feature and the fourth image feature corresponding to the OCT image all imply the image features of the fundus image; , the OCT image corresponds to the fundus image.

In this exemplary embodiment, the method of predicting the lesion in the eye image according to the fourth image feature is specifically: performing feature recognition on the fourth image feature to determine the feature part of the corresponding lesion position, and according to the feature part in the corresponding The location of the lesion is marked in the eye image.

It can be seen that the implementation of the lesion prediction method in the eye image shown in Figure 4 can generate the features of the fundus image corresponding to the OCT image through the upper branch of the unsupervised multi-modal fusion network (that is, the above-mentioned single-modal detection network). , improving the efficiency and accuracy of determining the location of the lesion; in addition, since this method can be applied to the fundus intelligent diagnosis system or the OCT intelligent diagnosis system, it can improve the diagnostic capabilities of various modal intelligent screening systems; in addition, Due to the strong scalability of the method, it can support at least one modality of eye images, which can improve the effect of eye disease screening.

Please refer to FIG. 5 , which schematically shows a schematic diagram of an eye image according to an embodiment of the present disclosure. As shown in FIG. 5 , an eye image 500 includes a fundus image 501 and an OCT image 502 . Wherein, the position of the black arrow in the fundus image 501 (also can be understood as the position of the eye lesion) corresponds to the OCT image 502, and the eye image 500 is the eye image involved in the embodiment of FIG. 4 . In addition, the fundus image 501 and the OCT image 502 can be used as the first image and the second image to train the single modality detection network.

Referring to the eye image shown in FIG. 5 , please refer to FIG. 6 . FIG. 6 schematically shows an architecture diagram of a method for adjusting parameters of a single-modal detection network according to an embodiment of the present disclosure. As shown in Figure 6, the architecture diagram of the parameter adjustment method of the single-mode detection network includes: a first image 601, a second image 602, N stacked convolutional layers and a nonlinear activation layer 603, and a maximum pooling layer 604, fusion layer 605, maximum value pooling layer 606, 3 convolutional layers + pooling layer + nonlinear activation layer 607, fully connected layer 608, first prior network 609, second prior network 610, the first Three a priori network 611 , a first a posteriori network 612 , a second a posteriori network 613 and a third a posteriori network 614 .

Specifically, the first distribution function corresponding to the first image 601 can be determined according to the N stacked convolutional layers and the nonlinear activation layer 603, the maximum pooling layer 604 and the first prior network 609, and according to the N The stacked convolutional layer and nonlinear activation layer 603 (where N is a positive integer), the maximum pooling layer 604 and the first posterior network 612 can determine the first image 601 and the second image 602 corresponding to the first Second distribution function; furthermore, the second distribution function can be sampled according to the first posteriori network 612, and the obtained sampling result is fused with the first image feature corresponding to the first image 601 to obtain the third image feature to be input to the next N stacked convolutional layers and nonlinear activation layers 603; wherein, the first image feature is obtained by processing the first image 601 by N stacked convolutional layers and nonlinear activation layers 603 and a maximum pooling layer 604 Furthermore, the sampling result of the second distribution function can be fused with the second image feature to obtain the fourth image feature, which can be input to the next N stacked convolutional layer and nonlinear activation layer 603; wherein, the first The two image features are obtained by processing the first image 601 and the second image 602 by N stacked convolutional layers and nonlinear activation layers 603 and the maximum value pooling layer 604; furthermore, the second prior network 610 Determine the third distribution function corresponding to the third image feature, and determine the fourth distribution function corresponding to the fourth image feature through the second posterior network 613; furthermore, the third loss function can be determined according to the third distribution function and the fourth distribution function Furthermore, the fourth distribution function can be sampled according to the second posterior network 613, and the sampling result can be fused with the third image feature to obtain the fifth image feature, which can be input to the next N stacked convolutional layers and Non-linear activation layer 603; furthermore, the sampling result of the fourth distribution function can be fused with the fourth image feature to obtain the sixth image feature; furthermore, the third prior network 611 can be used to determine the corresponding value of the fifth image feature The fifth distribution function, and determine the sixth distribution function corresponding to the sixth image feature, so as to determine the fourth loss function value according to the fifth distribution function and the sixth distribution function; furthermore, the sixth distribution can be calculated according to the third posterior network 614 The function performs sampling, and fuses the sampling result with the fifth image feature to obtain the image feature to be compared; furthermore, the feature processing of the image feature to be compared can be performed through 3 convolutional layers + pooling layer + nonlinear activation layer 607 , and perform global pooling on the feature processing results, and input the fully connected layer 608 to compare the image features to be compared with the labels to be compared, and perform training on the comparison results through the cross-entropy loss function.

Wherein, the first loss function value, the second loss function value and the third loss function value are determined by the KL loss function.

In addition, it should be noted that after any of the above "inputs to the next N stacked convolutional layers and nonlinear activation layers 603", a fusion layer 605 and a maximum value pooling layer 606 are required.

It can be seen that the parameter adjustment method of the single-mode detection network shown in Figure 3 can be implemented in combination with the architecture diagram shown in Figure 6, which can overcome the problem of poor parameter tuning effect to a certain extent, and then improve the processing effect of the network model on the input image ; and, the network parameters of the single-modal detection network can be adjusted through multi-modal fusion to realize the training of the single-modal detection network, improve the network training effect, and then improve the classification effect of the single-modal detection network for the input image and recognition effects.

Referring to the eye image shown in FIG. 5 , please refer to FIG. 7 . FIG. 7 schematically shows an architecture diagram of a method for predicting a lesion in an eye image according to an embodiment of the present disclosure. As shown in Figure 7, the architecture diagram of the lesion prediction method in eye images includes: a first image 701, N stacked convolutional layers and nonlinear activation layers 702, a maximum pooling layer 703, a fusion layer 704, Maximum pooling layer 705 , 3 convolutional layers + pooling layer + nonlinear activation layer 706 , fully connected layer 707 , first prior network 708 , second prior network 709 and third prior network 710 .

Specifically, the first distribution function corresponding to the first image 701 can be determined according to the N stacked convolutional layers and the nonlinear activation layer 702, the maximum pooling layer 703, and the first prior network 708; furthermore, the first distribution function corresponding to the first image 701 can be determined by The first prior network 708 samples the first distribution function, and fuses the sampling result with the first image feature corresponding to the eye image to obtain the second image feature, which is input to the next N stacked convolutional layers and non- Linear activation layer 702; Furthermore, the second distribution function corresponding to the second image feature can be determined according to the second prior network 708, and the second distribution function is sampled through the second prior network 708, and the sampling result is compared with the second The image features are fused to obtain a third image feature; furthermore, the third distribution function corresponding to the third image feature can be determined according to the third prior network 710, and the third distribution function is sampled through the third prior network 710, and The sampling result is fused with the third image feature to obtain the fourth image feature; furthermore, the feature processing of the fourth image feature can be performed through three convolutional layers + pooling layer + nonlinear activation layer 706, and the feature processing result Perform global pooling and input the fully connected layer 708 to compare the fourth image feature with the label to be compared to obtain the label corresponding to the first image 701 and the predicted lesion location.

In addition, it should be noted that after any of the above "inputs to the next N stacked convolutional layers and nonlinear activation layers 702", a fusion layer 701 and a maximum value pooling layer 705 are required.

It can be seen that, in combination with the architecture diagram shown in FIG. 7, the lesion prediction method in the eye image shown in FIG. The characteristics of the fundus image corresponding to the OCT image improve the efficiency and accuracy of determining the location of the lesion; in addition, since this method can be applied to the fundus intelligent diagnosis system or the OCT intelligent diagnosis system, it can improve various modes of intelligent screening. In addition, due to the strong scalability of this method, it can support at least one modality of eye images, which can improve the effect of eye disease screening.

Further, in this example embodiment, a parameter adjustment device 800 of a single-mode detection network is also provided. The apparatus 800 for adjusting parameters of a single-mode detection network can be applied to a server or a terminal device. Referring to FIG. 8, the parameter adjustment device 800 of the single-mode detection network may include: a distribution function determination unit 801, a loss function value determination unit 802, a feature fusion unit 803, and a parameter adjustment unit 804, wherein:

A distribution function determining unit 801, configured to extract features from the first image, and perform feature encoding on the extracted first image features to obtain a first distribution function, and perform feature encoding on the extracted second image features, To obtain the second distribution function; wherein, the first image feature corresponds to the first image, and the second image feature is obtained by feature fusion of the first image and the second image;

A loss function value determination unit 802, configured to determine a first loss function value according to the first distribution function and the second distribution function;

A feature fusion unit 803, configured to determine the features of the image to be compared according to the fusion of the sampling result of the second distribution function and the first image;

The parameter adjustment unit 804 is used to determine the second loss function value according to the comparison between the image feature to be compared and the label to be compared, and to adjust the network parameters of the single-modal detection network according to the first loss function value and the second loss function value Adjust until the loss function value converges; wherein, the loss function value includes a first loss function value and a second loss function value.

It can be seen that implementing the parameter adjustment device 800 of the single-mode detection network shown in FIG. The network parameters of the single-mode detection network are adjusted by mode fusion to realize the training of the single-mode detection network, improve the network training effect, and then improve the classification effect and recognition effect of the single-mode detection network for the input image. In this disclosure When the embodiments are applied to focus recognition, the accuracy of focus recognition for fundus images can be improved.

In an exemplary embodiment of the present disclosure, the distribution function determining unit 801 performs feature encoding on the extracted first image features to obtain the first distribution function as follows:

The distribution function determination unit 801 extracts the first image feature corresponding to the first image through a single-modal detection network;

The distribution function determining unit 801 performs feature encoding on the features of the first image to determine a first distribution function corresponding to the first image.

It can be seen that, implementing this exemplary embodiment, the encoder can perform feature encoding on the first image to determine the first distribution function, which can then be used to approximate its similarity with the second distribution function and improve the single-mode detection network. The effect of image feature generation.

In an exemplary embodiment of the present disclosure, the distribution function determining unit 801 performs feature encoding on the extracted second image features to obtain the second distribution function as follows:

The distribution function determination unit 801 fuses the first image and the second image through the multimodal detection network, and generates the second image features according to the fusion result;

The distribution function determining unit 801 performs feature encoding on the features of the second image to determine a second distribution function corresponding to the first image and the second image.

It can be seen that, implementing this exemplary embodiment, the encoder can perform feature encoding on the second image to determine the second distribution function, which can then be used to approximate its similarity with the first distribution function and improve the single-mode detection network. The effect of image feature generation.

In an exemplary embodiment of the present disclosure, the manner in which the feature fusion unit 803 determines the feature of the image to be compared according to the fusion of the sampling result of the second distribution function and the first image is as follows:

The feature fusion unit 803 fuses the sampling result of the second distribution function with the first image feature corresponding to the first image to obtain the third image feature, and fuses the sampling result of the second distribution function with the second image feature to obtain the first image feature Four image features;

The feature fusion unit 803 determines a third distribution function corresponding to the third image feature, and determines a fourth distribution function corresponding to the fourth image feature;

The feature fusion unit 803 determines a third loss function value according to the third distribution function and the fourth distribution function;

The feature fusion unit 803 fuses the sampling result of the fourth distribution function with the third image feature to obtain a fifth image feature, and fuses the sampling result of the fourth distribution function with the fourth image feature to obtain a sixth image feature;

The feature fusion unit 803 determines a fifth distribution function corresponding to the fifth image feature, and determines a sixth distribution function corresponding to the sixth image feature;

The feature fusion unit 803 determines a fourth loss function value according to the fifth distribution function and the sixth distribution function;

The feature fusion unit 803 fuses the sampling result of the sixth distribution function with the fifth image feature to obtain the image feature to be compared.

It can be seen that, implementing this exemplary embodiment, three loss function values can be determined to adjust the network parameters of the single-modal detection network according to the three loss function values, so as to improve the classification effect and recognition effect of the single-modal detection network on the input image .

In an exemplary embodiment of the present disclosure, the manner in which the parameter adjustment unit 804 determines the second loss function value according to the comparison between the features of the image to be compared and the label to be compared is specifically:

The parameter adjustment unit 804 performs feature processing on the features of the image to be compared, compares the features of the image to be compared with the label to be compared, and determines the second loss function value according to the comparison result; wherein, the feature processing includes volume Product processing, pooling processing, and nonlinear activation processing.

It can be seen that by implementing this exemplary embodiment, a loss function for adjusting network parameters can be determined through comparison with the tags to be compared, and then the image recognition effect of the network model can be improved through the loss function.

In an exemplary embodiment of the present disclosure, the loss function value determining unit 802 is further configured to determine the sum of the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value as Loss function value.

In an exemplary embodiment of the present disclosure, the loss function value determination unit 802 adjusts the network parameters of the single-mode detection network according to the first loss function value and the second loss function value until the loss function value converges. for:

The loss function value determination unit 802 adjusts the network parameters of the single-mode detection network according to the loss function value until the loss function value converges.

It can be seen that, implementing this exemplary embodiment, the network parameters can be adjusted through the above-mentioned loss function value, which can improve the effect of network training, and further improve the recognition effect of the single-modal detection network for the input image.

Furthermore, in this exemplary embodiment, an apparatus 900 for predicting lesions in eye images is also provided. The apparatus 900 for predicting lesions in eye images can be applied to a server or a terminal device. Referring to FIG. 9, the device 900 for predicting lesions in eye images may include: a function fitting unit 901, an image feature fusion unit 902, and an image feature acquisition unit 903, wherein:

A function fitting unit 901, configured to input the eye image into the single-mode detection network, and fit the first distribution function corresponding to the eye image according to the single-mode detection network;

An image feature fusion unit 902, configured to fuse the sampling result of the first distribution function with the first image feature corresponding to the eye image to obtain the second image feature;

The image feature acquisition unit 903 is configured to fit a second distribution function corresponding to the second image feature according to the single-modal detection network, and fuse the sampling result of the second distribution function with the second image feature to obtain a third image feature;

The image feature acquisition unit 903 is further configured to fit the third distribution function corresponding to the third image feature according to the single-modal detection network, and fuse the sampling result of the third distribution function with the third image feature to obtain the fourth image feature And predicting the lesion in the eye image according to the fourth image feature;

Wherein, the single-mode detection network is adjusted according to a parameter adjustment method of the single-mode detection network provided by the embodiments of the present disclosure.

It can be seen that implementing the lesion prediction device 900 in eye images shown in FIG. 9 can generate fundus images corresponding to OCT images through the upper branch of the unsupervised multi-modal fusion network (that is, the above-mentioned single-modal detection network). features, which improves the efficiency and accuracy of determining the location of the lesion; in addition, because this method can be applied to the fundus intelligent diagnosis system or the OCT intelligent diagnosis system, it can improve the diagnostic capabilities of various modal intelligent screening systems; in addition , due to the strong scalability of the method, it can support at least one modality of eye images, which can improve the effect of eye disease screening.

It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. Actually, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided to be embodied by a plurality of modules or units.

Since each functional module of the parameter adjustment device for a single-mode detection network in the exemplary embodiment of the present disclosure corresponds to the steps of the above-mentioned exemplary embodiment of the parameter adjustment method for a single-mode detection network, it is not disclosed in the embodiment of the disclosed device For details, please refer to the embodiment of the parameter adjustment method for a single-modal detection network mentioned above in this disclosure.

In addition, since each functional module of the device for predicting lesions in eye images in the exemplary embodiment of the present disclosure corresponds to the steps of the above-mentioned exemplary embodiment of the method for predicting lesions in eye images, there is no disclosure in the device embodiments of the present disclosure. For details, please refer to the embodiment of the lesion prediction method in eye images mentioned above in this disclosure.

It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. . Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that includes one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block in the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a A combination of dedicated hardware and computer instructions.

The units described in the embodiments of the present disclosure may be implemented by software or by hardware, and the described units may also be set in a processor. Wherein, the names of these units do not constitute a limitation of the unit itself under certain circumstances.

As another aspect, the present application also provides a computer-readable medium. The computer-readable medium may be included in the electronic device described in the above-mentioned embodiments; or it may exist independently without being assembled into the electronic device. middle. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by an electronic device, the electronic device is made to implement the methods described in the above-mentioned embodiments.

Other embodiments of the present disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, and these modifications, uses or adaptations follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure . The specification and examples are to be considered exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It should be understood that the present disclosure is not limited to the precise constructions which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

1. A parameter adjustment method of a single mode detection network, characterized in that, comprising: performing feature extraction on the first image, and performing feature encoding on the extracted first image features to obtain a first distribution function, and performing feature encoding on the extracted second image features to obtain a second distribution function; wherein , the first image feature corresponds to the first image, and the second image feature is obtained by feature fusion of the first image and the second image; determining a first loss function value based on the first distribution function and the second distribution function; Determining the features of the image to be compared according to the fusion of the sampling result of the second distribution function and the first image; Determine the second loss function value according to the comparison between the image features to be compared and the label to be compared, and perform network parameters of the single-mode detection network according to the first loss function value and the second loss function value Adjust until the loss function value converges; wherein, the loss function value includes the first loss function value and the second loss function value. 2. The method according to claim 1, wherein the extracted first image feature is subjected to feature encoding to obtain the first distribution function, comprising: Extracting first image features corresponding to the first image through a single-mode detection network; Perform feature encoding on the features of the first image to determine a first distribution function corresponding to the first image. 3. The method according to claim 1, wherein the extracted second image feature is subjected to feature encoding to obtain a second distribution function, comprising: Fusing the first image and the second image through a multimodal detection network, and generating second image features according to the fusion result; Perform feature encoding on the features of the second image to determine a second distribution function corresponding to the first image and the second image. 4. The method according to claim 1, wherein the characteristics of the image to be compared are determined according to the fusion of the sampling result of the second distribution function and the first image, including: Fusing the sampling result of the second distribution function with the first image feature corresponding to the first image to obtain a third image feature, and combining the sampling result of the second distribution function with the second image feature fusion to obtain the fourth image feature; determining a third distribution function corresponding to the third image feature, and determining a fourth distribution function corresponding to the fourth image feature; determining a third loss function value based on the third distribution function and the fourth distribution function; Fusion is performed according to the sampling result of the fourth distribution function and the third image feature to obtain a fifth image feature, and the sampling result of the fourth distribution function is fused with the fourth image feature to obtain a sixth image feature. image features; determining a fifth distribution function corresponding to the fifth image feature, and determining a sixth distribution function corresponding to the sixth image feature; determining a fourth loss function value based on the fifth distribution function and the sixth distribution function; The sampling result of the sixth distribution function is fused with the fifth image feature to obtain the image feature to be compared. 5. The method according to claim 1, wherein the second loss function value is determined according to the comparison between the image features to be compared and the label to be compared, comprising: Perform feature processing on the features of the images to be compared, and compare the feature-processed image features to be compared with the labels to be compared, and determine the second loss function value according to the comparison result; wherein, the feature processing includes Convolution processing, pooling processing, and nonlinear activation processing. 6. The method according to claim 4, further comprising: A sum of the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value is determined as a loss function value. 7. The method according to claim 6, wherein the network parameters of the single-mode detection network are adjusted according to the first loss function value and the second loss function value until the loss function value converges, including : Adjusting network parameters of the single-mode detection network according to the loss function value until the loss function value converges. 8. A lesion prediction method in an eye image, comprising: Input the eye image into a single-mode detection network, and fit the first distribution function corresponding to the eye image according to the single-mode detection network; Fusing the sampling result of the first distribution function with the first image feature corresponding to the eye image to obtain a second image feature; fitting a second distribution function corresponding to the second image feature according to the single-modal detection network, and fusing the sampling result of the second distribution function with the second image feature to obtain a third image feature; Fitting a third distribution function corresponding to the third image feature according to the single-modal detection network, and fusing the sampling result of the third distribution function with the third image feature to obtain a fourth image feature and predicting a lesion in the eye image according to the fourth image feature; Wherein, the single-mode detection network is adjusted according to the method described in any one of claims 1-6. 9. A parameter adjustment device for a single-mode detection network, characterized in that it comprises: A distribution function determination unit, configured to perform feature extraction on the first image, and perform feature encoding on the extracted first image features to obtain a first distribution function, and perform feature encoding on the extracted second image features to obtain A second distribution function is obtained; wherein, the first image feature corresponds to the first image, and the second image feature is obtained by feature fusion of the first image and the second image; a loss function value determining unit, configured to determine a first loss function value according to the first distribution function and the second distribution function; A feature fusion unit, configured to determine the features of the image to be compared according to the fusion of the sampling result of the second distribution function and the first image; A parameter adjustment unit, configured to determine a second loss function value based on the comparison between the image features to be compared and the label to be compared, and to perform single-modal analysis based on the first loss function value and the second loss function value The network parameters of the detection network are adjusted until the loss function value converges; wherein the loss function value includes the first loss function value and the second loss function value. 10. An electronic device, characterized in that it comprises: processor; and a memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the method according to any one of claims 1-8 by executing the executable instructions.