CN117316430A - Question answering method, device, equipment and medium for diagnosing ophthalmic diseases - Google Patents

Abstract

The embodiment of the application provides a question answering method, a question answering device, question answering equipment and a question answering medium for diagnosing ophthalmic diseases. The method comprises the following steps: acquiring a fundus image of a person to be detected; classifying the fundus image based on a fundus disease classification model to obtain an ophthalmic disease category corresponding to the fundus image; dividing the fundus image based on a fundus dividing model to obtain a fundus dividing result corresponding to the fundus image; and processing the ophthalmic disease related problems input by the user according to the ophthalmic disease category and the fundus segmentation result based on a language diagnosis question-answering model, and outputting the ophthalmic disease question-answering result of the person to be detected. The embodiment of the application can realize rapid diagnosis and treatment of the ophthalmic diseases, and improve the efficiency and accuracy of the ophthalmic medical diagnosis.

Description

Question-and-answer methods, devices, equipment and media for diagnosing ophthalmic diseases

Technical field

The present application relates to the field of image processing technology, and in particular to a question and answer method, device, equipment and medium for diagnosing ophthalmic diseases.

Background technique

With the continuous development of modern medicine, the diagnosis and treatment of eye diseases increasingly rely on high-tech means. There are many types of eye diseases with similar symptoms, so they require diagnosis by a professional doctor.

Current existing methods and systems for diagnosing eye diseases have the following problems:

1. Most ophthalmic disease diagnosis systems rely on professional doctors for diagnosis, the diagnosis efficiency is low, and there may be subjective differences in doctors’ diagnosis results.

2. Traditional eye disease diagnosis methods require patients to go to the hospital for diagnosis. The medical treatment process is cumbersome and the patient experience is poor. The existing eye disease diagnosis system lacks interaction with patients and cannot provide personalized treatment plans according to the patient's specific conditions. .

Contents of the invention

Embodiments of the present application provide a question and answer method, device, electronic equipment and storage medium for diagnosing ophthalmic diseases, so as to solve the problem of low efficiency in diagnosing ophthalmic diseases in the existing technology, cumbersome user consultation process, and inability to provide personalized information according to the specific situation of the patient. treatment options.

In order to solve the above technical problems, the embodiments of the present application are implemented as follows::

In a first aspect, embodiments of the present application provide a question-and-answer method for diagnosing ophthalmic diseases. The method includes:

Obtain the fundus image of the person to be tested;

Classify the fundus image based on a fundus disease classification model to obtain the ophthalmic disease category corresponding to the fundus image;

Perform segmentation processing on the fundus image based on a fundus segmentation model to obtain a fundus segmentation result corresponding to the fundus image;

Based on the language diagnosis question and answer model, the ophthalmic disease-related questions input by the user according to the ophthalmic disease category and the fundus segmentation result are processed, and the ophthalmic disease question and answer results of the person to be detected are output.

Optionally, before performing classification processing on the fundus image based on the fundus disease classification model to obtain the ophthalmic disease category corresponding to the fundus image, the method further includes:

Obtain sample fundus images used to train the fundus disease classification model;

Perform data enhancement processing on the sample fundus image based on a preset data enhancement strategy to obtain a processed fundus image;

The fundus disease classification model is trained based on the processed fundus images.

Optionally, the fundus image is segmented based on the fundus segmentation model to obtain a fundus segmentation result corresponding to the fundus image, including:

Input the ophthalmic disease category and the fundus image to the fundus segmentation model;

The fundus segmentation model is called to perform segmentation processing on the fundus image based on the ophthalmic disease category to obtain the fundus segmentation result.

Optionally, the calling the fundus segmentation model to perform segmentation processing on the fundus image based on the ophthalmic disease category to obtain the fundus segmentation result includes:

Calling the fundus segmentation model to perform feature extraction on the fundus image to obtain multi-scale fundus features of the fundus image;

Perform feature fusion processing on the multi-scale fundus features to obtain a fusion feature map;

The fusion feature map is processed based on the ophthalmic disease category to obtain the fundus segmentation result of fundus markers associated with the ophthalmic disease category.

Optionally, before the language-based diagnosis question and answer model processes ophthalmic disease-related questions input by the user according to the ophthalmic disease category and the fundus segmentation result, and outputs the ophthalmic disease question and answer result of the person to be detected, the method further includes: :

Based on the ophthalmology knowledge corpus, build a natural language instruction set;

Guide the language diagnosis question answering model to be trained to complete the answering task based on the natural language instruction set to obtain the language diagnosis question answering model;

Among them, the answering task refers to conducting knowledge questions and answers and disease interpretation based on user questions and personal diagnosis results.

Optionally, the language-based diagnosis question and answer model processes ophthalmological disease-related questions input by the user according to the ophthalmic disease category and the fundus segmentation result, and outputs the ophthalmic disease question and answer result of the person to be detected, including:

Obtain the eye disease-related questions input by the user;

The ophthalmic disease question and answer results corresponding to the ophthalmic disease related questions are obtained based on the language diagnosis question and answer model, and the ophthalmic disease question and answer results are output.

In a second aspect, embodiments of the present application provide a question-and-answer device for diagnosing ophthalmic diseases. The device includes:

The fundus image acquisition module is used to obtain the fundus image of the person to be tested;

An ophthalmic disease category acquisition module is used to classify the fundus image based on the fundus disease classification model and obtain the ophthalmic disease category corresponding to the fundus image;

A fundus segmentation result acquisition module is used to segment the fundus image based on a fundus segmentation model to obtain a fundus segmentation result corresponding to the fundus image;

The question and answer result output module is used to process the ophthalmological disease-related questions input by the user according to the ophthalmic disease category and the fundus segmentation result based on the language diagnosis question and answer model, and output the ophthalmic disease question and answer result of the person to be detected.

Optionally, the device also includes:

A sample image acquisition module, used to acquire sample fundus images for training the fundus disease classification model;

A processed image acquisition module, configured to perform data enhancement processing on the sample fundus image based on a preset data enhancement strategy to obtain a processed fundus image;

A classification model training module, configured to train the fundus disease classification model based on the processed fundus images.

Optionally, the fundus segmentation result acquisition module includes:

A fundus image input unit, configured to input the ophthalmic disease category and the fundus image to the fundus segmentation model;

A segmentation result acquisition unit is configured to call the fundus segmentation model to segment the fundus image based on the ophthalmic disease category to obtain the fundus segmentation result.

Optionally, the segmentation result acquisition unit includes:

The fundus feature acquisition subunit is used to call the fundus segmentation model to perform feature extraction on the fundus image to obtain multi-scale fundus features of the fundus image;

A feature map acquisition subunit is used to perform feature fusion processing on the multi-scale fundus features to obtain a fusion feature map;

A segmentation result acquisition subunit is configured to process the fusion feature map based on the ophthalmic disease category to obtain the fundus segmentation result of the fundus marker associated with the ophthalmic disease category.

Optionally, the device also includes:

Instruction set building module, used to build a natural language instruction set based on the ophthalmology knowledge corpus;

A question and answer model acquisition module, configured to guide the language diagnosis question and answer model to be trained to complete the answering task based on the natural language instruction set to obtain the language diagnosis question and answer model;

Among them, the answering task refers to conducting knowledge questions and answers and disease interpretation based on user questions and personal diagnosis results.

Optionally, the question and answer result output module includes:

A disease question obtaining unit, configured to obtain the ophthalmic disease-related questions input by the user according to the ophthalmic disease category and the fundus segmentation result;

A question and answer result output unit is configured to obtain the ophthalmic disease question and answer result corresponding to the ophthalmic disease related question based on the language diagnosis question and answer model, and output the ophthalmic disease question and answer result.

In a third aspect, embodiments of the present application provide an electronic device, including:

A memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements any of the above questions and answers for diagnosing ophthalmic diseases. method.

In the fourth aspect, embodiments of the present application provide a readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can perform any of the above-mentioned methods for diagnosing ophthalmology. Question-and-answer approach to disease.

In the embodiment of the present application, the fundus image of the person to be detected is obtained. Classify the fundus image based on the fundus disease classification model to obtain the ophthalmic disease category corresponding to the fundus image. The fundus image is segmented based on the fundus segmentation model to obtain the fundus segmentation result corresponding to the fundus image. Based on the language diagnosis question and answer model, the questions related to eye diseases input by the user based on the ophthalmological disease categories and fundus segmentation results are processed, and the question and answer results of the eye diseases of the person to be detected are output. The embodiments of this application realize automatic diagnosis and question answering of ophthalmic diseases through large model technology, which greatly improves the diagnosis efficiency and reduces the patient's medical burden. It is expected to improve the efficiency and accuracy of ophthalmic medical diagnosis and provide better medical services to patients. At the same time, this application can also provide personalized eye disease treatment plans based on the user's questions and answers.

The above description is only an overview of the technical solutions of the present application. In order to have a clearer understanding of the technical means of the present application, they can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present application more obvious and understandable. , the specific implementation methods of the present application are specifically listed below.

Description of drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. , for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a step flow chart of a question and answer method for diagnosing ophthalmic diseases provided by an embodiment of the present application;

Figure 2 is a step flow chart of a fundus disease classification model training method provided by an embodiment of the present application;

Figure 3 is a step flow chart of a method for obtaining fundus segmentation results provided by an embodiment of the present application;

Figure 4 is a step flow chart of another method for obtaining fundus segmentation results provided by an embodiment of the present application;

Figure 5 is a step flow chart of a method for obtaining a language diagnosis question and answer model provided by an embodiment of the present application;

Figure 6 is a step flow chart of an eye disease question and answer result output method provided by an embodiment of the present application;

Figure 7 is a schematic diagram of a diagnostic question and answer process provided by an embodiment of the present application;

Figure 8 is a schematic diagram of a segmentation model structure provided by an embodiment of the present application;

Figure 9 is a schematic structural diagram of a question and answer device for diagnosing ophthalmic diseases provided by an embodiment of the present application;

FIG. 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

Referring to Figure 1 , there is shown a step flow chart of a question and answer method for diagnosing ophthalmic diseases provided by an embodiment of the present application. As shown in Figure 1 , the question and answer method for diagnosing ophthalmic diseases may include: Step 101. 102, step 103 and step 104.

Step 101: Obtain the fundus image of the person to be detected.

Embodiments of the present application can be applied to scenarios in which large model technologies (i.e., fundus disease classification models, fundus segmentation models, and language diagnosis question and answer models) are combined to perform diagnosis and question answering of ophthalmic diseases on subjects to be examined.

When it is necessary to diagnose ophthalmic diseases of the subject to be examined, the fundus image of the subject to be examined can be obtained.

In some examples, the fundus image of the person to be tested may be a fundus image of the person to be tested taken by ophthalmic equipment in a hospital or other places. For example, when the person to be tested goes to a hospital or other places for treatment, the fundus images of the person to be tested can be captured through ophthalmic equipment in the hospital or other places.

In some examples, the fundus image of the person to be examined may be a fundus image of the person to be examined taken by the user. For example, when the person to be tested needs remote medical treatment, other users can use a shooting device (such as a mobile phone, etc.) to capture fundus images of the person to be tested.

It can be understood that the above examples are only examples to better understand the technical solutions of the embodiments of the present application, and are not intended to be the sole limitation of this embodiment.

After acquiring the fundus image of the person to be detected, step 102 is executed.

Step 102: Classify the fundus image based on a fundus disease classification model to obtain an ophthalmic disease category corresponding to the fundus image.

The fundus disease classification model refers to the ophthalmology disease category corresponding to the fundus image (for example, the fundus image is input into the fundus disease classification model to obtain its corresponding ophthalmology disease label, including the following common fundus diseases: glaucoma, macular degeneration, central retinal or branch arteries occlusion, central retinal or distributing vein occlusion, hypertensive retinopathy, diabetic retinopathy, age-related macular degeneration, leopard pattern lesions, high myopic retinopathy, etc.) models.

In this example, ResNet-50 can be used as the feature extraction network in the fundus disease classification model. The ResNet model can use residual learning to alleviate the low learning efficiency and inaccurate accuracy of the deep learning model as the number of layers increases. Effectively improve the problem, and at the same time, due to the deepening of the number of layers, it has better feature extraction capabilities and better model classification performance. The network performs well in many fields such as image classification, segmentation, and detection. Therefore, ResNet-50 can be used as the feature extraction network. In addition, the number of output units of the last fully connected layer can be replaced by the number of classification categories in the classification task.

It can be understood that the structure of the above fundus disease classification model is only an example to better understand the technical solution of the present application, and is not the only limitation on the structure of the fundus disease classification model in this embodiment.

The training process of the fundus disease classification model can be described in detail as follows in conjunction with Figure 2.

Referring to FIG. 2 , a flow chart of steps of a fundus disease classification model training method provided by an embodiment of the present application is shown. As shown in Figure 2, the fundus disease classification model may include: step 201, step 202 and step 203.

Step 201: Obtain sample fundus images used for training the fundus disease classification model.

In the embodiment of the present application, when training the fundus disease classification model, sample fundus images for training the fundus disease classification model may first be obtained.

In a specific implementation, the sample fundus images may include fundus images of patients with ophthalmic diseases, such as patients with glaucoma, macular degeneration, central or branch retinal artery occlusion, central retinal or distribution vein occlusion, hypertensive retinopathy, diabetic retinopathy, Fundus images of different eye diseases such as age-related macular degeneration, leopard pattern lesions, and high myopic retinopathy. Fundus images without ophthalmic diseases, that is, normal fundus images, can also be included. By adding normal fundus images to assist in model training, the prediction accuracy of the fundus disease classification model can be improved.

In this example, the sample fundus image may be a fundus image extracted from a medical database, or may be a fundus image downloaded from a medical website or other websites, etc. Specifically, the method of obtaining the sample fundus image may be determined according to business requirements, and this embodiment is not limited to this.

After obtaining the sample fundus image for training the fundus disease classification model, step 202 is performed.

Step 202: Perform data enhancement processing on the sample fundus image based on a preset data enhancement strategy to obtain a processed fundus image.

After obtaining the sample fundus image for training the fundus disease classification model, data enhancement processing can be performed on the sample fundus image based on the preset data enhancement strategy to obtain the processed fundus image.

In this example, the preset data enhancement strategy may include: random rotation, random horizontal/vertical flipping, and other data enhancement strategies. This embodiment does not limit the specific type of the preset data enhancement strategy.

The embodiment of the present application performs data enhancement processing on sample fundus images through preset data enhancement strategies during the training process of the fundus disease classification model, thereby increasing the general performance of the model.

After performing data enhancement processing on the sample fundus image based on the preset data enhancement strategy to obtain the processed fundus image, step 203 is performed.

Step 203: Train the fundus disease classification model based on the processed fundus images.

After performing data enhancement processing on the sample fundus image based on the preset data enhancement strategy to obtain the processed fundus image, the fundus disease classification model can be trained based on the processed fundus image. The specific training process can be as follows:

1. Divide the processed fundus images into several batches. The number of processed fundus images in each batch may be the same or different.

2. Input the processed fundus images of each batch into the fundus disease classification model to be trained in order to train the fundus disease classification model to be trained.

3. Obtain the predicted classification results of the sample fundus images output by the fundus disease classification model to be trained, and calculate the loss value by combining the predicted classification results and the pre-labeled real classification results.

4. When the calculated loss value is within the preset range, the trained fundus disease classification model to be trained can be used as the final fundus disease classification model.

When the fundus disease classification model is trained and the eye disease diagnosis of the subject is performed, the fundus image of the subject to be detected can be classified and processed through the fundus disease classification model to obtain the ophthalmic disease category corresponding to the fundus image.

After classifying the fundus image based on the fundus disease classification model to obtain the ophthalmic disease category corresponding to the fundus image, step 103 is executed.

Step 103: Segment the fundus image based on the fundus segmentation model to obtain a fundus segmentation result corresponding to the fundus image.

After the fundus image is classified based on the fundus disease classification model to obtain the ophthalmic disease category corresponding to the fundus image, the fundus image can be segmented based on the fundus segmentation model to obtain the fundus segmentation result corresponding to the fundus image. Among them, the fundus segmentation result is the segmentation result of each fundus marker in the fundus image, such as arterial and venous blood vessels, optic cup and optic disc, leopard pattern, arcuate spots, lesions, nerve fiber layer, etc.

In a specific implementation, after obtaining the ophthalmic disease category of the fundus image predicted by the fundus disease classification model, the fundus image can also be segmented based on the ophthalmic disease category assisted fundus segmentation model. This implementation process can be described in detail as follows in conjunction with Figure 3.

Referring to FIG. 3 , a flow chart of steps of a method for obtaining fundus segmentation results provided by an embodiment of the present application is shown. As shown in Figure 3, the fundus segmentation result acquisition method may include: step 301 and step 302.

Step 301: Input the ophthalmic disease category and the fundus image to the fundus segmentation model.

In the embodiment of the present application, after the ophthalmic disease category output by the fundus disease classification model is obtained, the ophthalmic disease category and the fundus image of the person to be detected can be input into the fundus segmentation model.

In this example, Unet++ can be used as the fundus segmentation model. It improves the skip connection method based on Unet. It can capture features at different levels and integrate them through feature superposition (as shown in Figure 8). This makes the feature map scale difference smaller during fusion. In addition, Unet++ also introduces a deep supervision strategy, so that the gradient can be propagated to each node when the model is updated, further improving the segmentation performance.

Of course, it is not limited to this. In specific implementation, other structures can also be used as the fundus segmentation model. This embodiment does not limit the specific structure of the fundus segmentation model.

After the ophthalmic disease category and the fundus image are input to the fundus segmentation model, step 302 is performed.

Step 302: Call the fundus segmentation model to perform segmentation processing on the fundus image based on the ophthalmic disease category to obtain the fundus segmentation result.

After the ophthalmic disease categories and fundus images are input into the fundus segmentation model, the fundus segmentation model can be called to segment the fundus images based on the ophthalmic disease categories to obtain the fundus segmentation results. Specifically, through the auxiliary segmentation method of the ophthalmic disease category, the segmentation model can pay more attention to the fundus area corresponding to the ophthalmic disease category, and improve the accuracy of fundus segmentation.

In this example, the processing process of the fundus segmentation model can be described in detail as follows with reference to Figure 4.

Referring to FIG. 4 , a flow chart of steps of another method for obtaining fundus segmentation results provided by an embodiment of the present application is shown. As shown in Figure 4, the fundus segmentation result acquisition method may include: step 401, step 402 and step 403.

Step 401: Call the fundus segmentation model to perform feature extraction on the fundus image to obtain multi-scale fundus features of the fundus image.

In the embodiment of the present application, the fundus segmentation model can be first called to perform feature extraction on the fundus image to obtain multi-scale fundus features of the fundus image. For example, extracting features of different scales such as 1*1, 3*3, 6*6, etc. in fundus images.

After calling the fundus segmentation model to perform feature extraction on the fundus image to obtain multi-scale fundus features of the fundus image, step 402 is executed.

Step 402: Perform feature fusion processing on the multi-scale fundus features to obtain a fusion feature map.

After calling the fundus segmentation model to perform feature extraction on the fundus image to obtain multi-scale fundus features of the fundus image, feature fusion processing can be performed on the multi-scale fundus features to obtain a fusion feature map. In specific implementation, feature superposition can be used to fuse multi-scale fundus features, so that the scale difference of the feature maps during fusion can be smaller.

After performing feature fusion processing on the multi-scale fundus features to obtain a fused feature map, step 403 is executed.

Step 403: Process the fusion feature map based on the ophthalmic disease category to obtain the fundus segmentation result of the fundus marker associated with the ophthalmic disease category.

After performing feature fusion processing on multi-scale fundus features to obtain a fused feature map, the fused feature map can be processed based on the ophthalmic disease category to obtain fundus segmentation results of fundus markers associated with the ophthalmic disease category. That is, the ophthalmic disease categories predicted by the fundus disease classification model assist the fundus segmentation model, so that the fundus segmentation model pays more attention to the fundus markers associated with the ophthalmic disease categories in the fusion feature map, so as to perform better segmentation and obtain the corresponding fundus segmentation results. .

After segmenting the fundus image based on the fundus segmentation model to obtain a fundus segmentation result corresponding to the fundus image, step 104 is executed.

Step 104: Process the ophthalmological disease-related questions input by the user according to the ophthalmic disease category and the fundus segmentation result based on the language diagnosis question and answer model, and output the ophthalmic disease question and answer results of the person to be detected.

The language diagnosis question and answer model refers to a model used to conduct corresponding interactive question and answer with the user (which can be the patient or the patient's relative, etc.) based on the patient's disease condition.

In this example, the language diagnosis question answering model can be but is not limited to ChatGLM2-6B. ChatGLM2-6B is an open source conversational language model that supports Chinese and English bilingual question and answer. It is based on the General Language Model (GLM) architecture and has 6.2 billion parameters. The training process of the language diagnosis question answering model can be described in detail as follows in conjunction with Figure 5.

Referring to FIG. 5 , a flow chart of steps of a method for obtaining a language diagnosis question and answer model provided by an embodiment of the present application is shown. As shown in Figure 5, the language diagnosis question and answer model acquisition method may include: step 501 and step 502.

Step 501: Construct a natural language instruction set based on the ophthalmology knowledge corpus.

In this embodiment, a natural language instruction set can first be constructed based on the ophthalmic knowledge corpus. Specifically, for the ophthalmology knowledge corpus reviewed and annotated by medical experts, a natural language instruction set can be constructed using a combination of artificial intelligence and ChatGPT. The instructions compiled from the natural language instruction set can be used to guide the model to complete the answering task.

After the natural language instruction set is constructed based on the ophthalmic knowledge prediction database, step 502 is executed.

Step 502: Instruct the language diagnosis question answering model to be trained to perform specific operations or complete answering tasks based on the natural language instruction set to obtain the language diagnosis question answering model.

After the natural language instruction set is constructed based on the ophthalmic knowledge prediction database, the language diagnosis question answering model to be trained can be guided to complete the answering task based on the natural language instruction set, thereby obtaining the language diagnosis question answering model.

Specifically, the answering task refers to conducting knowledge questions and answers, disease interpretation, etc. based on user questions and personal diagnosis results. Use P-tuning or full-parameter instruction fine-tuning technology on the instruction set to improve the response ability of the question and answer model to language instructions, so that it can generate answers that are consistent with corpus expert knowledge and human preferences.

After the fundus image is segmented based on the fundus segmentation model to obtain the fundus segmentation result corresponding to the fundus image, the ophthalmic disease category and the fundus segmentation result can be sent to the user, and the user can then input the ophthalmic disease according to the ophthalmic disease category and the fundus segmentation result. Related questions. Then, the language diagnosis question and answer model can obtain the corresponding eye disease question and answer results based on the eye disease-related questions input by the user, and output the eye disease question and answer results. Among them, the results of eye disease question and answer can include: treatment suggestions, causes of diseases, etc.

In specific implementation, the language diagnosis question and answer model can answer the questions entered by the user accordingly to provide personalized eye disease treatment plans. This implementation can be described in detail as follows in conjunction with Figure 6 .

Referring to FIG. 6 , a step flow chart of a method for outputting ophthalmic disease question and answer results provided by an embodiment of the present application is shown. As shown in Figure 6, the method for outputting ophthalmic disease question and answer results may include: step 601 and step 602.

Step 601: Obtain ophthalmic disease-related questions input by the user based on the ophthalmic disease category and the fundus segmentation result.

In the embodiment of the present application, after the ophthalmic disease category and the fundus segmentation result are obtained, the ophthalmic disease category and the fundus segmentation result can be sent to the user, and then the ophthalmological disease-related questions input by the user based on the ophthalmic disease category and the fundus segmentation result can be obtained . What eye diseases do you suffer from? How to treat? What is the cause of the disease, etc.

After obtaining the ophthalmic disease-related questions input by the user based on the ophthalmic disease category and the fundus segmentation result, step 602 is executed.

Step 602: Obtain the ophthalmic disease question and answer results corresponding to the ophthalmic disease related questions based on the language diagnosis question and answer model, and output the ophthalmic disease question and answer results.

After obtaining the eye disease-related questions input by the user, the eye disease question and answer results corresponding to the eye disease related questions can be obtained based on the language diagnosis question answering model, and the eye disease question and answer results corresponding to the eye disease related questions can be output.

The embodiments of this application realize automatic diagnosis and question answering of ophthalmic diseases through large model technology, which greatly improves the diagnosis efficiency and reduces the patient's medical burden. It is expected to improve the efficiency and accuracy of ophthalmic medical diagnosis and provide better medical services to patients. At the same time, this embodiment can also provide personalized ophthalmic disease treatment plans based on the user's questions and answers.

The above implementation process can be described completely as follows in conjunction with Figure 7.

Referring to FIG. 7 , a schematic diagram of a diagnostic question and answer process provided by an embodiment of the present application is shown.

As shown in Figure 7, when diagnosing eye diseases of the subject, data collection can be performed first, that is, the fundus image of the subject is obtained, and the fundus image of the subject is cleaned, that is, the ones that meet the conditions (that is, the quality is relatively high) fundus image (high fundus image). After obtaining the fundus image that meets the conditions, the fundus image can be input to the fundus disease classification model and the fundus marker segmentation model (ie, the fundus segmentation model in this example). The fundus disease classification model can output disease labels, such as glaucoma, macular degeneration, central or branch retinal artery occlusion, central retinal or distribution vein occlusion, hypertensive retinopathy, diabetic retinopathy, age-related macular degeneration, leopard pattern lesions, height Myopic retinopathy, etc. The fundus landmark segmentation model can output disease focus areas, such as arteriovenous vessels, optic cups and discs, leopard patterns, arcuate spots, lesions, nerve fiber layers, etc. Then, the person to be tested or other users can input disease-related questions, and the fundus knowledge question and answer model (ie, the language diagnosis question and answer model in this example) can combine the disease labels output by the fundus disease classification model and the diseases output by the fundus marker segmentation model. The focus area will be given corresponding answers, such as treatment suggestions, causes of disease, etc.

It can be understood that for the language diagnosis question and answer model provided in this embodiment, users who do not suffer from ophthalmic diseases can also perform corresponding questions and answers. For example, for normal users, they may need to understand the onset of certain ophthalmic diseases, such as glaucoma. Causes, treatment suggestions, etc. At this time, the user can input corresponding questions related to glaucoma, and the language diagnosis question and answer model can directly output corresponding question and answer results based on the questions entered by the user.

The question-and-answer method provided by the embodiment of the present application for diagnosing ophthalmic diseases obtains fundus images of the person to be detected. Classify the fundus image based on the fundus disease classification model to obtain the ophthalmic disease category corresponding to the fundus image. The fundus image is segmented based on the fundus segmentation model to obtain the fundus segmentation result corresponding to the fundus image. Based on the language diagnosis question and answer model, the questions related to eye diseases input by the user based on the ophthalmological disease categories and fundus segmentation results are processed, and the question and answer results of the eye diseases of the person to be detected are output. The embodiments of this application realize automatic diagnosis and question answering of ophthalmic diseases through large model technology, which greatly improves the diagnosis efficiency and reduces the patient's medical burden. It is expected to improve the efficiency and accuracy of ophthalmic medical diagnosis and provide better medical services to patients. At the same time, this application can also provide personalized eye disease treatment plans based on the user's questions and answers.

Referring to Figure 9, a schematic structural diagram of a question and answer device for diagnosing ophthalmic diseases provided by an embodiment of the present application is shown. As shown in Figure 9, the question and answer device 900 for diagnosing ophthalmic diseases may include the following modules:

The fundus image acquisition module 910 is used to acquire the fundus image of the person to be detected;

The ophthalmic disease category acquisition module 920 is used to classify the fundus image based on the fundus disease classification model and obtain the ophthalmic disease category corresponding to the fundus image;;

The fundus segmentation result acquisition module 930 is used to segment the fundus image based on the fundus segmentation model to obtain the fundus segmentation result corresponding to the fundus image;

The question and answer result output module 940 is used to process the ophthalmological disease-related questions input by the user according to the ophthalmic disease category and the fundus segmentation result based on the language diagnosis question and answer model, and output the ophthalmic disease question and answer result of the person to be detected.

Optionally, the device also includes:

A sample image acquisition module, used to acquire sample fundus images for training the fundus disease classification model;

A processed image acquisition module, configured to perform data enhancement processing on the sample fundus image based on a preset data enhancement strategy to obtain a processed fundus image;

A classification model training module, configured to train the fundus disease classification model based on the processed fundus images.

Optionally, the fundus segmentation result acquisition module includes:

A fundus image input unit, configured to input the ophthalmic disease category and the fundus image to the fundus segmentation model;

A segmentation result acquisition unit is configured to call the fundus segmentation model to segment the fundus image based on the ophthalmic disease category to obtain the fundus segmentation result.

Optionally, the segmentation result acquisition unit includes:

A fundus feature acquisition subunit is used to call the fundus segmentation model to perform feature extraction on the fundus image to obtain multi-scale fundus features of the fundus image;

A feature map acquisition subunit is used to perform feature fusion processing on the multi-scale fundus features to obtain a fusion feature map;

A segmentation result acquisition subunit is configured to process the fusion feature map based on the ophthalmic disease category to obtain the fundus segmentation result of the fundus marker associated with the ophthalmic disease category.

Optionally, the device also includes:

Instruction set building module, used to build a natural language instruction set based on the ophthalmology knowledge corpus;

A question and answer model acquisition module, used to guide the language diagnosis question and answer model to be trained to perform specific operations or complete answering tasks based on the natural language instruction set to obtain the language diagnosis question and answer model;

Among them, the specific operation means that the user can call, view or inquire about the segmentation results of specific fundus markers according to language instructions, and the answering task means to conduct knowledge questions and answers and condition interpretation based on the user's questions and personal diagnosis results.

Optionally, the question and answer result output module includes:

A disease question obtaining unit, configured to obtain the ophthalmic disease-related questions input by the user according to the ophthalmic disease category and the fundus segmentation result;

A question and answer result output unit is configured to obtain the ophthalmic disease question and answer result corresponding to the ophthalmic disease related question based on the language diagnosis question and answer model, and output the ophthalmic disease question and answer result.

The question and answer device provided by the embodiment of the present application for diagnosing ophthalmic diseases acquires fundus images of the person to be detected. Classify the fundus image based on the fundus disease classification model to obtain the ophthalmic disease category corresponding to the fundus image. The fundus image is segmented based on the fundus segmentation model to obtain the fundus segmentation result corresponding to the fundus image. Based on the language diagnosis question and answer model, the questions related to eye diseases input by the user based on the ophthalmological disease categories and fundus segmentation results are processed, and the question and answer results of the eye diseases of the person to be detected are output. The embodiments of this application realize automatic diagnosis and question answering of ophthalmic diseases through large model technology, which greatly improves the diagnosis efficiency and reduces the patient's medical burden. It is expected to improve the efficiency and accuracy of ophthalmic medical diagnosis and provide better medical services to patients. At the same time, this application can also provide personalized eye disease treatment plans based on the user's questions and answers.

Additionally, embodiments of the present application also provide an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is processed by the When the machine is executed, the above question and answer method for diagnosing ophthalmic diseases is implemented.

Figure 10 shows a schematic structural diagram of an electronic device 1000 according to an embodiment of the present invention. As shown in Figure 10, electronic device 1000 includes a central processing unit (CPU) 1001, which can be loaded into a random access memory (RAM) 1003 according to computer program instructions stored in a read-only memory (ROM) 1002 or from a storage unit 1008. computer program instructions to perform various appropriate actions and processes. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. CPU1001, ROM1002, and RAM1003 are connected to each other via a bus 1004. An input/output (I/O) interface 1005 is also connected to bus 1004.

Multiple components in the electronic device 1000 are connected to the I/O interface 1005, including: an input unit 1006, such as a keyboard, mouse, microphone, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk , optical disk, etc.; and communication unit 1009, such as network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunications networks.

The various processes and processes described above may be executed by the processing unit 1001. For example, the method of any of the above embodiments can be implemented as a computer software program, which is tangibly included in a computer-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 1000 via the ROM 1002 and/or the communication unit 1009 . When a computer program is loaded into RAM 1003 and executed by CPU 1001, one or more actions in the methods described above may be performed.

Embodiments of the present application also provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, each process of the above Q&A method embodiment for diagnosing ophthalmic diseases is implemented, and can achieve the same technical effect, so to avoid repetition, we will not repeat them here. Wherein, the computer-readable storage medium is such as read-only memory (ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc.

It should be noted that, in this document, the terms "comprising", "comprises" or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device that includes a series of elements not only includes those elements, It also includes other elements not expressly listed or inherent in the process, method, article or apparatus. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article or apparatus that includes that element.

Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or that contributes to the existing technology. The computer software product is stored in a storage medium (such as ROM/RAM, disk, CD), including several instructions to cause a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of this application.

The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Inspired by this application, many forms can be made without departing from the purpose of this application and the scope protected by the claims, all of which fall within the protection of this application.

Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of this application can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage media include: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application. should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims (14)

1. A question-answering method for diagnosing an ophthalmic disorder, the method comprising:

acquiring a fundus image of a person to be detected;

classifying the fundus image based on a fundus disease classification model to obtain an ophthalmic disease category corresponding to the fundus image;

dividing the fundus image based on a fundus dividing model to obtain a fundus dividing result corresponding to the fundus image;

and processing the ophthalmic disease related problems input by the user according to the ophthalmic disease category and the fundus segmentation result based on a language diagnosis question-answering model, and outputting the ophthalmic disease question-answering result of the person to be detected.

2. The method according to claim 1, further comprising, before classifying the fundus image based on the fundus disease classification model, obtaining an ophthalmic disease category corresponding to the fundus image:

Acquiring a sample fundus image for training a fundus disease classification model;

performing data enhancement processing on the sample fundus image based on a preset data enhancement strategy to obtain a processed fundus image;

training the fundus disease classification model based on the processed fundus image.

3. The method according to claim 1, wherein the performing the segmentation process on the fundus image based on the fundus segmentation model to obtain a fundus segmentation result corresponding to the fundus image includes:

inputting the ophthalmic disease category and the fundus image to the fundus segmentation model;

and calling the fundus segmentation model to segment the fundus image based on the ophthalmic disease category to obtain the fundus segmentation result.

4. A method according to claim 3, wherein said invoking the fundus segmentation model to segment the fundus image based on the ophthalmic disease category, results in the fundus segmentation result, comprises:

invoking the fundus segmentation model to perform feature extraction on the fundus image so as to obtain multi-scale fundus features of the fundus image;

performing feature fusion processing on the multi-scale fundus features to obtain a fusion feature map;

And processing the fusion feature map based on the ophthalmic disease category to obtain the fundus segmentation result of fundus markers associated with the ophthalmic disease category.

5. The method according to claim 1, further comprising, before the language diagnosis-based question-answering model processes the ophthalmic disease-related questions input by the user according to the ophthalmic disease category and the fundus segmentation result, outputting the ophthalmic disease-answering result of the subject:

constructing a natural language instruction set based on the ophthalmic knowledge corpus;

guiding a to-be-trained language diagnosis question-answer model to complete an answer task based on the natural language instruction set so as to obtain the language diagnosis question-answer model;

the answer task refers to knowledge question answering and illness state interpretation according to user questions and personal diagnosis results.

6. The method according to claim 1, wherein the language-based diagnosis question-answering model processes the ophthalmic disease-related questions inputted by the user according to the ophthalmic disease category and the fundus segmentation result, outputs the ophthalmic disease question-answering result of the subject, comprising:

acquiring the ophthalmic disease-related problems input by a user according to the ophthalmic disease category and the fundus segmentation result;

And acquiring an ophthalmic disease question-answering result corresponding to the ophthalmic disease-related problem based on the language diagnosis question-answering model, and outputting the ophthalmic disease question-answering result.

7. A question-answering apparatus for diagnosing an ophthalmic disorder, the apparatus comprising:

the fundus image acquisition module is used for acquiring fundus images of the person to be detected;

the ophthalmic disease category acquisition module is used for classifying the fundus images based on a fundus disease classification model to obtain ophthalmic disease categories corresponding to the fundus images;

the fundus segmentation result acquisition module is used for carrying out segmentation processing on the fundus image based on a fundus segmentation model to obtain a fundus segmentation result corresponding to the fundus image;

and the question-answering result output module is used for processing the ophthalmic disease related problems input by the user according to the ophthalmic disease category and the fundus segmentation result based on the language diagnosis question-answering model and outputting the ophthalmic disease question-answering result of the person to be detected.

8. The apparatus of claim 7, wherein the apparatus further comprises:

the sample image acquisition module is used for acquiring a sample fundus image for training a fundus disease classification model;

The processing image acquisition module is used for carrying out data enhancement processing on the sample fundus image based on a preset data enhancement strategy to obtain a processing fundus image;

and the classification model training module is used for training the fundus disease classification model based on the processed fundus image.

9. The apparatus according to claim 7, wherein the fundus segmentation result acquisition module includes:

a fundus image input unit for inputting the ophthalmic disease category and the fundus image to the fundus segmentation model;

and the segmentation result acquisition unit is used for calling the fundus segmentation model to carry out segmentation processing on the fundus image based on the ophthalmic disease category so as to obtain the fundus segmentation result.

10. The apparatus according to claim 9, wherein the division result acquisition unit includes:

the fundus characteristic acquisition subunit is used for calling the fundus segmentation model to perform characteristic extraction on the fundus image so as to obtain multi-scale fundus characteristics of the fundus image;

the characteristic map acquisition subunit is used for carrying out characteristic fusion processing on the multi-scale fundus characteristics to obtain a fusion characteristic map;

and the segmentation result acquisition subunit is used for processing the fusion feature map based on the ophthalmic disease category to obtain the fundus segmentation result of the fundus marker related to the ophthalmic disease category.

11. The apparatus of claim 7, wherein the apparatus further comprises:

the instruction set construction module is used for constructing a natural language instruction set based on the ophthalmic knowledge corpus;

the question-answering model acquisition module is used for guiding the to-be-trained language diagnosis question-answering model to complete an answer task based on the natural language instruction set so as to obtain the language diagnosis question-answering model;

the answer task refers to knowledge question answering and illness state interpretation according to user questions and personal diagnosis results.

12. The apparatus of claim 7, wherein the question-answer result output module comprises:

a disease-problem acquiring unit configured to acquire the ophthalmic-disease-related problem input by a user according to the ophthalmic disease category and the fundus segmentation result;

and the question and answer result output unit is used for acquiring the question and answer result of the ophthalmic disease corresponding to the ophthalmic disease related problem based on the language diagnosis question and answer model and outputting the question and answer result of the ophthalmic disease.

13. An electronic device, comprising:

a memory, a processor and a computer program stored on the memory and executable on the processor, which when executed by the processor implements the question-answering method for diagnosing ophthalmic disorders according to any one of claims 1 to 6.

14. A readable storage medium, characterized in that instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the question-answering method for diagnosing ophthalmic disorders according to any one of claims 1 to 6.