CN115696069A - Imaging method for 3D endoscope imaging system and 3D endoscope imaging system - Google Patents

Abstract

An imaging method for a 3D endoscope imaging system and a 3D endoscope imaging system, the method comprising: acquiring first image data acquired by a first image sensor and second image data acquired by a second image sensor, and acquiring Attitude data collected by the attitude sensor, where the attitude data is used to indicate the spatial positions corresponding to the first image data and the second image data; at least according to the attitude data, determine the currently applied display mode in the first display mode and the second display mode; when When it is determined to apply the first display mode, generate stereoscopic image data according to the first image data and the second image data, and control the display to display the stereoscopic image; when it is determined to apply the second display mode, generate stereoscopic image data according to the first image data and the second image data The two-dimensional image data is generated from the data, the display is controlled to display the two-dimensional image, and the two-dimensional image is displayed upright according to the posture data. Among them, the two-dimensional image integrates the information of the two-way image data, which can overcome the abnormal state.

Description

Imaging method for 3D endoscopic imaging system and 3D endoscopic imaging system

technical field

The present invention relates to the field of medical equipment, and more particularly to an imaging method for a 3D endoscope imaging system and the 3D endoscope imaging system.

Background technique

Endoscope can present the histological shape of internal organs and pathological changes in the patient's body during minimally invasive surgery, which is convenient for diagnosis and operation. It is one of the important tools for modern medical diagnosis and treatment. Traditional endoscopes only provide two-dimensional plane images, and doctors can only judge the depth information of various organs in the human cavity based on experience during surgery, so they are only suitable for simple operations. In recent years, a stereo vision endoscope capable of stereoscopically observing the area has been developed. Compared with traditional endoscopes, it can provide three-dimensional information and better reflect the real situation of the scene. It allows doctors to feel the visual effect of three-dimensional images during the operation, providing Realistic visual experience helps doctors to operate endoscope more accurately.

Displaying stereoscopic endoscopic images needs to collect two-way image signals. When switching from the display mode of stereoscopic images to the display mode of two-dimensional images, one of the two-way image signals is usually output. In case of abnormal conditions such as dirt, the content of the screen will be lost. Moreover, the stereoscopic effect of the stereoscopic endoscopic image depends on the horizontal parallax of the image. When the stereoscopic image is displayed, if the operator's head habitually follows the rotation of the image, the stereoscopic effect will be lost and the image will be ghosted.

Contents of the invention

A series of concepts in simplified form are introduced in the Summary of the Invention, which will be further detailed in the Detailed Description. The summary of the invention in the present invention does not mean to limit the key features and essential technical features of the claimed technical solution, nor does it mean to try to determine the protection scope of the claimed technical solution.

The first aspect of the embodiment of the present invention provides an imaging method for a 3D endoscopic imaging system, including:

Acquiring first image data collected by the first image sensor of the 3D endoscope imaging system and second image data collected by the second image sensor of the 3D endoscope imaging system, and obtaining attitude data collected by the attitude sensor, The gesture data is used to indicate the spatial positions corresponding to the first image data and the second image data;

determining a currently applied display mode among the first display mode and the second display mode based at least on the gesture data;

When it is determined that the display mode of the current application is the first display mode, generating stereoscopic image data according to the first image data and the second image data, and controlling the display to display a stereoscopic image according to the stereoscopic image data ;

When it is determined that the display mode of the current application is the second display mode, generate two-dimensional image data according to the first image data and the second image data, and control the display according to the two-dimensional image data A two-dimensional image is displayed, and the two-dimensional image is displayed upright according to the posture data.

In some embodiments, the method further includes determining a display mode of the current application in the first display mode and the second display mode according to a user instruction, wherein:

When the user instruction indicates to adopt the first display mode, determine the first display mode as the currently applied display mode to display the stereoscopic image;

When the user instruction indicates to adopt the second display mode, the second display mode is determined as the currently applied display mode to display the two-dimensional image.

In some embodiments, the 3D endoscope imaging system includes an endoscope, the endoscope includes an insertion part and an operation part, the first sensor and the second sensor are arranged in the insertion part, the insertion part The part is used to insert the patient's part to be observed, so as to obtain the first image data and the second image data of the patient's part to be observed, and the posture data includes the rotation angle of the insertion part;

The determining the display mode of the current application in the first display mode and the second display mode at least according to the gesture data includes:

When the difference between the rotation angle and the reference angle is less than a preset threshold, determining the first mode as the currently applied display mode to display the stereoscopic image;

When the difference between the rotation angle and the reference angle is greater than or equal to the preset angle, the second mode is determined as the currently applied display mode to display the two-dimensional image.

In some embodiments, the reference angle is an angle at which there is no vertical parallax between the first image data and the second image data.

In some embodiments, the generating two-dimensional image data according to the first image data and the second image data includes:

The brightness information of the two-dimensional image data is determined according to the first brightness information of the first image data and the second brightness information of the second image data.

In some embodiments, the determining the brightness information of the two-dimensional image data according to the first brightness information of the first image data and the second brightness information of the second image data includes:

detecting whether the first image data and the second image data are abnormal;

When an abnormality is detected in the first image data, adjusting the second brightness information of the second image data according to the first brightness information of the first image data to obtain the two-dimensional image data;

When an abnormality is detected in the second image data, the first brightness information of the first image data is adjusted according to the second brightness information of the second image data, so as to obtain the two-dimensional image data.

In some embodiments, the detecting whether the first image data and the second image data are abnormal includes:

determining a degree of similarity between said first image data and said second image data;

determining a first sharpness of the first image data and a second sharpness of the second image data when the similarity between the first image data and the second image data is less than a first threshold, If the first definition is lower than the second definition, it is determined that there is an abnormality in the first image data, and if the first definition is higher than the second definition, it is determined that the second image There is an anomaly in the data.

In some embodiments, the two-dimensional image data includes partial information in the first image data and partial information in the second image data.

The second aspect of the embodiment of the present invention provides an imaging method for a 3D endoscopic imaging system, the method comprising:

Acquiring first image data and second image data;

determining the display mode of the current application in the first display mode and the second display mode;

When it is determined that the display mode of the current application is the first display mode, generating stereoscopic image data according to the first image data and the second image data, and controlling the display to display a stereoscopic image according to the stereoscopic image data ;

When it is determined that the display mode of the current application is the second display mode, generate two-dimensional image data according to the first image data and the second image data, and control the display according to the two-dimensional image The data show two-dimensional images.

In some embodiments, the first image data is collected by the first image sensor of the 3D endoscopic imaging system, and the second image data is collected by the second image sensor of the 3D endoscopic imaging system of.

In some embodiments, the two-dimensional image data includes partial information in the first image data and partial information in the second image data.

In some embodiments, the method further includes: during the process of acquiring the first image data and the second image data, acquiring gesture data, the gesture data being used to indicate the first image data and the The spatial position corresponding to the second image data;

The determining the display mode of the current application in the first display mode and the second display mode includes: determining the display mode of the current application in the first display mode and the second display mode according to user instructions and/or the gesture data display mode.

In some embodiments, according to the user instruction, determining the display mode of the current application in the first display mode and the second display mode includes:

When the user instruction indicates to adopt the first display mode, determine the first display mode as the currently applied display mode to display the stereoscopic image;

When the user instruction indicates to adopt the second display mode, the second display mode is determined as the currently applied display mode to display the two-dimensional image.

In some embodiments, the 3D endoscope imaging system includes an endoscope, the endoscope includes an insertion part and an operation part, the first sensor and the second sensor are arranged in the insertion part, the insertion part The part is used to insert the patient's part to be observed, so as to obtain the first image data and the second image data of the patient's part to be observed, and the posture data includes the rotation angle of the insertion part;

According to the posture data, determining the display mode of the current application in the first display mode and the second display mode includes:

When the difference between the rotation angle and the reference angle is less than a preset threshold, determining the first mode as the currently applied display mode to display the stereoscopic image;

When the difference between the rotation angle and the reference angle is greater than or equal to the preset angle, the second mode is determined as the currently applied display mode to display the two-dimensional image.

In some embodiments, the reference angle is an angle at which there is no vertical parallax between the first image data and the second image data.

In some embodiments, the method further includes: during the process of acquiring the first image data and the second image data, acquiring gesture data, the gesture data being used to indicate the first image data and the The spatial position corresponding to the second image data;

The two-dimensional image data is rotated according to the posture data, so that the two-dimensional image is displayed upright.

In some embodiments, the method further includes: when receiving a setting instruction to display the two-dimensional image non-erectly, displaying the two-dimensional image in a non-erect state, making the orientation of the two-dimensional image Associated with the spatial positions corresponding to the first image data and the second image data.

In some embodiments, the method further includes: during the process of acquiring the first image data and the second image data, acquiring gesture data, the gesture data being used to indicate the first image data and the The spatial position corresponding to the second image data;

The stereoscopic image data is rotated according to the posture data, so that the stereoscopic image is displayed upright.

In some embodiments, the method further includes: when receiving a setting instruction to display the stereoscopic image non-erectly, displaying the stereoscopic image in a non-erect state, so that the direction of the stereoscopic image is the same as the direction of the stereoscopic image. The spatial positions corresponding to the first image data and the second image data are associated.

In some embodiments, the generating two-dimensional image data according to the first image data and the second image data includes:

The brightness information of the two-dimensional image data is determined according to the first brightness information of the first image data and the second brightness information of the second image data.

In some embodiments, the determining the brightness information of the two-dimensional image data according to the first brightness information of the first image data and the second brightness information of the second image data includes:

detecting whether the first image data and the second image data are abnormal;

When an abnormality is detected in the first image data, adjusting the second brightness information of the second image data according to the first brightness information of the first image data to obtain the two-dimensional image data;

When an abnormality is detected in the second image data, the first brightness information of the first image data is adjusted according to the second brightness information of the second image data, so as to obtain the two-dimensional image data.

In some embodiments, the adjusting the second brightness information of the second image data according to the first brightness information of the first image data to obtain the two-dimensional image data includes:

performing weighted summation of the average brightness of the first image data and the average brightness of the second image data to obtain a first weighted brightness;

Obtaining a first brightness coefficient according to the ratio of the first weighted brightness to the average brightness of the second image data;

adjusting the brightness value of each pixel in the second image data according to the first brightness coefficient to obtain the brightness value of each pixel in the two-dimensional image data;

The adjusting the first brightness information of the first image data according to the second brightness information of the second image data to obtain the two-dimensional image data includes:

performing weighted summation of the average brightness of the second image data and the average brightness of the first image data to obtain a second weighted brightness;

Obtaining a second brightness coefficient according to the ratio of the second weighted brightness to the average brightness of the first image data;

The brightness value of each pixel in the first image data is adjusted according to the second brightness coefficient, so as to obtain the brightness value of each pixel in the two-dimensional image data.

In some embodiments, when the average brightness of the first image data and the average brightness of the second image data are weighted and summed to obtain the first weighted brightness, the weight coefficient of the first image data is less than the weight coefficient of the second image data;

When performing weighted summation of the average brightness of the second image data and the average brightness of the first image data to obtain the second weighted brightness, the weight coefficient of the first image data is greater than that of the second image data weight coefficient of .

In some embodiments, the determining the brightness information of the two-dimensional image data according to the first brightness information of the first image data and the second brightness information of the second image data further includes:

When it is detected that neither the first image data nor the second image data is abnormal, adjusting the second brightness information of the second image data according to the first brightness information of the first image data, so as to The two-dimensional image data is obtained, or the first brightness information of the first image data is adjusted according to the second brightness information of the second image data, so as to obtain the two-dimensional image data.

In some embodiments, the generating two-dimensional image data according to the first image data and the second image data includes:

The color information of the two-dimensional image data is determined according to the first color information of the first image data and the second color information of the second image data.

In some embodiments, the determining the color information of the two-dimensional image data according to the first color information of the first image data and the second color information of the second image data includes:

detecting whether the first image data and the second image data are abnormal;

When an abnormality is detected in the first image data, adjusting the second color information of the second image data according to the first color information of the first image data to obtain the two-dimensional image data;

When an abnormality is detected in the second image data, the first color information of the first image data is adjusted according to the second color information of the second image data to obtain the two-dimensional image data.

In some embodiments, the adjusting the second color information of the second image data according to the first color information of the first image data to obtain the two-dimensional image data includes:

performing weighted summation of the average color value of the first image data and the average color value of the second image data to obtain a first weighted color value;

Obtaining a first color coefficient according to the ratio of the weighted color value to the average color value of the second image data;

adjusting the color value of each pixel in the second image data according to the first color coefficient to obtain the color value of each pixel in the two-dimensional image data;

The adjusting the first color information of the first image data according to the second color information of the second image data to obtain the two-dimensional image data includes:

performing weighted summation of the average color value of the second image data and the average color value of the first image data to obtain a second weighted color value;

Obtaining a second color coefficient according to the ratio of the second weighted color value to the average color value of the second image data;

The color value of each pixel in the second image data is adjusted according to the second color coefficient, so as to obtain the color value of each pixel in the two-dimensional image data.

In some embodiments, when the average color value of the first image data and the average color value of the second image data are weighted and summed to obtain a first weighted color value, the first image data the weight coefficient is smaller than the weight coefficient of the second image data;

When performing weighted summation on the average color value of the second image data and the average color value of the first image data to obtain a second weighted color value, the weight coefficient of the first image data is greater than that of the first image data The weight coefficient of the second image data.

In some embodiments, the determining the color information of the two-dimensional image data according to the first color information of the first image data and the second color information of the second image data further includes:

When it is detected that neither the first image data nor the second image data is abnormal, adjusting the second color information of the second image data according to the first color information of the first image data, so as to The two-dimensional image data is obtained, or the first color information of the first image data is adjusted according to the second color information of the second image data, so as to obtain the two-dimensional image data.

In some embodiments, the detecting whether the first image data and the second image data are abnormal includes:

determining a degree of similarity between said first image data and said second image data;

determining a first sharpness of the first image data and a second sharpness of the second image data when the similarity between the first image data and the second image data is less than a first threshold, If the first definition is lower than the second definition, it is determined that there is an abnormality in the first image data, and if the first definition is higher than the second definition, it is determined that the second image There is an anomaly in the data.

The third aspect of the embodiment of the present invention provides a 3D endoscope imaging system, including an endoscope and a camera host connected to the endoscope, the endoscope includes an insertion part and an operation part, and the insertion part is used for Insert into the patient's site to be observed;

The endoscope includes a first image sensor and a second image sensor for collecting first image data and second image data respectively;

The camera host is used to acquire the first image data and the second image data from the endoscope, so as to execute the above method.

In some embodiments, the 3D endoscopic imaging system further includes an attitude sensor disposed on the endoscope for collecting attitude data and sending the attitude data to the camera host, the attitude data It is used to indicate the spatial positions corresponding to the first image data and the second image data.

According to the imaging method for the 3D endoscope imaging system and the 3D endoscope imaging system according to the embodiments of the present invention, the display mode can be switched according to the user instruction and/or posture data, and the stereoscopic image can be ensured to have a better display mode in the first display mode. The three-dimensional effect, the two-dimensional image displayed in the second display mode integrates the information of the two image data, and can also display the organization information to the user when there is an abnormality in one of the images.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

In the attached picture:

Fig. 1 shows a schematic block diagram of a 3D endoscopic imaging system according to an embodiment of the present invention;

2 shows a schematic flowchart of an imaging method for a 3D endoscopic imaging system according to an embodiment of the present invention;

FIG. 3 shows a schematic diagram of determining an imaging mode according to a roll angle range according to an embodiment of the present invention;

Fig. 4 shows a schematic diagram of generating two-dimensional image data according to first image data and second image data in the presence of occlusion according to an embodiment of the present invention;

Fig. 5 shows a schematic diagram of generating two-dimensional image data according to first image data and second image data in the absence of occlusion according to an embodiment of the present invention;

Fig. 6 shows a schematic flowchart of an imaging method for a 3D endoscope imaging system according to another embodiment of the present invention.

Detailed ways

In order to make the objects, technical solutions and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Apparently, the described embodiments are only some embodiments of the present invention, rather than all embodiments of the present invention, and it should be understood that the present invention is not limited by the exemplary embodiments described here. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative effort shall fall within the protection scope of the present invention.

In the following description, numerous specific details are given in order to provide a more thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without one or more of these details. In other examples, some technical features known in the art are not described in order to avoid confusion with the present invention.

It should be understood that the invention can be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the/the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the terms "consists of" and/or "comprising", when used in this specification, identify the presence of stated features, integers, steps, operations, elements and/or parts, but do not exclude one or more other Presence or addition of features, integers, steps, operations, elements, parts and/or groups. As used herein, the term "and/or" includes any and all combinations of the associated listed items.

In order to thoroughly understand the present invention, a detailed structure will be provided in the following description to illustrate the technical solution proposed by the present invention. Alternative embodiments of the invention are described in detail below, however the invention may have other embodiments beyond these detailed descriptions.

In the following, a 3D endoscope imaging system according to an embodiment of the present application will first be described with reference to FIG. 1 , and FIG. 1 shows a schematic structural block diagram of a 3D endoscope imaging system 100 according to an embodiment of the present invention.

As shown in FIG. 1 , the 3D endoscope imaging system 100 includes at least an endoscope and a camera host 150 connected to the endoscope, and the endoscope includes an insertion part 130 and an operation part 160,

The insertion part 130 is used to insert the patient's site to be observed, and the insertion part 130 and the operation part 160 can be an integral structure or a detachable structure; the endoscope also includes a first image sensor and a second image sensor (not shown), respectively For collecting the first image data and the second image data, for example, the first image sensor and the second image sensor are arranged at the front end of the insertion part 130 of the endoscope. The camera host 150 is used to acquire the first image data and the second image data from the endoscope for image processing. In addition, the 3D endoscope imaging system 100 also includes a light source 110, a light guide 120, a cable 140, and an attitude sensor (not shown), the light source 110 is connected to the endoscope through the light guide 120, and the operating part 160 is The camera host 150 is connected, and the light source 110 is connected through the light guide 120 .

Wherein, the light source 110 is used to provide an illumination light source to the site to be observed. The light source 110 may include a visible light source and a special light source. Exemplarily, the light source can be an LED light source, which can provide multiple monochromatic lights in different wavelength ranges, a combination of multiple monochromatic lights, or a wide-spectrum white light source. The special light source can be a laser light source corresponding to the fluorescent reagent, such as near-infrared light. In some embodiments, before using the 3D endoscope imaging system for imaging, a fluorescent reagent is injected at the site to be observed, and the fluorescent reagent can generate fluorescence after absorbing the laser light generated by the laser light source.

The insertion part 130 of the endoscope includes a mirror tube, an image sensor, and an illumination optical path. The front end of the mirror tube is used to insert into the human body and penetrate deep into the part to be inspected. The illuminating light path is connected with the light guide 120, and is used for irradiating the light generated by the light source 110 to the part of the target object to be detected. The image sensor specifically includes a first image sensor and a second image sensor for converting optical signals into electrical signals, including but not limited to CCD sensors, CMOS sensors and the like. The image signal collected by the image sensor is sent to the camera host 150 for subsequent image processing after preliminary signal processing by the operation unit 160 , and the preliminary signal processing includes amplification, filtering and other processing. The optical axes of the first image sensor and the second image sensor can be arranged in parallel or at an angle. The first image data and the second image data collected by the first image sensor and the second image sensor can correspond to the stereo pair images observed by the left and right eyes of the human, thereby simulating the binocular stereo vision of human eyes.

The other end of the operation part 160 is connected to the camera host 150 through the cable 140 , and transmits the first image data and the second image data to the camera host 150 through the cable 140 for processing. In some embodiments, the operation unit 160 can also send the image data to the camera host 150 through wireless transmission.

In some embodiments, the camera host 150 is provided with a processor, and the processor acquires the image data output by the camera 160, processes the image data, and outputs the processed image data. Exemplarily, the 3D endoscopic imaging system 100 further includes a display 170, and the camera host 150 is connected to the display 170 through a video connection line, and is used for sending endoscopic images to the display 170 for display.

It should be noted that FIG. 1 is only an example of a 3D endoscopic imaging system 100, and does not constitute a limitation to the 3D endoscopic imaging system 100. The 3D endoscopic imaging system 100 may include more or more than those shown in FIG. Fewer components, or a combination of some components, or different components, for example, the 3D endoscopic imaging system 100 may also include a dilator, a smoke control device, an input and output device, a network access device, and the like.

An imaging method for a 3D endoscope imaging system according to an embodiment of the present invention will be described below with reference to FIG. 2 . The method can be implemented by the 3D endoscope imaging system described with reference to FIG. 1 , and specifically can be implemented by a camera host of the 3D endoscope imaging system. 2 is a schematic flow chart of an imaging method 200 for a 3D endoscope imaging system in an embodiment of the present invention, which specifically includes the following steps: In step S210, the first image sensor of the 3D endoscope imaging system The first image data collected and the second image data collected by the second image sensor of the 3D endoscope imaging system, and the posture data collected by the posture sensor are acquired, and the posture data is used to indicate the first data and the The spatial position corresponding to the second image data;

In step S220, at least according to the gesture data, determine the currently applied display mode among the first display mode and the second display mode;

In step S230, when it is determined that the currently applied display mode is the first display mode, generate stereoscopic image data according to the first image data and the second image data, and control the display to Data display stereoscopic image;

In step S240, when it is determined that the currently applied display mode is the second display mode, generate two-dimensional image data according to the first image data and the second image data, and control the display according to the The two-dimensional image data displays a two-dimensional image, and the two-dimensional image is displayed upright according to the posture data.

According to the imaging method 200 for the 3D endoscope imaging system in the embodiment of the present invention, the currently applied display mode can be determined according to the posture data, and then the two-dimensional image or the stereoscopic image can be determined to be displayed; wherein, the two-dimensional image refers to the first The information of the image data and the second image data has a clearer and more stable imaging effect; when displaying a two-dimensional image, the two-dimensional image is displayed upright according to the posture data, so that the two-dimensional image is closer to the observation effect of the human eye.

In the embodiment of the present invention, the first display mode refers to the display mode for displaying stereoscopic images, and the second display mode refers to the display mode for displaying two-dimensional images. For example, the user can manually choose to use the first display mode as needed mode or the second display mode as the display mode of the current application. When the user chooses to adopt the first display mode, generate stereoscopic image data according to the first image data and the second image data, and display the stereoscopic image on the display; The two-dimensional image data generates two-dimensional image data, and displays the two-dimensional image on the display.

The user can also choose to automatically switch the display mode of the current application. When the user chooses to automatically switch the display mode, the display mode of the current application is automatically determined between the first display mode and the second display mode according to the attitude data collected by the attitude sensor.

Wherein, the posture sensor can be arranged at any position of the endoscope, and moves synchronously with the first image sensor and the second image sensor, and specifically can include an accelerometer, a gyroscope, a magnetometer, and the like. Since the first sensor and the second sensor are arranged in the insertion part of the endoscope, when the insertion part rotates around the tube lens axis, the first image sensor and the second image sensor also rotate around the tube lens axis, therefore, the attitude data collected by the attitude sensor It includes the rotation angle of the insertion part, that is, the rotation angle of the insertion part around the tube lens axis. This rotation angle can also be called the roll angle of the insertion part.

As shown in Figure 3, when the rotation angle of the insertion part is 0° or 180°, there is only horizontal parallax between the first image data and the second image data, and there is no vertical parallax, which has the best stereoscopic effect; When the rotation angle of the part is in the first rotation range (-θ to θ) or the third rotation range (90°+θ to 180°+θ), the vertical parallax between the first image data and the second image data is small, The stereoscopic image formed by the first image and the second image also has a better stereoscopic effect. If the rotation angle of the insertion part is in the second rotation range (θ to 90°+θ) or the fourth rotation range (180°+θ to -θ), the vertical parallax will become larger, and it will not be possible to obtain a better stereoscopic image. Effect.

Therefore, determining the currently applied display mode according to the attitude data collected by the attitude sensor specifically includes: when the difference between the rotation angle and the reference angle is less than a preset threshold, determining the first mode as the currently applied display mode to display the three-dimensional Image; when the difference between the rotation angle and the reference angle is greater than or equal to the preset angle, determine the second mode as the currently applied display mode to display a two-dimensional image. Wherein, the reference angle may be an angle at which there is no vertical parallax between the first image data and the second image data, which are 0° and 180° in FIG. 3 .

As mentioned above, if it is determined according to the user instruction that the second display mode is currently applied, or when it is determined according to the gesture data that the second display mode is currently applied, two-dimensional image data is generated according to the first image data and the second image data, and the two-dimensional image data is generated according to the two-dimensional The image data displays a two-dimensional image on a display. In the embodiment of the present invention, the two-dimensional image data contains at least part of the information in the first image data and at least part of the information in the second image data, instead of only using one of the image data, even if one of the two image data In the event of occlusion or dirt, it can also avoid loss of screen content and ensure the stability of the imaging effect.

In the embodiment of the present invention, when the first image data and the second image data are fused to generate two-dimensional image data, the detail information of one of the image data may be fused with the global information of the other image data. Wherein, the global information of the image data may include brightness information. Specifically, when merging image data, the brightness information of the two-dimensional image data may be determined according to the first brightness information of the first image data and the second brightness information of the second image data. Since the brightness information of the two-dimensional image data is fused with the brightness information of the two images, when the image data for providing detail information is switched, the brightness of the finally displayed two-dimensional image still does not change too obviously.

When determining the image data used to provide detailed information, it may be detected whether the first image data and the second image data are abnormal, wherein the abnormality of the image data may include the image data being blocked or dirty. When there is no abnormality in the two-way image data, taking any one-way image to provide detailed information can ensure the clarity of the two-dimensional image, so the detail information of any one-way image data can be taken, and the brightness information of the two-way images can be fused at the same time, and finally To obtain the fused two-dimensional image data, that is, the second brightness information of the second image data can be adjusted according to the first brightness information of the first image data to obtain two-dimensional image data, or the second brightness information of the second image data can be adjusted The second brightness information adjusts the first brightness information of the first image data to obtain two-dimensional image data. Optionally, when there is no abnormality in the image data of the two channels, the sharpness of the two images may also be compared, and the image with a higher definition is selected to provide detailed information.

Conversely, when one of the images is abnormal, the image without abnormality can be used as an image providing detail information, and the other image data is used to provide brightness information to ensure the clarity of the two-dimensional image. Wherein, when an abnormality is detected in the first image data, the detailed information of the second image data is obtained, and the second brightness information of the second image data is adjusted according to the first brightness information of the first image data to obtain a two-dimensional image data. Similarly, when an abnormality is detected in the second image data, the detailed information of the first image data is obtained, and the first brightness information of the first image data is adjusted according to the second brightness information of the second image data to obtain a two-dimensional image data.

Referring to FIG. 4 , it shows a schematic diagram of generating two-dimensional image data according to the first image data and the second image data when the first image data is occluded. Because the first image data is occluded, the detail information of the second image data is taken, and the brightness information of the first image data and the second image data are fused to obtain the final two-dimensional image data, so that the two-dimensional image is displayed clearly, and The brightness is between the first image and the second image. Referring to FIG. 5 , it shows a schematic diagram of generating two-dimensional image data according to the first image data and the second image data when both the first image data and the second image data are occluded. At this time, obtaining the detail information of the first image data or the second image data can ensure the definition of the two-dimensional image data, and make the brightness of the two-dimensional image between the first image and the second image.

Exemplarily, detecting whether there is abnormality in the first image data and the second image data specifically includes: first, determining the similarity between the first image data and the second image data. If the similarity between the first image data and the second image data is less than the first threshold th1, it is considered that one of the images may be blocked/dirty; in this case, further calculate the first image data. The first definition and the second definition of the second image data, and consider that the lower-resolution image data is the blocked/dirty image data, that is, if the first definition is lower than the second definition, it is determined The first image data is abnormal, and if the first definition is higher than the second definition, it is determined that the second image data is abnormal.

Wherein, when determining the similarity between the first image data and the second image data, the similarity description between the first image data and the second image data can be performed based on the image structure similarity (Structure Similarity Index Measure, SSIM), and the calculation The formula is:

Among them, L and R represent the first image data and the second image data respectively, μ _L and μ _R are the pixel mean values of the first image data and the second image data respectively, σ _L and σ _R are the first image data and the second image data respectively The pixel standard deviation of the two image data, σ _LR is the pixel covariance of the first image data and the second image data, and c1 and c2 are correction constants. If SSIM(L, R)<th1, it is considered that there is abnormality in one path of image data among the first image data and the second image data. Of course, the evaluation index for similarity in the embodiment of the present invention is not limited to SSIM, and other image similarity judgment algorithms such as hash algorithm and histogram algorithm can also be used to calculate the similarity between the first image data and the second image data .

When calculating the sharpness of the first image data and the second image data, multiple different sharpness evaluation indexes can also be used. For example, the edge strength of the first image data and the second image data can be extracted based on the edge extraction algorithm of the Sobel operator, and the sharpness F of the first image data or the second image data can be calculated according to the edge strength, and the calculation formula is:

Wherein, img represents an input image, representing the first image data or the second image data. In addition, other image definition calculation algorithms such as Gaussian operator, Laplacian operator, and image variance can also be used to describe the definition of the first image data or the second image data, and the path of image data with lower definition is determined to exist Image data of abnormal conditions such as occlusion and dirt.

Fusing the brightness information of the two channels of image data may specifically include performing weighted summation of the brightness information of the first image data and the brightness information of the second image data, so as to obtain the final brightness information of the two-dimensional image data. If there is no abnormality in the first image data and the second image data, then in weighted summation, the weight W1 of the first image data and the weight W2 of the second image data are close, for example W1=0.5, W2=0.5, so, Even if the image data providing detailed information is switched, the brightness of the finally displayed two-dimensional image will not change significantly.

When the first image data is abnormal, the second brightness information of the second image data is adjusted according to the first brightness information of the first image data. Specifically, weighted summation is performed on the average brightness of the first image data and the average brightness of the second image data to obtain the first weighted brightness; according to the ratio of the first weighted brightness to the average brightness of the second image data, the first Brightness coefficient: adjust the brightness value of each pixel in the second image data according to the first brightness coefficient, so as to obtain the brightness value of each pixel in the two-dimensional image data.

Wherein, the average brightness of the first image data is the average value of the brightness of all pixels in the first image data, wherein the brightness of each pixel is the value of the brightness channel Y after converting the image data from the RGB domain to the YCbCr domain. When the first image data is abnormal image data, the brightness of the output second image data can be calculated by the following formula:

Yout＝Y2*(W1*mean(Y1)+W2*mean(Y2))/mean(Y2) (Formula 4)

Wherein, Y1 and Y2 respectively refer to brightness channels of the first image data and the second image data, which contain detailed information of the first image data or the second image data. mean(Y1) and mean(Y2) are the average brightness of the first image data and the average brightness of the second image data respectively, representing the global brightness information of the first image data and the second image data; W1 is the first image data Weight coefficient, W2 is the weight coefficient of the second image data. It can be seen from formula 4 that the luminance channel Yout of the final output two-dimensional image data contains the detail information (Y2) of the second image data, and integrates the global luminance information (mean(Y1) of the first image data and the second image data and mean(Y2)).

Further, since the brightness of the first image data that is occluded or dirty is obviously low, in order to prevent the first image data from reducing the brightness of the fused two-dimensional image too significantly, the average brightness of the first image data mean(Y1 ) and the average brightness mean(Y2) of the second image data are weighted and summed to obtain the first weighted brightness, the weight coefficient W1 of the second image data is greater than the weight coefficient W2 of the first image data, such as W1=0.05, W2 = 0.95.

Similarly, adjusting the first brightness information of the first image data according to the second brightness information of the second image data includes: performing weighted summation of the average brightness of the second image data and the average brightness of the first image data to obtain second weighted brightness; according to the ratio of the second weighted brightness to the average brightness of the first image data, a second brightness coefficient is obtained; according to the second brightness coefficient, the brightness value of each pixel in the first image data is adjusted to obtain a two-dimensional The luminance value of each pixel in the image data; for details, please refer to the above formula 4, which will not be repeated here.

In addition to the luminance information of the two channels of image data, in some embodiments, the color information of the two channels of image data can also be fused. Specifically, the color information of the two-dimensional image data may be determined according to the first color information of the first image data and the second color information of the second image data.

Similar to the fusion of brightness information, when the image fusion method based on color information is used, it can detect whether there is anomaly in the first image data and the second image data, and provide detailed information for the image data without abnormality, and take the image data for abnormality. One channel of image data provides color information. Specifically, when an abnormality is detected in the first image data, the second color information of the second image data is adjusted according to the first color information of the first image data to obtain two-dimensional image data; when the second image data is detected When the data is abnormal, the first color information of the first image data is adjusted according to the second color information of the second image data to obtain two-dimensional image data. When it is detected that there is no abnormality in both the first image data and the second image data, the second color information of the second image data can be adjusted according to the first color information of the first image data, or the second image data can be adjusted according to the second image data. The second color information of the first image data is adjusted to the first color information.

Wherein, when the second color information of the second image data is adjusted according to the first color information of the first image data, the average color value of the first image data and the average color value of the second image data can be weighted and summed , to obtain the first weighted color value; according to the ratio of the first weighted color value to the average color value of the second image data, the first color coefficient is obtained; according to the first color coefficient, the color value of each pixel in the second image data is processed Adjust to get the color value of each pixel in the two-dimensional image data, namely:

Cout＝C2*(W3*mean(C1)+W4*mean(C2))/mean(C2) (Formula 5)

Wherein, C1 and C2 respectively refer to the color channels of the first image data and the second image data, and specifically can be any color channel in the RGB channel, which contains the detailed information of the first image data or the second image data. mean(C1) and mean(C2) are the average color value of the first image data and the average color value of the second image data respectively, representing the global color information of the first image data and the second image data; W3 is the first image The weight coefficient of the data, W4 is the weight coefficient of the second image data.

Further, if there is an abnormality in the first image data, when performing weighted summation of the average color value C1 of the first image data and the average color value C2 of the second image data, the weight coefficient W3 of the second image data is greater than the first The weight coefficient W4 of the image data is used to prevent the color of the two-dimensional image from being too significantly affected by the abnormal first image data.

Conversely, if there is an abnormality in the second image data, the average color value of the first image data and the average color value of the second image data can be weighted and summed to obtain the second weighted color value; according to the second weighted color value and The ratio of the average color values of the second image data to obtain a second color coefficient; adjust the color value of each pixel in the first image data according to the second color coefficient to obtain the color value of each pixel in the two-dimensional image data. Wherein, the weight coefficient of the first image data is greater than the weight coefficient of the second image data. When there is no abnormality in the first image data and the second image data, the weight coefficient W3 of the first image data and the weight coefficient W4 of the second image data may be close to or equal to each other.

After the two-dimensional image data is obtained according to the first image data and the second image data, the display is controlled to display the two-dimensional image according to the two-dimensional image data, and the two-dimensional image is displayed upright according to the posture data collected by the posture sensor. Displaying the 2D image upright according to the attitude data collected by the attitude sensor may specifically include rotating the direction of the 2D image according to the rotation angle of the insertion part, so that the direction of the 2D image remains unchanged when the insertion part rotates around the tube lens axis. Referring to FIG. 3 , when the two-dimensional image is displayed upright, although the insertion part has deviated from the reference angle, the display direction of the two-dimensional image can be consistent with the display direction at the reference angle. The upright direction of the two-dimensional image is consistent with the upright direction of the stereoscopic image. In some embodiments, in the first display mode, the stereoscopic image is displayed upright. When switching from the first display mode to the second display mode , the two-dimensional image is also displayed upright, so that the switching process of the image is relatively smooth.

In some embodiments, when the second display mode is applied, the user may be allowed to choose whether to make the two-dimensional image be displayed upright. When receiving a setting instruction to display the two-dimensional image upright, the two-dimensional image is displayed upright according to the posture data. When receiving a setting instruction to display a two-dimensional image non-erect, display the two-dimensional image in a non-erect state, and associate the direction of the two-dimensional image with the spatial position corresponding to the first image data and the second image data, that is When the insertion part is rotated around the tube lens axis, the two-dimensional image displayed on the monitor is also rotated.

When the user chooses to apply the first display mode, or determines to apply the first display mode according to the attitude data collected by the attitude sensor, generate stereoscopic image data according to the first image data and the second image data, and control the display to display the stereoscopic image. The stereoscopic image includes a first image and a second image with a certain parallax, which are output to the user's left eye and right eye respectively, and the stereoscopic effect is realized through the horizontal parallax between the first image and the second image. Therefore, the stereoscopic image data combines all the information in the first image data and the second image data.

There may be multiple ways to output the stereoscopic image, for example, it may be output through dual monitors or single monitor. The dual-display output includes a helmet-mounted display, a dual-screen display, etc., so that the observer's left and right eyes respectively view the first image and the second image displayed on different displays. Single display output mainly includes active shutter stereoscopic display methods, passive polarization stereoscopic display methods, etc. The active shutter stereoscopic display method needs to be used in conjunction with liquid crystal light valve glasses, and the passive polarization stereoscopic display method needs to be used in conjunction with polarized glasses. Both of them realize the image separation of the left and right eyes through the cooperation of the display and the glasses.

When displaying a stereoscopic image, the stereoscopic image can be displayed upright according to the attitude data, so as to avoid the loss of the stereoscopic effect caused by the head of the operator following the rotation of the image. Specifically, the stereoscopic image can be rotated according to the rotation angle of the insertion part, so that the first image and the second image are always consistent with the image direction at the reference angle during the rotation of the insertion part around the tube lens axis, as shown in FIG. 3 .

Alternatively, when the display is controlled to display the stereoscopic image according to the stereoscopic image data, the user may also be allowed to choose whether to display the stereoscopic image upright. When receiving the setting instruction to display the stereoscopic image upright, the stereoscopic image is displayed upright according to the posture data as described above. When receiving a setting instruction to display the stereoscopic image non-erectly, the stereoscopic image is displayed in a non-erect state, and the direction of the stereoscopic image is associated with the spatial positions corresponding to the first image data and the second image data, that is, when When the user rotates the operation part and drives the insertion part to rotate, the stereoscopic image displayed on the display also rotates accordingly.

In summary, the imaging method 200 for a 3D endoscope imaging system according to the embodiment of the present invention can switch the display mode according to the posture data, and can ensure that the stereoscopic image has a better stereoscopic effect in the first display mode. The two-dimensional image displayed in the display mode integrates the information of the two image data, and can also display organizational information to the user when there is an abnormality in one of the images.

An imaging method for a 3D endoscope imaging system according to another embodiment of the present invention will be described below with reference to FIG. 6 . The method can be implemented by the 3D endoscope imaging system described with reference to FIG. 1 , and specifically can be implemented by a camera host of the 3D endoscope imaging system. FIG. 6 is a schematic flowchart of an imaging method 600 for a 3D endoscope imaging system in an embodiment of the present invention, which specifically includes the following steps:

In step S610, acquiring first image data and second image data;

In step S620, the display mode of the current application is determined among the first display mode and the second display mode;

In step S630, when it is determined that the currently applied display mode is the first display mode, generate stereoscopic image data according to the first image data and the second image data, and control the display to Data display stereoscopic image;

In step S640, when it is determined that the currently applied display mode is the second display mode, generate two-dimensional image data according to the first image data and the second image data, and control the display according to the The two-dimensional image data displays a two-dimensional image.

In some embodiments, in step S610, the first image data is collected by the first image sensor of the 3D endoscopic imaging system, and the second image data is collected by the second image sensor of the 3D endoscopic imaging system. There is a parallax between the first image data and the second image data, so that a stereoscopic effect can be produced when the first image and the second image are displayed simultaneously.

In step S620, a currently applied display mode is determined among the first display mode and the second display mode. Specifically, the display mode of the current application may be determined in the first display mode and the second display mode according to user instructions and/or gesture data. Specifically, when the user instruction indicates to adopt the first display mode, the first display mode is determined as the currently applied display mode to display the stereoscopic image. When the user instruction indicates to adopt the second display mode, the second display mode is determined as the currently applied display mode to display the two-dimensional image. When the user instruction indicates to automatically determine the display mode, the currently applied display mode is determined among the first display mode and the second display mode according to the gesture data.

Wherein, the pose data is acquired during the process of acquiring the first image data and the second image data, and is used to indicate the spatial positions corresponding to the first image data and the second image data. Exemplarily, the 3D endoscope imaging system includes an endoscope, the endoscope includes an insertion part and an operation part, the first sensor and the second sensor are arranged in the insertion part, and the insertion part is used to insert into the patient's part to be observed, so as to obtain First image data and second image data of a patient's site to be observed. The posture data can be collected by a posture sensor, and the posture sensor can be arranged at any position of the insertion part or the operation part. The posture data includes the rotation angle of the insertion part, and since the first image sensor and the second image sensor are arranged in the insertion part, the rotation angle of the insertion part can reflect the spatial positions of the first image sensor and the second image sensor.

For example, if the currently applied display mode is determined in the first display mode and the second display mode according to the gesture data, when the difference between the rotation angle of the insertion part and the reference angle is less than a preset threshold, the first The mode is determined as the currently applied display mode to display a stereoscopic image; when the difference between the rotation angle and the reference angle is greater than or equal to a preset angle, the second mode is determined as the currently applied display mode to display a two-dimensional image . Wherein, the reference angle may be an angle at which there is no vertical parallax between the first image data and the second image data. When the insertion part is at the reference angle, there is only horizontal parallax in the first image data and the second image data, and there is no vertical parallax, which has the best stereoscopic effect; when the difference between the rotation angle of the insertion part and the reference angle If the value is smaller than the preset angle, the vertical parallax between the first image data and the second image data is small, and the stereoscopic image composed of the first image and the second image also has a better stereoscopic effect, so the stereoscopic image can also be displayed . If the difference between the rotation angle of the insertion part and the reference angle is greater than the preset angle, the vertical parallax will become larger, and a better stereoscopic effect cannot be obtained when displaying a stereoscopic image, so it is switched to displaying a 2D image.

In step S630, when it is determined that the first display mode is applied, stereoscopic image data is generated according to the first image data and the second image data, and the display is controlled to display a stereoscopic image according to the stereoscopic image data. Exemplarily, the stereoscopic image includes a first image and a second image with a certain parallax, which are respectively output to the user's left eye and right eye, and the stereoscopic effect is achieved through the horizontal parallax between the first image and the second image . Therefore, the stereoscopic image data combines all the information in the first image data and the second image data. There may be multiple ways to output the stereoscopic image, for example, it may be output through dual displays or a single display, which is not limited in this embodiment of the present invention.

When displaying a stereoscopic image, the stereoscopic image can be displayed upright according to the attitude data, so as to avoid the loss of the stereoscopic effect caused by the head of the operator following the rotation of the image. Specifically, the stereoscopic image may be rotated according to the rotation angle of the insertion part, so that the first image and the second image are always consistent with the image directions at the reference angle during the rotation of the insertion part around the tube lens axis. Alternatively, when the display is controlled to display the stereoscopic image according to the stereoscopic image data, the user may also be allowed to choose whether to display the stereoscopic image upright. When receiving the setting instruction to display the stereoscopic image upright, the stereoscopic image is displayed upright according to the posture data as described above. When receiving a setting instruction to display the stereoscopic image non-erectly, the stereoscopic image is displayed in a non-erect state, and the direction of the stereoscopic image is associated with the spatial positions corresponding to the first image data and the second image data, that is, when When the user rotates the operation part and drives the insertion part to rotate, the stereoscopic image displayed on the display also rotates accordingly.

In step S640, when it is determined that the second display mode is applied, two-dimensional image data is generated according to the first image data and the second image data, and the display is controlled to display a two-dimensional image according to the two-dimensional image data. Exemplarily, the two-dimensional image data includes partial information in the first image data and partial information in the second image data.

In some embodiments, when generating two-dimensional image data according to the first image data and the second image data, brightness information of the two can be integrated, that is, according to the first brightness information of the first image data and the second brightness information of the second image data Two, brightness information, for determining the brightness information of the two-dimensional image data. Specifically, detect whether the first image data and the second image data are abnormal; when the first image data is detected to be abnormal, adjust the second brightness information of the second image data according to the first brightness information of the first image data , to obtain two-dimensional image data; when abnormality is detected in the second image data, the first brightness information of the first image data is adjusted according to the second brightness information of the second image data, so as to obtain two-dimensional image data. That is, the brightness information of the abnormal image data can be used to adjust the brightness information of the non-abnormal image, so that the two-dimensional image data contains the detail information of the non-abnormal image and the brightness information of the two images. When providing image data with detailed information, it is possible to avoid too strong changes in image brightness.

Exemplarily, detecting whether there is abnormality in the first image data and the second image data includes: determining the similarity between the first image data and the second image data; when the similarity between the first image data and the second image data When the sharpness is less than the first threshold, determine the first sharpness of the first image data and the second sharpness of the second image data, if the first sharpness is lower than the second sharpness, then determine that there is an abnormality in the first image data, if If the first definition is higher than the second definition, it is determined that the second image data is abnormal. Since both the first image data and the second image data are collected for the part to be observed, when there is no abnormality in the two, the similarity is relatively large, so the similarity between the first image data and the second image data can be When it is less than the first threshold, it is determined that there is an abnormality in one channel of image data. At this point, the sharpness of the two can be calculated, and the path of image data with lower sharpness is determined to be abnormal.

Wherein, adjusting the second brightness information of the second image data according to the first brightness information of the first image data to obtain two-dimensional image data includes: adjusting the average brightness of the first image data and the average brightness of the second image data Perform weighted summation to obtain the first weighted brightness; obtain the first brightness coefficient according to the ratio of the first weighted brightness to the average brightness of the second image data; calculate the brightness value of each pixel in the second image data according to the first brightness coefficient Adjustment is performed to obtain the brightness value of each pixel in the two-dimensional image data. Similarly, the first brightness information of the first image data is adjusted according to the second brightness information of the second image data to obtain two-dimensional image data, including: the average brightness of the second image data and the average brightness of the first image data The brightness is weighted and summed to obtain the second weighted brightness; according to the ratio of the second weighted brightness to the average brightness of the first image data, the second brightness coefficient is obtained; the brightness of each pixel in the first image data is calculated according to the second brightness coefficient The value is adjusted to obtain the brightness value of each pixel in the two-dimensional image data.

Wherein, when the average brightness of the first image data and the average brightness of the second image data are weighted and summed to obtain the first weighted brightness, the weight coefficient of the first image data is smaller than the weight coefficient of the second image data; When the average brightness of the second image data and the average brightness of the first image data are weighted and summed to obtain the second weighted brightness, the weight coefficient of the first image data is greater than the weight coefficient of the second image data. That is, the channel of image data without abnormality has a larger weight coefficient, so as to avoid the brightness of the fused two-dimensional image being too low due to the brightness of abnormal image data being too low.

In addition, when it is detected that there is no abnormality in both the first image data and the second image data, the second brightness information of the second image data can be adjusted according to the first brightness information of the first image data to obtain a two-dimensional image The first brightness information of the first image data may also be adjusted according to the second brightness information of the second image data to obtain two-dimensional image data.

In other embodiments, the color information of the two-dimensional image data may be determined according to the first color information of the first image data and the second color information of the second image data. Specifically, detect whether the first image data and the second image data are abnormal; when the first image data is detected to be abnormal, adjust the second color information of the second image data according to the first color information of the first image data , to obtain two-dimensional image data; when abnormality is detected in the second image data, the first color information of the first image data is adjusted according to the second color information of the second image data to obtain two-dimensional image data.

Wherein, the second color information of the second image data is adjusted according to the first color information of the first image data to obtain two-dimensional image data, including: the average color value of the first image data and the average value of the second image data The color values are weighted and summed to obtain the first weighted color value; according to the ratio of the weighted color value to the average color value of the second image data, the first color coefficient is obtained; according to the first color coefficient, each pixel in the second image data Adjust the color value of the pixel to obtain the color value of each pixel in the two-dimensional image data. Adjust the first color information of the first image data according to the second color information of the second image data to obtain two-dimensional image data, including: the average color value of the second image data and the average color value of the first image data Perform weighted summation to obtain a second weighted color value; obtain a second color coefficient according to the ratio of the second weighted color value to the average color value of the second image data; calculate each pixel in the second image data according to the second color coefficient Adjust the color value of the pixel to obtain the color value of each pixel in the two-dimensional image data.

When the average color value of the first image data and the average color value of the second image data are weighted and summed to obtain the first weighted color value, the weight coefficient of the first image data is smaller than the weight coefficient of the second image data; When performing weighted summation on the average color value of the second image data and the average color value of the first image data to obtain the second weighted color value, the weight coefficient of the first image data is greater than the weight coefficient of the second image data. That is, a path of image data without abnormality has a larger weight coefficient.

In addition, when it is detected that there is no abnormality in both the first image data and the second image data, the second color information of the second image data can be adjusted according to the first color information of the first image data to obtain a two-dimensional image The first color information of the first image data may also be adjusted according to the second color information of the second image data to obtain two-dimensional image data.

After obtaining the two-dimensional image data, the display is controlled to display the two-dimensional image according to the two-dimensional image data. In some embodiments, posture data may also be obtained during the process of obtaining the first image data and the second image data, and the two-dimensional image data may be rotated according to the posture data, so that the two-dimensional image is displayed upright. Wherein, the attitude data is used to indicate the spatial positions corresponding to the first image data and the second image data. Alternatively, when the display is controlled to display the two-dimensional image according to the two-dimensional image data, the user may also be allowed to choose whether to display the two-dimensional image upright. When receiving a setting instruction to display the two-dimensional image upright, the two-dimensional image is displayed upright according to the posture data as described above. When receiving a setting instruction to display the two-dimensional image non-erectly, display the two-dimensional image in a non-erect state, and associate the direction of the two-dimensional image with the spatial position corresponding to the first image data and the second image data , that is, when the user rotates the operating part and drives the insertion part to rotate, the two-dimensional image displayed on the display also rotates accordingly.

To sum up, the imaging method 600 for a 3D endoscope imaging system according to the embodiment of the present invention can switch between the first display mode and the second display mode, and in the first display mode, it can ensure that the stereoscopic image has a higher With a good stereoscopic effect, the two-dimensional image displayed in the second display mode integrates the information of the two image data, and can also display organizational information to the user when there is an abnormality in one of the image data.

Referring again to FIG. 1 , the embodiment of the present invention also provides a 3D endoscope imaging system 100, including an endoscope and a camera host 150 connected to the endoscope. The endoscope includes an insertion part 130 and an operation part 160. The insertion part 130 is used to insert the patient's site to be observed; the endoscope includes a first image sensor and a second image sensor, which are used to collect the first image data and the second image data respectively; the camera host 150 is used to obtain from the endoscope The first image data and the second image data execute the imaging method 200 for a 3D endoscope imaging system according to the embodiment of the present invention.

Further, the 3D endoscope imaging system 100 also includes an attitude sensor disposed on the endoscope for collecting attitude data and sending the attitude data to the camera host 150, the attitude data is used to indicate the first image data and the second image The corresponding spatial location of the data. Wherein, the attitude sensor includes at least one, specifically including an accelerometer, a gyroscope, a magnetometer, and the like. The posture sensor may be installed at any position of the insertion part 130 or the operation part 160 . The optical axes of the first image sensor and the second image sensor can be arranged in parallel or at an angle. The first image sensor and the second image sensor can simulate binocular stereo vision of human eyes. The camera host 150 can determine the currently applied display mode in the first display mode and the second display mode according to the user instruction or the attitude data collected by the attitude sensor, and when it is determined to apply the first display mode, according to the first image data and the second The image data generates stereoscopic image data, and controls the display to display the stereoscopic image; when it is determined to apply the second display mode, generates two-dimensional image data according to the first image data and the second image data, and controls the display to display the two-dimensional image.

The specific structure of the 3D endoscopic imaging system 100 and the specific steps of the imaging method 200 for the 3D endoscopic imaging system and the imaging method 600 for the endoscopic imaging system have been described above, and will not be described here Do repeat. The 3D endoscopic imaging system 100 of the embodiment of the present invention can switch the display mode according to the user instruction and/or the posture data of the posture sensor. In the first display mode, it can ensure that the stereoscopic image has a better stereoscopic effect, and in the second display mode The two-dimensional image shown below integrates the information of the two image data, and can also display organizational information to the user when there is an abnormality in one of the images.

Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above-described example embodiments are exemplary only and are not intended to limit the scope of the invention thereto. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another device, or some features may be omitted, or not implemented.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be understood that in the description of the exemplary embodiments of the invention, in order to streamline the disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure , or in its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the corresponding claims reflect, the inventive point lies in that the corresponding technical problem may be solved by using less than all features of a single disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

It will be appreciated by those skilled in the art that all features disclosed in this specification (including accompanying claims, abstract and drawings) and all features of any method or apparatus so disclosed may be used in any combination, except where the features are mutually exclusive. process or unit. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some modules according to the embodiments of the present invention. The present invention can also be implemented as an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

The above is only a specific embodiment of the present invention or a description of the specific embodiment, and the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily Any changes or substitutions that come to mind should be covered within the protection scope of the present invention. The protection scope of the present invention should be based on the protection scope of the claims.

Claims (32)

1. An imaging method for a 3D endoscopic imaging system, the method comprising:

acquiring first image data acquired by a first image sensor of the 3D endoscopic imaging system and second image data acquired by a second image sensor of the 3D endoscopic imaging system, and acquiring gesture data acquired by a gesture sensor, wherein the gesture data is used for indicating a spatial position corresponding to the first image data and the second image data;

determining a currently applied display mode in a first display mode and a second display mode according to at least the gesture data;

when the currently applied display mode is determined to be the first display mode, generating stereoscopic image data according to the first image data and the second image data, and controlling a display to display a stereoscopic image according to the stereoscopic image data;

when the currently applied display mode is determined to be the second display mode, generating two-dimensional image data according to the first image data and the second image data, controlling the display to display a two-dimensional image according to the two-dimensional image data, and enabling the two-dimensional image to be displayed in an upright mode according to the posture data.

2. The method of claim 1, further comprising determining a currently applied display mode among the first display mode and the second display mode according to a user instruction, wherein:

when the user instruction indicates that the first display mode is adopted, determining the first display mode as a currently applied display mode to display the stereoscopic image;

when the user instruction indicates that the second display mode is adopted, the second display mode is determined as the currently applied display mode so as to display the two-dimensional image.

3. The method according to claim 1, wherein the 3D endoscopic imaging system comprises an endoscope including an insertion portion and an operation portion, the first and second sensors being provided in the insertion portion, the insertion portion being used to be inserted into a site to be observed of a patient to acquire the first and second image data of the site to be observed of the patient, the posture data including a rotation angle of the insertion portion;

the determining a currently applied display mode in a first display mode and a second display mode according to at least the gesture data comprises:

when the difference value between the rotation angle and a reference angle is smaller than a preset threshold value, determining the first mode as a currently applied display mode to display the stereoscopic image;

and when the difference value between the rotation angle and the reference angle is greater than or equal to the preset angle, determining the second mode as a currently applied display mode to display the two-dimensional image.

4. The method of claim 3, wherein the reference angle is an angle at which there is no vertical disparity in the first image data and the second image data.

5. The method of claim 1, wherein generating two-dimensional image data from the first image data and the second image data comprises:

and determining the brightness information of the two-dimensional image data according to the first brightness information of the first image data and the second brightness information of the second image data.

6. The method of claim 5, wherein determining the luminance information of the two-dimensional image data from the first luminance information of the first image data and the second luminance information of the second image data comprises:

detecting whether the first image data and the second image data have abnormity or not;

when the first image data is detected to be abnormal, adjusting second brightness information of the second image data according to first brightness information of the first image data to obtain two-dimensional image data;

when the second image data is detected to be abnormal, adjusting first brightness information of the first image data according to second brightness information of the second image data to obtain the two-dimensional image data.

7. The method of claim 6, wherein said detecting whether an anomaly exists in the first image data and the second image data comprises:

determining a similarity between the first image data and the second image data;

when the similarity between the first image data and the second image data is smaller than a first threshold value, determining a first definition of the first image data and a second definition of the second image data, if the first definition is lower than the second definition, determining that the first image data is abnormal, and if the first definition is higher than the second definition, determining that the second image data is abnormal.

8. The method according to claim 1, wherein the two-dimensional image data includes partial information in the first image data and partial information in the second image data.

9. An imaging method for a 3D endoscopic imaging system, the method comprising:

acquiring first image data and second image data;

determining a currently applied display mode in the first display mode and the second display mode;

when the currently applied display mode is determined to be the first display mode, generating stereoscopic image data according to the first image data and the second image data, and controlling a display to display a stereoscopic image according to the stereoscopic image data;

and when the currently applied display mode is determined to be the second display mode, generating two-dimensional image data according to the first image data and the second image data, and controlling the display to display a two-dimensional image according to the two-dimensional image data.

10. The method of claim 9, wherein the first image data is acquired by a first image sensor of the 3D endoscopic imaging system and the second image data is acquired by a second image sensor of the 3D endoscopic imaging system.

11. The method of claim 9, wherein the two-dimensional image data includes partial information in the first image data and partial information in the second image data.

12. The method of claim 9, wherein the method further comprises: acquiring attitude data in the process of acquiring the first image data and the second image data, wherein the attitude data is used for indicating the spatial positions corresponding to the first image data and the second image data;

the determining a display mode of a current application in the first display mode and the second display mode includes: and determining the display mode of the current application in the first display mode and the second display mode according to a user instruction and/or the gesture data.

13. The method of claim 12, wherein determining a currently applied display mode among a first display mode and a second display mode according to the user instruction comprises:

when the user instruction indicates that the first display mode is adopted, determining the first display mode as a currently applied display mode to display the stereoscopic image;

when the user instruction indicates that the second display mode is adopted, the second display mode is determined as the currently applied display mode so as to display the two-dimensional image.

14. The method according to claim 12, wherein the 3D endoscopic imaging system includes an endoscope including an insertion portion and an operation portion, the first and second sensors being provided in the insertion portion, the insertion portion being used to be inserted into a site to be observed of a patient to acquire the first and second image data of the site to be observed of the patient, the posture data including a rotation angle of the insertion portion;

determining a currently applied display mode in a first display mode and a second display mode according to the attitude data, including:

when the difference value between the rotation angle and a reference angle is smaller than a preset threshold value, determining the first mode as a currently applied display mode to display the stereoscopic image;

and when the difference value between the rotation angle and the reference angle is greater than or equal to the preset angle, determining the second mode as a currently applied display mode to display the two-dimensional image.

15. The method of claim 14, wherein the reference angle is an angle at which there is no vertical disparity in the first image data and the second image data.

16. The method of claim 9, wherein the method further comprises: acquiring attitude data in the process of acquiring the first image data and the second image data, wherein the attitude data is used for indicating the spatial positions corresponding to the first image data and the second image data;

rotating the two-dimensional image data in accordance with the pose data to cause the two-dimensional image to be displayed upright.

17. The method of claim 16,

the method further comprises the following steps: when a setting instruction for causing the two-dimensional image to be displayed in a non-upright state is received, the two-dimensional image is displayed in a non-upright state, and the direction of the two-dimensional image is associated with the spatial position corresponding to the first image data and the second image data.

18. The method of claim 9, wherein the method further comprises: acquiring attitude data in the process of acquiring the first image data and the second image data, wherein the attitude data is used for indicating the spatial positions corresponding to the first image data and the second image data;

and rotating the stereo image data according to the attitude data so that the stereo image is displayed in an upright manner.

19. The method of claim 18,

the method further comprises the following steps: when a setting instruction for causing the stereoscopic image to be displayed in a non-upright state is received, the stereoscopic image is displayed in a non-upright state, and the direction of the stereoscopic image is associated with the spatial position corresponding to the first image data and the second image data.

20. The method of claim 9, wherein generating two-dimensional image data from the first image data and the second image data comprises:

and determining the brightness information of the two-dimensional image data according to the first brightness information of the first image data and the second brightness information of the second image data.

21. The method of claim 20, wherein determining the luminance information of the two-dimensional image data based on the first luminance information of the first image data and the second luminance information of the second image data comprises:

detecting whether the first image data and the second image data have abnormity or not;

when the first image data is detected to be abnormal, adjusting second brightness information of the second image data according to first brightness information of the first image data to obtain two-dimensional image data;

when the second image data is detected to be abnormal, adjusting the first brightness information of the first image data according to the second brightness information of the second image data to obtain the two-dimensional image data.

22. The method of claim 21, wherein the adjusting second luminance information of the second image data according to the first luminance information of the first image data to obtain the two-dimensional image data comprises:

carrying out weighted summation on the average brightness of the first image data and the average brightness of the second image data to obtain first weighted brightness;

obtaining a first brightness coefficient according to the ratio of the first weighted brightness to the average brightness of the second image data;

adjusting the brightness value of each pixel in the second image data according to the first brightness coefficient to obtain the brightness value of each pixel in the two-dimensional image data;

the adjusting first brightness information of the first image data according to second brightness information of the second image data to obtain the two-dimensional image data includes:

carrying out weighted summation on the average brightness of the second image data and the average brightness of the first image data to obtain second weighted brightness;

obtaining a second brightness coefficient according to the ratio of the second weighted brightness to the average brightness of the first image data;

and adjusting the brightness value of each pixel in the first image data according to the second brightness coefficient to obtain the brightness value of each pixel in the two-dimensional image data.

23. The method of claim 22, wherein when the average luminance of the first image data and the average luminance of the second image data are weighted and summed to obtain the first weighted luminance, the weight coefficient of the first image data is smaller than the weight coefficient of the second image data;

when the average brightness of the second image data and the average brightness of the first image data are subjected to weighted summation to obtain second weighted brightness, the weight coefficient of the first image data is larger than that of the second image data.

24. The method of claim 21, wherein determining the luminance information of the two-dimensional image data based on first luminance information of the first image data and second luminance information of the second image data, further comprises:

when it is detected that neither the first image data nor the second image data is abnormal, adjusting second brightness information of the second image data according to first brightness information of the first image data to obtain the two-dimensional image data, or adjusting first brightness information of the first image data according to second brightness information of the second image data to obtain the two-dimensional image data.

25. The method of claim 9, wherein generating two-dimensional image data from the first image data and the second image data comprises:

and determining color information of the two-dimensional image data according to the first color information of the first image data and the second color information of the second image data.

26. The method of claim 25, wherein determining color information for the two-dimensional image data based on the first color information for the first image data and the second color information for the second image data comprises:

detecting whether the first image data and the second image data have abnormity or not;

when the first image data is detected to be abnormal, adjusting second color information of the second image data according to first color information of the first image data to obtain two-dimensional image data;

when the second image data is detected to be abnormal, adjusting the first color information of the first image data according to the second color information of the second image data to obtain the two-dimensional image data.

27. The method of claim 26, wherein the adjusting second color information of the second image data according to first color information of the first image data to obtain the two-dimensional image data comprises:

weighted summation is carried out on the average color value of the first image data and the average color value of the second image data to obtain a first weighted color value;

obtaining a first color coefficient according to the ratio of the weighted color value to the average color value of the second image data;

adjusting the color value of each pixel in the second image data according to the first color coefficient to obtain the color value of each pixel in the two-dimensional image data;

the adjusting first color information of the first image data according to second color information of the second image data to obtain the two-dimensional image data includes:

performing weighted summation on the average color value of the second image data and the average color value of the first image data to obtain a second weighted color value;

obtaining a second color coefficient according to the ratio of the second weighted color value to the average color value of the second image data;

and adjusting the color value of each pixel in the second image data according to the second color coefficient to obtain the color value of each pixel in the two-dimensional image data.

28. The method of claim 27, wherein in weighted summing the average color value of the first image data and the average color value of the second image data to obtain a first weighted color value, the weight coefficient of the first image data is smaller than the weight coefficient of the second image data;

and when the average color value of the second image data and the average color value of the first image data are subjected to weighted summation to obtain a second weighted color value, the weight coefficient of the first image data is greater than the weight coefficient of the second image data.

29. The method of claim 28, wherein determining color information of the two-dimensional image data from first color information of the first image data and second color information of the second image data, further comprises:

when it is detected that neither the first image data nor the second image data is abnormal, adjusting second color information of the second image data according to first color information of the first image data to obtain the two-dimensional image data, or adjusting first color information of the first image data according to second color information of the second image data to obtain the two-dimensional image data.

30. The method of claim 21 or 25, wherein said detecting whether the first image data and the second image data have anomalies comprises:

determining a similarity between the first image data and the second image data;

when the similarity between the first image data and the second image data is smaller than a first threshold value, determining a first definition of the first image data and a second definition of the second image data, if the first definition is lower than the second definition, determining that the first image data is abnormal, and if the first definition is higher than the second definition, determining that the second image data is abnormal.

31. The 3D endoscope imaging system is characterized by comprising an endoscope and a camera host connected with the endoscope, wherein the endoscope comprises an insertion part and an operation part, and the insertion part is used for being inserted into a part to be observed of a patient;

the endoscope comprises a first image sensor and a second image sensor which are respectively used for acquiring first image data and second image data;

the camera host is configured to acquire the first image data and the second image data from the endoscope to perform the method of any of claims 1-30.

32. The 3D endoscopic imaging system of claim 31, further comprising a pose sensor disposed on the endoscope for collecting pose data indicating a spatial position corresponding to the first image data and the second image data and transmitting the pose data to the camera host.

Images