WO2024188159A1 - Imaging and display method for endoscope system, and endoscope system - Google Patents

Abstract

An imaging and display method for an endoscope system, and an endoscope system. The method comprises: acquiring a visible light image signal and a fluorescent image signal of a part to be observed; generating at least two endoscope images of different image types on the basis of the visible light image signal and the fluorescent image signal, wherein the endoscope images at least comprise a first endoscope image of a first image type, the first endoscope image is a three-dimensional endoscope image, the first endoscope image is generated according to left path image data and right path image data, and there is parallax between the left path image data and the right path image data; and simultaneously displaying the at least two endoscope images of different image types in at least two display regions of a display interface, respectively, wherein the display regions at least comprise a first display region, and the first endoscope image is displayed in the first display region. The present invention can achieve the simultaneous display of at least two endoscope images of different image types, and provides a three-dimensional visual effect.

Description

Imaging and display method for endoscope system and endoscope system

Citation of Related Applications

This application is based on the Chinese patent application filed on March 13, 2023, with application number 202310239872.2, and claims its priority. All its records are introduced here by reference.

Technical Field

The present invention relates to the field of medical equipment, and more particularly to an imaging and display method for an endoscope system and an endoscope system.

Background Art

Endoscopes can present the tissue morphology of a patient's internal organs and the pathological conditions in the body during minimally invasive surgery, facilitating diagnosis and implementation of surgery. They are one of the important tools for diagnosis and treatment in modern medicine. Traditional endoscopes only provide two-dimensional images, and doctors can only rely on experience to judge the depth information of various organs in the human body cavity during surgery, so they are only suitable for simple operations. In recent years, stereo endoscopes that can observe areas in three dimensions have been developed. Compared with traditional endoscopes that can provide three-dimensional information, they can better reflect the real situation of the scene, allowing doctors to feel the visual effects of three-dimensional images during surgery, providing realistic visual experience, and helping doctors to operate endoscopes more accurately.

Endoscopic imaging systems are usually capable of both visible light imaging and fluorescence imaging. Visible light imaging is imaging the tissue of the part to be observed under visible light irradiation, and fluorescence imaging is imaging the fluorescence generated by the excited fluorescent reagent injected into the tissue. Stereoscopic endoscopes can only display stereoscopic visible light images or stereoscopic fluorescence images. In stereoscopic display mode, users cannot view endoscopic images of different image types at the same time.

Summary of the invention

A series of simplified concepts are introduced in the Summary of the Invention, which will be further described in detail in the Detailed Description of the Invention. The Summary of the Invention does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the scope of protection of the claimed technical solution.

The present application provides an imaging and display method for an endoscope system, including:

Acquire a visible light image signal and a fluorescent image signal of a part to be observed, wherein the visible light image signal and the fluorescent image signal both include left image data and right image data, and there is parallax between the left image data and the right image data;

generating at least two endoscopic images of different image types based on the visible light image signal and the fluorescent image signal, the image types including visible light images, monochrome fluorescent images, color fluorescent images, monochrome fused images, and color fused images; wherein the endoscopic image at least includes a first endoscopic image of a first image type, the first endoscopic image is a stereoscopic endoscopic image, and the first endoscopic image is generated based on the left-path image data and the right-path image data;

The at least two endoscopic images of different image types are respectively displayed simultaneously in at least two display areas of the display interface, wherein the display areas at least include a first display area, and the first endoscopic image is displayed in the first display area.

In one embodiment, the endoscopic image further includes a second endoscopic image of a second image type, the second endoscopic image is a stereoscopic endoscopic image, and the second endoscopic image is generated based on the left image data and the right image data;

The display area also includes a second display area, and the second endoscopic image is displayed in the second display area.

In one embodiment, an area of the first display region is greater than an area of the second display region.

In one embodiment, the method further includes: performing parallax adjustment on the first endoscopic image to reduce the parallax of the first endoscopic image; or performing parallax adjustment on the second endoscopic image to increase the parallax of the second endoscopic image.

In one embodiment, the endoscopic image also includes a third endoscopic image of a third image type and a fourth endoscopic image of a fourth image type, the third endoscopic image and the fourth endoscopic image are both stereoscopic images, the third endoscopic image is generated based on the left image data and the right image data, and the fourth endoscopic image is generated based on the left image data and the right image data.

In one embodiment, the display area also includes a third display area and a fourth display area, the third endoscopic image is displayed in the third display area, the fourth endoscopic image is displayed in the fourth display area, and the area of the first display area is larger than the areas of the second display area, the third display area, and the fourth display area.

In one embodiment, the first display area has a first area, and the second display area, The third display area and the fourth display area both have a second area, and the first area is larger than the second area.

In one embodiment, the method further includes: performing parallax adjustment on the first endoscopic image to reduce the parallax of the first endoscopic image; or, performing parallax adjustment on the second endoscopic image, the third endoscopic image and the fourth endoscopic image respectively to increase the parallax of the second endoscopic image, the third endoscopic image and the fourth endoscopic image.

In one embodiment, it is characterized in that the image type of at least one of the first endoscopic image, the second endoscopic image, the third endoscopic image and the fourth endoscopic image is a monochrome fluorescence image or a color fluorescence image generated based on the fluorescence image signal, or is a monochrome fusion image or a color fusion image generated based on the visible light image signal and the fluorescence image signal.

In one embodiment, the endoscopic image further comprises a third endoscopic image of a third image type, the third endoscopic image being a two-dimensional endoscopic image;

The display area further includes a third display area in which the third endoscopic image is displayed.

In one embodiment, except for the first endoscopic image of the one image type, one or more of the endoscopic images of other image types are two-dimensional endoscopic images.

In one embodiment, the area of the first display region is larger than that of other display regions.

In one embodiment, the at least two display regions have the same area.

In one embodiment, the method further comprises:

In response to an operation instruction of a user, endoscopic images of different image types displayed in the different display areas are switched for display.

In one embodiment, the method further comprises:

An external image from an external image source is acquired, and the external image is displayed in a display area of the display interface.

In one embodiment, the method further comprises:

Acquire medical images from other medical devices and display the medical images in the display area of the display interface.

In one embodiment, the other medical equipment includes at least one of an ultrasound device, a monitoring device, and a ventilation device.

In one embodiment, the medical image includes at least one of an ultrasound examination image of an ultrasound device, a patient physiological parameter waveform of a monitoring device, and a breathing control parameter waveform of a ventilation device.

The present application also provides an endoscope system, including an endoscope, a light source, and a connection The endoscope comprises an insertion portion and an operation portion, wherein the insertion portion is used to be inserted into a patient's part to be observed;

The light source is used to provide illumination to the part to be observed;

The endoscope is provided with at least one image sensor, and the at least one image sensor is used to collect visible light image signals and fluorescent image signals of the part to be observed;

The camera host is used to obtain the visible light image signal and the fluorescent image signal from the at least one image sensor to perform the imaging and display method for an endoscope system in any of the above embodiments, and obtain at least two endoscopic images of different image types;

A display is used to display the at least two endoscopic images of different image types.

The present application also provides another imaging and display method for an endoscope system, including:

Acquiring visible light image signals and fluorescent image signals of the part to be observed;

generating at least two endoscopic images of different image types based on the visible light image signal and the fluorescent image signal, wherein the endoscopic images at least include a first endoscopic image of a first image type, the first endoscopic image is a stereoscopic endoscopic image, the first endoscopic image is generated based on left-path image data and right-path image data, and there is parallax between the left-path image data and the right-path image data;

The at least two endoscopic images of different image types are respectively displayed simultaneously in at least two display areas of the display interface, wherein the display areas at least include a first display area, and the first endoscopic image is displayed in the first display area.

The imaging and display method for an endoscope system and the endoscope system according to the embodiments of the present invention can simultaneously display at least two endoscopic images of different image types, thereby facilitating doctors to observe between different types of endoscopic images and providing a stereoscopic visual effect.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

In the attached picture:

FIG1 is a schematic block diagram of an endoscope system according to an embodiment of the present invention;

FIG. 2 is a schematic diagram showing an imaging and display method for an endoscope system according to an embodiment of the present invention. Sexual flow chart;

FIG3 is a schematic diagram showing a display interface according to an embodiment of the present invention;

4A-4C are schematic diagrams showing a display interface layout according to an embodiment of the present invention;

5A and 5B are schematic diagrams showing a display interface layout according to another embodiment of the present invention;

FIG6 is a schematic diagram showing the parallax sizes of different display areas according to an embodiment of the present invention;

FIG. 7 is a schematic diagram showing a principle of parallax adjustment according to an embodiment of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without paying creative work should fall within the protection scope of the present invention.

In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

It should be understood that the present invention can be implemented in different forms and should not be interpreted as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make the disclosure thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

The purpose of the terms used herein is only to describe specific embodiments and is not intended to be limiting of the present invention. When used herein, the singular forms "one", "an" and "said/the" are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "consisting of" and/or "comprising", when used in this specification, determine the presence of the features, integers, steps, operations, elements and/or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and/or groups. When used herein, the term "and/or" includes any and all combinations of the relevant listed items.

In order to fully understand the present invention, a detailed structure will be proposed in the following description to illustrate the technical solution proposed by the present invention. The optional embodiments of the present invention are described in detail as follows, but in addition to these detailed descriptions, the present invention may also have other implementations.

Below, an endoscope system according to an embodiment of the present invention is first described with reference to FIG. 1 , which shows a schematic structural block diagram of an endoscope system 100 according to an embodiment of the present invention.

As shown in FIG1 , the endoscope system 100 includes at least an endoscope and a camera host 150 connected to the endoscope. The endoscope includes an insertion portion 130 and an operating portion 160. The insertion portion 130 is used to be inserted into the patient's part to be observed. The insertion portion 130 and the operating portion 160 can be an integral structure or a detachable structure. The endoscope also includes at least one image sensor (not shown). Exemplarily, the image sensor can be arranged at the front end of the insertion portion 130 of the endoscope. The camera host 150 is used to obtain an image signal from the endoscope to perform signal processing on it. In addition, the endoscope system 100 also includes a light source 110, a light guide 120, a cable 140, and the light source 110 is connected to the endoscope through the light guide 120. The operating portion 160 is connected to the camera host 150 through the cable 140, and is connected to the light source 110 through the light guide 120.

Wherein, the light source 110 is used to provide an illumination light source to the part to be observed. The light source 110 may include a visible light source and a special light source. Exemplarily, the visible light source is an LED light source, which can provide multiple monochromatic lights of different wavelength ranges, a combination of multiple monochromatic lights, or a wide-spectrum white light source. The special light source can be a laser light source corresponding to a fluorescent agent, such as near-infrared light. In some embodiments, before imaging using an endoscopic system, a fluorescent agent is injected into the part to be observed, and the fluorescent agent can generate fluorescence after absorbing the laser generated by the laser light source.

The insertion part 130 of the endoscope includes a mirror tube, an image sensor and an illumination light path. The front end of the mirror tube is used to be inserted into the human body and penetrate into the part to be inspected. The illumination light path is connected to the light guide 120, and is used to irradiate the light generated by the light source 110 to the part to be inspected of the target object. The image sensor can specifically include a first image sensor and a second image sensor, which are used to convert optical signals into electrical signals, including but not limited to CCD sensors, CMOS sensors, etc. The image signal collected by the image sensor is sent to the camera host 150 for subsequent image processing after preliminary signal processing by the operating part 160, and the preliminary signal processing includes amplification, filtering and other processing. The optical axes of the first image sensor and the second image sensor can be set in parallel or at a certain angle. The left image signal and the right image signal collected by the first image sensor and the second image sensor can correspond to the stereo pair image observed by the left and right eyes of the human body, thereby simulating the binocular stereoscopic vision of the human eye.

The other end of the operation unit 160 is connected to the camera host 150 through the cable 140, and transmits the image signal to the camera host 150 for processing through the cable 140. In some embodiments, the operation unit 160 can also send the image data to the camera host 150 by wireless transmission.

In some embodiments, the camera host 150 is provided with a processor, which obtains the image signal output by the operation unit 160, processes the image signal, and outputs the processed image data. The endoscope system 100 further includes a display 170 , and the camera host 150 is connected to the display 170 via a video connection line, so as to send the endoscopic image to the display 170 for display.

It should be noted that FIG1 is merely an example of the endoscope system 100 and does not constitute a limitation on the endoscope system 100. The endoscope system 100 may include more or fewer components than those shown in FIG1, or a combination of certain components, or different components. For example, the endoscope system 100 may also include a dilator, a smoke control device, input and output devices, a network access device, etc.

Next, the imaging and display method for an endoscope system according to an embodiment of the present invention is described with reference to FIG2. The method can be implemented by the endoscope system described with reference to FIG1, and can be implemented by the camera host of the endoscope system by controlling other components. FIG2 is a schematic flow chart of an imaging and display method 200 for an endoscope system according to an embodiment of the present invention, which specifically includes the following steps:

In step S210 , a visible light image signal and a fluorescent image signal of a part to be observed are acquired.

In step S220, at least two endoscopic images of different image types are generated based on the visible light image signal and the fluorescent image signal, wherein the endoscopic image at least includes a first endoscopic image of a first image type, the first endoscopic image is a stereoscopic endoscopic image, and the first endoscopic image is generated based on left-path image data and right-path image data, and there is parallax between the left-path image data and the right-path image data. The "image type" described in the embodiments of the present application refers to a visible light image, a fluorescent image, or a type of image obtained by fusing the two, and these image types can be stereoscopic images or two-dimensional images.

In step S230, the at least two endoscopic images of different image types are respectively displayed simultaneously in at least two display areas of the display interface, wherein the display areas include at least a first display area, and the first endoscopic image is displayed in the first display area.

The imaging and display method 200 of the endoscope system of the embodiment of the present invention can simultaneously display endoscopic images of different image types, and display endoscopic images of at least one image type as a stereoscopic endoscopic image, so that the user can obtain a stereoscopic visual effect while viewing multiple types of endoscopic images.

Below, we will first explain the stereoscopic endoscopic image of an embodiment of the present invention. The endoscopic system achieves a stereoscopic imaging effect based on the binocular parallax principle. The binocular parallax principle means that when the human eye observes an object, the object is projected onto the retina of the left and right eyes respectively, forming two images with parallax. The human brain can automatically calculate the depth information of the object based on the parallax, thereby forming a stereoscopic visual effect. Based on the above principle, the display of the endoscope system alternately or simultaneously displays the left image and the right image with horizontal parallax, and uses the corresponding image separation technology to make the left eye image and the right eye image with horizontal parallax enter the human eye respectively. The left and right eyes of the observer are combined to make the human brain automatically fuse the stereoscopic effect and obtain depth information. Therefore, it can be understood that the stereoscopic endoscope image in the embodiment of the present invention refers to an image that can make the observer produce a stereoscopic visual effect, which essentially includes a left-path image output to the left eye and a right-path image output to the right eye.

There are many ways to output stereoscopic endoscopic images, for example, they can be output through dual monitors or single monitors. Dual monitor output includes helmet-mounted monitors, dual-screen monitors, etc., so that the observer's left and right eyes can view the left and right images displayed on different monitors respectively. Single monitor output mainly includes active shutter stereo display mode, passive polarization stereo display mode, etc. Active shutter stereo display mode needs to be used in conjunction with liquid crystal light valve glasses, and passive polarization stereo display mode needs to be used in conjunction with polarization glasses. Both of them realize left and right eye image separation through the cooperation of monitors and glasses.

Specifically, in step S210, a visible light image signal and a fluorescent image signal of the part to be observed are acquired. Exemplarily, the visible light image signal includes a left visible light image signal and a right visible light image signal, and the fluorescent image signal includes a left fluorescent image signal and a right fluorescent image signal. The left visible light image signal and the left fluorescent image signal may be acquired by the same image sensor or by different image sensors; similarly, the right visible light image signal and the right fluorescent image signal may be acquired by the same image sensor or by different image sensors. The image sensor for acquiring visible light image signals and fluorescent image signals may be disposed at the front end of the endoscope insertion portion. When the endoscope insertion portion is inserted into the patient's part to be observed, the image sensor approaches the part to be observed and acquires the visible light image signal and the fluorescent image signal of the part to be observed.

Among them, when collecting visible light image signals, the light source of the endoscopic system can provide multiple monochromatic lights of different wavelength ranges in time-sharing, and the provided monochromatic light can be blue light, green light, red light, etc.; or the light source can also provide a combination of multiple monochromatic lights, or a wide-spectrum white light source. When collecting fluorescent image signals, the light source provides a narrow-band laser to excite the fluorescent agent to produce fluorescence. Before using the endoscope system to image the part to be observed, a fluorescent agent is injected into the part to be observed. During the imaging process, special light illumination for exciting the fluorescent agent can be provided to the part to be observed, so that the fluorescent agent is excited to produce fluorescence; the part to be observed includes but is not limited to the blood circulation system, lymphatic system and tumor tissue, etc.

In step S220, at least two endoscopic images of different image types are generated based on the visible light image signal and the fluorescent image signal, that is, in the at least two endoscopic images of different image types, the endoscopic image of each image type can be generated based on the visible light image signal alone, can be generated based on the fluorescent image signal alone, or can be generated based on both the visible light image signal and the fluorescent image signal. Endoscopic images of different image types can provide different tissue-related information. For example, fluorescent images can show high contrast between diseased tissues such as tumor cells and normal tissue images, so that doctors can observe information such as tumor size and contour to achieve the purpose of precise treatment.

Specifically, the at least two endoscopic images of different image types include at least two of the following: a visible light image generated based on a visible light image signal, a fluorescent image generated based on a fluorescent image signal, and a fused image generated based on a visible light image signal and a fluorescent image signal.

The visible light image may be a color RGB image or a grayscale image. The fluorescence image may be a grayscale image or a monochrome image, or a color image obtained according to the mapping relationship between signal intensity and color value. After generating the visible light image and the fluorescence image, a fused image may be generated based on the visible light image and the fluorescence image. Specifically, the visible light image and the fluorescence image may be fused to obtain a fused image, which contains both the information of the visible light image and the information of the fluorescence image. The fused image may be generated by superimposing the layers of the visible light image and the fluorescence image. For example, the fluorescence image may be processed to have a certain degree of transparency, and then the fluorescence image may be superimposed on the visible light image to form a fused image. Alternatively, each pixel point on each fused image may be synthesized according to the pixel value of each pixel point on the visible light image and the pixel value of the corresponding pixel point in the fluorescence image. Specifically, certain weights may be assigned to the visible light image and the fluorescence image respectively, and the sum of the weights is 1. After generating the visible light image and the fluorescence image, the pixel values of each pixel point of the two are weighted summed to obtain the pixel value of each pixel point of the fused image. By changing the size of the weight, the proportion of visible light image information and fluorescent image information in the information reflected by the fused image can be adjusted.

Exemplarily, at least two endoscopic images of different image types may include different types of fused images. For example, a monochrome fused image may be obtained by fusing a monochrome fluorescent image (including but not limited to a green fluorescent image) with a visible light image; a color fused image may be obtained by fusing a color fluorescent image with a visible light image. In a specific example, as shown in FIG3 , at least two endoscopic images of different image types include a monochrome fused image displayed in a first display area 310, a visible light image 320 displayed in a second display area 320, a color fused image displayed in a third display area 330, and a fluorescent image displayed in a fourth display area 340.

The endoscopic image of the embodiment of the present invention includes at least one type of stereoscopic endoscopic image. That is, the at least two types of endoscopic images include at least a first endoscopic image of a first type, the first endoscopic image is a stereoscopic endoscopic image, the first endoscopic image is generated based on left-path image data and right-path image data, and there is parallax between the left-path image data and the right-path image data. Exemplarily, the first endoscopic image may refer to any one of a visible light image, a fluorescence image, or a fused image. indivual.

When generating the first endoscopic image, the left image data can be generated based on the first image signal collected by the first image sensor, and the right image data can be generated based on the second image signal collected by the second image sensor. Since the first image sensor and the second image sensor correspond to different viewing angles, there is parallax between the left image data and the right image data. Alternatively, a three-dimensional simulation can be performed based on at least one of the first image signal and the second image signal to obtain the left image data and the right image data with parallax.

Exemplarily, after the endoscopic image is generated based on the visible light image signal and the fluorescent image signal, the endoscopic image may be scaled, cropped, etc. to make the endoscopic image fit the size of the display area.

In step S230, the at least two endoscopic images of different image types are displayed simultaneously in at least two display areas of the display interface. The display area includes at least a first display area, and the first endoscopic image is displayed in the first display area, that is, at least the first display area in the display interface is used to display a stereoscopic endoscopic image. By simultaneously displaying at least two image types of endoscopic images in the display interface, and displaying at least one image type of endoscopic image as a stereoscopic endoscopic image, the user can receive information about endoscopic images of more image types while obtaining a stereoscopic visual effect.

In the display interface of the embodiment of the present invention, different display areas can be seamlessly spliced to make the layout of the display interface more compact; or, different display areas can be set with a certain interval. The position and size of each display area can be fixed, or the user can be allowed to adjust it to a certain extent.

In some embodiments, at least one of the at least two display areas is a main display area with a larger area, and the other display areas are secondary display areas, whose areas are smaller than those of the main display area, and the areas of the various secondary display areas may be equal or unequal. For example, in the display interface shown in FIG3 , the first display area 310 is the main display area, which is dominant in the display interface and can provide more details for the user; the second display area 320 , the third display area 330 , and the fourth display area 340 are secondary display areas, which are mainly for users to refer to. The second display area 320 , the third display area 330 , and the fourth display area 340 may be displayed on the same side of the first display area 310 , making it easier for users to view without frequently switching their sights. It should be noted that the main display area of the embodiment of the present invention is not limited to the first display area, that is, the endoscopic image displayed in the main display area may be a stereoscopic endoscopic image or a two-dimensional endoscopic image.

In another embodiment, at least two display areas have the same area, for example, referring to FIG5A , the first display area 510, the second display area 520, the third display area 530, and the fourth display area 540 may divide the display interface into four equal parts. When the areas of the display areas are the same, it is possible to avoid different parallaxes of the stereoscopic endoscopic images displayed in different display areas.

The number of display areas in the display interface may not be limited to four, but may have more or fewer display areas. For example, in FIG5B , the display interface includes three display areas, namely, a first display area 510, a second display area 520, and a third display area 530. The number of display areas may be fixed or may be set by the user as required.

The display interface of the embodiment of the present invention displays at least two endoscopic images of different image types, that is, in addition to the first endoscopic image, the endoscopic image displayed in the display interface also includes at least a second endoscopic image of a second image type. In some embodiments, the second endoscopic image is a stereoscopic endoscopic image, that is, in this embodiment, the display interface displays at least two stereoscopic endoscopic images of different image types. Similar to the first endoscopic image, the second endoscopic image is generated based on left-path image data and right-path image data, and there is parallax between the left-path image data and the right-path image data. The display area also includes a second display area, and the second endoscopic image is displayed in the second display area.

In some embodiments, the area of the first display area is larger than the area of the second display area, that is, stereoscopic endoscopic images of different image types are displayed in display areas of different areas, wherein the first display area may be a primary display area and the second display area may be a secondary display area. For example, the first endoscopic image displayed in the first display area may be 2880*2160 pixels, and the second endoscopic image displayed in the second display area may be 960*720 pixels.

As described above, for stereoscopic endoscopic images, the stereoscopic visual effect is generated by outputting the left image and the right image with parallax to the left and right eyes of the user respectively, so that the human brain automatically fuses the stereoscopic effect. As shown in FIG6, when stereoscopic endoscopic images of different image types are displayed in display areas of different areas, the first endoscopic image (e.g., a stereoscopic monochrome fusion image) displayed in the first display area with a larger area has a larger parallax △1, and the second endoscopic image (e.g., a stereoscopic visible light image) displayed in the second display area with a smaller area has a smaller parallax △2, so that the stereoscopic effects presented by the first display area and the second display area are inconsistent. When the user observes the first display area and the second display area, the convergence and adjustment states of the eyes are different. When the line of sight frequently rotates between observing the first display area and the second display area, the eyes will easily become tired and feel uncomfortable.

In order to solve the above problem, the embodiment of the present invention performs parallax adjustment on the first endoscopic image to The parallax of the first endoscopic image is reduced so that the parallax of the first endoscopic image is the same as or similar to the parallax of the second endoscopic image; or the parallax of the second endoscopic image is adjusted to increase the parallax of the second endoscopic image so that the parallax of the second endoscopic image is the same as or similar to the parallax of the first endoscopic image. By adjusting the parallax of the first endoscopic image and/or the second endoscopic image, the difference in the parallax of the stereoscopic endoscopic images displayed in different display areas can be reduced, thereby avoiding user discomfort and improving user experience.

The principle of parallax adjustment of the embodiment of the present invention is shown in FIG7 . Taking reducing the parallax of the first endoscopic image as an example, the parallax △1 of the first endoscopic image can be reduced to △2, or close to △2, by translating the right image data to the left and the left image data to the right. Similarly, when increasing the parallax of the second endoscopic image, the right image data can be translated to the right and the left image data can be translated to the left, thereby increasing the parallax △2 of the second endoscopic image to △1, or close to △1. Optionally, the parallax of the first endoscopic image can be reduced and the parallax of the second endoscopic image can be increased at the same time, so that after the parallax adjustment, the parallax of the two is close to the average value of △1 and △2.

In the above, the parallax adjustment is explained by taking the first endoscopic image and the second endoscopic image as examples. It can be understood that when more stereoscopic endoscopic images are displayed in the display interface, the parallax adjustment can also be performed based on a similar principle. For example, in FIG6, the display interface also includes a third display area and a fourth display area, the third display area is used to display a third endoscopic image of a third image type (e.g., a stereoscopic color fusion image), and the fourth display area is used to display a fourth endoscopic image of a fourth image type (e.g., a stereoscopic fluorescence image), and the third endoscopic image and the fourth endoscopic image are both stereoscopic images. Among them, the third endoscopic image is generated based on the left image data and the right image data, and there is a parallax between the left image data and the right image data, and the fourth endoscopic image is generated based on the left image data and the right image data, and there is a parallax between the left image data and the right image data.

In FIG. 6 , the first display area has a first area, and the second display area, the third display area, and the fourth display area all have a second area, and the first area is larger than the second area, that is, the first display area is the main display area, and the second display area, the third display area, and the fourth display area are secondary display areas. Since the first endoscopic image displayed in the first display area has a larger parallax, and the second endoscopic image, the third endoscopic image, and the fourth endoscopic image displayed in the second display area, the third display area, and the fourth display area have a smaller parallax, the first endoscopic image may be parallax-adjusted to reduce the parallax of the first endoscopic image; or, the second endoscopic image, the third endoscopic image, and the fourth endoscopic image may be parallax-adjusted to increase the parallax of the second endoscopic image, the third endoscopic image, and the fourth endoscopic image, respectively, so that the first endoscopic image, The second endoscopic image, the third endoscopic image and the fourth endoscopic image are the same or close to each other. Since the second display area, the third display area and the fourth display area have the same area, when the second endoscopic image, the third endoscopic image and the fourth endoscopic image are selected for parallax adjustment, the same parallax adjustment parameter can be used.

In some embodiments, in addition to the first display area and the second display area, at least one other display area can be used to display a two-dimensional endoscopic image. For example, at least the third endoscopic image displayed in the third display area can be a two-dimensional endoscopic image. Since a two-dimensional endoscopic image does not have parallax, even if the area of the third display area is not equal to that of the first display area or the second display area, there is no need to adjust the parallax of the third endoscopic image. Furthermore, in the display interface, only the first endoscopic image of the first image type may be a stereoscopic endoscopic image, and one or more of the endoscopic images of other image types may be two-dimensional endoscopic images. In this case, no parallax adjustment is required, and eye discomfort of the user due to different parallax sizes can be avoided.

In summary, referring to Figures 4A to 4C, assuming that the first endoscopic image is a stereoscopic monochrome fused image, displayed in the main display area; the visible light image, the fluorescence image (monochrome fluorescence image or color fluorescence image) and the color fused image are displayed in the secondary display area, then as shown in Figure 4A, the visible light image, the fluorescence image and the color fused image can all be stereoscopic endoscopic images, that is, all endoscopic images displayed in the display interface are stereoscopic endoscopic images; or, as shown in Figure 4B, the visible light image, the fluorescence image and the color fused image can all be two-dimensional endoscopic images, that is, only the first endoscopic image in the display interface is a stereoscopic endoscopic image, and endoscopic images of other image types are all two-dimensional endoscopic images; or, as shown in Figure 4C, the visible light image, the fluorescence image and the color fused image can include both stereoscopic endoscopic images and two-dimensional endoscopic images.

In some embodiments, the image types of endoscopic images displayed in different display areas can also be switched in response to the user's operation instructions, that is, endoscopic images of different image types displayed in different display areas are switched and displayed. For example, the user can switch the type of endoscopic images displayed in the main display area to meet the user's actual clinical needs. The switching method can be to rotate the endoscopic images displayed in each display area in a preset order, or the user can directly specify the endoscopic image displayed in the current display area.

In some embodiments, the endoscope system can also receive medical images sent by other medical devices through the communication connection between the endoscope system and other medical devices, and display the medical images from other medical devices in the display area of the display interface. Other medical devices include ultrasound equipment, monitoring equipment, ventilation equipment, etc. For example, in the four-split screen display interface, at least one display area can be allocated to the other. Other medical devices. Exemplarily, when the endoscope system establishes a communication connection with other medical devices, the display area is used to display medical images of other medical devices. When the endoscope system disconnects the communication connection with other medical devices, the display area can be used to display endoscope images, or no images may be displayed. For example, the medical image includes at least one of an ultrasound examination image of an ultrasound device, a patient physiological parameter waveform of a monitoring device (such as an electrocardiogram signal waveform, a blood pressure waveform, a blood oxygen waveform, etc.), and a respiratory control parameter waveform of a ventilation device (such as a ventilation pressure waveform).

To sum up, the imaging and display method 200 for an endoscope system according to an embodiment of the present invention can simultaneously display at least two endoscopic images of different image types, providing the user with more comprehensive information; at the same time, it can also produce a stereoscopic visual effect, provide a more realistic visual experience, and help the user perform surgery more accurately.

Referring again to FIG. 1 , an embodiment of the present invention further provides an endoscope system 100, comprising an endoscope, a light source 110 and a camera host 150 connected to the endoscope, wherein the endoscope comprises an insertion portion 130 and an operating portion 160, wherein the insertion portion 130 is used to be inserted into a patient's part to be observed; the light source 110 is used to provide illumination to the part to be observed; the endoscope is provided with at least one image sensor, wherein the at least one image sensor is used to collect visible light image signals and fluorescent image signals of the part to be observed, and the image sensor may be provided at the front end of the insertion portion 130; the camera host 150 is used to obtain visible light image signals and fluorescent image signals from the at least one image sensor, so as to execute the imaging and display method 200 for the endoscope system as described above, and obtain at least two endoscopic images of different image types; the endoscopic imaging system further comprises a display 170, which is used to display at least two endoscopic images of different image types.

The specific structure of the endoscope system 100 and the specific steps of the imaging and display method 200 for the endoscope system have been described above and will not be repeated here. The endoscope system 100 of the embodiment of the present invention can simultaneously display at least two endoscopic images of different image types and provide a stereoscopic visual effect.

Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

Those skilled in the art will appreciate that the units and algorithm steps of each example described in the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application. However, such implementation should not be considered as exceeding the scope of the present invention.

In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules according to the embodiments of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims (20)

An imaging and display method for an endoscope system, characterized in that the method comprises: Acquire a visible light image signal and a fluorescent image signal of a part to be observed, wherein the visible light image signal and the fluorescent image signal both include left image data and right image data, and there is parallax between the left image data and the right image data; generating at least two endoscopic images of different image types based on the visible light image signal and the fluorescent image signal, the image types including visible light image, monochrome fluorescent image, color fluorescent image, monochrome fusion image, and color fusion image; wherein the endoscopic image at least includes a first endoscopic image of a first image type, the first endoscopic image is a stereoscopic endoscopic image, and the first endoscopic image is generated based on the left-path image data and the right-path image data; The at least two endoscopic images of different image types are respectively displayed simultaneously in at least two display areas of the display interface, wherein the display areas include at least a first display area, and the first endoscopic image is displayed in the first display area. The method according to claim 1, characterized in that the endoscopic image further comprises a second endoscopic image of a second image type, the second endoscopic image is a stereoscopic endoscopic image, and the second endoscopic image is generated based on the left image data and the right image data; The display area also includes a second display area, and the second endoscopic image is displayed in the second display area. The method according to claim 2, characterized in that the area of the first display area is larger than the area of the second display area. The method according to claim 3 is characterized in that it also includes: performing parallax adjustment on the first endoscopic image to reduce the parallax of the first endoscopic image; or performing parallax adjustment on the second endoscopic image to increase the parallax of the second endoscopic image. The method as claimed in claim 2 is characterized in that the endoscopic image also includes a third endoscopic image of a third image type and a fourth endoscopic image of a fourth image type, the third endoscopic image and the fourth endoscopic image are both stereoscopic images, the third endoscopic image is generated based on the left image data and the right image data, and the fourth endoscopic image is generated based on the left image data and the right image data. The method according to claim 5, characterized in that the display area further comprises a third display area and a fourth display area, the third endoscopic image is displayed in the third display area, the fourth endoscopic image is displayed in the fourth display area, and the area of the first display area is is larger than the area of the second display area, the third display area and the fourth display area. The method according to claim 6 is characterized in that the first display area has a first area, the second display area, the third display area and the fourth display area all have a second area, and the first area is larger than the second area. The method as claimed in claim 7 is characterized in that the method further includes: performing parallax adjustment on the first endoscopic image to reduce the parallax of the first endoscopic image; or, performing parallax adjustment on the second endoscopic image, the third endoscopic image and the fourth endoscopic image respectively to increase the parallax of the second endoscopic image, the third endoscopic image and the fourth endoscopic image. The method according to any one of claims 5 to 7, characterized in that the image type of at least one of the first endoscopic image, the second endoscopic image, the third endoscopic image and the fourth endoscopic image is a monochrome fluorescence image or a color fluorescence image generated based on the fluorescence image signal, or is a monochrome fusion image or a color fusion image generated based on the visible light image signal and the fluorescence image signal. The method of claim 2, wherein the endoscopic image further comprises a third endoscopic image of a third image type, the third endoscopic image being a two-dimensional endoscopic image; The display area further includes a third display area, and the third endoscopic image is displayed in the third display area. The method of claim 1, wherein, except for the first endoscopic image of the first image type, one or more of the endoscopic images of other image types are two-dimensional endoscopic images. The method according to claim 11, characterized in that the area of the first display area is larger than the areas of other display areas. The method according to claim 1, characterized in that the at least two display regions have the same area. The method according to claim 1, characterized in that the method further comprises: In response to an operation instruction of a user, endoscopic images of different image types displayed in the different display areas are switched for display. The method according to claim 1, characterized in that the method further comprises: An external image from an external image source is acquired, and the external image is displayed in a display area of the display interface. The method according to claim 1, characterized in that the method further comprises: Acquire medical images from other medical devices and display the medical images in the display area of the display interface. The method according to claim 16, characterized in that the other medical equipment includes at least one of an ultrasound device, a monitoring device, and a ventilation device. The method as claimed in claim 16 is characterized in that the medical image includes at least one of an ultrasound examination image of an ultrasound device, a patient physiological parameter waveform of a monitoring device, and a breathing control parameter waveform of a ventilation device. An endoscope system, characterized in that it comprises an endoscope, a light source and a camera host connected to the endoscope, wherein the endoscope comprises an insertion portion and an operation portion, and the insertion portion is used to be inserted into a part to be observed of a patient; The light source is used to provide illumination to the part to be observed; The endoscope is provided with at least one image sensor, and the at least one image sensor is used to collect visible light image signals and fluorescent image signals of the part to be observed; The camera host is used to acquire the visible light image signal and the fluorescent image signal from the at least one image sensor to perform the method according to any one of claims 1 to 18 and obtain at least two endoscopic images of different image types; A display is used to display the at least two endoscopic images of different image types. An imaging and display method for an endoscope system, characterized in that the method comprises: Acquiring visible light image signals and fluorescent image signals of the part to be observed; generating at least two endoscopic images of different image types based on the visible light image signal and the fluorescent image signal, wherein the endoscopic images at least include a first endoscopic image of a first image type, the first endoscopic image is a stereoscopic endoscopic image, the first endoscopic image is generated based on left-path image data and right-path image data, and there is parallax between the left-path image data and the right-path image data; The at least two endoscopic images of different image types are respectively displayed simultaneously in at least two display areas of the display interface, wherein the display areas at least include a first display area, and the first endoscopic image is displayed in the first display area.