WO2025223515A1 - Imaging method and system and surgical robot system - Google Patents

Abstract

Provided are an imaging method and system and a surgical robot system. The method comprises: irradiating a target area using a light source, wherein the light source comprises a first light source and a second light source, the target area provides first light under the irradiation of the first light source, the target area provides second light under the irradiation of the second light source, the target area comprises a first object and a second object, and the first object and the second object provide light with different spectral characteristics under the irradiation of the second light source; acquiring an image of the target area, wherein the acquired image comprises a first area corresponding to the first object and a second area corresponding to the second object, and the first area and the second area have different image signals; and performing image enhancement processing on the acquired image to enhance the image signal of the first area and the image signal of the second area in the acquired image, thus obtaining an enhanced image.

Description

Imaging methods and systems and surgical robot systems

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority to Chinese Patent Application No. 202410513112.0, filed on April 25, 2024, the entire contents of which are hereby incorporated herein by reference.

Technical Field

This disclosure relates to the field of image processing technology, and in particular to imaging methods and systems and surgical robot systems.

Background Technology

In related technologies, for objects within a target area that are difficult to observe directly with the naked eye, a specially designed optical system can be used to excite and detect the special light emitted by the objects, thereby achieving visualization. However, the target area may include multiple objects, and related technologies struggle to accurately distinguish between different objects visually.

Summary of the Invention

In a first aspect, embodiments of this disclosure provide an imaging method, the method comprising: illuminating a target region with a light source, the light source including a first light source and a second light source, the target region providing first light under the illumination of the first light source, the target region providing second light under the illumination of the second light source, the target region including a first object and a second object, the first object and the second object providing light with different spectral characteristics under the illumination of the second light source; acquiring an image of the target region, the acquired image including a first region corresponding to the first object and a second region corresponding to the second object, the first region and the second region having different image signals; and performing image enhancement processing on the acquired image to enhance the image signals of the first region and the second region in the acquired image, thereby obtaining an enhanced image.

Secondly, embodiments of this disclosure provide an imaging system, the system comprising: a light source configured to illuminate a target region, the light source including a first light source and a second light source, the target region providing first light under illumination by the first light source and providing second light under illumination by the second light source, the target region including a first object and a second object, the first object and the second object providing light with different spectral characteristics under illumination by the second light source; an imaging unit configured to acquire an image of the target region, the acquired image including a first region corresponding to the first object and a second region corresponding to the second object, the first region and the second region having different image signals; and an image processor configured to perform image enhancement processing on the acquired image to enhance the image signals of the first region and the second region in the acquired image, thereby obtaining an enhanced image.

Thirdly, embodiments of this disclosure provide a surgical robot system, the system including the aforementioned imaging system, and further including a display unit that receives and displays images from the imaging system.

In this embodiment of the disclosure, the target area is imaged based on the first light and the second light. After obtaining the image of the target area, the image is further processed to enhance the difference between the image signal of the first area and the image signal of the second area. This allows the user to more intuitively distinguish the first area corresponding to the first object and the second area corresponding to the second object in the image, thereby improving the recognition between the first area and the second area.

It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

Attached Figure Description

The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to illustrate the technical solutions of this disclosure.

Figure 1 is a schematic diagram of an imaging system according to an embodiment of the present disclosure.

Figure 2 is a schematic diagram of the response intensity of the image sensor of this disclosure to light of different wavelengths.

Figure 3 is a schematic diagram of the pixel values of each channel of the pixel point before and after processing according to an embodiment of the present disclosure.

Figure 4 is a schematic diagram of an endoscope system according to an embodiment of the present disclosure.

Figure 5 is a schematic diagram of a surgical robot system according to an embodiment of the present disclosure.

Figure 6 is a schematic diagram of the imaging method according to an embodiment of the present disclosure.

Figure 7 is a schematic diagram of a computer device according to an embodiment of the present disclosure.

Detailed Implementation

Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of systems and methods consistent with some aspects of this disclosure as detailed in the appended claims.

The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and/or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, and to make the above-mentioned objectives, features and advantages of the embodiments of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

In related technologies, for objects within a target area that are difficult to observe directly with the naked eye, a specially designed optical system can be used to excite and detect the special light emitted by the object, thereby achieving visualization. In practical applications, the target area may be illuminated by multiple light sources, providing various types of light. Based on these multiple light sources, the target area can be imaged to obtain an image of the target area, thus enabling visualization of objects within the target area. The target area may include multiple objects, and different objects have different light absorption and reflection characteristics, resulting in different image signals in the image. Theoretically, based on the differences between the image signals corresponding to the regions of each object in the image, the regions corresponding to each object can be distinguished from the image. However, these differences are sometimes quite subtle, making it difficult to accurately distinguish the regions corresponding to each object in the image visually.

One specific implementation of the aforementioned visualization process is fluorescence imaging. Fluorescence imaging refers to the method of visualizing objects that cannot be directly distinguished by labeling them with fluorescent dyes, and then using a specially designed optical system to excite and detect fluorescence. Currently, fluorescence imaging technology has applications in many fields. In some applications, fluorescent dyes can be injected into the target area through injection, application, etc., and the target area can be illuminated with visible light and excitation light. The target area can reflect visible light under the illumination of the visible light source and emit fluorescence under the illumination of the excitation light source. Imaging can be performed based on the visible light and fluorescence from the target area to obtain an image of the target area, which can then be analyzed and processed. For example, in the field of surgery, the target area can be the surgical area, and the image of the target area can help doctors distinguish between normal tissues and diseased tissues, thereby improving surgical accuracy.

The target area may include multiple objects, and different objects may absorb fluorescent dyes to varying degrees. Therefore, under excitation light, the fluorescence intensity emitted by different objects may differ. For example, in the field of surgery, multiple objects may include normal in vivo tissue and diseased tissue. Typically, diseased tissue absorbs fluorescent dyes more readily than normal in vivo tissue; therefore, normal in vivo tissue emits weaker fluorescence under excitation light, while diseased tissue emits stronger fluorescence. This results in different image signals for regions corresponding to normal in vivo tissue and regions corresponding to diseased tissue in the target area image. Theoretically, different regions can be distinguished from the image based on the differences in their image signals. However, because image sensors have different transmittance for visible light and fluorescence, the differences between these image signals are not significant enough, making it difficult to visually and accurately distinguish the image regions corresponding to different objects.

It is understood that the above fluorescence imaging scenarios are merely illustrative examples. Other methods can be used to achieve visualization in other application scenarios, which will not be listed here. For ease of description, fluorescence imaging technology will be used as an example below.

To accurately distinguish image regions corresponding to different objects visually, embodiments of this disclosure provide an imaging system 10. Figure 1 shows a schematic diagram of an imaging system 10, which includes a light source 101, an imaging unit 102, and an image processor 103.

The light source 101 includes a first light source 101a and a second light source 101b, both of which can illuminate the target area S. The target area S receives first light under the illumination of the first light source 101a and second light under the illumination of the second light source 101b, the first and second lights having different wavelengths. The target area S includes a first object and a second object, which provide light with different spectral characteristics under the illumination of the second light source 101b.

In some embodiments, the first light source 101a may be a visible light source, and the second light source 101b may be an excitation source. The target region S provides visible light under the illumination of the visible light source and fluorescence under the illumination of the excitation source, with the visible light and fluorescence having different wavelengths. However, the first light source 101a being a visible light source and the second light source 101b being an excitation source is merely illustrative, and the first light source 101a and the second light source 101b are not limited to being a visible light source and an excitation source, respectively. In the following description, the embodiments of this disclosure will be described using the first light source 101a as a visible light source and the second light source 101b as an excitation source as an example. It can be understood that the principles of the embodiments of this disclosure are the same as when the first light source 101a and the second light source 101b are different light sources.

In some embodiments, the visible light source 101a may be a visible light source in a portion of the wavelength range from 380 nm to 780 nm. Depending on the specific requirements, different wavelengths of visible light La may be selected. Optionally, the visible light La may be yellow light in the wavelength range of 550 nm to 650 nm, or it may be red light in the wavelength range of 600 nm to 660 nm, or it may be blue light in the wavelength range of 380 nm to 470 nm.

In some embodiments, the excitation light source 101b can emit excitation light Lb, which can be any excitation light that can cause the target region S to emit fluorescence, such as infrared light, near-infrared light, or ultraviolet light.

In some embodiments, the target region S, when irradiated by visible light La emitted from visible light source 101a, can reflect visible light Lax, which is the first light. The target region S emits fluorescence Lby when irradiated by excitation light Lb emitted from excitation source 101b, which is the second light. The visible light Lax and the fluorescence Lby have different wavelengths. For example, the visible light Lax has a wavelength between 380 nm and 780 nm, while the fluorescence Lby has a wavelength between 800 nm and 900 nm. It is understood that the values in the above embodiments are merely illustrative and are not intended to limit this disclosure.

The target area S includes a first object O1 and a second object O2. Both the first object O1 and the second object O2 reflect visible light Lax when illuminated by visible light La emitted by visible light source 101a. The first object O1 and the second object O2 provide light with different spectral characteristics when illuminated by excitation source 101b. For example, the first object O1 does not emit fluorescence Lby when illuminated by excitation source 101b, while the second object O2 emits fluorescence Lby when illuminated by excitation source 101b.

In some embodiments, the target area S is a target area in a surgical scene, the first object O1 is tissue in the surgical scene that has not been fluorescently labeled, and the second object O2 is tissue in the surgical scene that has been fluorescently labeled. Thus, the first object O1 does not fluoresce under the illumination of the excitation light source 101b, while the second object O2 fluoresces under the illumination of the excitation light source 101b, thereby enabling the first object O1 and the second object O2 to provide light with different spectral characteristics under the illumination of the excitation light source 101b.

In some embodiments, fluorescent markers such as indocyanine green, fluorescein, and rhodamine can be used to fluorescently label tissues in a surgical setting. For example, the first object O1 is normal tissue within the surgical area, and the second object O2 is diseased tissue, such as tumor tissue, within the surgical area. In other examples, the first object O1 and the second object O2 can also be other types of objects. Taking the first object O1 as normal tissue and the second object O2 as diseased tissue as an example, the diseased tissue within the target area S can be labeled using fluorescent markers (such as indocyanine green, fluorescein, and rhodamine). Normal tissue does not fluoresce under the illumination of the excitation light source 101b, while the diseased tissue labeled with the fluorescent marker fluoresces under the illumination of the excitation light source 101b.

Imaging unit 102 acquires an image Is of target region S. In some embodiments, imaging unit 102 may include a lens 102a and an image sensor 102b. Visible light Lax and fluorescence Lby from target region S can be captured by lens 102a and imaged by image sensor 102b to obtain image Is. As an illustrative example, image sensor 102b may include, but is not limited to, a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) image sensor. Image Is includes a first region R1 corresponding to the first object O1 and a second region R2 corresponding to the second object O2. The first region R1 and the second region R2 have different image signals. In some embodiments, the image signals may include, but are not limited to, at least one of brightness, chroma, saturation, contrast, and boundary pixels of different regions.

Imaging unit 102 images the target region S based on the visible light Lax and fluorescence Lby provided by the target region S, and obtains the acquired image Is. Since the first object O1 and the second object O2 provide light with different spectral characteristics under the illumination of the excitation light source 101b, such as the first object O1 not emitting fluorescence under the illumination of the excitation light source 101b, and the second object O2 emitting fluorescence under the illumination of the excitation light source 101b, the first region R1 corresponding to the first object O1 in image Is does not include fluorescence signal (also called non-fluorescent region), and the second region R2 corresponding to the second object O2 in image Is includes fluorescence signal (also called fluorescent region), so that the first region R1 corresponding to the first object O1 and the second region R2 corresponding to the second object O2 in image Is have different image signals. For example, the first object O1 is tissue in a surgical scene that has not been fluorescently labeled, and the second object O2 is tissue in a surgical scene that has been fluorescently labeled. This is such that the first object O1 does not fluoresce under the illumination of the excitation light source 101b, while the second object O2 fluoresces under the illumination of the excitation light source 101b. As a result, the first region R1 corresponding to the first object O1 in the acquired image Is does not have a fluorescent signal and is called a non-fluorescent region, while the second region R2 corresponding to the second object O2 has a fluorescent signal and is called a fluorescent region. Thus, the first region R1 corresponding to the first object O1 and the second region R2 corresponding to the second object O2 in the image Is have different image signals.

In some embodiments, the image sensor 102b is a single image sensor, meaning the number of image sensors 102b used to image the target region S can be equal to one. This single image sensor may include multiple channels. In some embodiments, each channel may image the target region S based on both visible light (Lax) and fluorescence (Lby). In other embodiments, at least one of the multiple channels may image the target region S based on visible light (Lax), and at least one other channel may image the target region S based on fluorescence (Lby). Optionally, the multiple channels may include at least two of a red signal (R) channel, a green signal (G) channel, and a blue signal (B) channel. The multiple channels may form a Bayer array, where each channel can sense color information from the red, green, or blue bands of visible light (Lax), and at least one of the red, green, and blue signal channels may sense fluorescence (Lby), or each of the red, green, and blue signal channels may sense fluorescence (Lby). Optionally, the multiple channels may also include at least two of a red signal (R) channel, a first green signal (Gr) channel, a second green signal (Gb) channel, and a blue signal (B) channel. The multiple channels can form a Bayer array, where each channel can sense one of the color information in visible light Lax: red, first green, second green, or blue. At least one of the red, green, and blue signal channels can sense fluorescence Lby, or each of the red, green, and blue signal channels can sense fluorescence Lby. Figure 2 shows a schematic diagram of the image sensor 102b sensing visible light Lax and fluorescence Lby.

When a single image sensor is used to image the target region S based on the visible light Lax and fluorescence Lby provided by the target region S to obtain the acquired image Is, since the first object in the target region S only reflects visible light and does not emit fluorescence, and the second object reflects visible light and emits fluorescence, and the single image sensor can sense both visible light Lax and fluorescence Lby, the first region corresponding to the first object in the image Is acquired by the single image sensor includes visible light signals but does not include fluorescence signals, and is called the non-fluorescent region. The second region corresponding to the second object includes both visible light signals and fluorescence signals, and is called the fluorescent region. Thus, an image Is obtained by the single image sensor includes both fluorescent and non-fluorescent regions.

In some embodiments, if the visible light source 101a is red light in the band between 600nm and 660nm, the target area S provides red light and fluorescence under the illumination of red light and excitation light. The imaging unit 102 acquires an image Is of the target area S based on the red light and fluorescence provided by the target area S. Since the absorption rate of red light by the tissue in the target area is generally low in surgical scenarios, the image Is acquired by the imaging unit 102 based on the red light and fluorescence provided by the target area S has high brightness.

In some embodiments, if the visible light source 101a is blue light with a wavelength between 380nm and 470nm, the target area S provides blue light and fluorescence under the illumination of blue light and excitation light. The imaging unit 102 acquires an image Is of the target area S based on the blue light and fluorescence provided by the target area S. Since the tissue in the target area generally has a high absorption rate of blue light in surgical scenarios, the image Is acquired by the imaging unit 102 based on the blue light and fluorescence provided by the target area S has high contrast and clarity.

In some embodiments, if the visible light source 101a is yellow light with a wavelength between 550nm and 650nm, the target area S provides yellow light and fluorescence under the illumination of yellow light and excitation light. The imaging unit 102 acquires the image Is of the target area S based on the yellow light and fluorescence provided by the target area S. Since the absorption rate of yellow light by the tissue in the target area in the surgical scene is between that of blue light and red light, the image Is acquired by the imaging unit 102 based on the yellow light and fluorescence provided by the target area S can ensure both image brightness and image clarity and contrast.

In some embodiments, the imaging unit 102 further includes a filter disposed in the optical path before the image sensor 102b, for filtering the excitation light Lbx from the target region S. In this embodiment, the visible light Lax and/or fluorescence Lby from the target region S may also include the excitation light Lbx reflected from the target region S. When imaging the target region S, the excitation light Lbx will form stray light and interfere with the imaging process. Therefore, by filtering out the excitation light Lbx included in the visible light Lax and/or fluorescence Lby from the target region S by the filter disposed in the optical path before the image sensor 102b, the interference of the excitation light Lbx on the imaging process is reduced, and the imaging quality is improved.

Image sensor 102b can send image Is to image processor 103. Image processor 103 is configured to perform image enhancement processing on image Is to enhance the difference between the image signal of the first region R1 and the image signal of the second region R2 in image Is. The image enhancement processing includes, but is not limited to, at least one of the following: white balance processing, linear color correction, and nonlinear color mapping.

In some embodiments, the image processor 103 directly performs image enhancement processing on the image Is acquired by the imaging unit 102 to obtain an enhanced image. That is, the image processor 103 does not perform composite processing on the acquired image Is during the image enhancement processing. In this embodiment, the imaging unit 102 uses a single image sensor to image the target region S based on visible light Lax and fluorescence Lby provided by the target region S, resulting in an image Is that includes both fluorescent and non-fluorescent regions. During the image enhancement processing of the image Is acquired by the imaging unit 102, the image processor 103 only performs enhancement processing on this single image Is. In this embodiment, the image processor 103 obtains an enhanced image that visually distinguishes the first and second regions simply by directly performing image enhancement processing on the image Is obtained by the image sensor 102b, without requiring composite processing. In some embodiments, the enhanced image obtained by the image processor 103 is directly presented to the observer through a display device, allowing the observer to directly distinguish the first and second regions from the presented enhanced image. In other embodiments, after the image processor 103 obtains the enhanced image, it can be processed by image merging, color adjustment and other image processing before being presented to the observer through a display device.

In some embodiments, the image processor 103 performs white balance processing on the image Is, specifically by adjusting the ratio of each color channel in the image Is to amplify the image signal in the second region of the image Is, such as amplifying the fluorescence signal in the fluorescence region of the image Is, thereby amplifying the color difference between the image signal in the first region and the image signal in the second region of the image Is. Each color channel includes three color channels: R, G, and B.

In some embodiments, the linear color correction processing of the image processor 103 on the image Is specifically includes: correcting the image Is acquired by the imaging unit 102 using a linear color correction matrix to improve the color difference between the first region R1 and the second region R2 in the acquired image Is. In some examples, a mapping relationship between the color space of the input image and the target color space can be established. Typically, for an RGB three-channel sensor, linear color correction uses a 3×3 linear color correction matrix. Each element of the linear color correction matrix represents the weight relationship between the input channel and the target color channel. The corrected color value is obtained by multiplying each pixel value of the input image with the linear color correction matrix. In this embodiment, by performing linear color correction on the image Is, the color difference between the fluorescent and non-fluorescent regions in the image Is can be improved, and the color of the fluorescent region can be corrected to the target color, such as green or blue. The color of the non-fluorescent region can be corrected to black and white, or pseudo-color, or color with the highest possible color fidelity.

In some embodiments, the image processor 103 performs nonlinear color mapping processing on the image Is, specifically including: establishing a color mapping table; and performing a nonlinear transformation on the RGB values of the image Is based on the color mapping table and a preset nonlinear mapping algorithm to enhance the color difference between the first and second regions in the image Is, such as enhancing the color difference between fluorescent and non-fluorescent regions in the image Is. In some examples, nonlinear three-dimensional color mapping processing is used to perform nonlinear color mapping processing on the image Is. First, a color mapping table is established. Based on the RGB values of the image Is, and according to the color mapping table, a preset nonlinear transformation algorithm is used to output the target R’G’B’, thereby obtaining an enhanced image. In this embodiment, by using nonlinear three-dimensional color mapping, the color difference between fluorescent and non-fluorescent regions in the same image can be improved, and the color of the fluorescent region can be corrected to the target color, such as green or blue. The color of the non-fluorescent region can be corrected to black and white, or pseudo-color, or color with the highest possible color fidelity.

Figure 3 illustrates the signal intensity of each channel before and after image enhancement processing, using visible light (Lax) as the first light and fluorescence (Lby) as an example. The image processor 103 receives each frame image Is from the image sensor 102b. In the first region of image Is without fluorescence, the RGB channels of the pixels contain only visible light information. In the second region of image Is with fluorescence, the RGB channels of the pixels contain both visible light and fluorescence information (in Figure 3, fluorescence information is represented by the shaded areas on the R, G, and B channels of the second region). In some embodiments, image enhancement processing of image Is may involve adjusting the first region R1 to a colored region and the second region R2 to a green region. Alternatively, in some embodiments, the first region R1 may be adjusted to a grayscale region and the second region R2 to a green region. The above embodiment images the target region S based on visible light Lax and fluorescence Lby. After obtaining the image Is of the target region S, image enhancement processing is performed directly on the image Is, thereby enhancing the difference between the image signal of the first region R1 and the image signal of the second region R2. This allows the user to more intuitively distinguish the first region R1 corresponding to the first object O1 and the second region R2 corresponding to the second object O2 in the image Is, improving the recognition between the first region R1 and the second region R2.

In some embodiments, the visible light source 101a and the excitation source 101b can synchronously illuminate the target region S. The target region S receives mixed light under the synchronous illumination of the visible light source 101a and the excitation source 101b. This mixed light includes visible light Lax and fluorescence Lby provided by the target region S under the synchronous illumination of the visible light source 101a and the excitation source 101b. The lens 102a of the imaging unit 102 captures the mixed light from the target region S. The image sensor 102b images the target region S based on the mixed light captured by the lens 102a, obtaining an image Is. The image processor 103 performs image processing on the image Is obtained by the image sensor 102b to enhance the difference between the image signal of the first region R1 and the image signal of the second region R2. The image sensor 102b is a single image sensor.

In this embodiment, a visible light source 101a and an excitation source 101b are simultaneously used to illuminate the target region S. This ensures that the target region S provides both visible light (Lax) and fluorescence (Lby) at the same time. Consequently, in each frame image Is obtained by the image sensor 102b, the first region R1 and the second region R2 satisfy the following conditions: the first region R1 includes visible light information, and the second region R2 includes both visible light and fluorescence information. This eliminates motion blur and delay issues in each acquired frame, improving image quality. Simultaneously, the image processor 103 directly processes the image obtained by the image sensor 102b to enhance the difference between the image signals of the first region R1 and the second region R2. Since image enhancement processing is performed directly on the image obtained by the image sensor, without the processing and synthesis of multiple frames, the complexity of image enhancement processing is simplified. Furthermore, since image enhancement processing only needs to be performed on a single frame to process both visible light and fluorescence information, the complexity of image enhancement processing is further simplified, improving image processing efficiency. Since a single image sensor 102b can be used to obtain an image Is that includes both visible light and fluorescence information, it can effectively reduce hardware size and cost, and facilitate pipelined processing of multiple frames of images, thereby improving efficiency and flexibility in the image processing process.

In some embodiments, the imaging system 10 further includes a controller (not shown) configured to control the imaging unit 102 to image the target region S.

In some embodiments, the controller is further configured to adjust the intensity of the first light source 101a and/or the second light source 101b to enhance the difference between the image signal of the first region R1 and the image signal of the second region R2 of the image Is acquired by the imaging unit 102. For example, under illumination by the first light source 101a and the second light source 101b of the same intensity, if the signal intensity of the first light is relatively strong and the signal intensity of the second light is relatively weak, the intensity of the first light source 101a can be reduced, and/or the intensity of the second light source 101b can be increased.

This disclosure also provides an endoscope system, as shown in FIG4. The endoscope system includes a light source device 110, an imaging module 120, and an image processing device 130. The light source device 110 includes a light source 101 of the imaging system 10, the imaging module 120 includes an imaging unit 102 of the imaging system 10, and the image processing device 130 includes an image processor 103 of the imaging system 10.

In some embodiments, the endoscope system may further include a display unit (not shown) configured to receive and display images processed by the image processing device 130.

In some embodiments, the endoscope system may further include a controller (not shown) configured to control the imaging module 120 to image the target region S.

In some embodiments, the controller is further configured to control the light source device 110 to adjust the intensity of the first light source 101a and/or the second light source 101b to enhance the difference between the image signal of the first region R1 and the image signal of the second region R2 of the image Is acquired by the imaging unit 102. For example, under illumination by the first light source 101a and the second light source 101b of the same intensity, if the signal intensity of the first light is relatively strong and the signal intensity of the second light is relatively weak, the intensity of the first light source 101a can be reduced, and/or the intensity of the second light source 101b can be increased.

As shown in Figure 5, the imaging system 10 described above can be applied to a surgical robot system 20. The surgical robot system 20 may include the imaging system 10 and a display unit 201. The imaging system 10 can send the image Is processed by the image processor 103 to the display unit 201 for display. In some embodiments, the imaging unit 102 of the imaging system 10 may be an endoscope or a microscope. The surgical robot system 20 may include a robotic arm system 202, which includes one or more robotic arms 202a. The imaging unit 102 can be held on any one of the robotic arms 202a. By adjusting the end-effector position and orientation of the robotic arm 202a, the orientation of the imaging unit 102 can be changed, thereby controlling the imaging unit 102 to image the target region S in a specific orientation. The surgical robot system 20 may also include a console 203, on which the surgeon can operate to adjust the end-effector position and orientation of the robotic arm 202a. In some embodiments, the surgical robot system 20 may also include a controller (not shown) for controlling the imaging system 10 to image the target region S. The controller mentioned above can be the console 203 of the surgical robot system 20, or it can be the control unit integrated into the imaging unit 102. For example, when the imaging unit 102 is an endoscope, the controller can be the main unit of the endoscope.

This disclosure also provides an imaging method, as shown in FIG6, the imaging method including:

Step S1: Illuminate the target area S with a light source, which includes a first light source 101a and a second light source 101b. The target area S receives first light under the illumination of the first light source 101a and second light under the illumination of the second light source 101b. The target area S includes a first object O1 and a second object O2. The first object O1 and the second object O2 provide light with different spectral characteristics under the illumination of the second light source 101b. Illuminating the target area S with a light source can be done automatically or manually, so that the light source illuminates the target area S after it is turned on.

Step S2: Acquire an image Is of the target region S. The image Is includes a first region R1 corresponding to the first object O1 and a second region R2 corresponding to the second object O2. The first region R1 and the second region R2 have different image signals.

Step S3: Perform image enhancement processing on the acquired image Is to enhance the image signal of the first region R1 and the image signal of the second region R2 to obtain an enhanced image.

For details of the above method embodiments, please refer to the embodiments of the aforementioned imaging system 10, which will not be repeated here.

In some embodiments, the acquired image Is is not subjected to compositing processing during the process of performing image enhancement processing on the acquired image Is to obtain the enhanced image.

In some embodiments, during the image enhancement process of the acquired image Is to obtain an enhanced image, the acquired image Is is only one image during the image enhancement process. In some embodiments, acquiring the image Is of the target region includes: receiving mixed light from the target region S, the mixed light including first light and second light provided by the target region S under the synchronous illumination of the first light source 101a and the second light source 101b; and imaging the target region S based on the mixed light to obtain an image of the target region S.

In some embodiments, imaging the target region S based on mixed light to obtain an image of the target region S includes: imaging the target region S based on mixed light using a single image sensor to obtain an image of the target region S.

In some embodiments, the single image sensor includes a red signal channel, a green signal channel, and a blue signal channel. The red signal channel can sense the red band of the first light included in the mixed light, the green signal channel can sense the green band of the first light included in the mixed light, the blue signal channel can sense the blue band of the first light included in the mixed light, and at least one of the red, green, and blue signal channels can sense the second light in the mixed light.

In some embodiments, a single image sensor includes a red signal channel, a green signal channel, and a blue signal channel. The red signal channel can sense the red band of the first light included in the mixed light, the green signal channel can sense the green band of the first light included in the mixed light, and the blue signal channel can sense the blue band of the first light included in the mixed light. Each of the red, green, and blue signal channels can sense the second light in the mixed light.

In some embodiments, the target region S can reflect the excitation light Lb emitted by the second light source 101b. The excitation light Lbx formed by the reflection of the excitation light Lb by the target region S is filtered out before an image of the target region S is obtained by imaging the target region S based on the first light and the second light from the target region S.

In some embodiments, image enhancement processing includes at least one of the following: white balance processing, linear color correction, and nonlinear color mapping.

In some embodiments, white balance processing of image Is includes adjusting the ratios of each color channel in the acquired image Is to amplify the image signal of the second region in the acquired image. Each color channel includes three color channels: R, G, and B.

In some embodiments, linear color correction processing of image Is includes: correcting the acquired image using a linear color correction matrix to improve the color difference between the first region and the second region in the acquired image.

In some embodiments, performing nonlinear color mapping processing on image Is includes: establishing a color mapping table, and performing a nonlinear transformation on the acquired image based on the color mapping table and a preset nonlinear transformation algorithm to enhance the color difference between the first region and the second region in the acquired image.

In some embodiments, the image signal includes at least one of the following: brightness, chroma, saturation, contrast, and boundary pixels of different regions.

In some embodiments, the first light and the second light have different wavelengths.

In some embodiments, the first light is a portion of the visible light spectrum, with the visible light spectrum ranging from 380 nm to 780 nm, and the second light spectrum ranging from 800 nm to 900 nm.

In some embodiments, the first light is yellow light with a wavelength between 550nm and 650nm; or the first light is red light with a wavelength between 600nm and 660nm; or the first light is blue light with a wavelength between 380nm and 470nm.

In some embodiments, image enhancement processing is performed on image Is to enhance the difference between the image signal of the first region R1 and the image signal of the second region R2, including: adjusting the first region R1 to a color region and adjusting the second region R2 to a green region; or adjusting the first region R1 to a grayscale region and adjusting the second region R2 to a green region.

In some embodiments, the method further includes: adjusting the intensity of the first light source 101a and/or the second light source 101b to enhance the difference between the image signal of the first region R1 and the image signal of the second region R2.

This disclosure also provides a computer device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the foregoing embodiments.

Figure 7 illustrates a more specific hardware structure diagram of a computing device provided in an embodiment of this disclosure. The device may include: a processor 31, a memory 32, an input/output interface 33, a communication interface 34, and a bus 35. The processor 31, memory 32, input/output interface 33, and communication interface 34 are interconnected internally via the bus 35.

The processor 31 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The processor 31 may also include a graphics card, such as an Nvidia Titan X graphics card or a 1080Ti graphics card.

The memory 32 can be implemented as a read-only memory (ROM), random access memory (RAM), static storage device, dynamic storage device, etc. The memory 32 can store the operating system and other applications. When the technical solutions provided in the embodiments of this disclosure are implemented by software or firmware, the relevant program code is stored in the memory 32 and is called and executed by the processor 31.

Input/output interface 33 is used to connect input/output modules to realize information input and output. Input/output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

Communication interface 34 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB (Universal Serial Bus), network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.).

Bus 35 includes a pathway for transmitting information between various components of the device (e.g., processor 31, memory 32, input/output interface 33, and communication interface 34).

It should be noted that although the above-described device only shows the processor 31, memory 32, input/output interface 33, communication interface 34, and bus 35, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this disclosure, and not necessarily all the components shown in the figures.

This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory, read-only memory, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined in this article, computer-readable media do not include transient computer-readable media, such as modulated data signals and carrier waves.

As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this disclosure.

The systems, devices, modules, or units described in the above embodiments can be implemented by computer devices or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this disclosure, the functions of each module can be implemented in one or more software and/or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

The above description is merely a specific implementation of the embodiments of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this disclosure, and these improvements and modifications should also be considered within the protection scope of the embodiments of this disclosure.

Claims (35)

An imaging method, the method comprising: A target area is illuminated by a light source, which includes a first light source and a second light source. The target area receives a first light under the illumination of the first light source and a second light under the illumination of the second light source. The target area includes a first object and a second object. The first object and the second object receive light with different spectral characteristics under the illumination of the second light source. The target area is captured as an image, which includes a first area corresponding to the first object and a second area corresponding to the second object, wherein the first area and the second area have different image signals. The acquired image is subjected to image enhancement processing to enhance the image signals of the first region and the second region in the acquired image, thereby obtaining an enhanced image. According to the method of claim 1, wherein performing image enhancement processing on the acquired image to obtain an enhanced image includes: No image compositing process was performed on the acquired images during the image enhancement process to obtain the enhanced image. According to the method of claim 1 or 2, the acquired image is only one image in the image enhancement process. The method according to any one of claims 1-3, wherein acquiring the image of the target area includes: Receive mixed light from the target area, the mixed light including the first light and the second light provided by the target area under synchronous illumination by the first light source and the second light source; The acquired image is obtained by imaging the target area based on the mixed light. According to the method of claim 4, wherein the step of imaging the target region based on the mixed light to obtain the acquired image comprises: The acquired image is obtained by imaging the target area using a single image sensor based on the mixed light. According to the method of claim 5, the single image sensor includes a red signal channel, a green signal channel, and a blue signal channel, wherein the red signal channel is used to sense the red band in the mixed light, the green signal channel is used to sense the green band in the mixed light, the blue signal channel is used to sense the blue band in the mixed light, and at least one of the red signal channel, the green signal channel, and the blue signal channel is further used to sense the second light in the mixed light. According to the method of claim 5, the single image sensor includes a red signal channel, a green signal channel, and a blue signal channel, wherein the red signal channel is used to sense the red band in the mixed light, the green signal channel is used to sense the green band in the mixed light, the blue signal channel is used to sense the blue band in the mixed light, and the red signal channel, the green signal channel, and the blue signal channel are all further used to sense the second light in the mixed light. The method according to any one of claims 1-7, wherein, when the target region reflects the excitation light emitted by the second light source, the excitation light reflected by the target region is filtered out before acquiring an image of the target region to obtain the acquired image. The method according to any one of claims 1-8, wherein the image enhancement processing comprises at least one of the following: White balance processing, linear color correction, and non-linear color mapping. According to the method of claim 9, the white balance processing of the acquired image includes: Adjust the ratio of each color channel in the acquired image to amplify the image signal of the second region in the acquired image. According to the method of claim 9 or 10, the linear color correction processing of the acquired image includes: A linear color correction matrix is used to correct the acquired image to improve the color difference between the first region and the second region in the acquired image. The method according to any one of claims 9-11, wherein performing nonlinear color mapping processing on the acquired image comprises: A color mapping table is established, and the acquired image is subjected to nonlinear transformation based on the color mapping table and a preset nonlinear transformation algorithm to enhance the color difference between the first region and the second region in the acquired image. The method according to any one of claims 1-12, wherein the image signal in the acquired image includes at least one of the following: Brightness, chroma, saturation, contrast, and boundary pixels of different areas. The method according to any one of claims 1-13, wherein the first light includes a portion of the visible light spectrum, the second light is fluorescence, and the visible light and the fluorescence are in different spectrums. According to the method of claim 14, the visible light wavelength is between 380 nm and 780 nm, and the fluorescence wavelength is between 800 nm and 900 nm. According to the method of claim 15, wherein the visible light comprises yellow light in the wavelength range of 550 nm to 650 nm; or, The visible light includes red light in the wavelength range of 600 nm to 660 nm; or... The visible light includes blue light in the wavelength range of 380nm to 470nm. The method according to any one of claims 1-16, wherein the image enhancement processing of the acquired image comprises: Adjust the first area to a colored area and the second area to a green area; or... Adjust the first area to a grayscale area and the second area to a green area. The method according to any one of claims 1-17, wherein the method further comprises: Adjust the intensity of the first light source and/or the second light source to enhance the difference between the image signal of the first region and the image signal of the second region. An imaging system, the imaging system comprising: A light source configured to illuminate a target area, the light source including a first light source and a second light source, the target area providing first light under the illumination of the first light source and providing second light under the illumination of the second light source, the target area including a first object and a second object, the first object and the second object providing light with different spectral characteristics under the illumination of the second light source; An imaging unit is configured to acquire an image of the target region, the acquired image including a first region corresponding to the first object and a second region corresponding to the second object, the first region and the second region having different image signals; An image processor is configured to perform image enhancement processing on the acquired image to enhance the image signals of the first region and the second region in the acquired image, thereby obtaining an enhanced image. The system according to claim 19, wherein the image processor is further configured to not perform compositing processing on the acquired image during the process of performing image enhancement processing on the acquired image to obtain an enhanced image. The system according to claim 19 or 20, wherein the acquired image is only one image during the image enhancement process. The system according to any one of claims 19-21, wherein the imaging unit includes a lens and an image sensor, the lens receiving mixed light from the target region and imaging the mixed light onto the image sensor to obtain an image of the target region, the mixed light including the first light and the second light provided by the target region under synchronous illumination by the first light source and the second light source. The system according to claim 22, wherein the image sensor is a single image sensor. According to the system of claim 23, the single image sensor includes a red signal channel, a green signal channel, and a blue signal channel, wherein the red signal channel is used to sense the red band in the mixed light, the green signal channel is used to sense the green band in the mixed light, the blue signal channel is used to sense the blue band in the mixed light, and at least one of the red signal channel, the green signal channel, and the blue signal channel is further used to sense the second light in the mixed light. According to the system of claim 23, the single image sensor includes a red signal channel, a green signal channel, and a blue signal channel, wherein the red signal channel is used to sense the red band in the mixed light, the green signal channel is used to sense the green band in the mixed light, the blue signal channel is used to sense the blue band in the mixed light, and the red signal channel, the green signal channel, and the blue signal channel are all further used to sense the second light in the mixed light. The system according to any one of claims 22-25 is characterized in that the imaging unit further includes a filter disposed in the optical path before the image sensor for filtering the excitation light reflected by the target region before imaging the target region based on the first light and the second light from the target region to obtain an image of the target region. The system according to any one of claims 19-26, wherein the image enhancement processing includes at least one of the following: White balance processing, linear color correction, and non-linear color mapping. The system according to any one of claims 19-27, wherein the image processor is further configured to adjust the ratio of each color channel in the acquired image to amplify the image signal of the second region in the acquired image. The system according to any one of claims 19-28, wherein the image processor is further configured to correct the acquired image using a linear color correction matrix to improve the color difference between the first region and the second region in the acquired image. The system according to any one of claims 19-29, wherein the image processor is further configured to establish a color mapping table, and to perform a nonlinear transformation on the acquired image based on the color mapping table and a preset nonlinear transformation algorithm to enhance the color difference between the first region and the second region in the acquired image. The system according to any one of claims 19-30, wherein the first light is a portion of the visible light spectrum, the second light is fluorescence, and the visible light and the fluorescence spectrum are different. According to the system of claim 31, the visible light comprises yellow light in the wavelength range of 550 nm to 650 nm; or, The visible light includes red light in the wavelength range of 600 nm to 660 nm; or... The visible light includes blue light in the wavelength range of 380nm to 470nm. The system according to any one of claims 19-32, wherein the image processor is further configured to adjust the first region as a color region and adjust the second region as a green region; or, Adjust the first area to a grayscale area and the second area to a green area. A surgical robot system comprising an imaging system according to any one of claims 19-33, and further comprising a display unit that receives and displays images from the imaging system. The surgical robot system according to claim 34, wherein the surgical robot system further comprises: The controller is configured to control the imaging system to image the target area.