WO2025224161A1 - Method for providing different stereo image output signals using a medical visualisation system, medical visualisation system, and computer program product - Google Patents

Abstract

The invention relates to a method for providing different stereo image output signals (SBA1, SBA2) using a medical visualisation system (4), the method comprising the following steps: a. receiving at least one first stereo image signal (SB1) which has been generated by a stereo system (22) of a surgical microscope (10) and represents a three-dimensional image of an examination region (1); b. providing the first stereo image signal (SB1) as a first stereo image output signal (SBA1) or processing the first stereo image signal (SB1) to form a first stereo image output signal (SBA1); c. providing a further stereo image output signal (SBA2) which represents a further three-dimensional image of the examination region (1), wherein the first stereo image output signal (SBA1) is provided for the complete display of an image according to a first visualisation modality (VM1), and the further stereo image output signal (SBA2) is provided for the complete display of an image according to a further visualisation modality (VM2), wherein the first and the further visualisation modalities (VM1, VM2) are visualisation modalities (VM1, VM2) that are different from one another and are viewer-specific. The invention also relates to a medical visualisation system (4) and to a computer program product.

Description

Method for providing various stereo image output signals through a medical visualization system, medical visualization system and computer program product

The invention relates to a method for providing different stereo image output signals by means of a medical visualization system, a medical visualization system and a computer program product.

Surgical microscopes are used, among other things, to prepare for and perform medical operations on a patient. Such surgical microscopes are used by a user, e.g., a surgeon or an assistant, during a procedure to provide a magnified view of an area of examination, particularly in or on the patient's surgical site. For this purpose, a surgical microscope may include an objective lens or lens system to produce a true optical image of the area of examination. The objective lens may include optical elements for beam guidance, shaping, and/or direction. An optical element may, in particular, be a lens.

Surgical microscopes are used in medical facilities, as well as in laboratories and industrial applications. Examples of medical applications include neurosurgery, ophthalmic surgery, otolaryngology (ENT), plastic or reconstructive surgery, and orthopedic surgery. This list is not exhaustive. Generally, they are used in all areas of surgery where a magnified, high-resolution view of the surgical field is required to perform precise procedures.

A distinction can be made between analog and digital surgical microscopes. Unlike digital surgical microscopes, analog surgical microscopes do not capture images that are then displayed, for example, on a screen to magnify the examination area. Instead, they offer the user a directly visual magnification of the examination area. Here, radiation reflected or scattered from the area of application passes through the objective lens into at least one beam path and to at least one output section, through or into which the user looks to visually perceive the radiation and thus also the typically magnified representation of the examination area. An exemplary embodiment of a The output section is a so-called eyepiece, into or through which the user looks in order to optically perceive the examination area with at least one eye.

Digital surgical microscopes comprise at least one image acquisition device for microscopic imaging, which captures radiation in a beam path of the surgical microscope to generate a magnified image. This image can be displayed to the user or multiple users on one or more display devices, thus enabling high-resolution visualization. The image can be generated as a transmittable image signal that encodes or represents the image. Purely digital surgical microscopes, unlike analog surgical microscopes, do not have an output section for visually detectable radiation, specifically no eyepiece. The image signal can then be transmitted as a data signal, either wired or wirelessly. Digital surgical microscopes allow for the recording, storage, and further processing of images and videos. By applying image processing techniques, contrast, brightness, and other parameters can be adjusted to optimize the image quality of the generated images. Hybrid surgical microscopes can include at least one image acquisition device as well as at least one output section. For example, the radiation guided in a beam path of the operating microscope can be split with a beam splitter, whereby a first portion is directed to the output section and another portion is captured by the at least one image acquisition device.

Stereoscopic surgical microscopes are also well-known. These typically include two separate beam paths for beam guidance and provide the user with a depth perception of the examination area. The beams guided in the two beam paths can be visually detected by the user via output sections. Digital surgical microscopes alternatively or additionally include two image acquisition units, each capturing the beams in one of the beam paths to generate an image. Based on these two images, which can also be referred to as corresponding images, a three-dimensional image is then provided to the user via a suitable display device. The image acquisition units are components of a stereo (camera) system.

The operating microscope can form a medical visualization system, or the medical visualization system can include the operating microscope. The components of the medical visualization system described below may be components of the operating microscope or components designed differently from the operating microscope.

Another known method is the provision of an augmented representation of the examination area to a user. An augmented representation can, in particular, be a representation of the real examination area that is enhanced by computer, especially by adding or overlaying at least one virtual object and/or other additional information onto the representation of the real examination area. The augmented representation can be displayed to a user as an augmented image on a display device or provided in a visually perceptible manner via an output section.

To generate the augmented image, the additional information can be introduced into the beam path, for example, by being reflected. This information can be projected onto a projection element, such as a radiolucent disc, positioned in the beam path using a projection device of the operating microscope. The augmented image can then be generated by creating an image based on the beams into which the additional information has been introduced as described. Alternatively or cumulatively, the radiation representing the augmented image can also be provided by an output section for visual perception by a viewer.

Alternatively, an augmented image can be generated by computer-aided enhancement of an image of the real examination area, particularly through image processing. In this process, the additional information can be superimposed onto the image of the real examination area. With a stereoscopic operating microscope, it is possible to provide the user with two augmented representations. Generally, corresponding additional information can be introduced into each of the two beam paths of a stereoscopic operating microscope. With digital stereoscopic operating microscopes, for example, augmented images can be generated from the images produced by both image acquisition units. Thus, an augmented image can also be displayed to the user on a suitable display device or via an output section. Depth information, i.e., an augmented three-dimensional representation, is provided.

Additional information can be provided in the form of data that represents or encodes a geometric description of a space, particularly a three-dimensional space, especially one containing objects arranged within it. Additional information can also be information generated from such data, for example, information produced by rendering. Rendering, or image synthesis, is the process of computer-implemented generation of a photorealistic or non-photorealistic image from a 2D or 3D model. Multiple models can be defined in a scene file, which contains objects in a defined language or data structure. The scene file can contain geometry, viewpoint, texture, lighting, and shading information that describes the virtual scene. The data contained in the scene file is then passed to a rendering program, which processes it and outputs it to a digital image or raster graphics file. A software application or component that performs the rendering is called a rendering engine, rendering system, graphics engine, or simply a renderer.

A key requirement for augmentation is that the viewer of the augmented image, particularly a surgeon, is not disturbed or distracted by the augmentation during their work. It is especially desirable that a surgeon can operate ergonomically even when viewing an area with superimposed augmentation. For example, it is crucial that an object, such as a tumor, located spatially behind a surface of the examination area along a certain line of sight, is also represented in the augmented image in such a way that it does not mistakenly superimpose objects located in front of the surface, as this can impair the viewer's spatial perception. Furthermore, this can distort the stereoscopic depth perception when viewed through a stereoscopic operating microscope. This can be particularly true if augmented objects are superimposed in a stereoscopically perceptible manner, as the resulting three-dimensional perception can be confusing for the viewer. If several people are present who wish to view an image of the examination area, e.g., a first surgeon, a second surgeon, an assistant and/or an operating room nurse, it may be desirable to provide an image for each of these people.

US Patent 2022/079675 A1 discloses devices and methods for performing a surgical step or procedure with visual guidance using an optical head-mounted display (OHMD) with focus plane adjustment. A system configured for use with multiple OHMDs for multiple viewers is also disclosed. The system is configured so that all viewers view a single display through their OHMD. A disadvantage of the solution described in US Patent 2022/079675 A1 is that while a viewer can access individually desired information through a viewer's OHMD, this is only possible if they, like all other viewers, are viewing a common monitor/display through their OHMD. This results in the information displayed in the OHMD being superimposed on the image shown on the common display. The information displayed on the screen is identical for all viewers and is not personalized. Furthermore, the accuracy of the overlay of information through display in the OHMD is limited, which leads to an inaccurate representation of augmented information in certain scenarios.

The technical problem therefore arises of creating a method for providing different stereo image output signals through a medical visualization system, as well as a medical visualization system and a computer program product, which enables an improved, in particular space-saving, display of information that is individualized for multiple viewers, while also ensuring high accuracy of the display.

The solution to the technical problem is provided by the articles with the features of the independent claims. Further advantageous embodiments of the invention are described in the dependent claims. A method for providing various stereo image output signals via a medical visualization system is proposed. The medical visualization system can be configured as described above. In particular, the medical visualization system can be, or include, a stereo operating microscope. Operating microscopes and their technical characteristics were also briefly described above. The operating microscope can also be configured as an endoscope, specifically a stereo endoscope.

An operating microscope can comprise a microscope body. The objective lens described above can be integrated into the microscope body or attached to it, particularly in a detachable manner. The objective lens can be fixed in position relative to the microscope body. In addition to the objective lens, the microscope body can also have or incorporate at least one beam path for microscopic imaging and/or other optical elements for beam guidance, shaping, and/or deflection. In analog and hybrid operating microscopes, the microscope body can include at least one mounting interface for attaching an output element, such as an eyepiece, in particular in a detachable manner. The microscope body can comprise or form a housing, or be arranged within a housing. Components of the operating microscope, such as an image acquisition device for microscopic imaging, can be arranged in or on the housing.

The medical visualization system can include a stand for mounting the operating microscope. The operating microscope, in particular the microscope body, can be mechanically attached to the stand. The stand is designed to allow movement of the operating microscope in space, in particular with at least one degree of freedom, preferably with six degrees of freedom, where one degree of freedom can be translational or rotational. Furthermore, the stand can include at least one drive unit for moving the operating microscope. Such a drive unit can, for example, be a servo motor. Naturally, the stand can also include means for force/torque transmission, e.g., gear units. In particular, it is possible to control the at least one drive unit in such a way that the operating microscope performs a desired movement and thus a desired change of position in space, or assumes a desired position and/or orientation in space. For example, the at least one The drive unit can be controlled such that an optical axis of an objective lens of the operating microscope assumes a desired orientation. Furthermore, the at least one drive unit can be controlled such that a reference point of the operating microscope, e.g., a focal point, is positioned at a desired position in space. A target position can be specified by a user or another higher-level system. Methods for controlling the at least one drive unit as a function of a target position and a kinematic structure of the stand are known to those skilled in the art.

Furthermore, the medical visualization system can include one or even several display devices for showing the images. The display device can be used to show two- or three-dimensional images. Typical display devices are screens, especially 3D screens, head-mounted displays (HMDs), or digital eyepieces, which can also be referred to as booms.

Furthermore, the medical visualization system, in particular the operating microscope, may include one or more of the following elements:

• at least one white light lighting device,

• at least one infrared lighting device,

• at least one fluorescence illumination device for excitation of

Fluorescence radiation,

• at least one beam filter to provide excitation radiation with wavelengths from a broader spectrum, e.g. the spectrum of the white light illumination device,

• at least one fluorescence detection device for detecting fluorescence radiation,

• at least one filter device for filtering radiation from a broader spectrum, e.g. for detection by an image acquisition device for microscopic imaging,

• at least one image acquisition device of an optical position detection device, which can also be referred to as a surrounding camera,

• at least one gaze direction detection device,

• at least one position detection device for determining a pose, i.e. a position and/or orientation, at least of the operating microscope • at least one input device for operation,

• at least one interface for data transmission to or from another system or facility,

• at least one device for determining depth information, in particular with regard to the elements arranged in the detection range of the operating microscope, which may be designed, for example, as a distance sensor,

• at least one storage device for storing signals and/or information, in particular in a retrievable manner,

• a device for identifying a viewer.

An image acquisition device may, in particular, include a CMOS or CCD sensor. The detection range of the ambient camera may fully or at least partially encompass the detection range of the surgical microscope. Alternatively, the detection range of the surgical microscope may fully or at least partially encompass the detection range of the ambient camera.

In a fluorescence visualization mode, a filter device can be inserted into an observation beam path, providing the viewer with a filtered representation of the examination area. This radiation can be captured by the at least one image acquisition device for microscopic imaging. Alternatively, fluorescence radiation can also be captured by a separate acquisition device, such as a spectral camera. The fluorescence mode advantageously enables intraoperative tissue differentiation. Tumor tissue, in particular, can be visualized using fluorescence-based images. Nerve tissue, in particular, can be visualized using polarization-contrast-based images.

Medical visualization systems can be operated, for example, by manually controlling a component, particularly the operating microscope, or a corresponding input device; by voice control; by gesture control; by eye tracking; by image-based control; or by other operating methods. The medical visualization system or the operating microscope may include the necessary components. Image-based control may, in particular, generate operating or control signals by evaluating at least one Images include those produced by an image acquisition device for microscopic imaging or an image acquisition device of an optical position detection device.

Adjustable operating parameters of the medical visualization system or the surgical microscope can be formed by one or more of the following parameters:

• Magnification factor or zoom factor,

• Working distance or focus position,

• Detection range,

• Light intensity,

• Illumination spectrum.

The method comprises the following steps: a) Receiving at least one first stereo image signal generated by a stereo system of the operating microscope, representing a three-dimensional image of an examination area. A three-dimensional image can, in particular, be or comprise a stereo image pair, wherein the images of this image pair are stereoscopic images. The image signal can be received via a suitable interface, in particular by an evaluation unit of the medical visualization system. A stereo image signal can, in particular, encode or represent two corresponding images, i.e., a stereo image pair, generated by the image acquisition units of a stereo system of the medical visualization system. b) Providing the first stereo image signal as a first stereo image output signal or processing the first stereo image signal into a first stereo image output signal. A stereo image output signal encodes or represents a three-dimensional image, in particular a stereo image pair comprising two corresponding images of a stereo system, which is intended for output or display by a 3D display device. A 3D display device can be, for example, a digital eyepiece, a 3D screen, or an HMD (head-mounted display). The stereo image output signal can In particular, two corresponding two-dimensional images are encoded or represented, each displayed by a separate display device. It is conceivable that the corresponding images are output sequentially by the same display device or simultaneously by different display devices. The stereo image output signal can, in particular, represent a magnified three-dimensional image of the area under investigation.

Processing the initial stereo image signal can involve the application of image processing techniques to generate the stereo output signal from the stereo image signal, potentially taking into account additional signals or information. Such an image processing technique could, for example, be a filtering method. Augmentation can also occur during processing; for instance, additional information can be superimposed on the three-dimensional image, and the resulting image can then be provided as the initial stereo output signal. As explained in more detail below, processing can also include generating a three-dimensional reconstruction of the imaged area and creating the initial stereo output signal based on this reconstruction.

In particular, a virtual (3D) image or an augmented (3D) image can depict visible and/or hidden objects or elements, thus enabling their visual representation. c) Providing an additional stereo image output signal that represents another three-dimensional image of the area under investigation. This additional three-dimensional image can, in particular, depict the same area under investigation and be represented by another stereo image signal.

In particular, a further stereo image signal can be provided as a further stereo image output signal. This further stereo image (output) signal can be described as a stereo image (output) signal corresponding to the first stereo image (output) signal. The further three-dimensional image represented by the further stereo image output signal can depict the area under investigation with the same properties as the three-dimensional image represented by the first stereo image output signal. is represented, in particular with the same magnification properties.

The additional three-dimensional image can be generated by image acquisition devices, such as those of another stereo system of the operating microscope. This additional three-dimensional image can also be a white light image, a fluorescence contrast image, a polarization contrast image, an image generated from additional information, or an image combined from at least two of these modalities, for example, through augmentation. An image generated through augmentation can be displayed, in particular, by an AR (Augmented Reality) display device. An image generated purely from additional information can be displayed, in particular, by a VR (Virtual Reality) display device. It is possible that such images are generated by the image acquisition devices of the stereo system or by different image acquisition devices of the medical visualization system, especially the operating microscope.

The further three-dimensional image can also be simulated. For example, it can be a virtual image generated by virtual image acquisition devices, which can be, in particular, optical models of the image acquisition devices of the stereo system. Specifically, the further stereo image output signal can be generated depending on additional information.

It is also possible to process the first stereo image signal into a further stereo image output signal, which is then provided as a further stereo image output signal. Such processing was explained previously.

The corresponding stereo image signal can be generated, for example, by another stereo system of the medical visualization system, particularly the operating microscope, and represents a three-dimensional image produced by this additional stereo system. The corresponding stereo image signal can also be generated from additional information, which has already been explained. If the additional stereo image output signal is provided by processing the first stereo image signal, the method for providing it may differ from the method for providing the first stereo image signal from the first stereo image signal. The procedure can be carried out in various ways, in particular with different adjustable parameters, depending on which the processing takes place.

Image signals, in particular the image output signals, can then be transmitted to a display device, which is then controlled to output the image represented by the image (output) signal in a visually perceptible manner.

Additional information that can be displayed to a user in the form of an image or, for example, through augmentation, can include preoperatively generated information, such as preoperatively generated data, which can also be used for surgical planning. Such preoperatively generated data can be, in particular, volumetric data. Volumetric data can be provided as a point cloud, a voxel-based representation, or a mesh-based representation. The additional information can also be provided, in particular, as a transmittable signal.

Preoperative data can be generated, for example, using computed tomography or magnetic resonance imaging (MRI). Other imaging techniques, particularly ultrasound, X-ray, fluorescence, SPECT, or PET, can also be used. Such augmentation allows, for example, a tumor object or its contours, generated from preoperative information, to be superimposed on a white light image. Of course, it is also possible to display such preoperative information on its own, without augmentation.

Preoperative data generated using magnetic resonance imaging (MRI) can identify various tissue types, such as adipose tissue, muscle tissue, tumor tissue, as well as blood vessels and nerve pathways. Preoperative data generated using computed tomography (CT) can particularly depict bony structures.

Alternatively or additionally to the use of preoperatively generated information, it is possible to use intraoperative information, i.e., information recorded or generated during treatment, as supplementary information. This allows for the... For example, information can be collected and stored during an operation, which can then be used to display an image or to generate an augmented image. This is particularly advantageous when there are different visualization modalities that are activated at different times. For instance, in a fluorescence visualization modality, fluorescence information can be displayed as an image or superimposed on a white light image.

The additional information can be assigned a reference coordinate system, which means that the additional information can also include spatial information. This reference coordinate system can also be called a world coordinate system.

The first stereo image output signal is provided for the complete display of a three-dimensional image according to a first visualization modality. The second stereo image output signal is provided for the complete display of a three-dimensional image according to a further visualization modality, wherein the first and the further visualization modalities are distinct and viewer-specific. A complete display can be the display of the complete three-dimensional image represented by the stereo image signal according to the viewer-specific visualization modality, in particular by a single display device. This can especially mean that no more than one display device is required for a complete display according to the corresponding visualization modality.

A visualization modality refers to the way in which an image is displayed and/or what information is represented in the image. In particular, a visualization modality can include a display with predetermined color properties and/or predetermined transparency properties.

According to a visualization modality, for example, a white light image, a fluorescence contrast image, a polarization contrast image, an image generated from additional information, or an image combined from these modalities, in particular an augmented image, can be displayed on a display device. A white light image (VIS image) is, in particular, an image generated based on radiation with Wavelengths from a range of 360 nm to 780 nm were generated. A fluorescence contrast image is, in particular, an image generated with fluorescence-specific wavelengths, especially with predetermined fluorescence-specific wavelengths or wavelength ranges, for example, with wavelengths of 400 nm or 560 nm. A polarization contrast image is, in particular, an image generated with radiation of a predetermined polarization. An image generated from additional information has already been explained in more detail above and can, in particular, be an image generated from preoperative data information, which is generated, in particular, with a virtual image acquisition device. An image combined from at least two modalities can, in particular, be an image in which an image according to a first visualization modality is superimposed on an image according to another visualization modality, especially in a partial area. Thus, a combined image can be an augmented image.

Images can also be generated in different ways, tailored to each viewer, to suit various visualization modalities. This can be achieved by rendering using different rendering styles. These styles can include: transparent rendering (e.g., alpha-blending), animation-based rendering, texture-based rendering, and Fresnel-effect-based rendering.

To provide stereo image output signals, it may be necessary to identify different viewers. This can be achieved using a viewer identification device and appropriate identification methods. Alternatively, different viewers can register with the medical visualization system using viewer-specific information, for example, by operating a suitable input device. Image-based viewer identification is also conceivable. Image-based can mean that viewer identification is performed by evaluating at least one image, where the image can depict the operating room or a part thereof, and the viewer is positioned within the depicted area. Marker-based identification is also possible, where a marker assigned to a viewer is detected, particularly optically. In particular, the marker's identity can be... The viewer identity must be assigned, for example in the form of a computer-evaluable assignment.

A viewer identity can be assigned a viewer-specific visualization modality. Information about this assignment can be stored in a retrievable manner, particularly in a storage device of the medical visualization system or in a storage device of an external system. A viewer-specific visualization modality can be a visualization modality defined specifically for the viewer. This can be set, for example, by operating an input device, particularly in a manner associated with the viewer identity. This setting can be made preoperatively. However, it is also possible for the viewer-specific visualization modality to be set, particularly changed, intraoperatively. This can also be done by operating a corresponding input device.

To provide the stereo image output signals, it may be necessary to identify a viewer-specific display device on which the viewer-specific stereo image output signal is to be displayed. This can also be assigned to the viewer's identity. Suitable identification methods can also be used for this purpose. For example, it is conceivable to identify a viewer-specific display device depending on the viewer's pose and/or viewing direction, which can be determined as explained below. For example, depending on the pose and/or viewing direction, it can be determined which display device the viewer is currently looking at and identified as the viewer-specific display device. For this, it may also be necessary to determine the pose of a display device, e.g., with a tracking system.

It is also possible for different viewers to log in to the medical visualization system with information about their specific display device, for example, by using a suitable input device. Information about the assignment of a viewer identity to their specific display device can also be stored in a retrievable manner. Depending on the viewer identity and, if necessary, by evaluating the Based on the explained assignment(s), a viewer-specific visualization modality can thus be determined.

It is also possible to determine or record a viewer-specific pose, as well as a viewer-specific direction of gaze. This will be explained in more detail below.

Overall, this advantageously results in a simple and viewer-specific provision of various stereo image output signals, which can then be fully and individually output for a viewer by a common or by different, particularly viewer-specific, 3D display devices. In particular, it is not necessary to provide an individualized display by having the viewer look at a screen through an overhead display (OHMD). This also achieves improved display accuracy.

In another embodiment, a white light image, a fluorescence contrast image, a polarization contrast image, an image generated from additional information, or an image combined from these modalities is displayed according to a visualization modality. This has already been explained above. In particular, the various visualization modalities allow viewers to be provided with individually desired information. For example, a white light image enables good orientation, while a fluorescence contrast image is particularly suitable for improved visualization of, for example, tumorous tissue. A polarization contrast image is suitable for visualizing nerve tissue. An image generated from additional information also advantageously allows the display of preoperative or further intraoperative information.

In another embodiment, a viewer's pose and/or gaze direction is determined, and a viewer-specific stereo image output signal is provided depending on the pose and/or gaze direction. The described position detection device, which can also be referred to as a tracking system, can be used to determine the pose. A tracking system can be optical, electromagnetic, or operate in another way. The tracking system can be a marker-based tracking system that detects active or passive markers. Markers can be positioned on objects or subjects whose pose is to be detected by the tracking system. An optical tracking system An optical tracking system can include, in particular, optically detectable markers. Specifically, an optical tracking system can be a monoscopic position detection system. Here, the pose of an object can be determined by evaluating a two-dimensional image, specifically a two-dimensional image. In particular, the position can be determined by evaluating the intensity values of pixels (image points) of the two-dimensional image. It is also conceivable that the medical visualization system includes at least one image acquisition device of an optical position detection device, which can be, in particular, a component of the operating microscope. This can also be referred to as a peripheral camera and serves, in particular, for monoscopic position detection. A tracking system can also be part of an input device, whereby, for example, gesture control or gaze direction control is performed depending on information generated by the tracking system.

The pose of a viewer can therefore be captured, in particular by a tracking system, whereby at least one marker can be placed on the viewer. It is also conceivable, for example, to determine the pose of a body part of such a user, e.g., a hand, an arm, or a head. The position detection device can be a so-called registered device. This can mean that the pose information can be determined with respect to a reference coordinate system, in particular a common reference coordinate system.

Naturally, it is possible for the position detection device, or another position detection device, to also detect the pose of at least one other object or part thereof. An object can be, in particular, another component of the medical visualization system, especially a display device. However, an object can also be an item that is not part of the medical visualization system, e.g., a piece of equipment such as an operating table or a medical instrument. This makes it possible to determine the pose of the additional subject or object within a desired reference coordinate system.

The viewing direction of a viewer can be determined using the gaze direction detection device, and in particular image-based methods. For this purpose, medical devices can be used. A visualization system may include an image capture device, which is, for example, assigned to and/or designed to image the eyes of a viewer. The direction of gaze can then be determined by evaluating the generated images. Such methods are known to those skilled in the art. The device for determining/capturing the direction of gaze can also be a registered device; in particular, the gaze direction information can be determined in a reference coordinate system that differs from a device-specific coordinate system.

In particular, a stereo image output signal can be provided in such a way that the represented three-dimensional image is displayed with correct perspective, taking the microscopic image into account. In other words, the three-dimensional image can then be displayed as if the respective viewer, with their specific pose and/or viewing direction, were visually perceiving the examination area through the operating microscope. For this purpose, it may also be necessary to consider the pose of the operating microscope itself. Its determination will be explained in more detail below.

If a pose of the operating microscope is also determined, for example also with the tracking system, the viewer-specific stereo image output signal can be generated depending on the viewer's pose relative to the operating microscope.

For example, it is conceivable that different observers, from different positions and/or with different viewing angles, might wish to visually perceive the examination area magnified by the operating microscope. In other words, microscopic image information can be displayed depending on the observer's pose relative to the operating microscope.

The proposed approach then enables each of these viewers to have a perspective-correct representation of the three-dimensional images represented by the stereo image output signals, according to the viewer-specific visualization modality. It is also conceivable that, in different visualization modalities, viewers with different poses and/or viewing directions are each presented with a pose- and/or viewing-direction-dependent image, as explained. In a further embodiment, the provision of the viewer-specific stereo image output signal includes the generation of virtual images by virtual image acquisition devices, which are an optical model of the image acquisition devices of the stereo system. Virtual images can be generated, in particular, based on additional information, especially preoperatively and/or intraoperatively generated additional information. A virtual image can encode a texture, in particular with color and/or transparency information.

For example, virtual image acquisition devices can each generate an additional information signal, representing a virtual image created from or by the additional information. The virtual image acquisition devices can be optical models of the stereo system's image acquisition devices. This advantageously generates a perspectively correct and three-dimensionally perceptible representation of the additional information.

The additional information can be assigned a reference coordinate system, which means that the additional information can also include spatial information. This reference coordinate system can also be called a world coordinate system.

For this purpose, it may be necessary or useful to perform a registration between the reference coordinate system of the additional information and a reference coordinate system of the medical visualization system, in particular the operating microscope or an image acquisition device of the operating microscope. This registration can be carried out before the virtual images are generated. The registration establishes a spatial relationship between both the additional information and an image of the examination area, especially the three-dimensional image of the examination area, to a common reference coordinate system. This common reference coordinate system, which is also referred to below as the reference coordinate system, can be, in particular, the reference coordinate system of the additional information, the reference coordinate system of the medical visualization system, or a different reference coordinate system. Methods for registration are known to those skilled in the art. For example, model-based registration can be performed. In this case, features can be detected in an image that correspond to previously known features, e.g., geometric features in the supplementary information, and the registration can then be determined in a known manner depending on these corresponding features. The registration can, for example, be determined in the form of a transformation matrix that includes a rotation and/or translation component. An exemplary model-based registration can be an edge-based registration, where the corresponding features are formed, for example, by a property of at least one, preferably several, edges in both the image and the supplementary information. Topography-based registration can also be performed, particularly if a topography can be determined, e.g., with a stereo system of an operating microscope. In this way, topographic information can be determined in the at least one image, and corresponding features, points, or sections can then be detected in both the supplementary information and this topographic information, which can then be used to determine the registration.

The additional information can therefore be registered information. For the purposes of this invention, the property "registered" can mean that a spatial reference to the reference coordinate system is known, in particular in the form of a transformation matrix. A registered device can generate signals whose spatial reference to the reference coordinate system is known.

The additional information can be generated, in particular, through rendering. For example, a virtual image can be created through rendering, which is then used to display the augmentation and can be superimposed on a live-recorded image of the real examination area. The virtual image can also be provided as an image signal, which encodes or represents the virtual image. The viewer-specific stereo image output signal can therefore consist of, or comprise, two image signals.

The virtual image can be generated with a virtual image acquisition device, which can be a mathematical or physical and, in particular, computer-aided evaluable optical model of an image acquisition device. In particular A computer-implemented calculation of the pixels of the virtual image can be performed. This virtual image depends, among other things, on parameters of the (modeled) image acquisition device. In particular, the virtual image can be generated based on the intrinsic parameters of the image acquisition device for microscopic imaging, especially with these parameters. The virtual images generated by a virtual stereo system can additionally also be generated based on the extrinsic parameters of the two image acquisition devices of the (modeled) stereo system, especially with these parameters.

In other words, when evaluating the model to generate virtual images, the parameters of the stereo system's image acquisition device(s) used for microscopic imaging can be taken into account. This makes it possible to generate virtual images under the same conditions as real images.

The virtual image can also be generated based on a pose, i.e., a position and/or orientation, of the (modeled) image acquisition device of the stereo system. In particular, the pose of the stereo system's image acquisition device(s) can be taken into account when evaluating the model to generate the virtual images, utilizing the registration information described above. By considering the registration information, it is possible, for example, to determine which pose of the virtual image acquisition device in the reference coordinate system of the additional information, which can also be referred to as the render coordinate system, corresponds to the actual pose of the (modeled) image acquisition device of the stereo system, and this information can then be used for the rendering process. In other words, a pose of at least one virtual image acquisition device can be identical to the pose of the modeled image acquisition device in the reference coordinate system. Thus, it is possible to generate a virtual image that corresponds to the modeled image acquisition device in terms of both parameters and capture pose.

Thus, for the provision of virtual images, it may also be necessary to determine the current pose of the stereo system, in particular of its image acquisition devices. This pose can also be determined using the position acquisition device. A registration allows a relationship to be established between the reference coordinate system of the position acquisition device and the previously described reference coordinate systems, in particular the reference coordinate system. The additional information can be determined. Corresponding registration methods are known to those skilled in the art. This makes it possible to determine the pose of the stereo system in a desired reference coordinate system, in particular the reference coordinate system.

When a pose of the stereo system or the operating microscope is determined, the pose of the objective's optical axis or the position of a focal point can also be determined. If the operating microscope is mounted on a stand with at least one joint, its pose can also be determined as a function of the joint position, which can be detected, for example, by a sensor. This, in turn, allows the determination of the viewing angle/direction of the operating microscope under which the area of investigation was imaged.

The position of the stereo system and the additional information can define a previously described scene, i.e., a virtual spatial model that defines objects and their material properties, light sources, as well as the position and viewing direction of an observer, here the stereo system.

The described generation of virtual images advantageously enables a simple and perspective-correct representation of images generated from additional information in an individualized manner.

In another embodiment, the virtual images represent three-dimensional additional information. In particular, the virtual images can be corresponding images of a virtual stereo system. This also advantageously enables a simple and perspectively correct representation of additional information in an individualized manner.

In another embodiment, the three-dimensional additional information is generated from images of the stereo system's image acquisition devices. In particular, a reconstruction of the stereo system's detection range can be determined based on the images generated by the stereo system's image acquisition devices, and this reconstruction can be a three-dimensional, data-based representation of the detection range. For example, it is possible that during During operation of the stereo system of the operating microscope, corresponding images, i.e., stereo image pairs, are stored, which are then used for the reconstruction described. For this purpose, it may be necessary to generate at least a predetermined number of corresponding images.

This reconstruction, in turn, can provide the additional information. This makes it possible, in a simple and reliable way, to present viewers in different poses with a perspectively correct image, for example, in a white light visualization mode.

In a further embodiment, the additional stereo image output signal represents another three-dimensional image, which was generated at least partially or completely by another stereo system of the medical visualization system. Thus, the medical visualization system, in particular the surgical microscope, can comprise another stereo system. At least one, but preferably both, image acquisition devices of the additional stereo system can be different from the image acquisition devices of the (first) stereo system. It is conceivable that a pose of the additional stereo system relative to the (first) stereo system can be changed. In particular, the additional stereo system can be movably mounted on/in the surgical microscope. It is also conceivable that the first and the additional stereo systems are arranged on different elements of the medical visualization system, for example, on stands of different designs.

This advantageously enables the reliable and perspectively correct generation of images for different viewers, who, for example, each desire a representation of an image in a white light visualization modality, for different poses of these viewers.

For example, it is conceivable that the first stereo image signal is provided as a stereo image output signal and the subsequent stereo image output signal as a stereo image signal generated by the other stereo system.

In another embodiment, the various stereo image output signals are used to display the respective represented images by a common A display device is provided. On this shared display device, the corresponding three-dimensional images can be displayed or output in the respective viewer-specific visualization modality for at least two different viewers.

Preferably, the shared display device is a 3D screen. As explained in more detail below, the three-dimensional representation can be achieved in various ways. The essential point is that the shared display device presents images to a first viewer in the viewer-specific first visualization modality, while the shared display also presents images to a second viewer according to a viewer-specific second visualization modality.

This advantageously saves space, as multiple viewers can use a single display to view images according to their individual visualization preferences. In contrast to head-mounted displays, this also allows the viewer to perceive their surroundings more effectively.

In a further embodiment, the first stereo image output signal for displaying the represented images is provided with a first polarization encoding, and the second stereo image output signal for displaying the represented images is provided with a further polarization encoding, wherein the first and the further polarization encodings are different from each other. For example, it is possible that a first image encoded in the first stereo image output signal is provided for display with a first polarization, and a further image encoded by the first stereo image output signal is provided for display with a second polarization. Furthermore, a first image encoded by the further stereo image output signal can be provided for display with a third polarization, and a further image encoded by the further stereo image output signal can be provided for display with a fourth polarization. The first and further images are images of a stereo image pair. In this embodiment, it may be necessary for viewers to wear polarization filter glasses through which the images on the common display device are visually perceived, with the lenses of the polarization filter glasses being transparent to only one of the polarizations. In this case, the first polarization encoding consists of providing the corresponding images encoded in the first stereo image output signal for display with the first and second polarizations, and the subsequent polarization encoding consists of providing the corresponding images encoded in the further stereo image output signal for display with the third and fourth polarizations. The images encoded by the stereo image output signals can be displayed simultaneously or sequentially by the common display device. This advantageously results in a simple and reliable display of the respective represented images by a single display device.

In an alternative embodiment, the additional stereo image output signal for displaying the represented images is provided at a time that is later than the time of display of the images represented by the first stereo image output signal. In particular, it is possible to display a first image and a further image corresponding to the first image, represented by the first stereo image output signal, sequentially, especially with a predetermined time offset, using the common display device. Specifically, the first image can be displayed at a first time and the further corresponding image at a second time. It is also possible to display the first image and the further corresponding image represented by the additional stereo image output signal sequentially, especially with a predetermined time offset, on the common display device. Specifically, this first image can be displayed at a third time and the further corresponding image at a fourth time. Here, the fourth time can be after the third time, the third time after the second time, and the second time after the first time. In this embodiment, it may be necessary for viewers to wear so-called shutter glasses to view the common display device. These glasses, as is known to those skilled in the art, allow viewing through the left eye while simultaneously preventing viewing through the right eye, and at a later time, prevent viewing through the left eye while allowing viewing through the right eye. Also as is known, the operation of such shutter glasses is synchronized with the display times. This results in Advantageously, a simple and reliable display of the respective represented images is achieved through the common display device.

In a further alternative embodiment, the first stereo image output signal for displaying the represented images is provided with a first color encoding, wherein the second stereo image output signal for displaying the represented images is provided with a further color encoding, the first and the further color encodings being different from each other. Different color encodings can be based on different color bases.

A color base of a color space can be defined by a number, e.g. three, of primary colors that span the color space, whereby each color point, i.e., each color, in the color space can be defined by a combination of these three primary colors.

Different color bases can preferably span the same color spaces. Two color spaces can be considered essentially the same if they overlap at least in one area, particularly in a large part of the area, for example in an area > 70%, > 80% or > 90%, and especially preferably in an area that includes visible light.

Different color bases can differ in at least one, and preferably in all, primary colors. The primary colors specific to each color base can be chosen such that they can be separated from one another by a suitable color filter or combination of color filters. For example, a first color filter can be transparent to the primary colors of a first color base and opaque to the primary colors of at least one other color base. A further color filter can be transparent to the primary colors of the second color base but opaque to at least the primary colors of the first color base.

For example, it is possible that a first image encoded in the first stereo image output signal is provided for display according to a first color base, and a further image encoded by the first stereo image output signal is provided for display according to a second color base. Furthermore, a first image encoded by the further stereo image output signal can be provided for display according to a third color base, and a further image encoded by the further stereo image output signal can be provided for display according to a second color base. The images are provided for display according to a fourth color base. The first and subsequent images are again images of a stereo image pair. In this embodiment, it may be necessary for the viewers to wear color filter glasses through which the images on the common display device are visually perceived, the lenses of the color filter glasses being transparent only to the primary colors of one color base. Such color filter glasses can, in particular, be interference filter glasses.

In this case, the first color encoding consists of providing the corresponding images encoded in the first stereo image output signal for display according to the first and second color bases, and the subsequent color encoding consists of providing the corresponding images encoded in the further stereo image output signal for display according to the third and fourth color bases. The images encoded by the stereo image output signals can be displayed simultaneously or sequentially by the common display device. This advantageously results in a simple and reliable display of the respective represented images by a single display device.

In a further alternative embodiment, the first stereo image output signal for displaying the represented images is provided with a first angle encoding, and the second stereo image output signal for displaying the represented images is provided with a further angle encoding, wherein the first and the further angle encodings are different from each other. In this embodiment, the common display device can be configured to display a so-called lenticular image, i.e., as a display device with a lenticular grid. Alternatively, the common display device can be configured as a display device with a parallex barrier. These can be controlled in a manner known to those skilled in the art. Of course, other display technologies are also conceivable that output the respective encoded images at different angles. In particular, the common display device can be configured as a so-called two-view display or multi-view display.

For example, the first and the subsequent images corresponding to the first, which are encoded by the first stereo image output signal, can be combined with a corresponding The configured or set lens grid is output such that the first image can be perceived by the left eye and the corresponding second image by the right eye of a viewer. Similarly, the first image and the corresponding second image, encoded by the additional stereo image output signal, can be output at angles such that the first image can be perceived by the left eye of another viewer and the corresponding second image by the right eye of that viewer. In this embodiment, a pose and/or viewing direction of the respective viewer can preferably be determined, with the angle encoding then being adjusted depending on the pose and/or viewing direction. This also advantageously results in a simple and reliable display of the respective represented images by the common display device.

A further proposed medical visualization system comprises at least one interface for receiving at least one first stereo image signal, which was generated by a stereo system of an operating microscope of the medical visualization system and represents a three-dimensional image of an examination area. The visualization system further comprises at least one evaluation unit. The medical visualization system, in particular the evaluation unit, is configured to perform a method according to one of the embodiments described in this disclosure.

The evaluation unit can be configured as a computing unit or include one. A computing unit, in turn, can include at least one microcontroller and/or at least one integrated circuit, or be configured as such. The computing unit can, in particular, generate the virtual image. It is possible that the evaluation unit includes at least one graphics processing unit (GPU) for this purpose. The medical visualization system advantageously enables the execution of a method according to one of the embodiments described in this disclosure, with the advantages already explained.

In another embodiment, the medical visualization system comprises at least one device for identifying a viewer. The setup and its corresponding technical advantages have already been explained previously.

In another embodiment, the medical visualization system includes an additional stereo system. This, along with its corresponding technical advantages, has already been explained previously.

In another embodiment, the medical visualization system comprises at least one device for determining a viewer's pose and/or gaze direction. Reference can also be made to the previously provided explanations and the corresponding technical advantages.

In another embodiment, the medical visualization system comprises at least one 3D display device, wherein the display device is configured as a digital eyepiece, a 3D screen, or a head-mounted display. This, along with its corresponding technical advantages, has already been explained. A digital eyepiece can also be referred to as a boom. Such a digital eyepiece can be a head-mounted eyepiece. It can include two display devices for the stereoscopic display or representation of images. Alternatively, such an eyepiece can also be attached to a support device, in particular a stand, which may differ from the stand of the surgical microscope.

A further proposal is a computer program product comprising a computer program, wherein the computer program includes software means for executing one, several, or all steps of the method according to one of the embodiments described in this disclosure, when the computer program is executed by or in a computer or an automation system. The computer or automation system may include the evaluation device described above. The computer program product may, in particular, include means for performing the rendering, i.e., a rendering engine. The computer program product advantageously enables the execution of a method according to one of the embodiments described in this disclosure with the advantages already explained. The invention is explained in more detail using exemplary embodiments. The figures show:

Fig. 1 shows a schematic flowchart of a method according to the invention,

Fig. 2 shows a schematic flowchart of a method according to the invention in a further embodiment,

Fig. 3 shows a schematic flowchart of a method according to the invention in a further embodiment,

Fig. 4 shows a schematic flowchart of a method according to the invention in a further embodiment,

Fig. 5 shows a schematic top view of an application scenario.

Fig. 6 shows a schematic top view of another application scenario and

Fig. 7 shows a schematic block diagram of a medical visualization system according to the invention.

In the following, identical reference symbols denote elements with the same or similar technical characteristics.

Fig. 1 shows a schematic flowchart of a method according to the invention for providing various stereo image output signals SBA1, SBA2 by a medical visualization system 4 (see Fig. 7). In a receive step ES, a first stereo image signal SB1 is received, which was generated by a stereo system 3 of an operating microscope 10 (see Fig. 7) and which represents a three-dimensional image of an examination area. The first stereo image signal SB1 can, in particular, comprise a stereo image pair, wherein the images of this stereo image pair were generated by image acquisition devices 5, 5b of the stereo system 22. These images can be received via interfaces 21 of the medical visualization system 4. In a first provisioning step BS1, this first stereo image signal SB1 is provided as the first stereo image output signal SBA1. Alternatively, the first stereo image signal SB1 can be processed to produce the first stereo image output signal SBA1; in particular, an augmented three-dimensional image can be provided as the first stereo image output signal SBA1.

In a further provisioning step BS2, another stereo image output signal SBA2 is provided, representing another three-dimensional image of the examination area. This additional three-dimensional image can, in particular, depict the same examination area and be represented by another stereo image signal. It can depict the examination area with the same properties as the three-dimensional image represented by the first stereo image output signal SBA1. Thus, the additional stereo image output signal SBA2 can also be referred to as a stereo image output signal corresponding to the first stereo image signal SB1. This signal can also comprise a stereo image pair.

The additional three-dimensional image can be generated by image acquisition devices, e.g., by image acquisition devices of another stereo system of the operating microscope 10. In this case, the additional three-dimensional image can be received in a further reception step (not shown). The additional three-dimensional image can also be generated as a white light image, as a fluorescence contrast image, or as a polarization contrast image by a corresponding acquisition device 23. The additional three-dimensional image can also be an image generated from additional information ZI or an image combined from at least two of these modalities, e.g., by augmentation. An image generated from additional information ZI can be simulated. For example, it can be a virtual image generated by virtual image acquisition devices, which can be, in particular, optical models of the image acquisition devices 5, 5b of the stereo system 22. Thus, the additional stereo image output signal SBA2 can be generated depending on additional information ZI.

The first stereo image signal SB1 can also be processed into a further stereo image signal, which is then provided as a further stereo image output signal SBA2. Such processing was explained previously. An optional storage device 6 of the medical visualization system 1 is shown in dashed lines. Additional information ZI, stored in this storage device 6, can be retrieved to provide the further stereo image output signal SBA2. A detection device 23 for generating the further three-dimensional image represented by the further stereo image output signal SBA2 is also shown. This detection device 23 can, for example, be the further stereo system described above. Alternatively, the further detection device 23 can be a fluorescence detection device 14 or a polarization contrast imaging device.

The stereo image output signals SBA1 and SBA2 are provided such that the first stereo image output signal SBA1 is used to fully display an image according to a first visualization modality VM1, and the second stereo image output signal SBA2 is used to fully display an image according to a further visualization modality VM2 (see Fig. 2), wherein the visualization modalities VM1 and VM2 are different and viewer-specific. This has been explained above. Thus, the stereo image output signals SBA1 and SBA2 enable the full display of an image according to a viewer-specific visualization modality VM1 or VM2 on a display device 20, 20a, or 20b (see Fig. 5 and Fig. 6). In particular, an image according to a viewer-specific visualization modality VM1, VM2 does not need to be displayed by different display devices, which in particular means that two image output signals do not need to be generated to control the different display devices.

Fig. 2 shows a schematic flowchart of a further embodiment of a method according to the invention. In this embodiment, an identification step IS is performed before the execution of the method steps ES, BS1, BS2 shown in Fig. 1. In this identification step IS, various viewers 24a, 24b (see, for example, Fig. 5) are identified. Such different viewers 24a, 24b can, for example, be identified based on images. Identification can also be carried out depending on the input of user identities via a suitable input device 18 of the medical visualization system 1 (see Fig. 7). A storage device 6 is also shown, which can store assignment information in a retrievable manner, wherein the assignment information represents an assignment of a viewer identity to a visualization modality VM1, VM2. Thus, as a result of the In the identification step IS, various visualization modalities VM1 , VM2, ... are determined according to which the stereo image output signals SBA1, SBA2 are to be generated.

In a pose determination step PBS, a pose and/or a viewing direction of a viewer 24a, 24b is determined. This pose information PI can then be used to generate the stereo image output signals SBA1, SBA2. In particular, the stereo image output signals SBA1, SBA2 can be provided in such a way that a perspectively correct display is enabled for the respective viewer 24a, 24b on a common display device 20 or on the viewer-specific display device 20a, 20b.

It is also possible to determine the pose of the stereo system 22 of the operating microscope 10, whereby the stereo image output signals SBA1, SBA2, and in particular the additional stereo image output signal SBA2, are provided depending on the pose of the stereo system 22. This is particularly relevant if the additional stereo image output signal SBA2 represents a three-dimensional image generated from supplementary information ZI, especially from virtual images. In this case, virtual images can be generated in such a way that they correspond to the real images of the stereo system 22. This may require the registration of the supplementary information ZI.

Fig. 3 shows a schematic flowchart of a further embodiment of a method according to the invention. It illustrates that the first stereo image output signal SBA1 is displayed on a first display device 20a (see also Fig. 6), and the further stereo image output signal SBA2 is displayed on a further display device 20b, which is different from the first display device 20a. The display devices 20a and 20b are each viewer-specific. This can mean that the correspondingly represented images are displayed by the display devices 20a and 20b.

Fig. 4 shows a schematic flowchart of a method according to a further embodiment of the invention. In contrast to the embodiment shown in Fig. 3, the stereo image output signals SBA1, SBA2 are output on a common display device 20, which can in particular be a 3D screen. Fig. 5 shows a schematic top view of an application scenario. Depicted are an operating microscope 10, a common display unit 20, and a tracking system 17, in particular an optical tracking system. The operating microscope 10, the common display unit 20, and the tracking system 17 can be components of a medical visualization system 4 (see Fig. 7). Also shown are an operating table 25 and two viewers 24a, 24b, who view an examination area 1 (see Fig. 7) on the operating table 25 from different positions. The first stereo image output signal SBA1 can be provided such that, when the common display unit 20 is activated with the first stereo image output signal SBA1, a three-dimensional image corresponding to a white-light visualization modality is output to a first viewer 24a. The additional stereo image output signal SBA2 can be provided such that, when the common display unit 20 is activated with the additional stereo image output signal SBA2, a three-dimensional image corresponding to a fluorescence contrast or polarization contrast visualization modality is output to another viewer 24b. Alternatively, an image generated from additional information ZI can also be output to the additional viewer 24b. For example, it is possible to display a virtual 3D image generated from additional information ZI to the additional viewer 24b, which on the one hand corresponds to the 3D image generated by the operating microscope 20 and on the other hand is displayed to the additional viewer 24b in a perspectively correct manner according to their pose. For example, the image represented by the additional stereo image output signal SBA2 can depict the examination area 1 with the same magnification but with an angular offset of 90° relative to the 3D image generated by the operating microscope 10.

Exemplary methods for displaying the images encoded by the various stereo image output signals SBA1, SBA2 for the different viewers 24a, 24b have already been explained previously.

The pose of viewers 24a, 24b can be recorded by the tracking system 10. Also shown is a device 16 for determining the direction of gaze, which can, for example, be arranged on the common display device 20. Such a device 16 can, for example, include at least one image capture device for The image of the viewer 24a, 24b, in particular the eyes, is included. Depending on the viewing direction, a partial area of the display surface of the common display device 20 can also be determined in which the viewer-specific 3D image is to be displayed.

Fig. 6 shows another exemplary application scenario. In contrast to the embodiment shown in Fig. 5, the medical visualization system 4 does not include a common display device 20 for displaying images in different visualization modalities VM1, VM2 for different viewers 24a, 24b. Instead, the medical visualization system 4 includes different viewer-specific display devices 20a, 20b, which can be configured, for example, as HMDs or as digital eyepieces. Images according to a first visualization modality VM1 are displayed on a first display device 20a, while images according to a further visualization modality VM2 are displayed on a further display device 20b.

Fig. 7 shows a schematic block diagram of a medical visualization system 4 and an examination area 1. Also shown are an instrument 2 and a tumor object 3, which are arranged within the detection range of an operating microscope 10 of the medical visualization system 4. The tumor object 3 can be a hidden object.

Optional elements of the medical visualization system 4 are shown here with dashed lines. The medical visualization system 4 comprises image acquisition devices 5, 5b for microscopic imaging of the examination area 1 (see Fig. 3). These image acquisition devices 5, 5b can be part of a surgical microscope 10, which may include an objective 19 with a lens. This surgical microscope 10 is designed as a stereo surgical microscope, whereby the image acquisition devices 5, 5b form a stereo system 22. Also shown is a storage device 6 in which additional information (ZI) and/or assignment information can be stored. The medical visualization system 4 further comprises an evaluation device 7, which can receive and evaluate an image generated by the image acquisition devices 5, 5b, which can be transmitted to the evaluation device 7 as image signals via an interface 21. The evaluation device 7 can be used, in particular, to process the image shown in Fig. The receiving step ES and the provisioning steps BS1 and BS2 are carried out as explained above.

The figure further shows that the medical visualization system 4 can include a device 8 for determining depth information. This device can, for example, be designed as a distance sensor or include one. Also shown is an interface 9 of the medical visualization system 4 for data transmission with other, especially higher-level, systems.

Furthermore, the medical visualization system can include 4:

• at least one white light lighting device 11 ,

• at least one infrared lighting device 12,

• at least one fluorescence illumination device 13 for excitation of fluorescence radiation,

• at least one fluorescence detection device 14 for detecting fluorescence radiation,

• at least one surround-view camera 15,

• at least one device 16 for detecting the gaze direction of a viewer,

• at least one tracking system 17,

• at least one input device 18 for operating or controlling the medical visualization system,

• at least one display device 20.

The elements of the medical visualization system 4 can be connected via data and/or signal technology.

Not shown are beam filters for providing excitation radiation with wavelengths from a broader spectrum, e.g. the spectrum of the white light illumination device, or for filtering radiation from a broader spectrum.

The environmental camera 15 can be part of another tracking system, which serves in particular for the optical determination of the pose of instruments within the detection range of the environmental camera 15. The pose determination can be monoscopic. In particular, the determination can also be marker-based. Images from the environmental camera 15 can, in particular, The data is evaluated for object recognition in order to identify a sub-area with respect to which the size of the foreground area V and/or background area H is then changed.

Reference symbol list

1 examination area

2 Instrument

3 Tumor object

4 medical visualization system

5 Image acquisition device of an operating microscope

5b further image acquisition device of the operating microscope

6 Storage setup

7 Evaluation unit

8 Device for determining depth information

9 Interface

10 Operating microscope

11 White light lighting device

12 Infrared lighting device

13 Fluorescence lighting device

14 Fluorescence detection device

15 Surround camera

16 Device for gaze direction detection

17 Tracking system

18 Input device

19 Lens

20, 20a, 20b Display device

21 Interface

22 Stereo system

23 Recording device

24a, 24b Viewer

25 operating table

ES reception step

BS1, BS2 Determination step

IS identification step

PBS Pose Determination Step

SB1 first stereo image signal

SBA1 first stereo image output signal

SBA2 additional stereo image output signal ```

Claims

Patent claims 1. A method for providing different stereo image output signals (SBA1, SBA2) by a medical visualization system (4), comprising the steps of: a. Receiving at least one first stereo image signal (SB1) generated by a stereo system (22) of an operating microscope (10) and representing a three-dimensional image of an examination area (1), b. Providing a first stereo image output signal (SBA1), c. Providing a further stereo image output signal (SBA2) representing a further three-dimensional image of the examination area (1), wherein the first stereo image output signal (SBA1) is provided for the complete display of an image according to a first visualization modality (VM1) and the further stereo image output signal (SBA2) is provided for the complete display of an image according to a further visualization modality (VM2), wherein the first and the further visualization modalities (VM1, VM2) are different and viewer-specific visualization modalities. 2. Method according to claim 1, characterized in that the first stereo image signal (SB1) is provided as the first stereo image output signal (SBA1). 3. Method according to claim 1, characterized in that the first stereo image signal (SB1) is processed to the first stereo image output signal (SBA1). 4. Method according to claim 3, characterized in that augmentation takes place through the processing. 5. Method according to one of the preceding claims, characterized in that a white light image, a fluorescence contrast image, a polarization contrast image, an image generated from additional information (SI) or an image combined from these modalities is displayed according to a visualization modality. 6. The method of claim 5, characterized in that the image displayed according to the visualization modality is a fluorescence contrast image, a A polarization contrast image is an image generated from preoperative supplemental information (ZI) or an image combined from at least two modalities of a set comprising these modalities and further comprising a white light image. 7. Method according to one of the preceding claims, characterized in that the further three-dimensional image is a white light image, a fluorescence contrast image, a polarization contrast image, an image generated from preoperative additional information or an image combined from at least two of these modalities, in particular by augmentation. 8. Method according to one of the preceding claims, characterized in that a pose and/or a viewing direction of a viewer (24a, 24b) is determined, wherein a viewer-specific stereo image output signal (SBA1, SBA2) is provided depending on the pose and/or the viewing direction. 9. Method according to claim 8, characterized in that a pose of the operating microscope (10) is determined and the viewer-specific stereo image output signal (SBA1, SBA2) is generated depending on the pose of the viewer (24a, 24b) relative to the operating microscope (10). 10. Method according to claim 8 or 9, characterized in that the provision of the viewer-specific stereo image output signal (SBA1, SBA2) comprises the generation of virtual images by virtual image acquisition devices which are an optical model of image acquisition devices (5, 5b) of the stereo system (22). 11. Method according to claim 10, characterized in that the pose of the image acquisition device(s) (5, 5b) of the stereo system (22) is taken into account when evaluating the model for generating the virtual images. 12. Method according to claim 10 or 11, characterized in that the virtual images depict three-dimensional additional information. 13. Method according to claim 12, characterized in that the three-dimensional additional information is preoperative additional information. 14. Method according to claim 12 or 13, characterized in that the three-dimensional additional information is generated from images of the image acquisition devices (5, 5b) of the stereo system (22). 15. Method according to claim 8 or 9, characterized in that the further stereo image output signal (SBA2) represents a further three-dimensional image that was generated at least partially by a further stereo system of the medical visualization system (4). 16. Method according to one of the preceding claims, characterized in that the various stereo image output signals (SBA1, SBA2) are provided by a common display device (20) for displaying the respective represented images. 17. Method according to claim 16, characterized in that the first stereo image output signal (SBA1) for displaying the represented images is provided with a first polarization encoding, wherein the further stereo image output signal (SBA2) for displaying the represented images is provided with a further polarization encoding, wherein the first and the further polarization encoding are different from each other. 18. Method according to claim 16, characterized in that the further stereo image output signal (SBA2) for displaying the represented images is provided at a time that is later in time than the time for displaying the images represented by the first stereo image output signal (SBA1). 19. Method according to claim 16, characterized in that the first stereo image output signal (SBA1) is provided for displaying the represented images with a first color encoding, wherein the further stereo image output signal (SBA2) is provided for displaying the represented images with a further color encoding, wherein the first and the further color encoding are distinct from each other. are different. 20. Method according to claim 16, characterized in that the first stereo image output signal (SBA1) for displaying the represented images is provided with a first angle encoding, wherein the further stereo image output signal (SBA2) for displaying the represented images is provided with a further angle encoding, wherein the first and the further angle encoding are different from each other. 21. Medical visualization system (4), comprising at least one interface (21) for receiving at least one first stereo image signal (SB1) generated by a stereo system (22) of an operating microscope (10) of the medical visualization system (4) and representing a three-dimensional image of an examination area (1), and at least one evaluation device (7), wherein the medical visualization system (4) is configured to perform a method comprising the steps according to any one of claims 1 to 20. 22. Medical visualization system according to claim 21, characterized in that the medical visualization system (4) comprises at least one device for identifying a viewer. 23. Medical visualization system according to claim 21, characterized in that image-based viewer identification takes place. 24. Medical visualization system according to one of claims 21 to 23, characterized in that the medical visualization system (4) comprises a further stereo system. 25. Medical visualization system according to one of claims 21 to 24, characterized in that the medical visualization system (4) comprises at least one device (16) for determining a pose and/or a viewing direction of a viewer. 26. Medical visualization system according to one of claims 21 to 25, characterized in that the medical visualization system (4) includes at least one 3D display device (20, 20a, 20b) comprising the display device (20, 20a, 20b) being configured as a digital eyepiece, as a 3D screen or as a head-mounted display. 27. A computer program product comprising a computer program, wherein the computer program includes software means for performing one, several, or all steps of the method according to any one of claims 1 to 20, when the computer program is executed by or in a computer or an automation system. ...