WO2024115724A1 - Method for marking information in digital images - Google Patents

Abstract

A method for marking information in digital images of an imaging device, in particular an endoscope system, with the imaging device comprising an optical system having an optical unit and an image sensor for recording a series of digital images of at least one observed region of an extended object and with at least some of the optical unit being movable, comprises the steps of marking information in a digital image, determining and storing position information for the marked information and augmenting the landmark or movement instruction corresponding to the mark in subsequent digital images.

Description

Method for marking information in digital images

Technical area

The present invention relates to a method and a device for marking information in digital images of an imaging device in which at least part of the optics is movable.

Background

Imaging devices are used in almost all areas of daily life today. Applications range from documentation, support of visual information, to replacing direct visual perception by humans. With regard to the latter, due to the constant development in the areas of optics, sensors, as well as data transmission and processing, imaging devices in a wide variety of applications can replace direct visual access by humans. Replace people. Examples of this are remote-controlled systems, such as robot applications, or the use of endoscope systems in which small cameras, often in combination with tools, are inserted into areas inaccessible to humans, thus enabling minimally invasive interventions to be carried out.

By using at least partially movable optics, different viewing areas of an extended object can be captured or viewed.

A fundamental problem with the application is the orientation in the viewing areas recorded by the movement of the imaging device. This affects not only spatial orientation in general but also the retrieval of certain positions or objects beyond the various viewing areas. This is particularly evident in the example of minimally invasive medical interventions. Here, there is currently no control over how many objects have been introduced into the patient's body and how many objects leave the body (again). When reducing tissue in particular, there is a latent risk that tissue residues will be left behind in the body after the operation. This can be due to a lack of orientation, i.e. the object simply cannot be found again, and/or due to a lack of documentation.

Summary of the invention

The object of the present invention is to at least partially overcome the disadvantages known in the prior art. The above object is achieved by a method according to the invention according to claim 1, a device according to claim 14 and a computer program according to claim 15. Preferred embodiments of the invention are the subject of the corresponding subclaims.

Accordingly, the present invention discloses a method for marking information in digital images of an imaging device, in particular an endoscope system, wherein the imaging device has an optical system with an optic and an image sensor for recording a series of digital images of at least one viewing area of an extended object and at least a part of the optic is movable. The method according to the invention comprises recording a first digital image of a first viewing area in a first spatial position of the movable part of the optic, in which at least one first piece of information is contained, marking the first piece of information by setting at least one first landmark, determining position information of the first landmark, generating a Landmark linked first unique identifier, storing the unique identifier together with the associated position information, recording a second digital image of a second viewing area in a second spatial position of the movable part of the optics, determining the position of the first landmark in relation to the second viewing area, augmenting the first landmark onto the second digital image if the first information is contained in the second viewing area, or augmenting at least one movement instruction for guiding the movable part of the optics to a spatial position thereof in which the first information is contained in a third viewing area onto the second digital image if the first information is not contained in the second viewing area. Augmenting onto a digital image can take place on a suitable display unit, such as a monitor.

An imaging device in the sense of the present invention can be any technical and/or electronic device that is suitable for recording an image of a viewing area, further processing it and/or forwarding it and displaying it on a screen, for example. Medical imaging devices in the sense of the present invention can be all types of endoscopes known to those skilled in the art, in particular dual endoscopes and stereo endoscopes. An endoscope is a mostly narrow and elongated imaging device that is suitable for insertion into a cavity or through a mostly small opening and for recording an image of a viewing area within the cavity and/or the area behind the small opening using at least one camera or at least one image sensor.

The mobility of at least part of the optics of the imaging device in the sense of the present invention can mean the movement of the entire imaging device or of part of the imaging device, in particular in relation to the object to be viewed, provided that the movement covers an at least partially different viewing area of an extended object. Accordingly, the mobility of at least part of the optics also includes a movement of the object to be viewed or of a part of the object to be viewed in relation to the optics, provided that this movement covers an at least partially different viewing area of an extended object. The mobility includes in particular the possibilities for moving endoscopic devices known to those skilled in the art, in particular in the context of minimally invasive surgical interventions. In particular, the entire endoscope or parts of the endoscope can be moved, such as bending the endoscope tip. The mobility in the sense of the present invention can also be a rotation.

A viewing area in the sense of the present invention describes the region, volume or area which is to be viewed by means of the imaging device and of which a corresponding digital image is to be generated and can correspond to the field of view of the imaging device. Such a viewing area in the context of medical imaging devices can be, for example, an organ, a bone, a part of a human or animal body or another area of interest for a corresponding viewing.

Information within the meaning of the present invention is any aspect of a viewing area that can be marked by a landmark, in particular positions, structures, objects, tissues and the like.

Augmentation in the sense of the present invention is an enrichment of the representation of digital images with any digital information. Accordingly, augmentation can represent the display of graphic elements with an information content, in particular a graphic element corresponding to a landmark or a graphic element corresponding to a movement instruction.

A movement instruction in the sense of the present invention is information on how the imaging device or at least a part of the imaging device is to be moved in order to reach a predetermined position, in particular a position in which information associated with a landmark is contained in a viewing area. Furthermore, the movement instructions can also comprise control commands for a motorized movement of the imaging device; if these control commands are sent to a motorized imaging device and executed on it, the imaging device is guided to a position in which the corresponding marked information is in the field of view of the imaging device and is thus contained in a viewing area.

In embodiments of the method according to the invention, any amount of information can be marked and any amount of landmarks can be set accordingly. Each landmark can then be saved under its own unique identifier. If multiple pieces of information are marked, the corresponding landmarks can be categorized and saved with the resulting assignment. In embodiments of the method according to the invention, active landmarks, for example landmarks that have not yet been deleted or provided with a processing order, can be displayed, wherein the augmentation optionally displays an image of the information belonging to the landmark in addition to the number of active landmarks.

In embodiments of the method according to the invention, the movable part of the optics can have inertial sensors, and the method can comprise determining position information of the spatial position of the movable part of the optics based on inertial sensor data. The position information of the spatial position of the movable part of the optics can furthermore also comprise the spatial position in relation to a reference point. In a preferred embodiment, the imaging device can be an endoscope system and the reference point can be the trocar. In this case, it can also optionally be determined how far the endoscope system is guided through the trocar and thus the absolute position of a part of the endoscope system, such as the tip of the endoscope, in relation to the reference point can be determined and used as position information in the method according to the invention. In this case, the trocar can comprise corresponding sensors.

In embodiments of the method according to the invention, the tracking of landmarks can be carried out by means of inertial sensors, electromagnetic tracking, optical tracking, by stereo reconstruction and optional trajectory calculation, by depth determination based on artificial intelligence methods, by flow calculation based on artificial intelligence methods and/or other computer vision algorithms, whereby combinations of different or all methods are also possible.

Inertial sensors in the sense of the present invention include in particular acceleration sensors and rotation rate sensors which together form an inertial measuring unit by means of which up to six possible kinematic degrees of freedom can be recorded. This makes it possible to determine the movement and, derived from this, also the position of the element connected to the sensors, in particular the movable part of the optics. In the case of a rigid endoscope, the sensors can be arranged on any section of the endoscope. If the endoscope is designed to be movable, the sensors are preferably arranged on the endoscope tip.

For electromagnetic tracking, the movable part of the optics can be equipped with one or more field sensors and thus marked. Electromagnetic fields are generated by a field generator, which are detected by the field sensor(s). can be measured. On the basis of several field measurements, the position and orientation of the field sensor in space and thus also the orientation and position of the marked, movable part of the optics can be determined by means of a corresponding measuring and computing unit. By using several field sensors, different and/or additional instruments or objects can be marked and spatially tracked accordingly and used to mark information with landmarks. In further embodiments, objects independent of the imaging device, such as in the case of medical interventions, objects in the operating room and/or the patient to be treated, can also be marked with field sensors. In the latter case, for example, a movement and/or repositioning of the patient can be detected and the determination of the position of the landmarks or the imaging device can be included, or taken into account in the movement arrangements.

In preferred embodiments of the use of electromagnetic tracking and/or inertial sensors, a hand-eye calibration can be carried out on the software side, which recognizes how the image coordinate system is located in relation to the tracker coordinate system or to the position determined based on the inertial sensors, or in order to be able to convert the tracking coordinates or inertial sensor coordinates into image coordinates. For the calibration, at least one image of a known object can be taken. Its position can then be determined in the image using segmentation and registration methods, from which the calibration transformation can then be determined.

For stereo reconstructions, it can be calculated which pixel in the left image shows the same object point as the pixel in the right image. This requires camera calibration. Based on the stereo reconstruction, a 3D coordinate can be calculated for each pixel.

In preferred embodiments, the information resulting from the stereo reconstruction can be used in combination with the above-mentioned tracking (optical, electromagnetic, inertial sensor-based) to determine the exact 3D point of the marked information and thus to find it again more easily and with greater accuracy. Accordingly, the position of the landmark, and thus the information marked with the landmark, can be determined and documented more precisely.

In preferred embodiments, the stereo reconstruction can generate a 3D point cloud, optionally plus associated color information, wherein the Stereo reconstruction can preferably be carried out in real time. By fusing two point clouds that follow one another in time, an estimate of the camera movement can be made, which can then generate a trajectory that can be used to find information marked by landmarks or to augment corresponding information on the basis of landmarks. Accordingly, the tracking methods described above can be expanded or even replaced by the stereo reconstruction extended by trajectory calculation.

In preferred embodiments, similar to stereo reconstruction, depth can also be determined or estimated in monoscopic images. By applying appropriate mathematical methods, preferably based on artificial intelligence, the stereo reconstructions can be replaced or the application can be extended to monoscopic cameras. Accordingly, the position of landmarks can be determined and documented more precisely by determining the depth.

Optical tracking in the sense of the present invention is in particular the location of optical markers using a camera. Here, corresponding markings (markers) can be arranged on the part of the imaging device to be tracked, whereby a line of sight between the camera and the located marker is required. When using rigid endoscopes, corresponding markers can be arranged on sections of the endoscope that are not introduced into the work area, such as the inside of the body, and can be recorded by an external camera, such as a room camera. Since the geometry of the endoscope is known, the position of the camera and thus the field of view of the camera can be determined by tracking the corresponding markers. In preferred embodiments, other objects, in particular a patient, can also be marked and the corresponding position information can be included.

Flow calculations in the sense of the present invention can be based on computer vision algorithms known to those skilled in the art or on corresponding algorithms of artificial intelligence. The calculation of an optical flow in the image can be used to track landmarks over time and as a function of movement (in relation to the object to be viewed to the imaging device, i.e. with regard to the change in the viewing area). In preferred embodiments, the flow calculation can be used to increase the calculation speed and accuracy and/or to Adaptation to available computing power can be achieved using resolution pyramids.

In one embodiment of the method according to the invention, digital images of viewing areas can be assigned respective position information of the spatial position of the movable part of the optics. For example, a position of a digital image of a viewing area can be assigned to an absolute position in relation to a reference point.

In embodiments of the method according to the invention, the position information of the first landmark can be determined in relation to the spatial position of the movable part of the optics and/or correspond to the pixel position of the first landmark in the first digital image. In preferred embodiments, the position information of the set landmarks can include both the spatial position of the movable part of the optics, which corresponds to a viewing area in which the corresponding information is contained, and the pixel position of the corresponding landmark in the respective digital image of the respective viewing area. In this case, the pixel information can be based on stereo reconstruction and/or corresponding algorithmic depth determination as described above. The two position components, i.e. the spatial position and the pixel position, can be stored separately from one another and/or in combination with one another. In this way, absolute position values of the landmarks in relation to a reference point can be generated and also stored.

In embodiments of the method according to the invention, the position of the first landmark in relation to the second viewing area can be determined by calculating the optical flow based on a series of digital images and/or by tracking the position of the movable part of the optics. In preferred embodiments, the flow calculation can be carried out using resolution pyramids to increase the computing speed and accuracy and/or to adapt to the available computing power.

In further embodiments of the method according to the invention, the storage of the unique identifier together with the associated position information can also contain a description and/or categorization of the marked information. The description and/or categorization can be carried out via direct user inputs, for example via user inputs via corresponding interfaces. In preferred embodiments, the description and/or categorization can be carried out on a computer-assisted Object recognition, in particular based on object recognition supported by artificial intelligence. Here, after setting a landmark to mark information, the user can be shown an example of a description and/or categorization, which the user can then reject or confirm by making suitable entries. In preferred embodiments, suggestions for information to be marked can also be made to the user by means of object recognition, which the user can then reject or accept. By accepting, a corresponding landmark is set and the method according to the invention is continued.

In preferred embodiments, an artificial intelligence algorithm can be used to perform a segmentation of a digital image, in particular an endoscopic image. Through segmentation, the entirety of an object provided with a landmark by the user input can be recognized and used as a basis for further method steps. Accordingly, after selecting the landmark, the name of the corresponding object can be automatically suggested to the user. For this purpose, the predicted class of the pixel that was selected as a landmark is used. The landmark can then be associated with and stored with corresponding categorization and/or object information. Alternatively or in addition to this, the segmentation can also be used to mark an object in its entirety so that an area landmark can be created. Accordingly, augmenting movement arrangements can be determined so that the entire object is guided into the field of view of the imaging device. Combinations of point landmarks that correspond to the actual user input and area landmarks that correspond to the segmentation result are also possible. In preferred embodiments, segmentation is carried out using a U-Net structure of a convolutional neural network (CNN). This object detection allows objects in an image to be classified and their position to be determined at the same time.

In further embodiments of the method according to the invention, the at least one landmark can be deleted and/or the associated augmentation can be prevented.

In a further embodiment of the method according to the invention, the setting, deletion and/or prevention of the augmentation of the at least one landmark can be carried out by recognizing user inputs and/or by automatic object recognition. Suitable user inputs in the sense of the present invention can be made via keyboard, foot switch, joystick, voice recognition or gesture movement with an instrument in the field of view of a camera, in particular in the field of view of the imaging device. The selection of information for marking or the selection of already marked information can be related to the center of the viewing area or can be freely selected in the viewing area, wherein in preferred embodiments a marking element, such as a crosshair, is augmented in the image. In preferred embodiments, information or a landmark located in the area of the marking element can be set by the user input or selected for further processing such as deletion or categorization. In further preferred embodiments, in particular in the context of minimally invasive, endoscope-based interventions, instruments and/or tools introduced into the work area, in particular their tips, can also be used as the starting point for gesture control and the imaging device, such as the endoscope camera, can be used as a means of capturing the gesture.

In embodiments of the method according to the invention, the user inputs may include inputs via user interfaces, acoustic signals and/or direct gesture movements with an instrument in the viewing area or indirect gesture movements by moving the movable part of the optics.

In embodiments of the method according to the invention, landmarks can be automatically deleted, suggested for deletion, deactivated or suggested for deactivation if the associated object has been removed from the work area. For example, in the context of minimally invasive medical interventions, landmarks that are associated with objects to be removed can be treated accordingly if the respective objects have been removed from the work area. Corresponding recognition can be carried out via corresponding computer-based object recognition.

Active landmarks in the sense of the present invention are information markings that are provided with at least one further instruction that describes what is to be done with the corresponding marked information. For example, an object can be marked so that it is to be removed from the work area. Accordingly, deactivated landmarks are those for which the further instruction has already been carried out. Furthermore, landmarks can be provided with validity parameters, such as a validity period, object match or the like. In the event of deviations from the validity parameters, corresponding Information can be displayed in the database and/or augmented in digital images.

In embodiments of the inventive method, digital images of corresponding viewing areas that are separated in time can also be compared with one another. If significant changes, i.e. changes that are above a predefined threshold value, are detected, corresponding information, such as a warning, can be augmented in digital images.

In embodiments of the method according to the invention, digital images of different viewing areas can be combined to form a composite image by means of "stitching", i.e. creating a large digital image from various smaller digital images that show sections of the extended object. The digital images can be combined using overlapping areas of the individual digital images and/or based on the position or movement of the image display device. In preferred embodiments, viewing areas of the extended object in positions of the movable part of the optics in which no or inadequate digital images were recorded can be replaced in the composite image by placeholder images in the corresponding position. These placeholder images can be identified as such in the composite image, for example by displaying no content, a neutral color tone or the like. Augmentations can also be displayed in placeholder images if necessary and displayed across image boundaries.

The present invention is further directed to a device comprising an imaging device, in particular an endoscope system, wherein the imaging device has an optical system with an optics and an image sensor for recording a series of digital images of at least one viewing area of an extended object and at least a part of the optics is movable, and a control unit, wherein the device is configured to carry out the methods described above.

In certain embodiments, the controller is a computer system configured to perform one or more steps of one or more methods described or illustrated herein. In certain embodiments, one or more computer systems provide functionality described or illustrated herein. In certain embodiments, an execution on one or more computer systems executing software performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. The controller may include a storage device for data or instructions. For example, and not by way of limitation, the storage device may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. The storage device may include removable or non-removable (or fixed) media, where appropriate. The storage device may be located internally or externally of the computer system. In certain embodiments, the storage device is non-volatile solid state memory. In certain embodiments, the storage device includes read only memory (ROM). The storage device may include one or more storage controllers that facilitate communication between the processor and the storage device. Further, the storage device may include one or more storage devices. The controller may further be directly or indirectly connected to a display device.

The present invention is further directed to a computer program, wherein the computer program comprises instructions which, when the program is executed by a computer, cause the computer to carry out the method described above.

The scope of this disclosure includes all changes, substitutions, variations, alterations, and modifications to the embodiments described or illustrated herein that one of ordinary skill in the art would understand. The scope of this disclosure is not limited to the embodiments described or illustrated herein. In addition, although this disclosure describes and illustrates respective embodiments herein as comprising particular components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or permutation of any components, elements, features, functions, operations, or steps described or illustrated anywhere herein that one of ordinary skill in the art would understand. A reference in the appended claims to a method or apparatus or component of an apparatus or system being adapted, arranged, capable, configured, enabled, operable, or ready to perform a particular function further includes that apparatus, system, or component, whether or not it or that particular function is enabled, is turned on or enabled as long as that device, system, or component is adapted, arranged, capable, configured, enabled, operable, or ready to operate. In addition, although this disclosure describes or illustrates certain embodiments as providing certain advantages, certain embodiments may provide none, some, or all of these advantages.

Short description of the figures

Fig. 1 shows schematically the sequence of the method according to the invention.

Fig. 2 shows schematically a memory entry of the method according to the invention.

Fig. 3 shows the method for augmentation according to an embodiment.

The accompanying drawings illustrate exemplary embodiments of the invention and serve to explain the principles of the invention by way of example

Detailed description

The present invention will now be described in more detail with reference to the accompanying drawings. The method may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. It should be noted that the figures are intended to illustrate the general features of the methods utilized in certain embodiments. However, these figures may not precisely depict the exact structure or performance of a given embodiment. In addition, like reference numerals in the figures designate corresponding parts throughout the different views or embodiments.

The present invention also aims to enable a user, for example a doctor performing a minimally invasive operation, to set intraoperative landmarks. These landmarks mark information, such as certain objects that the doctor wants to remove during the operation. For example, a list of landmarks that the doctor wants to remove is stored. The position of the landmarks is maintained by optical tracking and/or 3D positioning according to the camera and object movement, even if the object leaves the camera's field of view. Figure 1 shows a schematic representation of the method according to the invention. According to the invention, digital images of a viewing area of an extended object are recorded using an imaging device, in particular a medical imaging device such as an endoscope system. By moving at least parts of the optics of the imaging device, different viewing areas can be recorded and thus different areas of the extended object can be recorded in the form of digital images. Digital images can be recorded in the form of irregularly recorded individual images or continuously, in particular as a video stream. At least one digital image is displayed to a user of the method, such as a doctor during a minimally invasive operation, via a suitable display device. Using suitable user inputs, as described in detail above, the user of the method can mark information that is contained in a (current) viewing area and thus in a (currently) displayed digital image. When information is marked and a landmark is set accordingly, the position of the landmark is determined via a control unit. This position determination can be carried out based on the pixel information in the digital image, based on absolute or relative position information of the imaging device, for example based on inertial sensor data, possibly in relation to a reference point. The position information is stored together with a (dynamically) generated, unique identifier assigned to the landmark in a suitable storage form, in particular a database.

During the application of the method, the optics of the imaging device or, depending on the system used, at least the movable part of the optics is moved by the user, whereby a digital image is recorded from a different position of the imaging device and thus at least a partially different viewing area. Based on the available position information of the imaging device and/or via optical flow analyses of successive digital images, the position of the landmark is determined in relation to the current, or currently displayed, viewing area.

If the marked information is contained in the current viewing area, i.e. in the field of view of the imaging device in its current position, information associated with the landmark is augmented onto the image. Alternatively, if the marked information is not contained in the current viewing area, i.e. in the field of view of the imaging device in its current position, one of the movement instructions is augmented onto the image. augmented. This movement instruction gives the user of the procedure the necessary information on how to move the imaging device in order to bring the marked information back into the field of view of the imaging device.

Figure 2 shows schematically which information assigned to a landmark can be stored in which form. Here, Figure 2a shows a data set as it can be generated in the process described in connection with Figure 3. Different landmarks are assigned unique identifiers (landmarks D) and these are shown with the corresponding coordinates in a digital image (image coordinates) of a viewing area, which in turn is described via the position of the imaging device (inertial sensor data). As shown in Figure 2b, further information can be stored. For example, a unique identifier assigned to the landmark, position information, category assignment, associated task and a status can be stored. Basically, the position information depends on the system-specific, available information, for example whether a system with inertial sensors is used and is shown purely as an example in the figure. Likewise, the associated task and the status depend on the respective application and are shown purely as an example in Figure 2.

Figure 3 shows a specific embodiment of the present invention in which a user of an endoscopic system, for example a surgeon, is enabled to set landmarks intraoperatively. These landmarks mark, for example, objects that the surgeon wants to remove during the operation. A list of landmarks that the surgeon wants to remove is stored. The position of the landmarks is tracked by optical tracking and/or 3D positioning according to the camera and object movement, even if the object leaves the camera's field of view.

In this embodiment, based on a minimally invasive medical procedure, a combination of inertial sensor data and optical tracking is used to track a set landmark. This example is based on typical endoscopic surgical equipment including an endoscope and camera, assuming a fixed trocar point and a 0° optic.

In a current live image shown in Figure 3a, the user sets a landmark at the center of the image. The landmark can be set as described in detail above. Internally, the inertial sensor position and the current live Image saved with the coordinate of the image center. The saved live image is labeled "0", the coordinate of the image center is (x_0, y_0), the inertial sensor data is R_0.

The subsequent live image "1" shown in Figure 3b is calculated with the saved image "0" using an optical flow algorithm. This calculates the shift of the image center and determines a new position (x_1, y_1) of the set landmark. This new position is displayed in the image. Analogously, an optical flow calculation of the images N+1 and N takes place for all subsequent frames, so that the coordinates (x_N+1, y_N+1) are determined from the previous coordinates (x_N, y_N) and the calculated flow.

After a certain number of frames N=M, it is no longer possible to calculate the optical flow. This can occur, for example, if the object belonging to the landmark leaves the image area or is obscured by further movement of the imaging device. If the endoscope moves too quickly, the flow cannot be determined. In this case, the inertial sensor data from the camera is used. As a rigid body, the camera basically has 6 degrees of freedom (3 translational and 3 rotational degrees of freedom). Assuming a fixed trocar point, only 3 rotational degrees of freedom and one translational degree of freedom (in-out) are variable. The inertial sensor measures the 3 rotational degrees of freedom. Since the set landmark is in the center of the image and a 0° optic is described, the coordinate of the landmark does not change during the selection process with a (fictitious) movement along the in-out axis. In other words, the object belonging to the landmark lies in the extension of the endoscope.

Accordingly, by calculating the current inertial sensor data R_M and the initial inertial sensor data R_0, the direction (but not the distance) in which the set landmark lies relative to the image center can be calculated. This direction is displayed in the image as shown in Figure 3c.

In each subsequent frame M+1, M+2, ..., the direction of the landmark is calculated and displayed by calculating the inertial sensor data. On the other hand, the current image is compared with image 0 and/or image N using an optical flow method (image 1 to image N-1 can be discarded if necessary, for example to save storage space) and an attempt is made to calculate the landmark coordinate. If this is successful, If this is displayed, the procedure is as per description b). If this is not successful, the procedure is as per description d).

As explained in detail above, the user has the option of deleting the set landmark or, if necessary, setting additional landmarks by means of gesture recognition, pressing a button, a foot pedal, etc.

Gesture control is a good option for setting or deleting landmarks. The doctor moves the inserted instruments so that the instrument tips follow a predefined movement trajectory. The endoscope camera captures the image. An artificial intelligence algorithm recognizes the instrument tip (a detection model that predicts a bounding box in the image is particularly suitable here) and records the movement of the instrument tip over a number of consecutive frames. If the movement corresponds to the predefined trajectory, the setting or deletion of the landmark is activated, for example. Defined trajectories that do not usually occur in a normal laparoscopic procedure are particularly suitable. For example, a "figure 8" movement can be defined to set a landmark and a circular movement can be used to delete the set landmark.

Opening and closing the jaw can also support gesture control, with a Kl model able to perform a binary classification. It is also conceivable that several instruments perform a gesture together.

The scope of this disclosure includes all changes, substitutions, variations, alterations, and modifications to the embodiments described or illustrated herein that one of ordinary skill in the art would understand. The scope of this disclosure is not limited to the embodiments described or illustrated herein. In addition, although this disclosure describes and illustrates respective embodiments herein as comprising particular components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or permutation of any components, elements, features, functions, operations, or steps described or illustrated anywhere herein that one of ordinary skill in the art would understand. A reference in the appended claims to a method or apparatus or component of an apparatus or system being adapted, arranged, capable, configured, enabled, operable, or ready to perform a particular function further includes that apparatus, system, or component, regardless of whether it or that particular function is activated, enabled, or enabled, as long as that device, system, or component is adapted, arranged, capable, configured, enabled, operable, or ready to operate. In addition, although this disclosure describes or illustrates certain embodiments as providing certain advantages, certain embodiments may provide none, some, or all of those advantages.

A method for marking information in digital images of an imaging device, in particular an endoscope system, wherein the imaging device has an optical system with an optic and an image sensor for recording a series of digital images of at least one viewing area of an extended object and at least a part of the optic is movable, comprising marking information in a digital image, determining and storing position information of the marked information and augmenting the landmark or movement instruction corresponding to the marking on subsequent digital images.

Claims

Expectations 1. A method for marking information in digital images of an imaging device, in particular an endoscope system, wherein the imaging device has an optical system with an optics and an image sensor for recording a series of digital images of at least one viewing area of an extended object and at least a part of the optics is movable, comprising: - Recording a first digital image of a first viewing area in a first spatial position of the movable part of the optics, in which at least one first piece of information is contained; Marking the first information by setting at least a first landmark; Determining position information of the first landmark; Generating a first unique identifier associated with the first landmark; Storing the unique identifier together with the associated position information; - taking a second digital image of a second viewing area in a second spatial position of the movable part of the optics; determining the position of the first landmark in relation to the second viewing area; - augmenting the first landmark onto the second digital image if the first information is contained in the second viewing area, or - Augmenting at least one movement instruction for guiding the movable part of the optics to a spatial position thereof in which the first information is contained in a third viewing area, onto the second digital image if the first information is not contained in the second viewing area. 2. The method of claim 1, wherein the movable part of the optics comprises inertial sensors and the method comprises determining position information of the spatial position of the movable part of the optics based on inertial sensor data. The method of claim 2, wherein determining the position information of the spatial position of the movable part of the optics also comprises the spatial position with respect to a reference point. The method of claim 3, wherein the imaging device is an endoscope system and the reference point is the trocar. The method of any of claims 2 to 4, wherein digital images of viewing areas are assigned respective position information of the spatial position of the movable part of the optics. The method of any of the preceding claims, wherein the position information of the first landmark is determined with respect to the spatial position of the movable part of the optics and/or corresponds to the pixel position of the first landmark in the first digital image. The method of any of the preceding claims, wherein determining the position of the first landmark with respect to the second viewing area is done via the calculation of the optical flow based on a series of digital images and/or via the tracking of the position of the movable part of the optics. Method according to one of the preceding claims, wherein the storage of the unique identifier together with the associated position information also contains a description and/or categorization of the marked information. Method according to one of the preceding claims, wherein the at least one landmark can be deleted and/or the associated augmentation can be prevented. Method according to one of the preceding claims, wherein the setting, deletion and/or prevention of the augmentation of the at least one landmark takes place by recognizing user inputs and/or by automatic object recognition. 11. The method of claim 10, wherein the user inputs include inputs via user interfaces, acoustic signals and/or direct gesture movements with an instrument in the viewing area or indirect gesture movements by moving the movable part of the optics. 12. Method according to one of the preceding claims, wherein digital images of different viewing areas are combined by means of stitching to form a composite image. 13. The method according to claim 12, wherein viewing areas of the extended object in positions of the movable part of the optics in which no or inadequate digital images were recorded are replaced in the composite image by placeholder images in the corresponding position. 14. Device comprising an imaging device, in particular an endoscope system, wherein the imaging device has an optical system with an optics and an image sensor for recording a series of digital images of at least one viewing area of an extended object and at least a part of the optics is movable, and a control unit, wherein the device is configured to carry out the method according to one of the preceding claims. 15. Computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to one of the preceding claims.