WO2025051775A1 - Method for detecting instruments in image data, and device for analysing image data - Google Patents

Abstract

The present invention relates to a method for detecting surgical instruments in medical image data, in particular for analysis and documentation purposes, said method comprising the steps of: - acquiring (101) image data (3a), preferably provided by means of an endoscope, in particular an endoscopic live image (40), - visually detecting (102) a surgical event (32) in the acquired image data, and - visually detecting (103a) at least one instrument (30a, 30b) in the acquired image data, wherein the method comprises detecting and assigning (103b) an instrument classification to the detected instrument (30a, 30b) based on the detection of the surgical event (32) and a determination of the distance between the detected surgical event (32) and an instrument end (31a, 31b) of the at least one instrument (30a, 30b).

Description

Method for detecting instruments in image data and device for analyzing image data

The present invention relates to a method for the visual recognition and classification of instruments in medical image data, in particular in an endoscopic live image.

Optical visualization systems such as endoscopes or microscopes for medical interventions are well known and enable the representation of a scene and/or a work area within a human body, for example, during a minimally invasive procedure. Such visualization systems output captured image or video data on suitable screens and are typically connected to an image recording device or system to record the captured image or video data, as well as any other device- and/or patient-specific data, and/or analyze them for further processing and/or evaluation.

Furthermore, methods and devices are known by which the image or video data acquired by an endoscope are processed for analysis and/or documentation purposes. For example, a visual recording of the surgical instruments used or surgical events occurring during the procedure can be performed. The resulting data and acquired information can then be used to monitor the course of treatment, to optimize future surgical interventions of a similar nature, or to train AI applications.

US 2019/0125455 A1 discloses a method for adjusting the operation of a surgical instrument using machine learning. The method includes collecting data during surgical procedures with surgical instruments, analyzing the collected data to determine a suitable operating setting of the surgical instrument, and adjusting operating parameters of the surgical instrument to improve the operation of the surgical instrument. US2018/0271603 A1 discloses a control system for surgical tools comprising an instrument management system, a database, a display means for displaying at least one surgical event, for determining the identity and 3D position of detected objects and instruments, and at least one computing unit for identifying a necessary medical application or action based on a surgical event.

EP 3 505 113 A1 discloses a surgical system for wireless communication between surgical devices, comprising a first surgical device with a control unit configured to be informed of situational events occurring in the vicinity of the first surgical device, according to data received from a database, a patient monitoring device or another surgical device coupled thereto.

US 2021/142487 A1 discloses a system for evaluating surgical scenes based on computer-assisted visual image analysis. The system includes the visual recognition of instruments and events and the determination and tracking of a position in the acquired image.

US 2019/0200985 A1 discloses a surgical instrument with a control circuit for monitoring a distance of the instrument from a tumor tissue, the instrument having an end effector for grasping tissue and a staple cartridge comprising staples insertable into the tissue and a sensor. The surgical instrument comprises a control circuit coupled to the sensor, the control circuit configured to receive the sensor signal and analyze the proximity of the sensor to cancerous tissue based on the sensor signal.

Based on the known prior art, it is the object of the present invention to provide a method and a device for the visual analysis of medical image data, which enables an optimized recognition and classification of instruments in the image data. This object is achieved by the detection method and the analysis device according to the independent claims. The dependent claims describe advantageous developments of the present invention. All combinations of at least two of the features disclosed in the claims, the description, and/or the figures fall within the scope of the invention.

In a first aspect, the present invention relates to a method for detecting surgical instruments in medical image data, in particular for analysis and documentation purposes, comprising the steps:

-Capturing image data, preferably provided by an endoscope, in particular an endoscopic live image,

-visual detection of an operative event in the acquired image data, and -visual detection of at least one instrument in the acquired image data, wherein the method comprises detection and assignment of an instrument classification to the acquired instrument based on the detection of the operative event and a distance determination between the detected operative event and an instrument end of the at least one instrument.

In this context, an instrument end is understood to mean, in particular, the distal end or a distal end region of an instrument. The instrument end is, in particular, a part of the instrument that can be visually detected by an image capture device that captures the image data. Alternatively or additionally, the instrument end can also be defined as a region of interest (ROI), i.e., a region in the image in which the instrument end is recognized or suspected. The instrument end can be visually detected and determined within the image, in particular by markings on the instrument end or by segmentation.

The present invention enables the preferably automatic determination and allocation of at least one attribute and in particular an instrument classification of a surgical instrument captured in the image data based on evaluation of the captured image data. In this context, visual acquisition and visual recognition are understood to mean software-based acquisition or recognition using known software algorithms and/or trained artificial intelligence (K1), which performs a visual analysis or evaluation of the acquired image data. This involves, for example, acquisition of contours and/or structures of an instrument and/or tissue and/or analysis of further image elements and/or image data, based on which an operative event or an instrument can be recognized, e.g., by comparison with previously stored data. Visual acquisition and recognition differs in particular from sensor- or antenna-based acquisition or recognition of objects in image data or a video image.

An operative event is understood to be an event that can be visually detected in image data, in particular a visually recognizable effect of an instrument application.

Image data is understood here to mean image and/or video data, and in particular image data provided by a video endoscope, exoscope or medical microscope.

In contrast to the prior art, the invention does not involve visual recognition of an attribute of a surgical instrument based on the image data of the captured instrument alone, but rather recognition and assignment of a classification or classification of the surgical instrument based on a distance determination between a respective instrument end and a detected surgical event. This enables an optimized and, in particular, more precise classification of the respective instrument. For example, different surgical instruments that have a similar contour/surface in image data and are therefore difficult or impossible to distinguish when analyzing the image data of the instrument alone can be more precisely identified or classified using an associated surgical event. The method preferably comprises the further step of storing and/or outputting the determined data, in particular the recognized and assigned instrument classification, preferably together with the acquired image data. For this purpose, an output means, such as a display, can advantageously be provided in which, in addition to the acquired image data such as a live endoscopic image, the determined instrument classification and/or the further acquired and/or determined data are presented to a user. Furthermore, the image data can preferably be stored together with the assigned data in a provided internal or external data storage device. The storage device can, for example, be a buffer connected to an image acquisition device, which transmits the data to a provided database at regular or irregular intervals, in particular for documentation and/or evaluation purposes.

In a preferred embodiment, when two or more instruments or associated instrument ends are visually detected in the captured image data, the instrument classification is recognized and assigned to the instrument whose instrument end has the shortest determined distance to the captured surgical event. Preferably, after a surgical event has been detected and recognized and after at least two instrument ends have been detected in the image data, the respective distance between the instrument ends and the surgical event is first determined. The instrument end that is detected closest to the surgical event is then assigned as the instrument classification described above. This is advantageously also done when at least two or more detected surgical events are detected, for which a distance determination to the respective surgical instruments visually detected in the image is then carried out for each surgical event. This allows a more precise classification of the instrument assigned to a surgical event when more than two surgical instruments are detected in the image data.

The distance determination between the operative event and the end of the detected instrument is preferably based on a distance or range measurement that can be directly visually detected from the image data. This can advantageously be a This can be a known software-based distance measurement between two image points, in particular between the respective centers of the instrument end and the recorded event, and/or an image analysis based on an additional depth measurement, which determines the distance between the image capture device and the scene, in particular the operative event, for individual image points. For example, the depth can be determined using a sensor such as a time-of-flight sensor. Alternatively, a depth can be assigned to individual image points in the scene using triangulation, for example using a stereo instrument. A depth map of the scene can be determined, in particular continuously, and this information can be used to determine the spatial distance between real points in the scene, in particular the instrument end and the operative event. The distance is then determined in three dimensions. This allows for efficient analysis of the image data with sufficient accuracy, without the need for complex additional devices such as sensors and/or antennas on the respective instruments.

Alternatively or additionally, a so-called region of interest (ROI) can be defined for the detected instrument end and the surgical event in the image. For example, a rectangle or other shape enclosing the instrument end and the ROI can be defined. The distance can then be a distance between specific points of these rectangles or shapes, for example, between the centers.

The step of determining the distance between the operative event and the end of the recorded instrument preferably occurs immediately after visual detection of the operative event and/or the detection of the instrument. This allows for a more precise association between a particular instrument and the operative event. In particular, this allows for a measurement to be taken as close as possible to the location of the operative event, thus avoiding any expansion or spatial change, or migration of the operative event in the image data, which could negatively affect the measurement.

Detection of an operative event preferably comprises event classification from the analyzed image data into one or more of the following event classifications: smoke development, a color or structural tissue change, a developing and/or changing tissue opening, a Change in the water content of the acquired tissue, blood development in the tissue, suture placement in the tissue, introduction of foreign material into the tissue, in particular introduction of metal, for example a clip, and/or spray mist or gas bubbles.

In the visual detection of a surgical event, the surgical event is preferably visually recorded and recognized using computer- or software-assisted image evaluation or analysis, in particular using known image evaluation algorithms. Advantageously, this involves evaluating the RGB image values, evaluating multispectral and/or hyperspectral image parameters, identifying contours/structures of the detected objects in the image, and/or evaluating using known algorithms for object recognition in image data, in particular comparing them with previously acquired and/or stored image data of known and thus predefined surgical events. For example, water content can be determined by evaluating hyperspectral or multispectral image parameters. Detection can also be performed by or with the support of artificial intelligence, for example, using a neural network, or based on machine learning. These are then trained to recognize these events based on observed surgical events.

In visual detection, an evaluation can also be performed over time or across multiple consecutive frames of a video image or image data. For example, a tissue change can be detected by evaluating the RGB values over time, particularly by detecting color changes, such as a change in tissue color from pink to black to detect tissue damage, or a change in image color information from white or gray to detect smoke.

The step of detecting an operative event may further preferably include assigning an event intensity and/or another event attribute. This can also be performed using the aforementioned, known computer- or software-assisted image evaluation or analysis. For example, a visual detection and subsequent assignment of a smoke intensity, a bleeding intensity and/or a tissue opening size. For this purpose, a respective intensity or size factor can be assigned to the event. For example, depending on the extent or density of smoke development in the tissue that can be visually detected in the image, a lower or higher intensity value can be determined and assigned to the surgical event.

In a preferred embodiment, the recognition and assignment of an instrument classification comprises a recognition and assignment of an instrument type and/or a further instrument attribute, in particular comprising the classification: HF instrument, non-HF instrument, forceps, scissors, suturing instrument, stapler, biopsy instrument, punch, clip setter and/or an ultrasound device.

According to the invention, the recognition and assignment of an instrument classification of the detected instrument is based on a detected surgical event in addition to the distance determination described above. The detected event comprises information or data on a detected event classification, a determined event intensity, and/or data on another event attribute. For example, by detecting smoke development on the tissue that can be detected in the image data and which was or is assigned to a detected instrument end by the distance determination, the corresponding instrument can be classified as a high-frequency (HF) instrument for the application of high-frequency current, for example for coagulation or cutting. This makes it possible to differentiate between HF instruments such as HF grasping forceps, which visually do not or only barely differ from grasping forceps without HF functionality.

In this case, the recognition and assignment of an instrument classification can be carried out based on the captured visual image data of the instrument, as well as on the evaluation of device and/or intervention data stored in a database for predefined instruments and/or instrument data stored for predefined interventions or predefined events. Thus, the recognition and assignment of an instrument classification can include the step of querying data stored and/or retrievable in an internal or external data storage device, which Assign surgical events to a particular instrument or instrument property. For example, the assignment can be stored in a memory that the surgical event of smoke development in tissue can be triggered by a surgical instrument with HF functionality, or that a resulting tissue opening or division can be triggered by scissors.

The method preferably includes the further step of determining a recognition and assignment of a type of medical treatment as an attribute for the analyzed image data. Thus, by recognizing the respective surgical event and/or the recognized instrument classification, an existing medical treatment or treatment type can be inferred, which can be assigned as an attribute to the acquired image data.

Furthermore, the method can comprise the step of reading out and/or evaluating device and/or instrument data which are used for the respective procedure and which are, for example, connected to a device for carrying out the procedure. For example, electronically readable device data can be analyzed and used for the further classification of an instrument, for the recognition of a surgical procedure and/or for the recognition of a type of medical treatment or an indication. If, for example, an HD rhinoscope is recognized as being connected to a corresponding device, it can be concluded that this is a diagnostic procedure because this endoscope does not have a working channel. If, for example, a cystoscope is recognized as being connected to a corresponding device, it can be assumed that this is an endourological procedure.

Based on this, it is also possible to conclude, prior to image analysis, whether a specific classification and/or surgical event is likely or unlikely. For example, if an endourological procedure is carried out underwater, then HF current is applied, rather than smoke, but bubbles are produced. Furthermore, there are purely diagnostic procedures, for example, in which HF instruments are not used. This has The advantage is that a respective image evaluation algorithm can be designed more efficiently, since only a limited number of instruments or instrument classifications are relevant for the respective procedure and not all of them.

Furthermore, the method may include the step of temporally recording each use of a classified surgical instrument and/or a surgical event. This may, in particular, include the respective time and duration of use or the duration of a detected event. The corresponding data may be assigned to the image data and used for further evaluation, in particular for classifying the instrument and/or detecting an event.

The method preferably further comprises calculating and assigning a validation value for the respective recognized surgical instrument, which is determined at least based on a distance value of the respective distance determination between a recognized surgical event and an instrument end. The validation value can serve to provide appropriate plausibility of the recognized instrument classification for the respective surgical instrument. Preferably, a higher validation value is assigned for a relatively short determined distance between the surgical event and the instrument end than for a relatively greater distance. This is based on the fact that for a directly nearby surgical event, the probability that the surgical instrument is responsible for the surgical event is higher than for a relatively greater distance between the instrument and the event. The validation value is preferably assigned to the image data and can be stored in a corresponding memory together with the other data and/or values.

The method may optionally include manual validation, for example, during or after the procedure, of the recognized instrument classification by means of user interaction. This may advantageously include output of the recognized instrument classification to a user, such as a treating physician, using designated output means and recording manual feedback using designated input means. In addition to Output of the recognized instrument classification can include output of additional associated data, in particular the output of an associated validation value. Further preferably, an optional manual validation can be performed only for the classification data or recognized instruments to which a relatively low validation value is assigned. This allows an optional manual review for less plausible data to be performed, which, on the one hand, improves the overall validity of the recorded data and, on the other hand, allows a manual review only for the data recognized as less valid, thus optimizing the overall efficiency for valid data capture.

In a further preferred embodiment, the method can also include the step of transmitting a feedback signal to a device connected to the detected instrument(s) based on the data acquired by the image analysis. This allows, in particular, the respective instrument to be controlled based on the data acquired by the image analysis. For example, if excessive tissue damage is detected during an RF current application, a corresponding control can be initiated to reduce the power of the device or generator connected to the RF instrument in order to prevent further tissue damage.

In a further aspect, the invention relates to a device for acquiring and analyzing medical image data, which is designed to carry out the method as described above, comprising:

- a connection unit for selectively connecting an image acquisition device for medical treatment, in particular an endoscope,

- a data acquisition unit for acquiring image data from at least the image acquisition device connected to the connection unit, and

- a computing unit which is designed to carry out a visual detection and recognition of an operative event in the acquired image data, wherein the computing unit is further designed to carry out a distance determination between the detected operative event and an instrument end of at least one instrument visually detected in the image data, as well as a detection and assignment of an instrument classification to the detected instrument based thereon. In a preferred embodiment, the device comprises a documentation unit for recording image data provided by the data acquisition unit and the data determined by the computing unit, in particular including the determined instrument classification. The documentation unit can advantageously be configured to record data depending on a determined instrument classification. This allows for selective data acquisition and documentation only for data that is of greater interest for subsequent documentation or analysis, thus enabling more efficient use of data and computing capacity.

The documentation unit is preferably designed to link the respective image data with the data acquired and/or determined by the computing unit and to store them as associated data or as a correspondingly linked data set. The determined and/or assigned data can be stored as metadata of the respective image data. This allows the image data to be clearly assigned to the acquired data at a later point in time and, in particular, to avoid human error during the separate recording of image data known from the prior art on the one hand and during the respective examination of additionally acquired data sets on the other. The documentation unit can also be designed to store the respectively assigned data as metadata of the image data in predefined frames or, more preferably, in each individual frame of the image data.

Furthermore, the computing unit can be configured to adjust and/or adapt the data acquisition unit, in particular the recording parameters that influence data recording, and to control the data acquisition unit depending on the determined instrument classification and/or the additionally acquired data. For example, a frame rate, an image section, and/or other image parameters of the acquired image signal can be adjusted or adapted depending on the determined and/or assigned data.

The computing unit advantageously comprises a memory unit which is used to store and/or assign attributes such as an instrument classification, an event event classification, an event intensity, a type of medical treatment, an assigned validation value for a respectively recorded event and/or a respectively recorded surgical instrument. The computing unit can in particular comprise one or more lookup tables, by means of which, for example, predefined and/or changeable attributes are assigned to a respectively stored surgical event, which can be used, for example, for instrument classification.

In a preferred embodiment, the data acquisition unit is configured to acquire additional device data and/or parameter data from devices connected to the device in a wired or wireless manner. The data acquisition unit is also advantageously configured to acquire provided patient data and/or vital parameters of a patient. The computing device can advantageously be configured to determine attributes of the respective surgical instrument or the respective medical procedure from a combination of the acquired device data and/or parameter data and the acquired image data.

In a preferred embodiment, the device further comprises a data transmission unit, which is designed for preferably bidirectional data exchange with an external network and/or an external server unit. The data transmission unit can have a LAN and/or WLAN interface for connection to an internal or external network, such as a hospital LAN or the Internet. The device can be designed for the acquisition of data acquisition commands from the external server unit and/or for the automated and/or non-automated transmission of recordings from the data acquisition unit to the external server unit. The transmission to an external network and/or to an external server unit can take place in real time and/or with a time delay relative to the recording of the documentation unit. Particularly preferably, the respective transmission of the acquired data records can take place at predefined time intervals or at predefined transmission times. The respective intermediate storage of the data can take place in external or internal storage units. The device described above can be designed as a separate or integral component of an endoscopy device or an endoscopy system.

To avoid repetition, reference is made here to the above-mentioned statements on the method according to the invention, which are to be considered equally as disclosed and claimable for the device according to the invention, and vice versa.

Details, advantageous effects and details of the present invention are explained below with reference to the schematic, merely exemplary drawings.

Showing:

Fig.1: a schematic representation of a preferred embodiment of the device according to the invention for analyzing medical image data;

Fig. 2: an exemplary live image of an endoscopic procedure with analysis of the acquired image data; and

Fig. 3: a preferred embodiment of the method according to the invention as a schematically simplified block diagram.

Fig. 1 shows a preferred embodiment of the device 10 according to the invention for analyzing acquired medical image data. The device 10 initially comprises a connection unit 1 for selectively connecting an image acquisition device 2 for a medical treatment or a medical procedure. This is, in particular, an endoscope, more preferably a video endoscope. The connection unit can have known plug connections, which serve for the wired connection of the endoscope to the device.

The connection unit 1 can also be designed to connect further devices, such as an input unit 7, for example a keyboard and/or a mouse, a display unit 8 for outputting a captured video image to a treating physician. For this purpose, the connection unit 1 can have wired or wireless interfaces for the respective connection and/or communication of the devices 7, 8.

The device further comprises a data acquisition unit 3 for acquiring at least image data 3a provided by the image acquisition device 2. In addition, the data acquisition unit 3 can be configured to acquire device data 3b from at least the image acquisition device 2 connected to the connection unit 1. The device data 3b can comprise, for example, a device identification number, a device identifier, or the like, based on which an identification and/or recognition of the endoscope 2 connected to the device 10 is possible.

The data acquisition unit 3 can additionally acquire patient data and/or vital parameters 3c associated with the patient. The patient data and/or vital parameters 3c can either be entered manually using appropriate input means 7 or transmitted automatically by other devices connected to the device.

The device further comprises a computing unit 4, which is configured to perform a visual detection and recognition of a surgical event in the acquired image data 3a. Furthermore, the computing unit 4 is configured to determine the distance between the detected surgical event 32 and an instrument end 31a, 31b of at least one surgical instrument 30a, b visually detected in the image data 3a (see Fig. 2), as well as to perform a detection and assignment of an instrument classification to the detected instrument based thereon. The computing unit 4 preferably comprises an internal memory 4a, on which suitable software algorithms are stored and/or retrievable.

The device 10 preferably further comprises a documentation unit 5 for recording image data 3a, device data 3b and/or the further patient data and/or vital parameters 3c provided by the data acquisition unit 3. For this purpose, the documentation unit 5 advantageously comprises an internal or external Storage unit 5a, in particular a non-volatile memory, on which the supplied data can be stored.

The computing unit 4 can further advantageously be designed to control the documentation unit 5 for activating and/or deactivating a recording, and/or for setting and/or adapting recording parameters influencing the data recording depending on the determined and/or assigned attributes.

In contrast to pure data logging of all data during an examination with an endoscope, this means that a recording can, for example, only be activated or deactivated if a predefined instrument classification and/or a predefined attribute of the surgical instrument or of a recognized surgical procedure is met or not met. This can be particularly advantageous for data collection for training AI applications, since, for example, only data is recorded that appears to be useful for the respective desired application. In addition to or as an alternative to the aforementioned configuration, the computing unit 4 can be configured to enable manual activation of the documentation unit 5, in particular by a treating physician. In this way, data recording could be triggered in particular if, for example, there is interest in a certain type of treatment, so that the corresponding image data is also saved for the next available procedure that is similar to this.

The device 10 further preferably comprises a data transmission unit 6, which is designed for preferably bidirectional data exchange with an external network and/or an external server unit 20. The data transmission unit 6 can have a LAN and/or WLAN interface for connection to an internal or external network, such as a hospital LAN or the Internet. This allows the device to transmit the recorded data to an external network and/or an external server unit 20. The respective transmission can take place during the actual recording of data and thus during a medical treatment. Alternatively, Each transmission can also take place at predefined time intervals. The device 10 can further be configured to capture data capture commands by the external server unit 20 and/or to transmit records from the documentation unit 5 to the external server unit 20 in an automated and/or non-automated manner.

With reference to Figs. 2 and 3, a preferred embodiment of the method according to the invention is described below, which can be carried out with the device 10 described above.

In a first step 101, the image data provided by the endoscope 2 is acquired. In particular, a live image 40 of the endoscope is acquired, as shown in Fig. 2, and is output, for example, on a display unit 8 of the device 10.

In a further step 102, an operative event 32 is detected in the acquired image data. This step preferably initially comprises a visual detection 102a, i.e., a spatial localization of an operative event 32 in the image data. This is performed using a corresponding software-based algorithm in the acquired image data and can be displayed or output in the live image 40 as a preferably rectangular, frame-like boundary 32a.

Simultaneously and/or subsequently, a visual recognition 102b of the surgical event in the image data takes place. Using a corresponding algorithm, the acquired image data is preferably compared with stored and/or retrievable image data of predefined surgical events and, based on this, the event is classified, for example, into one or more of the following event classifications: smoke development, a color or structural tissue change, a developing and/or changing tissue opening, a change in tissue water content, blood development, suture placement, the introduction of foreign material into the tissue, in particular the introduction of metal, for example a clip, and/or a spray mist or gas bubbles. The detected event classification, for example, the presence of smoke, can be output in a text field A assigned to frame 42. Optionally, this step can also include a detection and assignment of an event intensity and/or another event attribute in a corresponding step 102c. For example, the intensity of the event, for example, a smoke intensity, can be classified by appropriate image analysis and output as associated information in a text field B assigned to frame 32a.

In a further step 103a, the surgical instruments 30a, 30b are visually detected in the image data. This step preferably comprises the detection, i.e., a spatial localization of the surgical instruments 30a, 30b visually recognizable in the image data and their respective associated instrument ends 31a, 31b. This is performed using a corresponding software-based algorithm by analyzing the acquired image data and can be displayed or output in the live image 40 as a preferably rectangular, frame-like boundary 33a, 33b.

In a further step 103b, the surgical instruments 30a, 30b are visually recognized in the image data. A corresponding algorithm can first be used to compare the acquired image data with stored and/or retrievable image data of predefined surgical instruments and their instrument ends, and then, based on this, to recognize an instrument type. The corresponding recognized instrument type, for example, a hook or a gripper, can then be output in a text field C1, C2 assigned to the frame 33a, 33b.

Step 103b further comprises detecting and assigning an instrument classification to the instrument. According to the invention, this is done based on the detection of the operative event in the preceding step 102 and a distance determination between the detected operative event and a respective instrument end. First, the respective distance between the instrument ends 31a, 31b and the operative event 32 is determined. The distance determination between the operative event 32 and the respective instrument end 31a, 31b of the detected instrument is preferably carried out using a software-based distance measurement between two image points, in particular between the respective centers of the instrument end 31a, 31b and the detected event 32, and/or based on an additional depth measurement, which methods are known per se. The instrument end 31a, 31b, which is detected closest to the operative event 32, is then assigned as the relevant instrument classification. In this case, the instrument end 31a, 31b is closer to the operative event 32 than the instrument end 31b, so that the former is assigned to the event 32.

Based on the detected surgical event 32, the instrument 30a most closely associated with the event can now be classified. For example, by detecting the surgical event as smoke development in the tissue, it can be concluded that the corresponding instrument is to be classified as a high-frequency or HF instrument. In particular, HF instruments such as HF grasping forceps, which are visually indistinguishable or only barely distinguishable from grasping forceps without HF functionality, can be differentiated from such instruments. Further examples of instrument classification include: HF instrument, non-HF instrument, forceps, scissors, suture instrument, stapler, biopsy instrument, punch, clip applicator, and/or an ultrasound device.

The corresponding classification is preferably performed by retrieving and/or evaluating data stored, for example, in look-up tables. The recognized instrument classification is also assigned to the image data, i.e., stored in a corresponding memory for later evaluation and/or documentation purposes. Furthermore, the recognized instrument classification can also be output in the live image, for example, in a text field D assigned to frame 33a. This enables optimized instrument recognition in medical image data such as an endoscopic live image. In particular, an instrument classification can be performed that would not be possible using purely optical or visual image evaluation.

In a further preferred step 104, the recorded and recognized data are stored, in particular for documentation and/or evaluation purposes. The acquired and recognized data, in particular the detected operational event and/or a detected intensity of the event, can be used not only for pure analysis and/or documentation purposes but also to control the device 10 and/or a device connected to the device. For example, based on the detected event intensity, such as smoke intensity, an RF power of an RF device connected to the device can also be controlled. For example, in the case of a relatively high event intensity, such as very heavy smoke development, the current strength of the device connected to the device can be reduced by appropriate control.

List of reference symbols

1 connection unit

2 image capture device

3 Data acquisition unit

3a Image data

3b Device data

3c Patient data

4 Control unit

4a Storage unit

5 Documentation unit

5a Storage unit

6 Data transmission unit

7 Input devices

8 Display unit

10 Device

20 server units

30a, b surgical instrument

31a,b Instrument end

32 operational event

32a visual boundary event

33a, b visual boundary of instrument end

40 Live image endoscope

101 Recording operational events

102 Detection of operational event

103a Recording Instrument

103b Recognition Instrument Classification

104 Storage

Claims

Patent claims 1 . A method for detecting surgical instruments in medical image data, in particular for analysis and documentation purposes, comprising the steps: - capturing (101) image data (3a) preferably provided by means of an endoscope, in particular an endoscopic live image (40), - visual detection (102) of an operative event (32) in the acquired image data, and - visual detection (103a) of at least one instrument (30a, 30b) in the detected image data, characterized in that the method comprises a recognition and assignment (103b) of an instrument classification to the detected instrument (30a, 30b) based on the recognition of the operative event (32) and a distance determination between the recognized operative event (32) and an instrument end (31a, 31b) of the at least one instrument (30a, 30b). 2. Method according to one of the preceding claims, characterized in that, when two or more instruments (30a, 30b) are visually detected in the detected image data, the recognition and assignment of the instrument classification takes place for the instrument whose instrument end (31a, 31b) has the shortest determined distance to the detected operative event (32). 3. Method according to claim 1 or 2, characterized in that the determination of a distance between the operative event (32) and the instrument end (31a, 31b) of the detected instrument (30a, 30b) is carried out based on a distance measurement that can be directly visually detected from the image data, in particular between respective centers of an instrument end and a detected event, and/or based on additional depth measurement data. 4. Method according to one of the preceding claims, characterized in that the determination of a distance between the operative event (32) and the instrument end (31 a, 31 b) of the detected instrument (30a, 30b) takes place immediately after visual detection of the operative event and/or detection of the instrument. 5. Method according to one of the preceding claims, characterized in that the visual detection of the operative event (32) comprises an event classification into one or more of the following event classifications: a smoke development, a color or structural tissue change, a developing and/or changing tissue opening, a change in the water content of the tissue, a blood development, a suture formation, an introduction of foreign material into the tissue, in particular a metal introduction, for example a clip, and/or a spray mist or gas bubbles. 6. Method according to one of the preceding claims, characterized in that the visual recognition of the operative event (32) comprises an assignment of an event intensity and/or a further event attribute, in particular comprising a smoke intensity, a bleeding intensity and/or a tissue opening size. 7. Method according to one of the preceding claims, characterized in that the recognition and assignment of an instrument classification comprises a recognition and assignment of an instrument type and/or a further instrument attribute, in particular comprising HF instrument, non-HF instrument, forceps, scissors, suturing instrument, stapler, biopsy instrument, punch, clip setter and/or an ultrasound device. 8. Method according to one of the preceding claims, characterized in that the recognition and assignment of an instrument classification of the detected instrument (30a, 30b) is additionally based on a recognized event classification, an event intensity, and/or a further event attribute. 9. Method according to one of the preceding claims, characterized in that the visual recognition of an operative event (32) is carried out by means of computer-assisted image evaluation algorithms, in particular by evaluating the RGB image values over time, by evaluating multispectral and/or hyperspectral image parameters and/or by using algorithms for object recognition in image data. 10. Method according to one of the preceding claims, characterized in that the recognition and assignment of an instrument classification of the detected instrument (30a, 30b) is carried out not only on the basis of the detected visual image data of the instrument but also on the basis of the evaluation of device and/or intervention data stored in a database for predefined instruments. 11. Method according to one of the preceding claims, characterized in that the method comprises the further step of determining a recognition and assignment of a type of medical treatment as an attribute for the analyzed image data. 12. Method according to one of the preceding claims, characterized in that the method comprises calculating and assigning a validation value for the respective instrument (30a, 30b), which is determined at least based on a distance value of the respective distance determination between a recognized operative event (32) and an instrument end (31a, 31b). 13. Method according to one of the preceding claims, characterized in that the method comprises a manual validation of the recognized instrument classification by means of user interaction, in particular comprising an output of the recognized instrument classification to a user by means of provided output means (8) and a recording of a manual feedback by means of provided input means (7). 14. Method according to one of the preceding claims, characterized in that the method comprises the further step (104) of storing the recorded and recognized data, for example the recognized event, the instrument classification, the distance or other attributes for documentation and/or evaluation purposes. 15. Device for analyzing medical image data, which is designed to carry out the method according to one of the preceding claims, comprising: - a connection unit (1) for selectively connecting an image capture device (2) for medical treatment, in particular an endoscope, - a data acquisition unit (3) for acquiring image data (3a) from at least the image acquisition device (2) connected to the connection unit (1), and - a computing unit (4) which is designed to carry out a visual detection and recognition of an operative event (32) in the captured image data (3a), characterized in that the computing unit (4) is further designed to carry out a distance determination between the detected operative event (32) and an instrument end (31a, 31b) of at least one instrument (30a, 30b) visually detected in the image data (3a), and a detection and assignment of an instrument classification to the detected instrument based thereon. 16. Device according to claim 15, characterized in that the device comprises a documentation unit (5) for recording image data (3a) provided by the data acquisition unit (3) and the data determined by the computing unit (4), in particular comprising the determined instrument classification, wherein the documentation unit is preferably designed for recording depending on a determined instrument classification.