JP2025129144A - Endoscopic device for acquiring images of the upper gastrointestinal tract and method for controlling the same - Google Patents

Abstract

To provide an endoscope device for acquiring upper gastrointestinal tract images and a method of controlling the same.SOLUTION: The invention belongs to the field of medical imaging equipment and automation control technology, and particularly relates to an endoscope device associated with imaging of the upper gastrointestinal tract using an endoscope, and to a method of controlling the endoscope device. The method includes the steps of: acquiring an image of the upper gastrointestinal tract from an image sensor; sensing at least one first body part from the image based on a pre-trained model; calculating relative position information between the sensed first body part and a distal end portion; generating a first control signal related to the position of the first body part based on the relative position information; and transmitting the first control signal to a drive unit.SELECTED DRAWING: Figure 2

Description

The present invention relates to the fields of medical imaging equipment and automated control technology, and more particularly to an endoscopic device used for imaging the upper gastrointestinal tract using an endoscope.

An endoscope is a general term for medical instruments that are inserted into the body to observe organs without performing surgery or autopsies (pathological autopsies). Endoscopes are inserted into the human body, shining light into it and visualizing the light reflected from the surface of the inner wall. Endoscopes are categorized by purpose and body part, and can be broadly divided into rigid endoscopes, in which the endoscopic tube is made of metal, and flexible endoscopes, such as gastrointestinal endoscopes.

Today, when endoscopists perform endoscopic examinations and discover a lesion, they must perform additional actions such as inserting instruments to perform a tissue examination and pressing buttons on the scope. In doing so, they may let go of the hand holding the scope, causing the scope to shake and the lesion to move out of the field of view.

Such endoscopic techniques primarily rely on the manual operation of medical professionals to adjust the tip of the endoscope and acquire images of the patient's internal body parts. This process is highly dependent on the operator's skill and experience, which makes it difficult to acquire accurate and clear images of the desired body part. In particular, unnecessary movements and difficulty in precise adjustment during the process of operating the endoscope can prevent the acquisition of sufficient images of specific parts necessary for accurate diagnosis.

In addition, endoscopic technology has limitations in precisely adjusting the position and angle of the endoscope's tip, making it particularly difficult to obtain images of areas such as the upper gastrointestinal tract, which has many curves. These limitations reduce the efficiency and accuracy of endoscopic diagnosis in terms of early detection and accurate location of diseases.

The present invention has been proposed to solve the above-mentioned problems, and the problem that the present invention aims to solve is to provide technology for controlling the position and direction of the tip of an endoscope based on an artificial neural network.

To achieve the above-mentioned object, according to one embodiment of the present specification, a method for controlling an endoscopic device includes the steps of acquiring an image relating to the upper gastrointestinal tract from an image sensor, detecting at least one first body part from the image based on a pre-trained model, calculating relative position information between the detected first body part and the distal end, generating a first control signal for steering in correspondence with the first body part based on the relative position information, and transmitting the first control signal to a drive unit.

The step of acquiring images of the upper gastrointestinal tract includes acquiring a first image at a first point and acquiring a second image at a second point, and the step of calculating the relative position information also includes calculating i) an angle change of the tip between the first point and the second point, and ii) a target rotation angle of the tip based on the change between the first image and the second image.

The method further includes identifying at least one second body part from the image based on the pre-trained model, calculating relative pose information between the identified second body part and the tip, generating a second control signal related to photographing the second body part based on the relative pose information, and transmitting the second control signal to the driver.

The step of calculating the relative pose information also includes the steps of identifying brightness differences in the image based on the acquired image and brightness information related to lighting, estimating the distance between the tip and the second body part based on the identified brightness differences, and calculating the relative pose information based on the estimated distance.

The step of acquiring images relating to the upper gastrointestinal tract includes acquiring a first image at a first point and acquiring a second image at a second point, and the step of calculating the relative pose information also includes estimating the distance between the tip and the second body part based on i) the angle change of the tip between the first point and the second point, and ii) the change between the first image and the second image, and calculating the relative pose information.

The pre-trained models may also include classification models and detection models trained using a dataset of labeled images of a first body part and a second body part related to the upper gastrointestinal tract.

The step of generating the second control signal also includes the steps of identifying at least one imaging location corresponding to the identified second body part, generating a second control signal based on the at least one imaging location and the relative position information, and capturing an image when the position of the tip corresponds to the imaging location.

The method may further include adjusting the tension of at least one wire and controlling the rotation angle of the tip based on the first control signal.

The drive unit further includes a step of generating torque feedback and transmitting the torque feedback to the operating unit.

The torque feedback may have a positive correlation with the target rotation angle to which the tip must move in response to the first control signal related to the image capture.

An endoscopic device according to one embodiment of the present specification, which aims to achieve the above-mentioned object, includes a tip portion equipped with an image sensor, a drive unit that controls the rotation angle of the tip portion, and a control unit that detects at least one first body part from the image based on a pre-trained model, calculates relative position information between the detected first body part and the tip portion, generates a first control signal related to the position of the first body part based on the relative position information, and transmits the first control signal to the drive unit.

The control unit may acquire a first image at a first point and a second image at a second point, and calculate a target rotation angle of the tip based on i) the change in angle of the tip between the first point and the second point, and ii) the change between the first image and the second image.

The control unit may identify at least one second body part from the image based on the pre-trained model, calculate relative pose information between the identified second body part and the tip, generate a second control signal related to photographing the second body part based on the relative pose information, and transmit the second control signal to the driving unit.

The control unit may identify brightness differences in the image based on the acquired image and brightness information related to lighting, estimate the distance between the tip and the second body part based on the identified brightness differences, and calculate the relative pose information based on the estimated distance.

The control unit may acquire a first image at a first location and a second image at a second location, and estimate the distance between the tip and the second body part based on i) the change in angle of the tip between the first location and the second location, and ii) the change between the first image and the second image, and calculate the relative pose information.

The pre-trained models may also include classification models and detection models trained using a dataset of labeled images of a first body part and a second body part related to the upper gastrointestinal tract.

The control unit may identify at least one imaging location corresponding to the identified body part, generate a second control signal based on the at least one imaging location and the relative position information, and capture an image when the position of the tip corresponds to the imaging location.

The endoscopic device further includes a bending section connected to the tip section, and the control section controls the drive section based on the control signal to adjust the tension of at least one wire connected to the drive section, and adjusts the rotation angle of the tip section based on the bending of the bending section.

The endoscope device further includes an operation unit having a bending steering unit, and the control unit can generate torque feedback based on the drive unit and transmit the torque feedback to the bending steering unit.

The torque feedback may have a positive correlation with the target rotation angle to which the tip must move in response to the first control signal related to the image capture.

According to one embodiment of the present invention, the control unit of the endoscope device controls the position of the tip, allowing for accurate and efficient acquisition of images of specific body parts. This reduces the difficulty of endoscopic operation and improves the accuracy of medical diagnoses.

The effects of this embodiment are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the claims.

1 is a diagram schematically illustrating an endoscopic device according to an embodiment of the present invention. 10A to 10C are diagrams illustrating a process in which a bending section is controlled by a driving section and a wire according to an embodiment of the present invention. 1 is a flowchart illustrating an operation for generating a learning model according to one embodiment of the present invention. 1 is a flowchart illustrating the operation of an endoscopic device, in accordance with one embodiment of the present invention. 10 is a flowchart illustrating the operation of an endoscopic device in accordance with another embodiment of the present invention. 10 is a flowchart illustrating the operation of an endoscopic device according to yet another embodiment of the present invention. 10 is a flowchart illustrating an operation of calculating relative pose information of an endoscopic device in accordance with an embodiment of the present invention. 10 is a flowchart illustrating an operation of calculating relative pose information of an endoscopic device according to another embodiment of the present invention. 10 is a flowchart illustrating an operation of calculating relative pose information of an endoscopic device according to yet another embodiment of the present invention. 1 is a flowchart illustrating operations involved in capturing images in an endoscopic device, in accordance with one embodiment of the present invention. 10 is a flowchart illustrating operations relating to torque feedback in an endoscopic device, in accordance with one embodiment of the present invention. FIG. 1 is a block diagram illustrating a block configuration of a computer device according to an embodiment of the present invention.

The terms used in this specification are merely used to describe particular embodiments and are not intended to limit the scope of other embodiments. The singular expressions also include the plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in this specification. Among the terms used in this specification, commonly defined terms should be interpreted as meanings that are identical to or similar to the meanings they have in the context of the relevant technology, and should not be interpreted as ideal or overly formal unless expressly defined in this specification. In some cases, even if a term is defined in this specification, it should not be interpreted as excluding embodiments of the present specification.

Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention. However, the technical concept of the present invention may be embodied in various forms and is not limited to the embodiments described herein. In describing the embodiments disclosed herein, if a detailed description of related publicly known technology is deemed to obscure the gist of the technical concept of the present invention, the detailed description of that publicly known technology will be omitted. Identical or similar components will be designated by the same reference numerals, and redundant description thereof will be omitted.

The term "module" used in this embodiment refers to a component that performs a specific function, whether performed by software or hardware such as an FPGA (field programmable gate array) or an application-specific integrated circuit (ASIC). However, the term "module" is not limited to being performed by software or hardware. The "module" may exist in the form of data stored on an addressable recording medium, or may be embodied by an instruction word, configuring one or more processors to execute a specific function.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to operate in a desired manner or may, independently or collectively, instruct the processing device. The software and/or data may be permanently or temporarily embodied in some type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. The software may also be distributed across network-linked computer systems, stored or executed in a distributed manner. The software and data may be stored in one or more computer-readable storage media. The software may be read into main memory from other computer-readable media, such as a data storage device, or from another device via a communications interface. Software instructions stored in main memory may cause a processor to perform the processes or steps described in detail below. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to implement processes consistent with the principles of the invention. Thus, embodiments consistent with the principles of the invention are not limited to any specific combination of hardware circuitry and software.

The terms used in this application are merely used to describe particular embodiments and are not intended to limit the present invention. The singular terms include the plural terms unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "have" specify the presence of a specified feature, number, step, operation, component, part, or combination thereof, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. Terms such as "first" and "second" may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

The term "learning model" as used herein includes all forms of algorithms or methodologies used to learn or understand specific patterns or structures from data. That is, the term "learning model" includes not only machine learning models such as regression models, decision trees, random forests, support vector machines, K-nearest neighbors, naive phase, and clustering algorithms, but also deep learning models such as neural networks, convolutional neural networks, recurrent neural networks, Transformer-based neural networks, GANs (generative adversarial networks), and autoencoders. A "learning model" refers to a set of learned parameters or weights used to predict or classify an output for a specific input. The model can be trained through methods such as supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. It also includes not only single models, but also various learning methods and structures such as ensemble models, multimodal models, and models via transfer learning. Such learning models can be pre-trained on a computer device separate from the computer device that predicts the output for an input, and then used on another computer device.

The learning model according to one embodiment of the present invention may include at least one model related to object classification, object detection, and position estimation.

Figure 1 illustrates a schematic diagram of an endoscopic device according to one embodiment of the present invention.

Referring to FIG. 1, an endoscopic device 100 according to one embodiment of the present invention is a flexible endoscope, specifically a gastrointestinal endoscope. The endoscopic device 100 also includes a configuration capable of acquiring medical images of the inside of the gastrointestinal tract, and a configuration capable of inserting a tool and performing treatment or procedures while viewing the medical images, if necessary.

The endoscope device 100 also includes an output unit 110, a control unit 120, a drive unit 130, a scope 140, and an operation unit 160.

The output unit 110 may also include a display that displays medical images. The output unit 110 may output visualized information using a display module such as a liquid crystal display (LCD), thin film transistor-liquid crystal display (TFT LCD), organic light-emitting diode (OLED), flexible display, or 3D display, or may include a display module that may implement a touch screen, supporting functions such as image display, image enlargement, and image reduction. Furthermore, the touch screen function allows the user to manipulate the image and obtain necessary information. The output unit 110 may display medical images acquired by the scope 140 or medical images processed by the control unit 120.

The driving unit 130 may provide the necessary power when the scope 140 is inserted into or moves within the body. For example, the driving unit 130 may include multiple motors connected to wires inside the scope 140 and a tension adjustment unit that adjusts the tension of the wires. The driving unit 130 may control the power of each of the multiple motors to control the scope 140 in various directions. Specifically, the driving unit 130 may control the power of each of the multiple motors to adjust the tension of the wires, bend the bending portion 142, and adjust the rotation angle of the tip portion 143.

The scope 140 also includes an insertion section 141, a bending section 142, and a tip section 143. The insertion section 141 is the part that is inserted into the body and can be steered by the bending section 142 to move to internal organs.

The bending portion 142 is connected to the insertion portion 141 and can adjust the direction in which the scope 140 enters the body. The bending portion 142 can adjust its rotation angle in response to a user command or a control signal from the control unit.

The tip 143 is located at the tip of the scope 140 and can perform various operations in response to commands from the user or control signals from the control unit. The tip 143 also includes an image sensor 151, a nozzle 152, a light 153, a lens 154, and a working channel 155.

The image sensor 151 can capture images of the endoscopic device. For example, the image sensor 151 can be a CMOS (complementary metal-oxide-semiconductor) sensor or a CCD (charge-coupled device) sensor.

Nozzle 152 can spray solutions, medications, etc. to clean lens 154. Nozzle 152 can also inject medications required for tissue examination or treatment into the body.

The lighting 153 may emit a light source at a constant illuminance so that the image sensor 151 can capture an image. Information regarding the brightness of the lighting 153 may also be stored in advance in the control unit 120.

Lens 154 may focus light so that image sensor 151 can capture a suitable image. Such a lens 154 may also include a wide-angle or zoom function.

The working channel 155 may refer to a channel for transmitting a separate instrument or sampling tool into the human body.

The operation unit 160 may refer to the user interface that actually operates the endoscope. The operation unit 160 also includes the bending and steering unit 161 and various buttons, dials, and levers for controlling the various functions of the endoscope. The user can input user commands based on the configuration provided in such operation unit 160.

The bending steering unit 161 may be used to adjust the bending portion 142. The bending steering unit 161 may be embodied in the form of a rotary knob, joystick, or lever, and the user may rotate or move it to steer the direction of the bending portion 142. The bending steering unit 161 may receive torque feedback from the drive unit 130. For example, the torque feedback may be a physical signal based on a control signal from the drive unit 130 that replicates the force generated when the tip portion 143 comes into contact with internal human tissue or is based on a pre-trained model.

The control unit 120 may control the overall operation of the endoscopic device 100 and perform the operation of the endoscopic device according to one embodiment. The control unit 120 may control the movement of the scope 140 via the driving unit 130 connected to the scope 140. The control unit 120 may perform various control operations for capturing images of the inside of the digestive tract through the scope 140. The control unit 120 may perform various processes on the medical images acquired through the scope 140.

According to one embodiment, the control unit 120 may acquire an image of the upper gastrointestinal tract from an image sensor, detect at least one body part from the image based on a pre-trained model, calculate relative position information between the body part and the distal end, generate a control signal for image capture based on the identified body part and relative pose information, and transmit the control signal to the driver. Such relative position information may refer to steering information for moving the distal end from a first position, which is the current position of the distal end, to a second position. For example, the control unit 120 may use the relative position information to calculate a target rotation angle by which the distal end can be steered toward a body part that is located approximately x and y coordinates away on the image. The control unit 120 may also generate a control signal based on the target rotation angle to control the driver, and adjust the pitch and yaw of the distal end to steer the distal end toward a body part that is located approximately x and y coordinates away on the image.

According to one embodiment, the control unit 120 may acquire an image related to the upper gastrointestinal tract from an image sensor, identify at least one body part from the image based on a pre-trained model, calculate relative pose information between the body part and the tip, generate a control signal related to image capture based on the identified body part and the relative pose information, and transmit the control signal to the driver. Such relative pose information may refer to information for moving from a first pose, which is the current pose of the tip, to a second pose for effectively capturing the identified body part. A pose also includes the position and orientation of an object in space. For example, the position may be expressed in the form of x, y, and z coordinates in a coordinate system, and the orientation may be expressed in the form of pitch (roll around x-axis), yaw (roll around y-axis), and roll (roll around z-axis).

The control unit 120 may also include a central processing unit (CPU), random access memory (RAM), read-only memory (ROM), a system bus, etc. The control unit 120 may be implemented by a single CPU or multiple CPUs (or a digital signal processor (DSP), system-on-chip (SoC)). In one embodiment, the control unit 120 may be implemented by a digital signal processor (DSP) that processes digital signals, a microprocessor, or a time controller (TCON). However, the control unit 120 may include or be defined by one or more of the following terms: a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a communication processor (CP), or an ARM processor, without being limited thereto. The control unit 120 may also be implemented as a system-on-chip (SoC) or large-scale integration (LSI) with a built-in processing algorithm, or as a field programmable gate array (FPGA). Furthermore, the control unit 120 may include a neural processing unit (NPU), a graphics processing unit (GPU), and a tensor processing unit (TPU).

Figure 2 is a diagram illustrating the process by which the bending portion is controlled by the drive unit and wires according to one embodiment of the present invention. For ease of explanation, the operation of controlling the direction of the bending portion 142 using two motors 200 is illustrated. However, it will be clear to those skilled in the art that it is also possible to use multiple motors to adjust the tension of each wire and adjust the rotation angle of the tip portion.

Referring to FIG. 2, the endoscopic device 100 also includes a motor 200, a first wire 210, and a second wire 220 included in the drive unit 130. To bend the bending portion 142 to one side, the endoscopic device 100 can control the motor 200 to increase the tension of the first wire 210 and decrease the tension of the second wire 220. To bend the bending portion 142 to the other side, the endoscopic device 100 can control the motor 200 to decrease the tension of the first wire 210 and increase the tension of the second wire 220.

The endoscope device 100 can adjust the rotation angle of the tip portion 143 in this manner.

FIG. 3 is a flowchart illustrating operations for generating a learning model for object detection according to one embodiment of the present invention. The learning model in FIG. 3 may correspond to a pre-trained model. While such operations are disclosed as being learned by a separate computer device for convenience of explanation, it will be clear to one skilled in the art that they may be performed by the endoscopic device 100 or a separate computer device.

Referring to FIG. 3, in step S310, the computer device may review the endoscopic image and tag or label specific body parts. Based on user input, the computer device may label parts of the upper gastrointestinal tract in the image with bounding boxes. For example, the upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum. The endoscopic image may also include images of the oral cavity, pharynx, esophagus, stomach, and duodenum.

In one embodiment, the computer device may generate a dataset including labeled images in step S320. Such a dataset may include various lighting conditions, viewing angles, body part conditions, etc.

In one embodiment, the computer device may train a neural network model using the generated dataset in step S330. The computer device may select the neural network model and train the model based on the dataset. For example, such a model may be an object detection model, including a convolutional neural network.

In another embodiment, a computer device may generate a learning model for object classification. Such a learning model for object classification may also be a model for identifying objects for image capture.

In another embodiment, in step S310, the computer device may review the endoscopic image and tag or label a specific body part. Based on user input, the computer device may label the image with a part of the upper gastrointestinal tract. Such a part of the upper gastrointestinal tract may refer to the body part for which the image was taken.

In one embodiment, the computer device may generate a dataset including labeled images in step S320. Such a dataset may include various lighting conditions, viewing angles, body part conditions, etc.

In one embodiment, the computer device may train a neural network model using the generated dataset in step S330. The computer device may select the neural network model and train the model based on the dataset. For example, such a model may be an object classification model, including a convolutional neural network.

According to yet another embodiment, the computer device may generate a learning model using endoscopic images and control history information corresponding to the user's clinical skills. Such control history information may refer to control history information used to observe the endoscopic image and adjust the direction, angle, depth, etc. of the endoscope. According to another embodiment, the computer device may acquire endoscopic images and control history information related to the upper gastrointestinal tract in step S310. For example, the upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum. Furthermore, the endoscopic images may include images of the lumen of the oral cavity, pharynx, esophagus, stomach, and duodenum.

In yet another embodiment, the computer device may generate a data set including endoscopic images and control history information in step S320.

In yet another embodiment, the computer device may train a neural network model using the generated dataset in step S330. The computer device may select a neural network model and train the model based on the dataset. For example, such a model may include a classification model such as a convolutional neural network (CNN), and may perform image processing using at least one of a recurrent neural network (RNN) and a sequence processing model such as a long short-term memory (LSTM) network to perform sequence processing of control history information.

Figure 4A is a flowchart illustrating the operation of an endoscopic device according to one embodiment of the present invention.

Referring to FIG. 4A, the endoscopic device may acquire an image of the upper gastrointestinal tract from an image sensor in step S410a. For example, the distal end of the endoscopic device may enter the upper gastrointestinal tract and acquire an image in which an optical signal is converted into an electrical signal from the image sensor.

In one embodiment, the endoscopic device may detect at least one body part from the image based on a pre-trained model in step S420a. The endoscopic device may detect body parts based on a pre-trained model labeled with a bounding box for the upper gastrointestinal tract and confirm the position of the at least one body part within the image.

In one embodiment, an endoscopic device may calculate relative position information between a body part and the tip of the endoscopic device in step S430a. The endoscopic device may calculate the relative position information based on an image change caused by an angle change of the tip. The endoscopic device may calculate such relative position information based on disparity and the angle change of the tip.

In one embodiment, in step S440a, an endoscopic device may generate a control signal for steering relative to the body part based on the relative position information.

In one embodiment, the endoscopic device may transmit a control signal to the driver in step S450a. In one embodiment, the endoscopic device may adjust the tension of at least one wire to control the rotation angle of the tip so that it can be steered in response to the detected body part in step S450a.

As a specific example, an endoscopic device according to an embodiment may acquire a first image at a first point and acquire a second image at a second point rotated by a certain angle Δθ. If the time difference between the first image and the second image is ΔL, then θ _target for steering the distal end, which is separated by L _target from the center coordinates of the bounding box of the detected body part, to the detected body part can be expressed as follows:
[Equation 1]
θ _target = (Δθ/ΔL)L _target

Since Δθ can be calculated from the encoder information of the drive unit and ΔL can be calculated from the time difference between the images, the relationship between the target distance on the image and the target angle of tip rotation by the drive unit can be derived.

Equation 1 can be expressed as a coordinate system. That is, the rotation of the endoscopic device relative to the direction in which the tip advances into the body is called roll, and the drive unit can control the pitch and yaw to steer the endoscopic device toward a body part that is distant by the x and y coordinates on the image.

Figure 4B is a flowchart illustrating the operation of an endoscopic device according to another embodiment of the present invention.

Referring to FIG. 4B, the endoscopic device may acquire an image related to the upper gastrointestinal tract from an image sensor in step S410b. For example, the distal end of the endoscopic device may enter the upper gastrointestinal tract, and an image in which an optical signal is converted into an electrical signal may be acquired from the image sensor.

In another embodiment, the endoscopic device may identify at least one body part from the image based on a pre-trained model in step S420b. The endoscopic device may classify body parts based on a pre-trained model labeled for the upper gastrointestinal tract and identify at least one body part within the image.

In another embodiment, an endoscopic device may calculate relative pose information including the distance between the body part and the tip of the endoscopic device in step S430b. For example, the endoscopic device may calculate relative pose information based on the brightness of the lighting, image changes due to changes in the angle of the tip, and the movement trajectory of the tip. The operation of the endoscopic device to calculate such relative pose information may correspond to Figures 5, 6, and 7.

In another embodiment, an endoscopic device may generate a control signal for image capture based on the identified body part and relative pose information in step S440b. Such a control signal may also include a signal for adjusting the tip of the endoscope.

An endoscopic device according to another embodiment may transmit a control signal to the driver in step S450b. An endoscopic device according to one embodiment may adjust the tension of at least one wire and control the rotation angle of the tip portion in step S450b to correspond to at least one pre-set imaging point related to the identified body part.

In another embodiment, an endoscopic device may capture an image based on the identified body part and relative pose information in step S460b. The endoscopic device may capture an image when the distal end is positioned at at least one pre-set capture position related to the identified body part.

Figure 4C is a flowchart illustrating the operation of an endoscopic device according to yet another embodiment of the present invention.

Referring to FIG. 4C, the endoscopic device may acquire an image related to the upper gastrointestinal tract from an image sensor in step S410c. For example, the distal end of the endoscopic device may enter the upper gastrointestinal tract, and an image in which an optical signal is converted into an electrical signal may be acquired from the image sensor.

In yet another embodiment, an endoscopic device may generate a control signal for image capture based on a pre-trained model in step S420c, using control history information corresponding to the endoscopic image and the user's clinical skills. The endoscopic device may input the endoscopic image into a classification model and a learning model, which is a sequence processing model, and generate a control signal. Such a control signal may also include a signal for adjusting the tip of the endoscope.

In yet another embodiment, the endoscopic device may transmit a control signal to the driver in step S430c. In step S430c, the endoscopic device may adjust the tension of at least one wire and control the rotation angle of the tip portion so as to correspond to at least one pre-set imaging point related to the identified body part.

In yet another embodiment, an endoscopic device may capture an image based on the identified body part and relative pose information in step S440c. The endoscopic device may capture an image when the distal end is positioned at at least one pre-set imaging position related to the identified body part.

Figure 5 is a flowchart illustrating the operation of calculating relative pose information of an endoscopic device according to one embodiment of the present invention. The operation of the endoscopic device in Figure 5 may correspond to step S430b in Figure 4B. Such relative pose information may refer to information for moving from a first pose, which is the current pose of the tip, to a second pose for effectively capturing an identified body part.

Referring to FIG. 5, in step S510, the endoscope device may identify brightness differences in the image based on the acquired image and brightness information related to the illumination. The endoscope device may analyze the difference between the stored brightness information related to the illumination and the brightness information of the image captured by the image sensor and the illumination. The endoscope device may also identify brightness patterns indicated by the characteristics of internal tissue, the intensity and direction of the illumination.

In one embodiment, an endoscopic device may calculate relative pose information based on the identified brightness difference in step S520. The endoscopic device may estimate the distance between the tip and a specific body part at the current pose of the tip based on the identified brightness difference, and may calculate position information for the tip to move and direction information for the image sensor provided in the tip to capture images based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part. The relative pose information may also include the path, direction, and required angle change of the tip through the inside of the body.

In one embodiment, an endoscopic device can project a specific pattern of light based on structured light, rather than just brightness differences, and calculate relative pose information based on changes in the pattern. For example, the endoscopic device can estimate distance based on changes in the pattern, and calculate position information for the movement of the tip and direction information for the image sensor equipped at the tip to capture images based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part.

In one embodiment, an endoscopic device uses multiple image sensors, analyzes the pixel difference between two images, and can calculate the distance from the tip to the lesion.

Figure 6 is a flowchart illustrating the operation of calculating relative pose information of an endoscopic device according to another embodiment of the present invention. The operation of the endoscopic device in Figure 6 may correspond to step S430b in Figure 4B.

Referring to FIG. 6, the endoscope device may acquire a first image at a first location in step S610.

In another embodiment, the endoscopic device may acquire a second image at a second location in step S620.

In another embodiment, an endoscopic device may calculate relative pose information in step S630 based on the change in the angle of the tip and the change in pixels on the screen while moving from the first point to the second point.

For example, an endoscope device may analyze an input image using convolution operations. The endoscope device may calculate the location of identified body parts using bounding box coordinates based on a pre-trained model that is trained by extracting features from the input image through various convolution layers. The coordinates consist of left, top, right, and bottom, and the output bounding box coordinates may be converted to the center coordinates of the bounding box through post-processing.

The center coordinates of such a bounding box can be expressed as Equation 2.
refers to the center coordinate on the x-axis, taking the average value of the boundaries between the left (l) and right (r),
indicates the center coordinate on the y-axis, taking the average value of the upper (t) and lower (b) boundaries.

The endoscopic device can calculate the angle change (Δθ) of the tip while moving from the first point to the second point based on the encoder information, and can calculate the distance between the identified body part and the tip using the pixel change (ΔL) in the image.

This can be shown as in Equation 3.
where θ is the distance between the lesion and the tip, R is the radius of rotation of the tip, Δθ is the angular change of the endoscope tip, f is the focal length, ΔB is the displacement of the image sensor, and ΔL is the pixel change on the screen, which refers to the disparity of the same object on the two images. The endoscopic device may calculate a target angle to which the tip of the endoscope should move based on the calculated rotation angle using a polynomial trajectory. The endoscopic device may analyze the difference between the calculated target movement angle and the current angle of the tip and generate a control signal.

The operation of the endoscope device described above can be realized not only through a single image sensor, but also through stereo vision based on multiple image sensors. ΔB is also the distance between the image sensors.

The endoscopic device can estimate the distance and, based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part, calculate position information for the movement of the tip and directional information for the image sensor equipped at the tip to take images.

Figure 7 is a flowchart illustrating the operation of calculating relative pose information of an endoscopic device according to yet another embodiment of the present invention. The operation of the endoscopic device in Figure 7 may correspond to step S430b in Figure 4B.

Referring to FIG. 7, the endoscope device may acquire movement trajectory data of the distal end in step S710. Such movement trajectory data is data on the movement of the distal end inside the body, and may include at least one of sensor-based tracking data, such as a magnetic field sensor, gyroscope, and accelerometer, and image-based tracking data based on an image sensor.

In yet another embodiment, an endoscopic device may calculate relative pose information between the body part and the tip based on the movement trajectory of the tip and the acquired image in step S720. For example, the endoscopic device may calculate relative pose information for moving the tip from a first pose, which is the current position, to a second pose, which is the target position, based on position information based on the motor encoder value and a depth estimation algorithm.

Figure 8 is a flowchart illustrating operations related to image capture by an endoscopic device according to one embodiment of the present invention. The operations of the endoscopic device in Figure 8 may correspond to steps S440b through S460b in Figure 4B and steps S420c through S440c in Figure 4c.

Referring to FIG. 8, the endoscopic device may identify at least one pre-defined imaging position for the identified body part in step S810. Such at least one pre-defined imaging position is a predetermined position for imaging or treating the body part, and may refer to a position previously set by the user based on the user's clinical skills, or a position where structural information of the identified body part can be obtained.

In one embodiment, an endoscopic device may generate a control signal for controlling the rotation of the tip portion based on at least one imaging position and relative pose information in step S820. Specifically, the endoscopic device may calculate relative pose information between the body part and the endoscopic device, and generate a control signal for positioning the tip portion at at least one imaging position based on the calculated relative pose information and the distance and direction to at least one pre-set imaging position.

In one embodiment, the endoscopic device transmits a control signal to the driving unit in step S830, and the driving unit adjusts the tension of at least one wire based on the control signal, bends the bending portion, and controls the rotation angle of the tip portion.

In one embodiment, in step S840, when the distal end portion is positioned at each of at least one imaging point, the endoscopic device may capture an image based on an image sensor at the position of each of the at least one imaging point.

Figure 9 is a flowchart illustrating operations related to torque feedback in an endoscopic device according to one embodiment of the present invention. Such torque feedback can be a physical signal based on a drive control signal that replicates the forces generated when the tip contacts internal human tissue or can be based on a pre-trained model.

Referring to FIG. 9 , the endoscopic device may generate torque feedback related to the movement of the distal end in step S910. Such torque feedback may be generated based on a control signal related to image capture. The torque feedback is generated based on information related to the direction and amount of rotation of the distal end of the endoscopic device, and the movement distance and the magnitude of the torque feedback may have a positive correlation.

The endoscope device may transmit the torque feedback to the operation unit in step S920. For example, the endoscope device may control the distal end according to the rotation angle that must be controlled for image capture, and the bending steering unit may transmit torque feedback in the direction in which the distal end is steered, thereby providing the feedback to the user.

Figure 10 is a block diagram illustrating the block configuration of a computer device according to one embodiment of the present invention.

The computer device 1000 also includes a memory 1010 and a processor 1020. The computer device 1000 may be a separate device from the endoscope device or may be included in the control unit. It may execute one or more sets of instructions that cause any one or more of the methodologies described herein to be performed.

The memory 1010 may store a set of instructions, including system-related instructions that cause any one or more of the methodology functions described herein to be performed, and instructions related to the user interface. The memory 1010 temporarily or permanently stores data such as basic programs, applications, and configuration information for device operation. The memory 1010 may include RAM, ROM, and a permanent mass storage device such as a disk drive, but the present invention is not limited thereto. Such software components may be loaded from a computer-readable recording medium separate from the memory 1010 using a drive mechanism. Such separate computer-readable recording media may include computer-readable recording media such as a floppy drive, disk, tape, DVD (digital versatile disc)/CD-ROM (compact disc read only memory) drive, and memory card. In one embodiment, the software components may be loaded into the memory 1010 via a communication unit rather than a computer-readable recording medium. Additionally, the memory 1010 may provide stored data at the request of the processor 1020. According to an embodiment of the present invention, the memory 1010 may store configuration information.

The processor 1020 controls the overall operation of the computing device. The processor 1020 may also be configured to process instructions by performing basic arithmetic, logic, and input/output operations. The instructions may be provided to the processor 1020 by the memory 1010. For example, the processor 1020 may be configured to execute instructions received from program code stored in a storage device such as the memory 1010. For example, the processor 1020 may control the device to perform operations according to the various embodiments described above.

In accordance with one embodiment of the present invention, the processor 1020 may review an endoscopic image and tag or label specific body parts. Based on user input or the like, the computing device may label or classify parts of the upper gastrointestinal tract in the image with bounding boxes. For example, such upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum.

According to one embodiment of the present invention, the processor 1020 may generate a dataset containing labeled images. Such a dataset may also include a variety of lighting conditions, viewing angles, body part conditions, etc.

According to one embodiment of the present invention, the processor 1020 may use the generated dataset to train a neural network model. The computer device may select the neural network model and train the model based on the dataset. For example, such a model may be at least one of an object detection model and an object classification model, and may also include a convolutional neural network.

According to another embodiment of the present invention, the processor 1020 may generate a learning model using endoscopic images and control history information corresponding to the user's clinical skills. Such control history information may refer to control history information used to observe the endoscopic images and adjust the direction, angle, depth, etc. of the endoscope. The processor 1020 may acquire endoscopic images and control history information related to the upper gastrointestinal tract. For example, such upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum.

In other embodiments, the processor 1020 may generate a data set including endoscopic images and control history information.

In another embodiment, the processor 1020 may use the generated dataset to train a neural network model. The computing device may select the neural network model and train the model based on the dataset. For example, such models may include classification models such as a convolutional neural network (CNN) for image processing, and at least one of sequence processing models such as a recurrent neural network (RNN) and a long short-term memory (LSTM) network for sequence processing of control history information.

As mentioned above, even though the present embodiment has been described using limited embodiments and drawings, those skilled in the art will appreciate that various modifications and variations may be made from the foregoing description. For example, the described techniques may be performed in a different order than described, and/or the described components, such as systems, structures, devices, and circuits, may be combined or fused in a different manner than described, or may be substituted or substituted by other components or equivalents, and still achieve suitable results.

Therefore, other implementations, other embodiments, and equivalents of the claims are intended to fall within the scope of the claims.

100: Endoscope device 110: Output unit 120: Control unit 130: Drive unit 140: Scope 141: Insertion unit 142: Bending unit 143: Distal end portion 151: Image sensor 152: Nozzle 153: Lighting 154: Lens 155: Working channel 160: Operation unit 161: Bending and steering unit 200: Motor 210: First wire 220: Second wire 1000: Computer device 1010: Memory 1020: Processor

Claims (20)

1. A control method for an endoscope apparatus, comprising:
acquiring an image relating to the upper gastrointestinal tract from an image sensor;
detecting at least one first body part from the image based on a pre-trained model;
calculating relative position information between the sensed first body part and the tip;
generating a first control signal for steering the first body part relative to the first body part based on the relative position information;
transmitting the first control signal to a driver. The step of acquiring an image relating to the upper gastrointestinal tract comprises:
acquiring a first image at a first location;
acquiring a second image at a second location;
The step of calculating the relative position information includes:
2. The method of claim 1, further comprising: calculating a target rotation angle of the tip based on: i) an angular change of the tip between the first point and the second point; and ii) a change between the first image and the second image. identifying at least one second body part from the image based on the pre-trained model;
calculating relative pose information between the identified second body part and a tip part;
generating a second control signal for imaging the second body part based on the relative pose information;
The method of claim 1 , further comprising: transmitting the second control signal to the driver. The step of calculating the relative pose information includes:
identifying brightness differences in the image based on the acquired image and brightness information related to lighting;
estimating a distance between the tip and the second body part based on the identified brightness difference;
and calculating the relative pose information based on the estimated distance. The step of acquiring an image relating to the upper gastrointestinal tract comprises:
acquiring a first image at a first location;
acquiring a second image at a second location;
The step of calculating the relative pose information includes:
i) estimating a distance between the tip and the second body part based on the change in angle of the tip between the first point and the second point, and ii) the change in angle between the first image and the second image;
and calculating the relative pose information. The method of claim 3, wherein the pre-trained models include a classification model and a detection model trained on a dataset of labeled images of a first body part and a second body part related to the upper gastrointestinal tract. The step of generating the second control signal comprises:
identifying at least one image capture location corresponding to the identified second body part;
generating the second control signal based on the at least one photographing location and the relative pose information;
and capturing an image when the tip position corresponds to the at least one image capture location. The method of claim 1, further comprising adjusting the tension of at least one wire and controlling the rotation angle of the tip based on the first control signal. The method of claim 1, further comprising the step of: the drive unit generating torque feedback; and transmitting the torque feedback to the operating unit. The method of claim 9, wherein the torque feedback has a positive correlation with a target rotation angle that the tip must move due to the first control signal. In an endoscope device,
a tip portion having an image sensor;
a drive unit for controlling the rotation angle of the tip portion;
a control unit that detects at least one first body part from the image based on a pre-trained model, calculates relative position information between the detected first body part and the tip end, generates a first control signal for steering in correspondence with the first body part based on the relative position information, and transmits the first control signal to a drive unit. The control unit
capturing a first image at a first location and a second image at a second location;
The endoscope device according to claim 11, wherein a target rotation angle of the tip portion is calculated based on i) an angle change of the tip portion between the first point and the second point, and ii) a change between the first image and the second image. The control unit
12. The endoscope device according to claim 11, further comprising: identifying at least one second body part from the image based on the pre-trained model; calculating relative pose information between the identified second body part and the tip portion; generating a second control signal related to imaging of the second body part based on the relative pose information; and transmitting the second control signal to the drive unit. The control unit
The endoscopic device according to claim 13, further comprising: identifying a brightness difference in the image based on the acquired image and brightness information related to lighting; estimating a distance between the tip and the second body part based on the identified brightness difference; and calculating the relative pose information based on the estimated distance. The control unit
capturing a first image at a first location and a second image at a second location;
14. The endoscope device according to claim 13, wherein the distance between the tip and the second body part is estimated based on i) an angle change of the tip between the first point and the second point, and ii) a change between the first image and the second image, and the relative pose information is calculated. The endoscope device of claim 13, wherein the pre-trained models include a classification model and a detection model trained using a dataset of labeled images of a first body part and a second body part related to the upper gastrointestinal tract. The control unit
The endoscopic device according to claim 13, further comprising: identifying at least one imaging point corresponding to the identified body part; generating the second control signal based on the at least one imaging point and the relative pose information; and capturing an image when the position of the tip corresponds to the at least one imaging point. further comprising a curved portion connected to the tip portion;
The control unit
The endoscope device according to claim 11, wherein the drive unit is controlled based on the control signal, the tension of at least one wire connected to the drive unit is adjusted, and the rotation angle of the tip portion is adjusted based on the bending of the bending portion. Further including an operating unit having a bending steering unit,
The control unit
The endoscope device according to claim 11 , wherein torque feedback is generated based on the drive unit, and the torque feedback is transmitted to the bending steering unit. The endoscope device of claim 19, wherein the torque feedback has a positive correlation with the target rotation angle to which the distal end portion must move in response to the first control signal.