WO2025122777A1

WO2025122777A1 - Self-calibration of a multi-sensor system

Info

Publication number: WO2025122777A1
Application number: PCT/US2024/058715
Authority: WO
Inventors: Ma Luo; Kevin E. Mark; Mohammad MIRI; Jeannine E. ELLIOTT; Faisal I. Bashir; Dany JUNIO
Original assignee: Medtronic Navigation Inc
Current assignee: Medtronic Navigation Inc
Priority date: 2023-12-07
Filing date: 2024-12-05
Publication date: 2025-06-12
Anticipated expiration: 2026-06-07

Abstract

A system or method may include receiving first data about an environment from a first imaging device (704), receiving second data about the environment from a second imaging device (708), where a perspective of the environment from the second imaging device is different from a perspective of the environment from the first imaging device, identifying at least one common spatial point in the environment using both the first data and the second data (712), and performing a registration process for the first imaging device and/or the second imaging device using the at least one common spatial point (716).

Description

SELF-CALIBRATION OF A MULTI-SENSOR SYSTEM

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of and priority to U.S. Provisional Application No. 63/607,490 filed on December 7, 2023, entitled “SELF-CALIBRATION OF A MULTI-SENSOR SYSTEM”, the entirety of which is hereby incorporated herein by reference.

FIELD OF INVENTION

[0002] The present disclosure is generally directed to image registration and utilizing image registration to facilitate calibration of one or more sensors in a multi-sensor system.

BACKGROUND

[0003] Imaging may be used by a medical provider for diagnostic and/or therapeutic purposes during a surgery or surgical procedure. A particular challenge faced with certain imaging devices is that they require calibration and may further require re-calibration during a surgical procedure. If re-calibration of an imaging device is required during a surgical procedure, the surgical procedure itself may be delayed, which may unnecessarily extend the surgical procedure. Another challenge with currently-available calibration techniques is that a physical apparatus, such as a checkerboard, is required to facilitate calibration. Requiring use of a physical apparatus for calibration imposes workflow burdens and difficulties on surgical staff.

[0004] Even target-free on-the-fly calibration of a stereo endoscope (e.g., multiple imaging sensors in a single housing) can suffer from the drawbacks described above. For instance, stereo endoscopes are re-processed between procedures resulting in calibration parameter shift. Sub-optimal calibration parameters result in inaccurate registration of subtissue structure on an organ surface. Re-calibrating the endoscope in the operating room will result in a delay of care.

BRIEF SUMMARY

[0005] A calibration procedure is proposed herein that overcomes many, if not all, of the above-mentioned shortcomings. In some embodiments, a calibration solution is proposed in which computer vision is coupled with machine learning. Specifically, but without limitation, one or more common spatial points are observed from multiple different perspectives. The one or more common spatial points may be observed from different imaging devices or similar types of imaging sensors. Alternatively or additionally, the one or more common spatial points may be observed by a single device with multiple imaging sensors (e.g., a multi-camera imaging system). One such example of a single device with multiple imaging sensors is a stereo endoscope, which is ubiquitous in Robotic Assisted Surgical (RAS) systems. By observing the same spatial point or a collection of spatial points (e.g., two, three, four, or more spatial points) from multiple imaging devices, spatial transforms or relations between and among the imaging devices can be computed to facilitate calibration of intrinsic and/or extrinsic calibration parameters of the multicamera imaging system. While examples will be described in connection with observing one or more common spatial points, it should be appreciated that embodiments of the present disclosure contemplate using one or a plurality of common spatial points to facilitate calibration. Illustratively and without limitation, the plurality of common spatial points may be distributed across a line, a surface, a plurality of lines, or a plurality of surfaces of an object. Utilizing more spatial points to facilitate calibration may support a more accurate calibration process, but it should be appreciated that embodiments contemplated herein can function with one common spatial point or a collection of common spatial points, lines, or surfaces without departing from the scope of the present disclosure.

[0006] In some embodiments, the imaging device(s) may include depth cameras, visible light cameras, Red Green Blue (RGB) cameras, infrared cameras, ultrasound devices, endoscopes, or any other type of image sensing device. The one or more common spatial points may be detecting using traditional image processing techniques and/or via machine learning, which can yield usable information (e.g., human pose estimations, placements of navigational and/or surgical systems) from multiple sensors simultaneously in a single coordinate space (e.g., a three dimensional (3D) coordinate space).

[0007] Embodiments of the present disclosure can provide a solution for touchless registration of a surgical navigation system. Alternatively or additionally, the calibration techniques proposed herein may be used to support assisted port placement for an endoscopic robot. A multi-sensor setup as depicted and described herein can significantly improve the field of view in any surgical procedure using two or more imaging devices. Furthermore, the quantity of input data may be increased to help improve the delivery of therapy and to improve the experience of the patient and surgical staff alike.

[0008] In some embodiments, a system is provided that includes a computer vision system. The computer vision system may include: (a) multiple sensors (e.g., depth camera, RGB camera, etc.) that can obtain data such as but not limited to depth data, RGB data, infrared data, ultrasound data, etc.; (b) the sensors can be placed at different angles to reduce line of sight issues; (c) estimating physical key points such as facial landmarks and human pose estimations with traditional as well as machine learning methods.

[0009] The system may also include a registration unit that is capable of relating one or more physical/spatial points via a transform, thereby facilitating self-calibration for the imaging device(s) used within the system. The registration process may be repeated for different poses to further enhance the accuracy of transform estimation. Using the transform(s) above to relate the spatial coordinates and data among different sensors to fuse data, the system proposed herein can also reduce line of sight issues as well as broaden the overall system capability by increasing the total amount of imaging data available to a surgical navigation system.

[0010] Example aspects of the present disclosure include a system, including: a first imaging device to collect first data about an environment; a second imaging device to collect second data about the environment from a perspective that differs from the first imaging device; a processor; and a memory coupled to the processor and storing data thereon that, when processed by the processor, enables the processor to: receive the first data and the second data; identify at least one common spatial point in the environment using both the first data and the second data; and perform a registration process for the first imaging device and/or the second imaging device using the at least one common spatial point during the registration process.

[0011] In some embodiments, the registration process relates the first data and the second data for at least one common spatial point via a transformation matrix, a mathematical model, or a machine learning-based model comprising at least one of a set of transformation matrices and a dense displacement field representing a non-linear relation between the first data and the second data.

[0012] In some embodiments, the memory stores further data for processing by the processor that, when processed, enables the processor to: perform a registration process by computing the transformation matrix, reprojecting the detected features from first imaging device into the second imaging device and vice versa, and computing the reprojection error, and computing the root means square value of all such reprojections for all detected features in both imaging devices and repeat the registration process until the reprojection error from the optimized transformation matrix is no longer improved and is better than reprojection error provided by a predetermined number of previous transformation matrices [0013] In some embodiments, the first imaging device comprises at least one of a depth camera, a visible light camera, a Red Green Blue (RGB) camera, an infrared camera, an ultrasound device, and a stereo or monocular endoscope.

[0014] In some embodiments, the memory stores further data for processing by the processor that, when processed, enables the processor to: automatically calibrate at least one of the first imaging device and the second imaging device.

[0015] In some embodiments, an operating parameter of the first imaging device and/or the second imaging device is defined and set when automatically calibrated.

[0016] In some embodiments, the common spatial features comprise: features on anatomical elements of a patient or keypoints on surgical tools or 6D pose of the CAD model of the surgical tools.

[0017] In some embodiments, the at least one common spatial point comprises a point on a surgical navigation tracker.

[0018] In some embodiments, the at least one common spatial point comprises a point on a surgical robot and wherein the surgical robot comprises one or more detachable arms having their own reference frame.

[0019] In some embodiments, the memory stores further data for processing by the processor that, when processed, enables the processor to: register the first data and the second data in a common coordinate space.

[0020] In some embodiments, the memory stores further data for processing by the processor that, when processed, enables the processor to: develop one or more estimated transform matrices to relate the at least one common spatial point between the first data and the second data; fuse the first data and the second data to enable simultaneous use of the first imaging device and the second imaging device during a surgical procedure; and provide the one or more estimate transform matrices to a machine learning model as training data.

[0021] According to other aspects, a system is provided that includes: a processor; and a memory coupled to the processor and storing data thereon that, when processed by the processor, enables the processor to: receive first data from a first imaging device; receive second data from a second imaging device; identify at least one common spatial point in the first data and the second data; and register the first data with the second data using the at least one common spatial point, wherein the first data is registered with the second data by applying a transformation matrix to the first data that moves the first data into a coordinate system of the second data. [0022] In some embodiments, at least one of the first imaging device and the second imaging device comprises at least one of a depth camera, a visible light camera, a Red Green Blue (RGB) camera, an infrared camera, an ultrasound device, and an endoscope. [0023] In some embodiments, the memory stores further data for processing by the processor that, when processed, enables the processor to: automatically calibrate at least one of the first imaging device and the second imaging device.

[0024] In some embodiments, both the first imaging device and the second imaging device are calibrated via the transformation matrix, a mathematical model, or a machine learning-based model comprising at least one of a set of transformation matrices and a dense displacement field representing a non-linear relation between the first data and the second data.

[0025] According to other aspects, a method is provided that includes: receiving first data about an environment from a first imaging device; receiving second data about the environment from a second imaging device, wherein a perspective of the environment from the second imaging device is different from a perspective of the environment from the first imaging device; identifying at least one common spatial point in the environment using both the first data and the second data; and performing a registration process for the first imaging device and/or the second imaging device using the at least one common spatial point.

[0026] In some embodiments, the registration process relates the first data and the second data for at least one common spatial point via a transformation matrix.

[0027] In some embodiments, at least one of the first imaging device and the second imaging device comprises at least one of a depth camera, a visible light camera, a Red Green Blue (RGB) camera, an infrared camera, an ultrasound device, and an endoscope. [0028] In some embodiments, the at least one common spatial point comprises one or more of a point on an anatomical element of a patient, a point on a surgical navigation tracker, a feature on one or more surgical tools, and a point on one or more arms of a surgical robot.

[0029] In some embodiments the method further includes: calibrating at least one of the first imaging device and the second imaging device using the at least one common spatial point.

[0030] Any aspect in combination with any one or more other aspects.

[0031] Any one or more of the features disclosed herein.

[0032] Any one or more of the features as substantially disclosed herein. [0033] Any one or more of the features as substantially disclosed herein in combination with any one or more other features as substantially disclosed herein.

[0034] Any one of the aspects/features/embodiments in combination with any one or more other aspects/features/embodiments.

[0035] Use of any one or more of the aspects or features as disclosed herein.

[0036] It is to be appreciated that any feature described herein can be claimed in combination with any other feature(s) as described herein, regardless of whether the features come from the same described embodiment.

[0037] The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims.

[0038] The phrases “at least one”, “one or more”, and “and/or” are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions “at least one of A, B and C”, “at least one of A, B, or C”, “one or more of A, B, and C”, “one or more of A, B, or C” and “A, B, and/or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together. When each one of A, B, and C in the above expressions refers to an element, such as X, Y, and Z, or class of elements, such as Xl-Xn, Yl-Ym, and Zl-Zo, the phrase is intended to refer to a single element selected from X, Y, and Z, a combination of elements selected from the same class (e.g., XI and X2) as well as a combination of elements selected from two or more classes (e.g., Y1 and Zo).

[0039] The term “a” or “an” entity refers to one or more of that entity. As such, the terms “a” (or “an”), “one or more” and “at least one” can be used interchangeably herein. It is also to be noted that the terms “comprising”, “including”, and “having” can be used interchangeably.

[0040] The preceding is a simplified summary of the disclosure to provide an understanding of some aspects of the disclosure. This summary is neither an extensive nor exhaustive overview of the disclosure and its various aspects, embodiments, and configurations. It is intended neither to identify key or critical elements of the disclosure nor to delineate the scope of the disclosure but to present selected concepts of the disclosure in a simplified form as an introduction to the more detailed description presented below. As will be appreciated, other aspects, embodiments, and configurations of the disclosure are possible utilizing, alone or in combination, one or more of the features set forth above or described in detail below.

[0041] Numerous additional features and advantages of the present disclosure will become apparent to those skilled in the art upon consideration of the embodiment descriptions provided hereinbelow.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS [0042] The accompanying drawings are incorporated into and form a part of the specification to illustrate several examples of the present disclosure. These drawings, together with the description, explain the principles of the disclosure. The drawings simply illustrate preferred and alternative examples of how the disclosure can be made and used and are not to be construed as limiting the disclosure to only the illustrated and described examples. Further features and advantages will become apparent from the following, more detailed, description of the various aspects, embodiments, and configurations of the disclosure, as illustrated by the drawings referenced below.

[0043] Fig. 1 A is a block diagram of a system according to at least one embodiment of the present disclosure;

[0044] Fig. IB is a block diagram illustrating an alternative configuration of the system illustrated in Fig. 1 A;

[0045] Fig. 2 illustrates components of a system in a surgical environment in accordance with at least one embodiment of the present disclosure;

[0046] Fig. 3 illustrates an object subjected to imaging by two different imaging devices in accordance with at least one embodiment of the present disclosure;

[0047] Fig. 4 A illustrates an image of an object from a perspective of a first imaging device in accordance with at least one embodiment of the present disclosure;

[0048] Fig. 4B illustrates an image of the object from a perspective of a second imaging device in accordance with at least one embodiment of the present disclosure;

[0049] Fig. 5 illustrates two point clouds in a common coordinate system in accordance with at least one embodiment of the present disclosure;

[0050] Fig. 6 illustrates a transformation matrix in accordance with at least one embodiment of the present disclosure;

[0051] Fig. 7 is a flowchart illustrating a first method in accordance with at least one embodiment of the present disclosure;

[0052] Fig. 8 is a flowchart illustrating a second method in accordance with at least one embodiment of the present disclosure; [0053] Fig. 9 is a flowchart illustrating a third method in accordance with at least one embodiment of the present disclosure; and

[0054] Fig. 10 is a flowchart illustrating a fourth method in accordance with at least one embodiment of the present disclosure.

DETAILED DESCRIPTION

[0055] It should be understood that various aspects disclosed herein may be combined in different combinations than the combinations specifically presented in the description and accompanying drawings. It should also be understood that, depending on the example or embodiment, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, and/or may be added, merged, or left out altogether (e.g., all described acts or events may not be necessary to carry out the disclosed techniques according to different embodiments of the present disclosure). In addition, while certain aspects of this disclosure are described as being performed by a single module or unit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of units or modules associated with, for example, a computing device and/or a medical device.

[0056] In one or more examples, the described methods, processes, and techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Alternatively or additionally, functions may be implemented using machine learning models, neural networks, artificial neural networks, or combinations thereof (alone or in combination with instructions). Computer-readable media may include non-transitory computer-readable media, which corresponds to a tangible medium such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0057] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple Al l, A12, A12X, A12Z, or A13 Bionic processors; an ARM processor, or any other general purpose microprocessors), graphics processing units (e.g., Nvidia GeForce RTX 2000-series processors, Nvidia GeForce RTX 3000-series processors, Nvidia AGX-series SOM's, Nvidia ClaraHoloscan based platforms, AMD Radeon RX 5000-series processors, AMD Radeon RX 6000-series processors, or any other graphics processing units), application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0058] Before any embodiments of the disclosure are explained in detail, it is to be understood that the disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The disclosure is capable of other embodiments and of being practiced or of being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Further, the present disclosure may use examples to illustrate one or more aspects thereof. Unless explicitly stated otherwise, the use or listing of one or more examples (which may be denoted by “for example,” “by way of example,” “e.g.,” “such as,” or similar language) is not intended to and does not limit the scope of the present disclosure.

[0059] During surgeries or surgical procedures, imaging device(s) may be used to support the surgeon or other operating room personnel. The imaging device(s) may be calibrated prior to use during the surgical procedure. Conditions may exist which disrupt the calibration of the imaging device(s) during the surgical procedure, which requires a recalibration of the imaging device(s). Maintaining an accuracy/calibration of the imaging device(s) is particularly important when the imaging device(s) are used in connection with supporting semi-automated or fully-automated surgical navigation. Maintaining an accuracy/calibration of the imaging device(s) is also important when the imaging device(s) are used to support robot-assisted surgical procedures (e.g., because machine vision is utilized during the surgical procedure to ensure a surgical plan is followed). Accurate imaging device(s) also help support robot-assisted port placement.

[0060] Frustratingly, calibration and re-calibration of imaging device(s) may require attention of operating room personnel, which may delay a surgical procedure. Further still, some calibration or re-calibration processes require use of a physical apparatus, such as a checkerboard, which imposes workflow burdens and difficulties. Instead of focusing on the patient or the surgical procedure, operating room personnel or surgeons are required to focus on calibrating or re-calibrating the imaging device(s).

[0061] Embodiments of the present disclosure contemplate a solution to the calibration and re-calibration issues mentioned above. Specifically, embodiments of the present disclosure provide a solution using computer vision with machine learning, whereby observing the same spatial points, spatial transforms or relations between and among imaging device can be computed (e.g., self-calibration). As will be described herein, the computer vision may be powered by depth cameras, visible light cameras, RGB cameras, infrared cameras, ultrasound devices, endoscopes (e.g., stereo or monocular endoscopes), and/or other possible sensing technologies. One or more common spatial points may be detected using traditional imaging processing techniques or via machine learning, which can yield usable information (e.g., human pose estimations, placements of navigational and/or surgical systems) from multiple imaging devices simultaneously in a single 3D coordinate space. Processing sensor data from multiple angles will improve the robustness of a system of multiple cameras to occlusions by reducing the issue of line of sight. Additionally, by combining data from different imaging devices, the overall image richness (quality) and image coverage (quantity) is improved.

[0062] An advantage of utilizing multiple imaging devices as described herein is that there may be a reduction of the hardware built-in error. When there are multiple sensors from different angles, the overall error for the whole system can be affected by each sensor's hardware however the built-in sensor's hardware error is usually in the form of Normal distribution and having multiple sensors for the whole system will create nonoverlapping Normal curves for the error which could reduce the overall system error. [0063] Embodiments of the present disclosure provide technical solutions to one or more of the problems of (1) cumbersome calibration or re-calibration of imaging devices, (2) inaccurate registration, and (3) inaccurate correspondence matching.

[0064] Turning first to Figs. 1 A and IB, various configurations of a system 100 will be described according to at least some embodiments of the present disclosure. The system 100 may be used to register a patient to a surgical navigation system coordinate system; to register patient exam data to a camera or other imaging device; to control, pose, and/or otherwise manipulate a surgical mount system and/or surgical tools attached thereto; to support automated/robotic port placement; and/or to carry out one or more other aspects of one or more of the methods disclosed herein. The system 100 is illustrated to include a computing device 102, one or more imaging devices 112, a robot 114, a navigation system 118, a database 134, and/or a cloud or other network 136. Systems according to other embodiments of the present disclosure may comprise more or fewer components than the system 100. For example, the system 100 may not include the robot 114, one or more components of the computing device 102, the database 134, and/or the cloud 136.

[0065] The computing device 102 is illustrated to include a processor 104, a memory 106, a communication interface 108, and a user interface 110. Computing devices according to other embodiments of the present disclosure may comprise more or fewer components than the computing device 102.

[0066] The processor 104 of the computing device 102 may be any processor described herein or any similar processor. The processor 104 may be configured to execute instructions stored in the memory 106, which instructions may cause the processor 104 to carry out one or more computing steps utilizing or based on data received from the imaging device 112, the robot 114, the navigation system 118, the database 134, and/or the cloud 136. The processor 104 may be or comprise one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple Al l, A12, A12X, A12Z, or A13 Bionic processors; or any other general purpose microprocessors), graphics processing units (e.g., Nvidia GeForce RTX 2000-series processors, Nvidia GeForce RTX 3000-series processors, AMD Radeon RX 5000-series processors, AMD Radeon RX 6000-series processors, or any other graphics processing units), application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.

[0067] The memory 106 may be or comprise RAM, DRAM, SDRAM, other solid-state memory, any memory described herein, or any other tangible, non-transitory memory for storing computer-readable data and/or instructions. The memory 106 may store information or data useful for completing, for example, any step of the methods described herein, or of any other methods. The memory 106 may store, for example, instructions and/or machine learning models that support one or more functions of the computing device 102, the imaging devices 112, the navigation system 118, and/or the like. For instance, the memory 106 may store content (e.g., instructions and/or machine learning models) that, when executed by the processor 104, enable image processing 120, segmentation 122, transformation 124, registration 126, and/or self-calibration 128. Such content, if provided as in instruction, may, in some embodiments, be organized into one or more applications, modules, packages, layers, or engines. Alternatively or additionally, the memory 106 may store other types of content or data (e.g., machine learning models, artificial neural networks, deep neural networks, etc.) that can be processed by the processor 104 to carry out the various method and features described herein. Thus, although various contents of memory 106 may be described as instructions, it should be appreciated that functionality described herein can be achieved through use of instructions, algorithms, and/or machine learning models. The data, algorithms, and/or instructions may cause the processor 104 to manipulate data stored in the memory 106 and/or received from or via the imaging device 112, the robot 114, the database 134, and/or the cloud 136. [0068] The communication interface 108 may be used for receiving image data or other information from an external source (such as the imaging device 112, the robot 114, the navigation system 118, the database 134, the cloud 136, and/or any other system or component not part of the system 100), and/or for transmitting instructions, images, or other information to an external system or device (e.g., another computing device 102, the imaging device 112, the robot 114, the navigation system 118, the database 134, the cloud 136, and/or any other system or component not part of the system 100). The communication interface 108 may comprise one or more wired interfaces (e.g., a USB port, an Ethernet port, a Firewire port) and/or one or more wireless transceivers or interfaces (configured, for example, to transmit and/or receive information via one or more wireless communication protocols such as 802.1 la/b/g/n, Bluetooth, NFC, ZigBee, and so forth). In some embodiments, the communication interface 108 may be useful for enabling the device 102 to communicate with one or more other processors 104 or computing devices 102, whether to reduce the time needed to accomplish a computing-intensive task or for any other reason.

[0069] The computing device 102 may also comprise one or multiple user interfaces 110. The user interface(s) 110 may be or comprise a keyboard, mouse, trackball, monitor, television, screen, touchscreen, and/or any other device for receiving information from a user and/or for providing information to a user. The user interface(s) 110 may be used, for example, to receive a user selection or other user input regarding any step of any method described herein. Notwithstanding the foregoing, any required input for any step of any method described herein may be generated automatically by the system 100 (e.g., by the processor 104 or another component of the system 100) or received by the system 100 from a source external to the system 100. In some embodiments, the user interface 110 may be useful to allow a surgeon or other user to modify instructions to be executed by the processor 104 according to one or more embodiments of the present disclosure, and/or to modify or adjust a setting of other information displayed on the user interface 110 or corresponding thereto.

[0070] Although the user interface 110 is shown as part of the computing device 102, in some embodiments, the computing device 102 may utilize a user interface 110 that is housed separately from one or more remaining components of the computing device 102. In some embodiments, the user interface 110 may be located proximate one or more other components of the computing device 102, while in other embodiments, the user interface 110 may be located remotely from one or more other components of the computer device 102.

[0071] The imaging device(s) 112 may be operable to image anatomical feature(s) (e.g., a bone, veins, tissue, organs, nerves, etc.) and/or other aspects of patient anatomy to yield image data (e.g., image data depicting or corresponding to a bone, veins, tissue, morgan, nerve, etc.). “Image data” as used herein refers to the data generated or captured by an imaging device 112 or similar sensor (e.g., first sensor(s) 142 and/or second sensor(s) 144), including in a machine-readable form, a graphical/visual form, and in any other form. In some cases, the image data may be or comprise first image data 130, second image data 132, first sensor data 146, and/or second sensor data 148 generated by one or more 3D imaging device(s) (e.g., an O-arm, a C-arm, a G-arm, a CT scanner, a depth camera, etc.), one or more 2D imaging device(s) (e.g., an emitter/detector pair, an endoscope, a visible light camera, an RGB camera, an infrared camera, an ultrasound device, etc.), one or more image pickup sensors, one or more proximity sensors, one or more motion sensors, combinations thereof, and the like.

[0072] In various examples, the image data may comprise data corresponding to an anatomical feature of a patient, or to a portion thereof. Alternatively or additionally, the image data may comprise data corresponding to a surgical navigation tracker, an environment of an operating room, or the like. The image data may be or comprise a preoperative image, an intraoperative image, a postoperative image, or an image taken independently of any surgical procedure. The imaging device(s) 112, including the first imaging device(s) 138 and/or second imaging device(s) 140, may be capable of taking a 2D image or a 3D image to yield the image data. The imaging device(s) 112 may be or comprise, for example, a stereo camera, an ultrasound scanner (which may comprise, for example, a physically separate transducer and receiver, or a single ultrasound transceiver), an O-arm, a C-arm, a G-arm, or any other device utilizing X-ray -based imaging (e.g., a fluoroscope, a CT scanner, or other X-ray machine), a CBCT imaging device, a magnetic resonance imaging (MRI) scanner, an optical coherence tomography (OCT) scanner, an endoscope, a microscope, an optical camera, a thermographic camera (e.g., an infrared camera), a radar system (which may comprise, for example, a transmitter, a receiver, a processor, and one or more antennae), or any other imaging device 112 suitable for obtaining images of an anatomical feature of a patient. The imaging device(s) 112 may be contained entirely within a single housing, or may comprise a transmitter/emitter and a receiver/ detector that are in separate housings or are otherwise physically separated. In some embodiments, a first imaging device 112 may be used to obtain first image data 130 (e.g., a first image), and a second imaging device 112 may be used to obtain second image data 132 (e.g., a second image). The first image data 130 and the second image data 132 may be in the same format or different formats, depending upon the capabilities of the first imaging device 138 and second imaging device 140.

[0073] In some examples (and as discussed in further detail below) the first imaging device 138 and/or second imaging device 140 can be an imaging device that is used for navigation (e.g., in conjunction with the navigation system 118). The first imaging device may be or comprise, for example, any of the example imaging devices described above (an ultrasound scanner, an O-arm, a C-arm, a G-arm, or any other device utilizing X-ray-based imaging, a magnetic resonance imaging scanner, an OCT scanner, an endoscope, a microscope, an optical camera, a thermographic camera, a radar system, a stereo camera, etc.). Alternatively or additionally, the first imaging device 138 and/or second imaging device 140 may be an imaging device used for registration (e.g., to facilitate alignment of patient scan data with the patient in the surgical environment). Each of the first imaging device 138 and second imaging device 140 may be configured to be registered to one another and, in some embodiments, may support self-calibration of the other imaging device.

[0074] The imaging device(s) 112 (e.g., the first imaging device 138 and/or second imaging device 140) may be operable to generate image data in the form of still images and/or a stream of image data. For example, the imaging device(s) 112 may be configured to operate with an open shutter, or with a shutter that continuously alternates between open and shut so as to capture successive images. For purposes of the present disclosure, unless specified otherwise, image data may be considered to be continuous and/or provided as an image data stream if the image data represents two or more frames per second. In some embodiments, reference markers (e.g., navigation markers) may be placed on the imaging device(s) 112 and/or any other object in the surgical space. The reference markers may be tracked by the navigation system 118, and the results of the tracking may be used by an operator of the system 100 or any component thereof.

[0075] In some embodiments, the imaging device(s) 112 may be or comprise a stereo camera, a depth camera, a visible light camera, an RGB camera, an infrared camera, an ultrasound camera, and/or an endoscope. Imaging device(s) 112 may include one or many image sensors. In situations where the imaging device(s) 112 includes multiple image sensors, each image sensor may generate and send image information to the computing device 102, which may use image processing 120 to generate an image (e.g., image data) from the image information generated by the image sensor. The image sensors may be physically spaced apart or otherwise separated from one another in the imaging device(s) 112, such that each image sensor captures a different view of an object imaged by the imaging device(s) 112.

[0076] The robot 114 may be any surgical robot or surgical robotic system. The robot 114 may be or comprise, for example, the Mazor X™ Stealth Edition robotic guidance system. Alternatively or additionally, the robot 114 may be or comprise the Hugo™ RAS system. The robot 114 may be configured to position the imaging device(s) 112 at one or more precise position(s) and orientation(s), and/or to return the imaging device(s) 112 to the same position(s) and orientation(s) at a later point in time. The robot 114 may additionally or alternatively be configured to manipulate a surgical tool (whether based on guidance from the navigation system 118 or not) to accomplish or to assist with a surgical task (e.g., assisted port placement, implant placement, therapy delivery, etc.). In some embodiments, the robot 114 may be configured to hold and/or manipulate an anatomical element during or in connection with a surgical procedure. The robot 114 may comprise one or more robotic arms 116. In some embodiments, the robotic arm 116 may comprise a first robotic arm and a second robotic arm, though the robot 114 may comprise more than two robotic arms. In some embodiments, one or more of the robotic arms 116 may be used to hold and/or maneuver the imaging device 112. In embodiments where the imaging device 112 comprises two or more physically separate components (e.g., a transmitter and receiver), one robotic arm 116 may hold one such component, and another robotic arm 116 may hold another such component. Each robotic arm 116 may be positionable independently of the other robotic arm. The robotic arms 116 may be controlled in a single, shared coordinate space, or in separate coordinate spaces. Alternatively or additionally, the robot 114 may include one or more endoscopes and endoscopic devices. [0077] The robot 114, together with the robotic arm 116, may have, for example, one, two, three, four, five, six, seven, or more degrees of freedom. Further, the robotic arm 116 may be positioned or positionable in any pose, plane, and/or focal point. The pose includes a position and an orientation. As a result, an imaging device(s) 112, surgical tool, or other object held by the robot 114 (or, more specifically, by the robotic arm 116) may be precisely positionable in one or more needed and specific positions and orientations. [0078] The robot 114 may include a robot with multiple detached arms 116 in their own reference frames. The multiple imaging devices 112 might be mounted on each of the detached arm or on the tower or on an operating room ceiling, etc. The system 100 may be configured to detect the keypoints on each of the detached robotic arms 116 from each of the imaging sensors. The system 100 may use the correspondence of the same keypoints or features viewed from different camera perspectives to optimize the intrinsic and extrinsic camera calibration parameters.

[0079] The robotic arm(s) 116 may comprise one or more sensors that enable the processor 104 (or a processor of the robot 114) to determine a precise pose in space of the robotic arm (as well as any object or element held by or secured to the robotic arm). Alternatively or additionally, as illustrated in Fig. IB, first sensor(s) 142 and/or second sensor(s) 144 may be part of the robot 114 or may be connected to the robot as part of an endoscope.

[0080] In some embodiments, reference markers (e.g., navigation markers) may be placed on the robot 114 (including, e.g., on the robotic arm 116), the imaging device(s) 112, or any other object in a surgical space. The reference markers may be tracked by the navigation system 118, and the results of the tracking may be used by the robot 114 and/or by an operator of the system 100 or any component thereof. In some embodiments, the navigation system 118 can be used to track other components of the system (e.g., imaging device(s) 112) and the system can operate without the use of the robot 114 (e.g., with the surgeon manually manipulating the imaging device(s) 112 and/or one or more surgical tools, based on information and/or instructions generated by the navigation system 118, for example). In some embodiments, the system may work without placing any reference markers on the robot 114. In this case, the system may detect the keypoints, skeleton or 6D pose of the robotic arms or physical objects in the scene from imaging device(s) 112. The system may use the detected features from imaging device(s) 112 to optimize the intrinsic and extrinsic calibration parameters.

[0081] The navigation system 118 may provide navigation during an operation. The navigation system 118 may be any now-known or future-developed navigation system, including, for example, the Medtronic Stealth Station™ S8 surgical navigation system or any successor thereof. The navigation system 118 may utilize information from the imaging device(s) 112 or other sensor(s) 142, 144 for tracking one or more reference markers, navigated trackers, patient anatomy, or other objects within the operating room or other room in which some or all of the system 100 is located. In some embodiments, the navigation system 118 may comprise one or more electromagnetic sensors. In various embodiments, the navigation system 118 may be used to track a position and orientation (e.g., a pose) of the imaging device(s) 112, the robot 114 and/or the robotic arm 116, a surgeon, the patient, patient anatomy, and/or one or more surgical tools (or, more particularly, to track a pose of a surgical navigation tracker attached, directly or indirectly, in fixed relation to the one or more of the foregoing). The navigation system 118 may include a display for displaying one or more images from an external source (e.g., the computing device 102, imaging device 112, or other source) or for displaying an image and/or video stream from the one or more cameras or other sensors of the navigation system 118. In some embodiments, the system 100 can operate without the use of the navigation system 118. The navigation system 118 may be configured to provide guidance to a surgeon or other user of the system 100 or a component thereof or to any other element of the system 100 (e.g., the robot 114) regarding, for example, a pose of one or more anatomical elements, whether or not a tool is in the proper trajectory, and/or how to move a tool into the proper trajectory to carry out a surgical task according to a preoperative or other surgical plan.

[0082] The database 134 may store information that correlates one coordinate system to another (e.g., a patient coordinate system to a navigation coordinate system or vice versa). The database 134 may additionally or alternatively store, for example, one or more surgical plans (including, for example, pose information about a target and/or image information about a patient’s anatomy at and/or proximate the surgical site, for use by the robot 114, the navigation system 118, and/or a user of the computing device 102 or of the system 100); one or more images useful in connection with a surgery to be completed by or with the assistance of one or more other components of the system 100; and/or any other useful information. The database 134 may be configured to provide any such information to the computing device 102 or to any other device of the system 100 or external to the system 100, whether directly or via the cloud 136. In some embodiments, the database 134 may be or comprise part of a hospital image storage system, such as a picture archiving and communication system (PACS), a health information system (HIS), and/or another system for collecting, storing, managing, and/or transmitting electronic medical records including image data.

[0083] The cloud 136 may be or represent the Internet or any other wide area network (e.g., Amazon Web Services (AWS®), Microsoft Azure®, or other cloud-computing services). The computing device 102 may be connected to the cloud 136 via the communication interface 108, using a wired connection, a wireless connection, or both. In some embodiments, the computing device 102 may communicate with the database 134 and/or an external device (e.g., a computing device) via the cloud 136.

[0084] The system 100 or similar systems may be used, for example, to carry out one or more aspects of any of the methods 700, 800, 900, and/or 1000 described herein. The system 100 or similar systems may also be used for other purposes.

[0085] Referring now to Fig. 2, an example of a surgical environment in which components of the system 100 may be deployed will be described in accordance with at least some embodiments of the present disclosure. Specifically, the surgical environment 200 is shown to include a surgical space 212 in which the robot 114 may support a surgical procedure of a patient 220. The robot 114 may be operated with the assistance of the navigation system 118 and computing device 102. As mentioned above, image(s) of the surgical space 212 may be captured by the imaging device(s) 112, which may include the first imaging device 138 and/or second imaging device 140. Each imaging device 138, 140 may have a different field of view 232a, 232b, which may result in each imaging device providing a different perspective of the surgical space 212. In some embodiments, the different perspectives provided by each imaging device 138, 140 can be exploited to self-calibrate each imaging device 138, 140 and/or other sensors in the surgical environment 200. Calibration of imaging devices 138, 140 may be performed preoperatively, intraoperatively, or postoperatively without departing from the scope of the present disclosure. In some embodiments, a common spatial point or set of common spatial points/features (e.g., point(s), lines, edges, surfaces, or the 6D pose) may be identified in image(s) 204 obtained from the first imaging device 138 and second imaging device 140. The common spatial point may be used to perform a registration process for one or both imaging devices 138, 140, which may also facilitate calibration of one or both imaging devices 138, 140.

[0086] The surgical space 212 may correspond to a sterilized environment in which a surgical procedure is being performed. For instance, the surgical space 212 may have a table 216 on which a patient 220 lies during the surgical procedure. The robot 114 may be positioned near the table 216 and/or patient 220 to support the surgical procedure. One or more anatomical elements 224a-N of the patient may be subject to the surgical procedure. Illustratively, but without limitation, the anatomical elements 224a-N may include bony anatomical elements, organs, soft-tissue anatomical elements, etc. Alternatively or additionally, the surgical space 212 may correspond to an examination room.

[0087] As shown in Fig. 2, surgical navigation trackers 228 may be positioned at various locations in the surgical space 212. In some embodiments, the surgical navigation trackers 228 may be similar or identical to tracking devices placed on various components of system 100. The surgical navigation trackers 228 may correspond to objects having known geometric properties (e.g., size, shape, etc.) that are visible by the first imaging device 138 and/or second imaging device 140. Multiple surgical navigation trackers 228 may be attached to a common instrument (e.g., a tracking array) in a known pattern or relative configuration. In some embodiments, a tracking array having multiple surgical navigation trackers 228 may be attached to one or more objects such as a surgical instrument, the robot 114, a robotic arm 116, the table 216, an anatomical element 224a-N, or the like. [0088] The first imaging device 138 and/or second imaging device 140 may be configured to capture one or more images 204 of the surgical space 212. Such images may include some or all of the surgical navigation trackers 228 positioned within the surgical space 212. As shown in Fig. 2, the first imaging device 138 may have a first field of view 232a that intersects the surgical space 212 and the second imaging device 140 may have a second field of view 232b that intersects the surgical space 212, but is taken from a different point of origin. The first field of view 232a may intersect the second field of view 232b and one or more common spatial points may be captured by both fields of view 232a, 232b, albeit from different perspectives. In some embodiments, it may be desirable to have all surgical navigation trackers 228 located within both fields of view 232a, 232b. While only first and second imaging devices 138, 140 are illustrated, it should be appreciated that the system 100 may include two, three, four, or more imaging devices, sensors, or the like. [0089] During initial setup of the surgical space 212, the surgical navigation trackers 228 may be positioned within one, some or all fields of view 232a, 232b. One or multiple surgical navigation trackers 228 may correspond to a common spatial point in the surgical space 212. When viewed from different imaging devices 138, 140, the surgical navigation tracker(s) 228 may be used to register the first imaging device 138 with the second imaging device 140, or vice versa, using the image processing 120, segmentation 122, transformation 124, registration 126, and/or self-calibration 128. Multiple views of the surgical navigation tracker(s) 228 may also support self-calibration of the first imaging device 138 and/or second imaging device 140.

[0090] The first imaging device 138 and/or second imaging device 140 may be fixed in a predetermined location, may be connected to a moveable object (e.g., a cart or tripod), or may be moveable. Moveable versions of the first imaging device 138 and/or second imaging device 140 may be moveable under automated/robot 114 operation or may be moveable by a person. In either scenario, a moveable imaging device may be rotated and/or translated.

[0091] The images 204 captured by the imaging device(s) 112 may be static images or video images. In some aspects, the images 204 may be stored as a multimedia file 208 that includes video (or video and sound). While the images 204 will be described as optical images, it should be appreciated that any type of image 204 can be captured and the type of image 204 may depend on the type of imaging device 112 used to capture the image(s) 204. Non-limiting examples of image 204 types include optical images, x-ray images, CT images, MRI images, ultrasound images, infrared images, etc. The system 100 may support acquiring image data to generate or produce images (e.g., images 204, multimedia file 208, etc.) of the patient 220, the anatomical elements 224a-N, or other objects within the surgical space 212. Images 204 may include first image data 130, second image data 132, first sensor data 146, and/or second sensor data 148.

[0092] Referring now to Figs. 3, 4A, and 4B, additional aspects of the system 100 will be described in accordance with at least some embodiments of the present disclosure. As shown in Fig. 3, a first imaging device 304 and a second imaging device 308 may each capture an image (e.g., generate image data) of a common object 312. The first imaging device 304 may be or comprise the first imaging device 138 and/or the first sensor 142. The second imaging device 308 may be or comprise the second imaging device 140 and/or the second sensor 144.

[0093] The first imaging device 304 may capture first image data 130 of the object 312 using a first field of view 316. The second imaging device 308 may capture second image data 132 using a second field of view 320. In some embodiments, the object 312 may correspond to a common spatial point or set of common spatial points/features in the environment that is expressed by the first image data 130 and the second image data 132. In some embodiments, a specific location on the object 312 (e.g., a specific anatomical element or point on a specific anatomical element) may correspond to a common spatial point in the environment. Non-limiting examples of a common spatial point on the object 312 may include a tip of the nose, a corner of the mouth, a center of a pupil, or the like. Where the object 312 is not inclusive of human anatomy, the object 312 may include a surgical navigation tracker 228 and the common spatial point may correspond to a center of a surgical navigation tracker 228 (or a tracker array).

[0094] As shown in Figs. 4A and 4B, the first image data 130 may include a first image 404 of the object 312 from a first perspective whereas the second image data 132 may include a second image 408 of the object 312 from a second perspective or angle. One or more common points on the object 312 that are visible in the first image 404 and second image 408 may be used to relate the coordinate system of the first imaging device 304 with the coordinate system of the second imaging device 308. In some embodiments, a plurality of spatial points are detected using image processing 120. The transformation 124, registration 126, and self-calibration 128 may then be used to register the imaging devices 304, 308, and to calibrate one or both imaging devices 304, 308. When properly calibrated, images obtained from the imaging devices 304, 308, can also yield usable information for a surgical procedure (e.g., human pose estimations, placements of navigational and/or surgical systems), and such information can be used simultaneously in a single 3D coordinate space.

[0095] As mentioned above, registration 126 and self-calibration 128 may be facilitated by applying a transformation matrix that transforms the coordinate system of the first imaging device 304 and the coordinate system of the second imaging device 308 into a common coordinate system. The transformation matrix may be determined by execution of the transformation 124 instruction set/machine learning model(s).

[0096] As can be seen in Figs. 5 and 6, a transformation function 600 may include a transformation matrix 608 developed by the transformation 124, which may help register a first coordinate space 604 with a second coordinate space 612. In some embodiments, the transformation matrix 608 may be determined using one or multiple common spatial points identified in the first image data 130 and the second image data 132. As a nonlimiting example, common spatial point(s) may be identified by generating a first point cloud using the first image data 130 and a second point cloud using the second image data 132. The first and second point clouds may be represented as overlay ed point clouds 500 in common spatial coordinate system. The transformation 124 and registration 126 may cooperate with one another to adjust the overlay of the point clouds so that a maximum number of points in the first point cloud and the second point cloud overlap with one another. Alternatively or additionally, the transformation 124 and registration 126 may cooperate with one another to adjust the overlay of the point clouds so that a distance between points in the overlayed point clouds 500 is minimized. Whether maximizing or minimizing, the process of overlaying and testing different registrations may be iterated until an accuracy of the transformation matrix 608 is no longer improved as compared to an accuracy provided by a predetermined number of previous transformation matrices. [0097] Iteratively adjusting the overlay of the first point cloud and the second point cloud may ultimately result in the two point clouds being successfully overlayed with one another, which can then result in the determination of the transformation matrix 608. In some embodiments, registration 126 may relate the first image data 130 and the second image data 132 for one or more common spatial points via the transformation matrix 608, which may facilitate registration and self-calibration.

[0098] Referring now to Fig. 7, details of a method 700 for performing a registration process will be described in accordance with at least some embodiments of the present disclosure.

[0099] The method 700 (and/or one or more steps thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s) 104 of the computing device 102 described above. The at least one processor may be part of a robot (such as a robot 114) or part of a navigation system (such as a navigation system 118). A processor other than any processor described herein may also be used to execute the method 700. The at least one processor may perform the method 700 by executing elements stored in a memory such as the memory 106. The elements stored in memory and executed by the processor may cause the processor to execute one or more steps of a function as shown in method 700. One or more portions of a method 700 may be performed by the processor executing any of the contents of memory, such as an image processing 120, a segmentation 122, a transformation 124, a registration 126, and/or a self-calibration 128.

[0100] The method 700 comprises collecting or receiving first data from a first imaging device (step 704). The first data may include first image data 130 and/or first sensor data 146. The first data may be collected from a first imaging device about an environment from a first perspective. The first data may include one or more images of the environment and may depict a patient, patient anatomy, one or more objects, a surgical navigation tracker, elements in a room, and the like. In some examples, the first data may include a first point cloud representing objects within a field of view of the first imaging device. In some embodiments, the first data is retrieved from the database 134 and rendered to the user interface 110.

[0101] The method 700 may also comprise collecting or receiving second data from a second imaging device (step 708). The second data may include second image data 132 and/or second sensor data 148. The second data may be collected from a second image device about the environment from a second perspective, which is different from the first perspective of the first imaging device. The second data may include one or more images of the environment and may depict a patient, patient anatomy, one or more objects, a surgical navigation tracker, elements in a room, and the like. In some embodiments, the second data may include a second point cloud representing objects within a field of view of the second imaging device. In some embodiments, the second data may be retrieved from the database 134 and rendered to the user interface 110.

[0102] The method 700 may further include identifying at least one common spatial point using both the first data and the second data (step 712). In some embodiments, the at least one common spatial point may be identified using a machine learning model that is trained to identify particular points of patient anatomy. Alternatively or additionally, the at least one common spatial point may be identified using a machine learning model that is trained to track a surgical navigation tracker or a tracker array. Alternatively or additionally, the at least one common spatial point may be identified by overlaying (and optionally iteratively overlaying) the first point cloud and the second point cloud until at least a predetermined number of points in the point clouds overlap or are sufficiently close to one another.

[0103] The method 700 may also include performing a registration process for the first imaging device and/or the second imaging device using the at least one common spatial point (step 716). In some embodiments, the registration process may relate the first data and the second data for at least one common spatial point via a transformation matrix. [0104] The present disclosure encompasses embodiments of the method 700 that comprise more or fewer steps than those described above, and/or one or more steps that are different than the steps described above. [0105] Referring now to Fig. 8, a method 800 of performing self-calibration will be described in accordance with at least some embodiments of the present disclosure.

[0106] The method 800 (and/or one or more steps thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s) 104 of the computing device 102 described above. The at least one processor may be part of a robot (such as a robot 114) or part of a navigation system (such as a navigation system 118). A processor other than any processor described herein may also be used to execute the method 800. The at least one processor may perform the method 800 by executing elements stored in a memory such as the memory 106. The elements stored in memory and executed by the processor may cause the processor to execute one or more steps of a function as shown in method 800. One or more portions of a method 800 may be performed by the processor executing any of the contents of memory, such as an image processing 120, a segmentation 122, a transformation 124, a registration 128, and/or a self-calibration 128.

[0107] The method 800 comprises receiving first data about an environment from a first perspective (step 804). The first data may be received from or generated by a first imaging device. The method 800 also comprises receiving second data about the environment from a second perspective (step 808). The second data may be received from or generated by a second imaging device. The first perspective may be different from the second perspective. The environment may include a surgical environment in which a patient is situated. Accordingly, the first data and/or the second data may include image data that comprises at least one of preoperative images, intraoperative images, and postoperative images.

[0108] The method 800 may also include relating one or more common points in space between the first data and the second data using a transformation matrix (step 812). The transformation matrix may be generated by transformation 124, which may be or comprise one or more algorithms, machine learning models, artificial intelligence models, combinations thereof, and/or the like capable of transforming coordinates associated with one coordinate system into coordinates associated with another, different coordinate system. Alternatively or additionally, both the first data and the second data may be comapped to a different (e.g., third) coordinate system. In some embodiments, the transformation 124 may comprise one or more transformation matrices that transform coordinates associated with a first imaging device and a second imaging device into coordinates associated with the common point in space. In some embodiments, the transformation 124 may generate an initial transformation matrix based off historical data of other similar surgeries or surgical procedures.

[0109] The method 800 also comprises overlaying the first data and the second data (step 816). In some embodiments, the overlay may be facilitated by using the transformation matrix used in step 812. Thereafter, an iterative registration and verification process may be performed where additional transformation matrices are used to overlay the first data and the second data. An overlay may be considered valid when an accuracy of a transformation matrix used to perform the overlay no longer improves an accuracy of the overlay as compared to other transformation matrices (steps 820, 824). [0110] Once the overlay has been validated and a final transformation matrix is selected, the method 800 may include self-calibrating the first imaging device and/or the second imaging device using the transformation matrix (step 828). As part of self-calibrating the first imaging device and/or the second imaging device, one or more operating parameters of the first imaging device and/or second imaging device may be automatically set/calibrated. Calibration of imaging device(s) is important in computer vision crucial in various applications such as 3D reconstruction, object tracking, augmented reality, and image analysis. Accurate calibration ensures precise measurements and reliable analysis by correcting distortions and estimating intrinsic and extrinsic camera parameters. Calibration may include the process of determining specific parameters of the imaging device to complete operations with specified performance measurements. In some embodiments, calibration includes estimating one or more characteristics (e.g., intrinsic parameters and/or extrinsic parameters). Calibrating intrinsic parameters may allow mapping between pixel coordinates and coordinates in an image frame (e.g., optical center, focal length, and radial distortion coefficients of a lens). Calibrating extrinsic parameters may describe an orientation and/or location of the imaging device relative to the at least one common spatial point. Calibration of one or more imaging devices may ensure that all data received from the imaging device(s) provides an accurate relationship between the at least one common spatial point and a corresponding two dimensional projection in an image acquired by the imaging device.

[OHl] The present disclosure encompasses embodiments of the method 800 that comprise more or fewer steps than those described above, and/or one or more steps that are different than the steps described above.

[0112] Referring now to Fig. 9, another method 900 of performing self-calibration will be described in accordance with at least some embodiments of the present disclosure. [0113] The method 900 (and/or one or more steps thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s) 104 of the computing device 102 described above. The at least one processor may be part of a robot (such as a robot 114) or part of a navigation system (such as a navigation system 118). A processor other than any processor described herein may also be used to execute the method 900. The at least one processor may perform the method 900 by executing elements stored in a memory such as the memory 106. The elements stored in memory and executed by the processor may cause the processor to execute one or more steps of a function as shown in method 900. One or more portions of a method 900 may be performed by the processor executing any of the contents of memory, such as an image processing 120, a segmentation 122, a transformation 124, a registration 128, and/or a self-calibration 128.

[0114] The method 900 starts by initiating a self-calibration process for a first imaging device and/or a second imaging device (step 904). Initiation of the self-calibration process may be initiated automatically (e.g., after expiration of a time, in response to a predetermined condition occurring, in response to an accuracy of an image falling below a predetermined threshold, etc.). Alternatively or additionally, self-calibration may be initiated manually by a surgeon or operating room personnel pressing a button on the user interface 110 which initiates the self-calibration process.

[0115] After the self-calibration process is initiated, the method 900 continues by collecting data of an environment from a first imaging device and a second imaging device (step 908). The data collected of the environment may include first image data from the first imaging device and second image data from the second imaging device.

[0116] Using the data from the first imaging device and the second imaging device, the method 900 continues by identifying at least one common spatial point in the environment (step 912). In some embodiments, the at least one common spatial point is identified using machine learning (e.g., by providing the data from both imaging devices to a machine learning model trained to identify particular points or common spatial points).

[0117] The method 900 further includes determining a transformation matrix that relates the at least one common point between the data from the first imaging device and the data from the second imaging device (step 916). The transformation matrix may then be used to calibrate at least one of the first imaging device and the second imaging device (step 920). In some embodiments, both imaging devices are calibrated using the transformation matrix by registering data from the first imaging device and data from the second imaging device in a common coordinate space.

[0118] The present disclosure encompasses embodiments of the method 900 that comprise more or fewer steps than those described above, and/or one or more steps that are different than the steps described above.

[0119] Referring now to Fig. 10, a registration method 1000 will be described in accordance with at least some embodiments of the present disclosure.

[0120] The method 1000 (and/or one or more steps thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s) 104 of the computing device 102 described above. The at least one processor may be part of a robot (such as a robot 114) or part of a navigation system (such as a navigation system 118). A processor other than any processor described herein may also be used to execute the method 1000. The at least one processor may perform the method 1000 by executing elements stored in a memory such as the memory 106. The elements stored in memory and executed by the processor may cause the processor to execute one or more steps of a function as shown in method 1000. One or more portions of a method 1000 may be performed by the processor executing any of the contents of memory, such as an image processing 120, a segmentation 122, a transformation 124, a registration 128, and/or a self-calibration 128.

[0121] The method 100 starts by initiating a registration process (step 1004). Initiation of the registration process may be initiated automatically (e.g., after expiration of a time, in response to a predetermined condition occurring, in response to an accuracy of an image falling below a predetermined threshold, etc.). Alternatively or additionally, registration may be initiated manually by a surgeon or operating room personnel pressing a button on the user interface 110 which initiates the registration process.

[0122] After the registration process is initiated, the method 1000 continues by collecting first image data and second image data (step 1008). The image data may include sensor data and may comprise one or more images of an environment, such as a surgical environment.

[0123] The method 1000 may also include using one or more transform estimations to relate a common spatial point between the first image data and the second image data (step 1012). In some embodiments, the one or more transform estimations may include one or more transformation matrices that are applied to the first image data and/or the second image data to map the image data into common spatial coordinates. [0124] The method 1000 may further include fusing the first image data and the second image data (1016). Fusing the first image data and the second image data may support registration of the first imaging device, the second imaging device, or both.

[0125] The method 1000 may further include an optional step of providing the one or more transform estimations used during registration to a machine learning model as training data (step 1020). In particular, one or more transform matrices may be provided as training data to one or more machine learning models that are being trained to register and/or calibrate imaging devices using a common spatial point, as described herein.

[0126] The present disclosure encompasses embodiments of the method 1000 that comprise more or fewer steps than those described above, and/or one or more steps that are different than the steps described above.

[0127] As noted above, the present disclosure encompasses methods with fewer than all of the steps identified in Figs. 7, 8, 9, and 10 (and the corresponding description of the methods 700, 800, 900, and 1000), as well as methods that include additional steps beyond those identified in Figs. 7, 8, 9, and 10 (and the corresponding description of the methods 700, 800, 900, and 1000). The present disclosure also encompasses methods that comprise one or more steps from one method described herein, and one or more steps from another method described herein. Any correlation described herein may be or comprise a registration or any other correlation.

[0128] The foregoing is not intended to limit the disclosure to the form or forms disclosed herein. In the foregoing Detailed Description, for example, various features of the disclosure are grouped together in one or more aspects, embodiments, and/or configurations for the purpose of streamlining the disclosure. The features of the aspects, embodiments, and/or configurations of the disclosure may be combined in alternate aspects, embodiments, and/or configurations other than those discussed above. This method of disclosure is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed aspect, embodiment, and/or configuration. Thus, the following claims are hereby incorporated into this Detailed Description, with each claim standing on its own as a separate preferred embodiment of the disclosure.

[0129] Moreover, though the foregoing has included description of one or more aspects, embodiments, and/or configurations and certain variations and modifications, other variations, combinations, and modifications are within the scope of the disclosure, e.g., as may be within the skill and knowledge of those in the art, after understanding the present disclosure. It is intended to obtain rights which include alternative aspects, embodiments, and/or configurations to the extent permitted, including alternate, interchangeable and/or equivalent structures, functions, ranges or steps to those claimed, whether or not such alternate, interchangeable and/or equivalent structures, functions, ranges or steps are disclosed herein, and without intending to publicly dedicate any patentable subject matter.

Claims

CLAIMS What is claimed is:

1. A system, comprising: a first imaging device to collect first data about an environment; a second imaging device to collect second data about the environment from a perspective that differs from the first imaging device; a processor; and a memory coupled to the processor and storing data thereon that, when processed by the processor, enables the processor to: receive the first data and the second data; identify at least one common spatial point in the environment using both the first data and the second data; and perform a registration process for the first imaging device and/or the second imaging device using the at least one common spatial point during the registration process.

2. The system according to claim 1, wherein the registration process relates the first data and the second data for at least one common spatial point via a transformation matrix, a mathematical model, or a machine learning-based model comprising at least one of a set of transformation matrices and a dense displacement field representing a non-linear relation between the first data and the second data.

3. The system according to claims 1 or 2, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: perform a registration process by computing the transformation matrix, reprojecting the detected features from first imaging device into the second imaging device and vice versa, and computing the reprojection error, and computing the root means square value of all such reprojections for all detected features in both imaging devices and repeat the registration process until the reprojection error from the optimized transformation matrix is no longer improved and is better than reprojection error provided by a predetermined number of previous transformation matrices

4. The system according to any preceding claim, wherein the first imaging device comprises at least one of a depth camera, a visible light camera, a Red Green Blue (RGB) camera, an infrared camera, an ultrasound device, and a stereo or monocular endoscope.

5. The system according to any preceding claim, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: automatically calibrate at least one of the first imaging device and the second imaging device.

6. The system according to claim 5, wherein an operating parameter of the first imaging device and/or the second imaging device is defined and set when automatically calibrated.

7. The system according to any preceding claim, wherein the common spatial features comprise: features on anatomical elements of a patient or keypoints on surgical tools or

6D pose of the CAD model of the surgical tools.

8. The system according to any preceding claim, wherein the at least one common spatial point comprises a point on a surgical navigation tracker.

9. The system according to any preceding claim, wherein the at least one common spatial point comprises a point on a surgical robot and wherein the surgical robot comprises one or more detachable arms having their own reference frame.

10. The system according to any preceding claim, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: register the first data and the second data in a common coordinate space.

11. The system according to any preceding claim, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: develop one or more estimated transform matrices to relate the at least one common spatial point between the first data and the second data; fuse the first data and the second data to enable simultaneous use of the first imaging device and the second imaging device during a surgical procedure; and provide the one or more estimate transform matrices to a machine learning model as training data.

12. A method, comprising: receiving first data about an environment from a first imaging device; receiving second data about the environment from a second imaging device, wherein a perspective of the environment from the second imaging device is different from a perspective of the environment from the first imaging device; identifying at least one common spatial point in the environment using both the first data and the second data; and performing a registration process for the first imaging device and/or the second imaging device using the at least one common spatial point.

13. The method according to claim 12, wherein the registration process relates the first data and the second data for at least one common spatial point via a transformation matrix.

14. The method according to claim 12 or 13, wherein at least one of the first imaging device and the second imaging device comprises at least one of a depth camera, a visible light camera, a Red Green Blue (RGB) camera, an infrared camera, an ultrasound device, and an endoscope.

15. The method according to any of claims 12 to 14, wherein the at least one common spatial point comprises one or more of a point on an anatomical element of a patient, a point on a surgical navigation tracker, a feature on one or more surgical tools, and a point on one or more arms of a surgical robot, the method further comprising: calibrating at least one of the first imaging device and the second imaging device using the at least one common spatial point.