WO2024044573A1 - Methods and apparatus for synthetic surgical data generation - Google Patents

Abstract

A system and method are disclosed for generating a synthetic surgical data. The systems and methods may receive a surgical video data and/or radiological image data. The surgical video and radiological image data may be analyzed to recognize included anatomies and/or pathologies. Based on the recognized anatomies and pathologies, statistics may be determined to describe ranges of variation of the anatomies and pathologies. Synthetic surgical data including synthetic surgical video and synthetic radiological data may be generated based on the determined statistics.

Description

METHODS AND APPARATUS FOR SYNTHETIC SURGICAL DATA GENERATION

CLAIM OF PRIORITY

[0001] This application claims priority to U.S. provisional patent application no. 63/400,032, titled “METHODS AND APPARATUS FOR SYNTHETIC SURGICAL DATA GENERATION”, and filed on August 22, 2022, which is herein incorporation by reference in its entirety.

INCORPORATION BY REFERENCE

[0002] All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference.

FIELD

[0003] The present embodiments relate generally to surgical data and more specifically to generating synthetic surgical data.

BACKGROUND

[0004] The scientific community, and medical researchers in particular, desire access to medical data in order to study trends, test hypotheses, train, develop software tools, and the like. The number of actual available medical records may be limited. In some cases, identifying patient information may not be removed or redacted from the medical data making widespread distribution of the data difficult due to patient privacy requirements.

[0005] For example, variability of surgical procedures, and error rates in such surgical procedures, can be very high (e.g., greater than 40%). Further most training requires actual patients, and measurement and documentation of surgical procedures is typically still done manually on paper. What is needed are apparatuses, e.g., systems and devices, including in particular software, that may be used in real time with medical procedures and for medical training to allow improved consistency of treatment, reduction in error rates and may help close the knowledge gap for physicians, including surgeons.

[0006] In particular, it would be helpful to provide systems and apparatuses that may generate synthetic patient data that may be used to help understand actual (or potential) patient data and/or to help train physicians. SUMMARY OF THE DISCLOSURE

[0007] Described herein are apparatuses, systems, and methods to generate one or more surgical outputs (e.g., reports), including an initial surgical report. The surgical output may be generated based on one or more neural networks. The surgical output may include imaging (video, stills, or both) data, including actual and/or synthetic imaging data. The surgical output may describe detected surgical procedures. The surgical output may be reviewed and corrected by the patient’s surgeon. These corrections may be used to generate a billing report that may be used to generate a bill for a surgical operation. In general, these surgical reports may be referred to as initial surgical reports, although they may be reviewed and/or subsequently finalized and//or modified.

[0008] The methods and apparatuses (e.g., systems, including software) described herein may be part of an interoperative guidance an assistance system that may reduce variability in the quality of patient care and may reduce time in the operating room (OR) and/or may reduce complication rates. These methods and apparatuses may provide Al-powered surgical training and/or feedback to allow surgeons to better understand the patient’s anatomy as well as actual or potential conditions and complications before, during or after a surgical procedure. In some examples these methods and apparatuses may use synthetic patient data.

[0009] Any of the methods and apparatuses (e.g., systems, including software) described herein may be used to generate a surgical outputs describing detected surgical features and potential activities. Any of the methods may include receiving a surgical video of a surgical procedure performed on a patient, identifying one or more surgical tools in the surgical video, detecting surgical activity within the surgical video, and determining one or more activities based on the identified surgical tools and the detected surgical activities.

[0010] Any of the methods described herein may also include recognizing a patient’s anatomy in the surgical video, where determining the one or more surgical activities is based, at least in part, on the recognized patient’s anatomy. In some examples, recognizing the patient’s anatomy may include executing a neural network trained to recognize anatomy in a surgical video. In general, these methods may include generating synthetic user data, based on real user/patient data, which may be useful for research, training and/or treatment.

[0011] In any of the methods described herein, the methods and apparatuses may include identifying the one or more surgical tools includes executing a neural network trained to identify surgical tools in a surgical video. Any of the methods may further include recognizing a pathology in a surgical video, where determining the one or more surgical activities is based, at least in part, on the recognized pathology. In some examples, recognizing the pathology may include executing a neural network trained to recognize pathology in a surgical video. [0012] In any of the methods described herein, the surgical activities may include a video clip of the detected surgical activity. Still, in any of the methods, the surgical activities may include a descriptive text based at least in part on the detected surgical activity.

[0013] Any of the methods described herein may include generating an initial surgical output based at least in part on the one or more determined surgical activities. In any of the methods, the surgical video may be captured with an orthoscopic camera.

[0014] In any of the methods described herein, detecting surgical activity may include executing a neural network trained to detect surgical activity in a surgical video.

[0015] Any of the systems described herein may include one or more processors and a memory configured to store instructions that, when executed by the one or more processors, cause the system to receive a surgical video of a surgical procedure performed on a patient, identify one or more surgical tools in the surgical video, detect surgical activity within the surgical video, and determine one or more surgical activities based on the identified surgical tools and the detected surgical activities.

[0016] Any of the methods described herein may provide an initial surgical report describing detected surgical and/or treatment activities. The methods may include determining a plurality of video clips from a surgical video of a surgical procedure performed on a patient, determining a plurality of recommended key frames from the plurality of video clips, and determining one or more surgical activities based on the plurality of key frames. Any of these may include synthetic data that is matched to the actual patient anatomy and/or patient treatment, e.g., based on statistical matching to real-world data. Thus surgical complications and/or conditions may be generated based on statical likelihood from one or more patient features (e.g., age, race, health, etc.).

[0017] Any of the methods and apparatuses descried herein may further include detecting a plurality of surgical phases from the plurality of video clips, where the key frames are based, at least in part, on the plurality of surgical phases. In some examples, any of the methods may further include recognizing one or more stages within at least one of the plurality of surgical phases, where the key frames are based, at least in part, on the one or more stages.

[0018] In any of the methods, the key frames may include diagnostic key frames, site preparation key frames, suture passing key frames, anchor insertion key frames, post treatment key frames, or a combination thereof. Furthermore, in any of the methods described herein may further include generating an initial surgical output (including synthetic data) based at least in part on the key frames.

[0019] In any of the methods described herein may include recognizing patient anatomy in one or more of the key frames, where the billable activities are based, at least in part, on the recognized patient anatomy. In some examples, recognizing patient anatomy may include executing a neural network trained to recognize patient anatomy.

[0020] In any of the methods described herein may include recognizing a pathology in one or more of the key frames, where the billable activities are based, at least in part, on the recognized pathology. In some examples, recognizing the pathology may include executing a neural network trained to recognize patient pathology.

[0021] Any of the methods described herein may include recognizing a surgical tool in one or more of the key frames, where the billable activities are based, at least in part, on the recognized surgical tool. In some examples, recognizing the surgical tool may include executing a neural network trained to recognize one or more surgical tools.

[0022] Any of the systems described herein may include one or more processors and a memory configured to store instructions that, when executed by the one or more processors, cause the system to determine a plurality of video clips from a surgical video of a surgical procedure performed on a patient, determine a plurality of recommended key frames from the plurality of video clips, and determine one or more billable activities based on the plurality of key frames.

[0023] Any of the methods described herein may include receiving video clips of an operation performed on a patient, determining any modifications to billable activities based, at least in part, on the video clips, and generating a billing report based, at least in part, on the determined modifications.

[0024] In any of the methods described herein, determining any modifications to surgical activities may include verifying that at least one of the video clips include a particular surgical procedure. In any of the methods, determining any modifications to surgical activities may include verifying that at least one of the video clips include a particular surgical tool, patient anatomy, or pathology.

[0025] In any of the methods described herein, verifying may include executing a neural network trained to recognize surgical tools, patient anatomy, or pathology. In any of the methods, generating the billing report may include mapping detected surgical activity to surgical procedures.

[0026] Any of the systems described herein may include one or more processors, and a memory configured to store instructions that, when executed by the one or more processors, cause the system to receive video clips of an operation performed on a patient, determine any modifications to billable activities based, at least in part, on the video clips, and generate a billing report based, at least in part, on the determined modifications. [0027] All of the methods and apparatuses described herein, in any combination, are herein contemplated and can be used to achieve the benefits as described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

[0028] A better understanding of the features and advantages of the methods and apparatuses described herein will be obtained by reference to the following detailed description that sets forth illustrative embodiments, and the accompanying drawings of which:

[0029] FIG. 1 shows an example system for generating synthetic medical images.

[0030] FIG. 2 is a flowchart showing an example method for generating synthetic medical images.

[0031] FIG. 3 shows example surgical video images. .

[0032] FIG. 4 shows another example of surgical video images.

[0033] FIG. 5 shows a block diagram of a device that may be one example of any device, system, or apparatus that may provide any of the functionality described herein.

DETAILED DESCRIPTION

[0034] Described herein are systems and methods for generating synthetic surgery or surgery related images. The generated images may include video images and/or radiological images. The radiological images may include x-ray images, ultrasound images, or any other feasible images.

[0035] The generated images may be based on actual patient video images or radiological images. Any feasible patient images may be analyzed for anatomies or pathologies. In some cases, the analysis may be based on execution of one or more trained neural networks. Furthermore, a processor or processing device may determine one or more statistics associated with any identified anatomies or pathologies. Based on the determined statistics, the processor or processing device may generate synthetic video and/or radiological images. In some cases, the synthetic video or radiological image generation may be based on an execution of trained generative adversarial networks.

[0036] FIG. 1 shows an example system 100 for generating synthetic medical images. The system 100 may include a compute node 110. The compute node 110 may include a processor, computer, or the like. The compute node 110 may be, for example, located in or near a surgeon’s medical office or clinic. In another example, the compute node 110 may be a remote, virtual, or cloud-based processor, computer, or the like remotely located with respect to the surgeon, doctor, or other clinician. Generally, the compute node 110 may include, one or more processors, memory (including dynamic, non-volatile, mechanical, solid-state, or the like), and any number of interfaces (including user interfaces), communication interfaces (serial, parallel, wired, wireless, and the like).

[0037] The system 100 may include surgical video data 140 and radiological image data 150. The surgical video data 140 may include surgical video associated with a variety of surgeries that may be associated with a variety of different patients. In some examples, the surgical video data 140 may include arthroscopic video images from any feasible number of patients. In some other examples, the surgical video data 140 may include any other feasible (e.g., non-arthroscopic) video images.

[0038] The radiological image data 150 may include x-ray data, ultrasound data, or any other feasible radiological data that may be associated with a variety of different patients and a variety of different surgeries.

[0039] The compute node 110 may generate statistical models 160 from the surgical video data 140 and the radiological image data 150. The statistical models may be built, and the networks (GANs) may be trained during a learning phase. For example, the statistical models 160 may describe variations of features that are denoted or identified within the surgical video data 140 and the radiological image data 150. For example, the statistical models 160 may include statistical models of pathological variations determined from the surgical video data 140. In another example, the statistical models 160 may include statistical models of morphological variations determined from the radiological image data 150.

[0040] During an execution phase, a given patient's radiological images may be used to synthesize pathology. This could be used for report generation purposes. The report to the care provider may therefore use synthesized images.

[0041] The compute node 110 may use the statistical models 160 to generate synthesized images 130. For example, the compute node 110 may generate a set of synthesized surgical video images based on one or more of the statistics models 160. Thus, features within the synthesized surgical videos may be within in a range of variations included in the surgical video data 140.

[0042] The compute node 110 may use the statistical models 160 to generate a set of synthesized radiological images. Features within the synthesized radiological images may be within a range of variations included in the radiological image data 150. In any of the methods and apparatuses described herein, features may be enlarged or expanded, e.g., to expand the range of morphological variations, even beyond the actual variations in the dataset. For example, these methods and apparatuses may synthesize pathology images for a larger person (e.g., a person who is 7’2”) even when the dataset only contains actual images for people who are smaller (e.g., under 6’). These methods and apparatuses may also produce new combinations, i.e., large tears on people who are slightly built, etc.

[0043] FIG. 2 is a flowchart showing an example method 200 for generating synthetic medical images. Some examples may perform the operations described herein with additional operations, fewer operations, operations in a different order, operations in parallel, and some operations differently. The method 200 is described below with respect to the system 100 of FIG. 1, however, the method 200 may be performed by any other suitable system or device.

[0044] The method 200 begins in block 210 as surgery videos are obtained. For example, the compute node 110 may obtain or receive the surgical video data 140 of FIG. 1. The surgery videos may be actual surgical videos that were recorded during a patient’s surgery. In some examples, the videos may be arthroscopic videos recorded from arthroscopic cameras.

[0045] Next in block 212, anatomy recognition is performed. For example, the compute node 110 can identify various anatomical parts and/or features visible and/or identifiable from the surgery video. In some examples, the compute node 110 may execute a neural network trained to identify patient anatomy from the surgery video (from the surgical video data 140).

[0046] Next, in block 214, pathology recognition is performed. For example, the compute node 110 may identify various patient pathologies that may be visible and/or identifiable from the surgery video. Example pathologies may include, but are not limited to, tendon damage, tom ligaments, rotator cuff injury, meniscus damage, and the like. In some examples, the compute node 110 may execute a neural network trained to identify pathologies from the surgery video (from the surgical video data 140).

[0047] Next, in block 216, pathology quantification and classification is performed. For example, the compute node 110 may quantify and classify any of the pathologies identified in block 216. There are a few of ways that the apparatus/method may quantify/classify pathologies. For example, some classifications are categorical; major tear, minor tear, etc. Some are binary, partial thickness tears, which implies that the tendons are frayed but some residual tendons are still holding; as opposed to total tears where we can see through the tendons. These methods and apparatuses may report on the measured sizes of tears and defects (e.g., 5mm partial thickness, 5mm cartilage lesion, 5mm x 6mm cartilage defect, etc.). In some examples, the compute node 110 may execute a neural network trained to quantify and classify from the surgery video (from the surgical video data 140) based on the identified pathologies in block 214.

[0048] Next, in block 218, statistical models of pathological variations are generated. For example, the compute node 110 may determine statistics regarding anatomies and pathologies that are included within the surgery videos. The statistics may describe, at least in part, the ranges of various features (anatomies and pathologies) that have been identified and/or recognized from within the surgery videos.

[0049] Turning to block 220, radiological images are obtained. For example, the compute node 110 may obtain or receive radiological image data 150. The radiological images may be actual radiological images (x-rays, ultrasounds, and the like) that were collected during diagnosis or treatment of a patient.

[0050] Next, in block 222 radiological anatomy recognition is performed. For example, the compute node 110 may identify various anatomical parts and/or features visible and/or identifiable from the obtained radiological images. In some examples, the compute node 110 may execute a neural network trained to identify patient anatomy from the radiological images (from the radiological image data 150).

[0051] Next, in block 224 radiological pathology recognition is performed. For example, the compute node 110 may identify various patient pathologies that may be visible and/or identifiable from the radiological images. Example pathologies may include, but are not limited to, tendon damage, torn ligaments, rotator cuff injury, meniscus damage, and the like. In some examples, the compute node 110 may execute a neural network trained to identify pathologies from the radiological images (from the radiological image data 150).

[0052] Next in block 228 statistical models of morphological variations are generated. Morphological variations may include variations identified from the literature, such as small variations in the curvature of the glenoid and humeral heads, i.e., the socket and ball joints in the shoulder (and the hip). Even in the absence of a significant number of real examples to train the models, these methods and apparatuses may start with a normal patient’s radiological image, use computer vision algorithms to alter the curvatures (within realistic limits) and synthesize optical images based on these images. Other variations could be straightforward patient height / body structure related. For example, larger people have larger bones (and joint spaces), so the field of view might look slightly different.

[0053] For example, the compute node 110 may determine statistics regarding anatomies and pathologies that are included within the radiological images. The statistics may describe, at least in part, the ranges of various features (anatomies and pathologies) that have been identified and/or recognized from within the radiological images. For example, statistical models may capture the variations in the size of the bones, curvature of structures such as the humeral heads, glenoids, condyle, etc., and these variations may be controlled for appropriate factors.

[0054] Next, in block 230 synthetic images are generated. Synthetic surgery videos and/or synthetic radiological images may be generated that are consistent with the statistical models of pathological variations (block 218) and statistical models of morphological variations (block 228). Advantageously, the synthesized images although representative of actual pathologies, are not directly attributed to any one patient or individual. Thus, the synthesized images can effectively anonymize surgery videos and radiological images.

[0055] In some variations, the synthetic images may be generated through an execution of a neural network. For example, one or more generative adversarial networks (GANs) may be executed to generate any feasible surgery or radiological images.

[0056] Note that in some embodiments, the surgical video (such as the surgical video data 140) may be processed independently from radiological images (such as the radiological image data 150). In other words, operations associated with blocks 210, 212, 214, 216, and 218 may be performed separately from operations associated with blocks 220, 222, 224, and 228. For example, in some cases the method 200 may process only surgical video data or only radiological image data. In some other cases, the method 200 may process both surgical video data and radiological image data.

[0057] FIG. 3 shows example surgical video images 300. A first image 310 shows a reference image. Thus, the first image 310 may be an actual surgical video image associated with an individual patient. A second image 320 is a synthetic image that is generated in accordance with the method 200. FIG. 3 is an example of the ACL in the knee joint. The apparatus (e.g., system) may take a reference pathology, i.e., the image on the left in FIG. 3, and produces a variation based on other images it has seen. The apparatus may generate a plurality of different candidate images; some of these candidates may look more or less realistic. The system may reject unrealistic images and settles on images that look sufficient different from the original image but retains attributes (in an abstract sense) of the original. This may be combined and variations in the radiological images may be used to synthesize data to cover variations which would not be seen easily in real-life.

[0058] The second image 320 may include similar features (anatomies) that are included within the first image 310. However, the features of the first image 310 may be modified based on the statistics determined through the method 200.

[0059] FIG. 4 shows example surgical video images 400. A first image 410 shows a reference image. Thus, the first image 410 may be an actual surgical video image associated with an individual patient. A second image 420 is a synthetic image that is generated in accordance with the method 200. FIG. 4 is an example of how the system can generate variations in a meniscal tear in the knee joint.

[0060] FIG. 5 shows a block diagram of a device 500 that may be one example of any device, system, or apparatus that may provide any of the functionality described herein. The device 500 may include a transceiver 520, a processor 530, and a memory 540. [0061] The transceiver 520, which is coupled to the processor 530, may be used to interface with any other feasible device and/or network. For example, the transceiver 520 may include any feasible wired and/or wireless interface to transmit and/or receive data. In some examples, the transceiver 520 may include a wired transceiver that includes a wired network (Ethernet or the like) interface. In some other examples, the transceiver 520 may include a wireless transceiver that conforms to Wi-Fi, Bluetooth, Zigbee, Long Term Evolution (LTE) or other wireless protocols.

[0062] The processor 530, which is also coupled to the memory 540, may be any one or more suitable processors capable of executing scripts or instructions of one or more software programs stored in the device 500 (such as within memory 540).

[0063] The memory 540 may include image data 542 that may include surgical video data (such as the surgical video data 140 of FIG. 1) and/or radiological image data (such as the radiological image data 150). The device 500 may obtain or receive the image data (e.g., the surgical video data and/or the radiological image data 150) through the transceiver 520.

[0064] The memory 540 may also include synthetic image data 543. The synthetic image data 543, in turn, may include synthetic surgery video data and/or synthetic radiological image data that may be generated in accordance with the method 200 of FIG. 2.

[0065] The memory 540 may also include a non-transitory computer-readable storage medium (e.g., one or more nonvolatile memory elements, such as EPROM, EEPROM, Flash memory, a hard drive, etc.) that may store the following software modules: an anatomy recognizer module 544 to recognize and/or identify patient anatomy; a pathology recognizer module 545 to recognize and/or identify pathologies; a pathology quantification and classification module 546 to quantify and classify any feasible pathology; a statistical model generation module 547 to generate statistical models; and an image generation module 548 to generate surgical and/or radiological images.

[0066] Each software module includes program instructions that, when executed by the processor 530, may cause the device 500 to perform the corresponding function(s). Thus, the non-transitory computer-readable storage medium of memory 540 may include instructions for performing all or a portion of the operations described herein.

[0067] The processor 530 may execute the anatomy recognizer module 544 to identify and/or recognize patient anatomies within surgical video clips and/or radiological image data. In some examples, execution of the anatomy recognizer module 544 may cause the processor 530 to execute a neural network trained to identify and/or recognize any feasible patient anatomy.

[0068] The processor 530 may execute the pathology recognizer module 545 to identify and/or recognize pathologies within surgical video clips and/or radiological image data. In some examples, execution of the pathology recognizer module 545 may cause the processor 530 to execute a neural network trained to identify and/or recognize any feasible pathology.

[0069] The processor 530 may execute the pathology quantification and classification module 546 to quantify and/or classify any recognized (identified) pathologies within the surgical video data and/or the radiological image data. In some examples, execution of the pathology quantification and classification module 546 may cause the processor 530 to execute a neural network trained to quantify and/or classify any recognized (identified) pathologies.

[0070] The processor 530 may execute statistical model generation module 547 to generate any feasible statistical models in accordance with the recognized anatomy and pathology from the surgical video data and the radiological image data.

[0071] The processor 530 may execute the image generation module 548 to generate synthetic surgery videos and synthetic radiological images. For example, execution of the image generation module 548 may cause the processor 530 to execute one or more neural networks trained to generate surgery videos and radiological images based on one or more statistical models. In some examples, execution of the image generation module 548 may cause the processor 530 to execute one or more GAN neural networks trained to iteratively generate surgery videos and radiological images.

[0072] In general, as mentioned above, any of the methods and apparatuses may include creating synthetic data that may match real-world statistical distributions for the patient-specific context. For example, these methods and apparatuses may include generating individual readings (attributes) of patient health datasets; replicating the statistical distributions present in the real-world data; combining the readings into synthetic patient records which feed into the datasets. In some examples, these methods and apparatuses may include matching distributions should balance how and what parameters could be combined. This may work with quantified and categorical data attributes. In any of these examples the methods or apparatuses may include generating synthetic data to mimic subject responses to various dosings, which may rely on assumptions about the drug toxicities, statistical models of subject reactions, etc.

[0073] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein and may be used to achieve the benefits described herein.

[0074] The process parameters and sequence of steps described and/or illustrated herein are given by way of example only and can be varied as desired. For example, while the steps illustrated and/or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed. The various example methods described and/or illustrated herein may also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.

[0075] Any of the methods (including user interfaces) described herein may be implemented as software, hardware or firmware, and may be described as a non-transitory computer-readable storage medium storing a set of instructions capable of being executed by a processor (e.g., computer, tablet, smartphone, etc.), that when executed by the processor causes the processor to control perform any of the steps, including but not limited to: displaying, communicating with the user, analyzing, modifying parameters (including timing, frequency, intensity, etc.), determining, alerting, or the like. For example, any of the methods described herein may be performed, at least in part, by an apparatus including one or more processors having a memory storing a non-transitory computer-readable storage medium storing a set of instructions for the processes(s) of the method.

[0076] While various embodiments have been described and/or illustrated herein in the context of fully functional computing systems, one or more of these example embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the example embodiments disclosed herein.

[0077] As described herein, the computing devices and systems described and/or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those contained within the modules described herein. In their most basic configuration, these computing device(s) may each comprise at least one memory device and at least one physical processor.

[0078] The term “memory” or “memory device,” as used herein, generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and/or computer-readable instructions. In one example, a memory device may store, load, and/or maintain one or more of the modules described herein. Examples of memory devices comprise, without limitation, Random Access Memory (RAM), Read Only Memory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives (SSDs), optical disk drives, caches, variations or combinations of one or more of the same, or any other suitable storage memory. [0079] In addition, the term “processor” or “physical processor,” as used herein, generally refers to any type or form of hardware-implemented processing unit capable of interpreting and/or executing computer-readable instructions. In one example, a physical processor may access and/or modify one or more modules stored in the above-described memory device. Examples of physical processors comprise, without limitation, microprocessors, microcontrollers, Central Processing Units (CPUs), Field-Programmable Gate Arrays (FPGAs) that implement softcore processors, Application- Specific Integrated Circuits (ASICs), portions of one or more of the same, variations or combinations of one or more of the same, or any other suitable physical processor.

[0080] Although illustrated as separate elements, the method steps described and/or illustrated herein may represent portions of a single application. In addition, in some embodiments one or more of these steps may represent or correspond to one or more software applications or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks, such as the method step.

[0081] In addition, one or more of the devices described herein may transform data, physical devices, and/or representations of physical devices from one form to another. Additionally or alternatively, one or more of the modules recited herein may transform a processor, volatile memory, non-volatile memory, and/or any other portion of a physical computing device from one form of computing device to another form of computing device by executing on the computing device, storing data on the computing device, and/or otherwise interacting with the computing device.

[0082] The term “computer-readable medium,” as used herein, generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media comprise, without limitation, transmission-type media, such as carrier waves, and non-transitory-type media, such as magnetic-storage media (e.g., hard disk drives, tape drives, and floppy disks), optical -storage media (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), and BLU-RAY disks), electronic-storage media (e.g., solid-state drives and flash media), and other distribution systems.

[0083] A person of ordinary skill in the art will recognize that any process or method disclosed herein can be modified in many ways. The process parameters and sequence of the steps described and/or illustrated herein are given by way of example only and can be varied as desired. For example, while the steps illustrated and/or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed. [0084] The various exemplary methods described and/or illustrated herein may also omit one or more of the steps described or illustrated herein or comprise additional steps in addition to those disclosed. Further, a step of any method as disclosed herein can be combined with any one or more steps of any other method as disclosed herein.

[0085] The processor as described herein can be configured to perform one or more steps of any method disclosed herein. Alternatively or in combination, the processor can be configured to combine one or more steps of one or more methods as disclosed herein.

[0086] When a feature or element is herein referred to as being "on" another feature or element, it can be directly on the other feature or element or intervening features and/or elements may also be present. In contrast, when a feature or element is referred to as being "directly on" another feature or element, there are no intervening features or elements present. It will also be understood that, when a feature or element is referred to as being "connected", "attached" or "coupled" to another feature or element, it can be directly connected, attached or coupled to the other feature or element or intervening features or elements may be present. In contrast, when a feature or element is referred to as being "directly connected", "directly attached" or "directly coupled" to another feature or element, there are no intervening features or elements present. Although described or shown with respect to one embodiment, the features and elements so described or shown can apply to other embodiments. It will also be appreciated by those of skill in the art that references to a structure or feature that is disposed "adjacent" another feature may have portions that overlap or underlie the adjacent feature.

[0087] Terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. For example, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as "/".

[0088] Spatially relative terms, such as "under", "below", "lower", "over", "upper" and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device in the figures is inverted, elements described as "under" or "beneath" other elements or features would then be oriented "over" the other elements or features. Thus, the exemplary term "under" can encompass both an orientation of over and under. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Similarly, the terms "upwardly", "downwardly", "vertical", "horizontal" and the like are used herein for the purpose of explanation only unless specifically indicated otherwise.

[0089] Although the terms “first” and “second” may be used herein to describe various features/elements (including steps), these features/elements should not be limited by these terms, unless the context indicates otherwise. These terms may be used to distinguish one feature/element from another feature/element. Thus, a first feature/element discussed below could be termed a second feature/element, and similarly, a second feature/element discussed below could be termed a first feature/element without departing from the teachings of the present invention.

[0090] Throughout this specification and the claims which follow, unless the context requires otherwise, the word “comprise”, and variations such as “comprises” and “comprising” means various components can be co-jointly employed in the methods and articles (e.g., compositions and apparatuses including device and methods). For example, the term “comprising” will be understood to imply the inclusion of any stated elements or steps but not the exclusion of any other elements or steps.

[0091] In general, any of the apparatuses and methods described herein should be understood to be inclusive, but all or a sub-set of the components and/or steps may alternatively be exclusive, and may be expressed as “consisting of’ or alternatively “consisting essentially of’ the various components, steps, sub-components or sub-steps.

[0092] As used herein in the specification and claims, including as used in the examples and unless otherwise expressly specified, all numbers may be read as if prefaced by the word "about" or “approximately,” even if the term does not expressly appear. The phrase “about” or “approximately” may be used when describing magnitude and/or position to indicate that the value and/or position described is within a reasonable expected range of values and/or positions. For example, a numeric value may have a value that is +/- 0.1% of the stated value (or range of values), +/- 1% of the stated value (or range of values), +/- 2% of the stated value (or range of values), +/- 5% of the stated value (or range of values), +/- 10% of the stated value (or range of values), etc. Any numerical values given herein should also be understood to include about or approximately that value unless the context indicates otherwise. For example, if the value "10" is disclosed, then "about 10" is also disclosed. Any numerical range recited herein is intended to include all sub-ranges subsumed therein. It is also understood that when a value is disclosed that "less than or equal to" the value, "greater than or equal to the value" and possible ranges between values are also disclosed, as appropriately understood by the skilled artisan. For example, if the value "X" is disclosed the "less than or equal to X" as well as "greater than or equal to X" (e.g., where X is a numerical value) is also disclosed. It is also understood that the throughout the application, data is provided in a number of different formats, and that this data, represents endpoints and starting points, and ranges for any combination of the data points. For example, if a particular data point “10” and a particular data point “15” are disclosed, it is understood that greater than, greater than or equal to, less than, less than or equal to, and equal to 10 and 15 are considered disclosed as well as between 10 and 15. It is also understood that each unit between two particular units are also disclosed. For example, if 10 and 15 are disclosed, then 11, 12, 13, and 14 are also disclosed.

[0093] Although various illustrative embodiments are described above, any of a number of changes may be made to various embodiments without departing from the scope of the invention as described by the claims. For example, the order in which various described method steps are performed may often be changed in alternative embodiments, and in other alternative embodiments one or more method steps may be skipped altogether. Optional features of various device and system embodiments may be included in some embodiments and not in others. Therefore, the foregoing description is provided primarily for exemplary purposes and should not be interpreted to limit the scope of the invention as it is set forth in the claims.

[0094] The examples and illustrations included herein show, by way of illustration and not of limitation, specific embodiments in which the subject matter may be practiced. As mentioned, other embodiments may be utilized and derived there from, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Such embodiments of the inventive subject matter may be referred to herein individually or collectively by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept, if more than one is, in fact, disclosed. Thus, although specific embodiments have been illustrated and described herein, any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.

Claims

CLAIMS What is claimed is:

1. A method of generating synthetic surgical data, the method comprising: receiving surgical videos of a surgical procedure performed on a patient; determining statistical models of pathological variations based on the surgical videos; receiving radiological images associated with the patient; determining statistical models of morphological variations based on the radiological images; and generating synthetic surgical and radiological images based at least in part on the statistical models of pathological variations and statistical models of the morphological variations.

2. The method of claim 1, wherein generating the synthetic surgical images comprises executing one or more generative adversarial neural networks trained to generate the surgical images based on the statistical models of pathological variations.

3. The method of claim 1, wherein generating the synthetic radiological images comprises executing one or more generative adversarial neural networks trained to generate the radiological images based on the statistical models of morphological variations.

4. The method of claim 1, further comprising: identifying one or more anatomies within the surgical videos; and identifying one or more pathologies within the surgical videos, wherein determining the statistical models of pathological variations is based on the one or more anatomies identified within the surgical videos and the one or more pathologies identified in the surgical videos.

5. The method of claim 4, wherein identifying one or more anatomies within the surgical video comprises executing a neural network trained to recognize anatomies in the surgical video.

6. The method of claim 4, wherein identifying one or more pathologies within the surgical video comprises executing a neural network trained to recognize pathologies in the surgical video.

7. The method of claim 4, further comprising quantizing and classifying the one or more pathologies identified within the surgical videos, wherein the statistical models of the pathological variations is based on the quantized and classified pathologies:

8. The method of claim 1, further comprising: identifying one or more anatomies within the radiological images; and identifying one or more pathologies within the radiological images, wherein determining the statistical models of morphological variations is based on the one or more anatomies identified within the radiological images and the one or more pathologies identified in the radiological images.

9. The method of claim 8, wherein identifying one or more anatomies within the radiological images comprises executing a neural network trained to recognize anatomies within the radiological images.

10. The method of claim 8, wherein identifying one or more pathologies within the radiological images comprises executing a neural network trained to recognize pathologies within the radiological images.

11. A system comprising: one or more processors; and a memory configured to store instructions that, when executed by the one or more processors, cause the system to: receive a surgical video of a surgical procedure performed on a patient; determine statistical models of pathological variations based on the surgical videos; receive radiological images associated with the patient; determine statistical models of morphological variations based on the radiological images; and generate synthetic surgical and radiological images based at least in part on the statistical models of pathological variations and statistical models of the morphological variations.

12. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a device, cause the device to: receive a surgical video of a surgical procedure performed on a patient; determine statistical models of pathological variations based on the surgical videos; receive radiological images associated with the patient; determine statistical models of morphological variations based on the radiological images; and generate synthetic surgical and radiological images based at least in part on the statistical models of pathological variations and statistical models of the morphological variations. A method of generating synthetic surgical data, the method comprising: receiving surgical videos of a surgical procedure performed on a patient; determining statistical models of pathological variations based on the surgical videos; and generating synthetic surgical images based at least in part on the statistical models of pathological variations. A method of generating synthetic surgical data, the method comprising: receiving radiological images associated with the patient; determining statistical models of morphological variations based on the radiological images; and generating synthetic radiological images based at least in part on the statistical models of the morphological variations.