WO2024243372A2

WO2024243372A2 - Improved visualization for ultrasound-assisted procedure

Info

Publication number: WO2024243372A2
Application number: PCT/US2024/030689
Authority: WO
Inventors: Hyungsoon Im; Matthew Leming; Ralph Weissleder
Original assignee: General Hospital Corp
Current assignee: General Hospital Corp
Priority date: 2023-05-23
Filing date: 2024-05-23
Publication date: 2024-11-28
Anticipated expiration: 2025-11-23
Also published as: WO2024243372A3

Abstract

Systems and methods are provided for guiding a biopsy. The system includes an echograph system that provides an ultrasound image of a region of interest, a processor, an output device, and a non-transitory computer readable medium. Executable instructions on the medium provide a convolutional neural network that generates a predicted area for a biopsy needle from the ultrasound image. The convolutional neural network is trained on a plurality of ultrasound images with corresponding labelled images having an area representing the biopsy needle indicated by a human expert using a loss function that, for a given ultrasound training image and the corresponded labelled image, penalizes mismatches between the predicted area generated from the given ultrasound training image and the area indicated by the human expert on the corresponding labelled image. An output interface provides the ultrasound image to the output device with the predicted area for the biopsy needle indicated.

Description

IMPROVED VISUALIZATION FOR ULTRASOUND-ASSISTED PROCEDURE

Related Applications

[0001] This application claims priority to U.S. Provisional Patent No. 63/503,913, filed May 23, 2023, and entitled “Artificial Intelligence Based Intervention for Ultrasound,” which is hereby incorporated by reference in its entirety.

Technical Field

[0002] This invention relates to medical imaging, and more particularly, to improved visualization for ultrasound-assisted procedure.

Background

[0003] Devices used to collect medical images in a clinical setting come with a tradeoff of resolution and image quality for practical usability. For instance, while MRIs provide very high resolution and high-quality images, magnetic resonance imaging (MRI) devices are large and expensive; in contrast, while functional nearinfrared spectroscopy (fNIR) devices are very portable and can be used daily, they provide little in the way of image resolution. Ultrasound, as a medical imaging method, is less constrained; it has both very large devices that can collect high- quality images and handheld devices that are more practically useful in the day-to- day. However, handheld devices nearly always output lower-resolution images than their counterparts. This lower resolution makes location of tools and structures within the ultrasound image more difficult.

Summary of the Invention

[0004] In one implementation, a system is provided for guiding a procedure. The system includes an echograph system that provides an ultrasound image of a region of interest, a processor, a device, and a non-transitory computer readable medium storing instructions executable by the processor for providing an augmented representation of the image of the region of interest. The executable instructions are executable to provide a convolutional neural network that generates a predicted area and trajectory for a needle from the ultrasound image. The convolutional neural network is trained on a plurality of ultrasound images with corresponding labelled images having an area representing the needle indicated by a human expert using a loss function that, for a given ultrasound training image and the corresponded labelled image, penalizes mismatches between the predicted area generated from the given ultrasound training image and the area indicated by the human expert on the corresponding labelled image. An output interface provides the ultrasound image to the output device with the predicted area and trajectory for the needle indicated.

[0005] In another implementation, a method is provided for guiding a procedure. An ultrasound image of a region of interest is received and a predicted area for a needle and a landmark anatomical structure is generated from the ultrasound image from a convolutional neural network. The convolutional neural network is trained on a plurality of ultrasound images with corresponding labelled images having areas representing the needle and the landmark anatomical structure indicated by a human expert using a loss function that, for a given ultrasound training image and the corresponded labelled image, penalizes mismatches between the predicted areas generated from the given ultrasound training image and the areas indicated by the human expert on the corresponding labelled image. The ultrasound image is displayed to a user with the predicted area and trajectory for the needle and the predicted area for the landmark anatomical structure highlighted at an associated display.

[0006] In a further implementation, a system is provided for guiding a biopsy. The system includes an echograph system that provides an ultrasound image of a region of interest, a processor, a display, and a non-transitory computer readable medium storing instructions executable by the processor for providing an augmented representation of the image of the region of interest. The executable instructions are executable to provide a first convolutional neural network that generates a predicted area for a biopsy needle from the ultrasound image. The convolutional neural network is trained on a plurality of ultrasound images with corresponding labelled images having an area representing the biopsy needle indicated by a human expert using a loss function that, for a given ultrasound training image and the corresponded labelled image, penalizes mismatches between the predicted area generated from the given ultrasound training image and the area indicated by the human expert on the corresponding labelled image. A second convolutional neural network generates an upsampled representation of the ultrasound image, and a user interface displays the upsampled representation of the ultrasound image to a user with the predicted area for the biopsy needle highlighted at an associated display.

Brief Description of the Drawings

[0007] The foregoing and other features of the present disclosure will become apparent to those skilled in the art to which the present disclosure relates upon reading the following description with reference to the accompanying drawings, in which:

[0008] FIG. 1 illustrates one example of a system for guiding a procedure;

[0009] FIG. 2 illustrates an example of a system for guiding a biopsy;

[0010] FIG. 3 illustrates a method for guiding a procedure;

[0011] FIG. 4 illustrates another method for guiding a biopsy; and

[0012] FIG. 5 is a schematic block diagram illustrating an exemplary system of hardware components capable of implementing examples of the systems and methods disclosed herein.

Detailed Description

[0013] In the context of the present disclosure, the singular forms “a,” “an” and “the” can also include the plural forms, unless the context clearly indicates otherwise. The terms “comprises” and/or “comprising,” as used herein, can specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups. [0014] As used herein, the term “and/or” can include any and all combinations of one or more of the associated listed items.

[0015] Additionally, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, a “first” element discussed below could also be termed a “second” element without departing from the teachings of the present disclosure. The sequence of operations (or acts/steps) is not limited to the order presented in the claims or figures unless specifically indicated otherwise.

[0016] As used herein, the term “substantially identical” or “substantially equal” refers to articles or metrics that are identical other than measurement error.

[0017] As used herein, an “outline” of a region is a line that covers at least seventy-five percent of the circumference of the region in a manner that clearly delineates the region.

[0018] As used herein, a “handheld ultrasound imager” is an imager system in which each of the ultrasound probe and the display can be held simultaneously by a human being.

[0019] As used herein, an “upsampled image” is a representation of an image having a higher resolution than the original image.

[0020] The systems and methods described herein provide an improved display of a region of interest associated with a procedure to assist a physician or other medical professional. The systems and methods provide an automated identification and highlighting of a needle to allow the medical professional to more readily target a region while avoiding structures for which penetration by the needle would be detrimental. In one implementation, one or more landmark anatomical structures for the procedure can also be highlighted to assist the physician in locating the target region.

[0021] FIG. 1 illustrates one example of a system 100 for guiding a procedure. An echograph system 102 is configured to capture an ultrasound image of a region of interest. In one implementation, the echograph system 102 is a handheld ultrasound imaging system. The system 100 further includes a processor 104, an output device 106, and a non-transitory computer readable medium 110 that stores instructions executable by the processor for providing an augmented representation of the image of the region of interest.

[0022] The executable instructions stored on the computer readable medium 110 provide a convolutional neural network 112 that generates a predicted area and trajectory for a needle from the ultrasound image. The convolutional neural network 112 can be trained on a plurality of ultrasound images with corresponding labelled images having an area representing the needle indicated by a human expert. In one implementation, the convolutional neural network 112 is trained using a loss function that, for a given ultrasound training image and the corresponded labelled image, penalizes mismatches between the predicted area generated from the given ultrasound training image and the area indicated by the human expert on the corresponding labelled image. For example, the loss function can be implemented using a Jaccard function or a Dice function that evaluates the degree of overlap between the predicted area and the labelled area. In one example, the convolutional neural network 1 12 can predict an area for each of the needle and one or more anatomical structures of interest, such as a landmark anatomical structure for the procedure. In such a case, the convolutional neural network 112 can be trained with labelled images that also include these regions.

[0023] In one implementation, the convolutional neural network 112 can be implemented as two sets of convolutional units. A first set of convolutional units each includes a plurality of convolutional layers, an activation layer, and a pooling layer. Accordingly, each of the first set of convolutional units tends to shrink the size of the input and increase the number of channels, such that each unit has an input with a larger size and a smaller number of channels than its output. The output of the first set of convolutional units is provided to a second set of convolutional units, each including a plurality of convolutional layers, an activation layer, and an upscaling layer. As a result, each of the second set of convolutional units tends to expand the size of the input and decrease the number of channels, such that each unit has an input with a smaller size and a larger number of channels than its output. In one example, each of the convolutional units is implemented as a block with two convolutional layers, each of them followed by an activation layer, with batch normalization. The activation layer can use any appropriate activation function including a linear function, a sigmoid function, a hyperbolic tangent, a rectified linear unit (RELU), or a softmax function. In the first set of convolutional layers, maxpooling operations are used in an encoding path to downsample the feature maps resolution, while bilinear upsampling operations followed by convolutional blocks were applied in the second set of convolutional units to recover the original image size.

[0024] An output interface 114 provides the ultrasound image to an output device with the predicted area and trajectory for the needle indicated to an output device 106, such as a display or a robotic surgery device. It will be appreciated that the displayed images are intended to guide a live procedure, and thus the convolutional neural network 112 must identify the needle and any other target regions sufficiently rapidly to maintain a video feed to the user or a real-time indication to any automated systems. To facilitate real-time identification of the relevant objects in the procedure, the images provided to the convolutional neural network during operation and training can be resized to 256x256 images. In practice, the number of parameters (e.g., link weights) needed to implement the convolutional neural network can be maintained at under seventy million.

[0025] FIG. 2 illustrates another example of a system 200 for guiding a biopsy. A handheld ultrasound imaging system 202 is configured to capture an ultrasound image of a region of interest. The system 200 further includes a processor 204, a display 206, and a non-transitory computer readable medium 210 that stores instructions executable by the processor for providing an augmented representation of the image of the region of interest. The executable instructions stored on the computer readable medium 210 provide a resizing component 212 that generates a version of the image having a reduced resolution and a first convolutional neural network (CNN) 214 that generates a representation of the image having an enhanced resolution, and the captured ultrasound image is provided to each of the resizing component and the first convolutional neural network. [0026] The resizing component 212 applies a filter to the ultrasound image to reduce the size of the image to a desired resolution, for example, via a downsampling algorithm, such as a sine-based image resampler or a box filter algorithm. The output of the resizing component is provided to a second convolution neural network 216. The second convolutional neural network 216 can be trained on a plurality of ultrasound images with corresponding labelled images having an area representing at least the biopsy needle and a landmark anatomical structure that is indicated by a human expert. In one example, for a thyroid biopsy, the trachea layer and the anterior strap muscle layer can be labelled. In another example, for a carotid biopsy, the common carotid layer, the sternocleidomastoid muscle layer, and the thyroid layer can be labelled. In one implementation, the convolutional neural network 216 is trained using a loss function that, for a given ultrasound training image and the corresponded labelled image, penalizes mismatches between the predicted area generated from the given ultrasound training image and the area indicated by the human expert on the corresponding labelled image. To facilitate this process, a region definition component 217 can process the labelled image to indicate all pixels inside of the indicted area based on a complete or substantially complete outline of the area provided by the human expert. In one implementation, between one thousand five hundred and two thousand five hundred labelled images can be used for training. The output of the second convolutional neural network 216, representing the location and trajectory of the biopsy needle and a landmark anatomical structure within the image, is provided to an overlay component 218.

[0027] The first convolutional neural network 214 generates an upsampled representation of the captured ultrasound image. In one implementation, the ultrasound image can be preprocessed using various augmentation methods at a preprocessing component 220. The image can also be subjected to an elastic transform to adjust a shape of the image, as ultrasound images can have standard non-rectangular shapes and printed text that cannot be readily separated from images. In one example, a motion detection algorithm can be used to crop images and mask nonmoving pixels, both for training samples and during operation. [0028] The first convolutional neural network 214 is trained as a generative adversarial network (GAN), specifically the generative network in the GAN. The GAN includes the first convolutional neural network 214, which generates candidates during training, a second generative network that generates lower resolution versions of the generated candidates for comparison to the original input to the first neural network, and a discriminative network that evaluates the candidates to determine if they are actual training examples or outputs of the first convolutional neural network. The discriminative network can be implemented with a similar architecture to the second convolution neural network 216.

[0029] The first convolutional neural network 214 learns to map from a latent space to a data distribution of interest from a set of training samples, while the discriminative network distinguishes candidates produced by the generator from the true data distribution. In the illustrated implementation, the set of training samples can include a first set of low-resolution ultrasound images taken at a handheld ultrasound device and second set of higher resolution ultrasound images taken at a portable ultrasound device. In practice, there is no need to match the content of images between the first set of ultrasound images and the second set of ultrasound images, but in one implementation, each pair of images represents views of the same anatomical structure from two different patents. The generative network's training objective is to increase the error rate of the discriminative network. In the illustrated implementation, for a mapping G . X^ Y that translates low resolution ultrasound image Xto high resolution representation Zand the image discriminator DY that attempts to distinguish between real and fake images, the generator function is given by:

[0030] A cyclic consistency function, which ensures that images can be translated back to their original input, is given by:

Lcyc(G, F) = E_x~pdata(X)[IIF(G(x)) - x//1 ] + E_x~_pdata(x)[HG(F (y)) - y//1] Eq. 2

These are combined to a final loss function: L(G, F, DY, X, Y) = L_GAN G, Dy, X, Y ) + L_GAN F, DX, Y, X) + AL_cyc G, F)

Eq. 3

[0031] The loss function relies on both of the discriminator loss, which attempts to discriminate between real instances and fake instances, thus incentivizing the network to create realistic outputs, and cycle consistency loss, which incentivizes the network to be able to both translate an image to a new domain and translate it back to a reconstruction that looks as similar to the input as possible, thus disallowing the network from creating output images that look nothing like the input.

[0032] The higher-resolution representation of the input image generated at the first convolutional neural network 214 is provide to the overlay component 218 which generates a combined image from the outputs of the first convolutional neural network and the second convolutional neural network 216. Specifically, the overlay component 218 generates an image using the higher resolution representation as a base, with the locations indicated by the second convolutional neural network 216 highlighted. This image can be provided to the user at the display 206 via a user interface 222. It will be appreciated that this provides the user with both a higher representation version of the image for recognizing important anatomical landmarks as well as decision support in locating the biopsy needle and important anatomical landmarks. As a result, the safety and efficiency of the biopsy procedure can be enhanced.

[0033] In view of the foregoing structural and functional features described above, example methods will be better appreciated with reference to FIGS. 3 and 4. While, for purposes of simplicity of explanation, the example methods of FIGS. 3 and 4 are shown and described as executing serially, it is to be understood and appreciated that the present examples are not limited by the illustrated order, as some actions could in other examples occur in different orders, multiple times and/or concurrently from that shown and described herein. Moreover, it is not necessary that all described actions be performed to implement a method.

[0034] FIG. 3 illustrates a method 300 for guiding a procedure. At 302, an ultrasound image of a region of interest is received. In one implementation, the ultrasound innage of the region of interest is generated using a handheld ultrasound imager. At 304, a predicted area and trajectory for a needle and a landmark anatomical structure are generated from the ultrasound image at a convolutional neural network. The convolutional neural network is trained on a plurality of ultrasound images with corresponding labelled images having areas representing the needle and the landmark anatomical structure indicated by a human expert using a loss function that, for a given ultrasound training image and the corresponded labelled image, penalizes mismatches between the predicted areas generated from the given ultrasound training image and the areas indicated by the human expert on the corresponding labelled image. In one implementation, the plurality of ultrasound images are generated by allowing a user to outline the needle and the landmark anatomical structure on each of the plurality of ultrasound images and selecting all pixels within the outline as the areas indicated by the human expert. At 306, the ultrasound image is displayed to a user with the predicted areas for the needle and the landmark anatomical structure highlighted at an associated display.

[0035] FIG. 4 illustrates another method 400 for guiding a biopsy. At 402, an ultrasound image of a region of interest is received. In one implementation, the ultrasound image of the region of interest is generated using a handheld ultrasound imager. At 404, a predicted area and trajectory for a biopsy needle and a landmark anatomical structure are generated from the ultrasound image at a first convolutional neural network. The first convolutional neural network is trained on a plurality of ultrasound images with corresponding labelled images having areas representing the biopsy needle and the landmark anatomical structure indicated by a human expert using a loss function that, for a given ultrasound training image and the corresponded labelled image, penalizes mismatches between the predicted areas generated from the given ultrasound training image and the areas indicated by the human expert on the corresponding labelled image. In one implementation, the plurality of ultrasound images are generated by allowing a user to outline the biopsy needle and the landmark anatomical structure on each of the plurality of ultrasound images and selecting all pixels within the outline as the areas indicated by the human expert. [0036] At 406, an upsampled representation of the ultrasound image is generated at a second convolutional neural network. In one implementation, the second convolutional neural network is a generative component of a generative adversarial network trained on a first set of images, having a resolution substantially equal to a resolution of the received ultrasound image and a second set of images having a resolution greater than the resolution of the received ultrasound image. At 408, the upsampled representation of the ultrasound image is displayed to the user at an associated display with the predicted area for the biopsy needle and the landmark anatomical structure highlighted.

[0037] FIG. 5 is a schematic block diagram illustrating an exemplary system 500 of hardware components capable of implementing examples of the systems and methods disclosed herein. The system 500 can include various systems and subsystems. The system 500 can be a personal computer, a laptop computer, a workstation, a computer system, an appliance, an application-specific integrated circuit (ASIC), a server, a server BladeCenter, a server farm, etc.

[0038] The system 500 can include a system bus 502, a processing unit 504, a system memory 506, memory devices 508 and 510, a communication interface 512 (e.g., a network interface), a communication link 514, a display 516 (e.g., a video screen), and an input device 518 (e.g., a keyboard, touch screen, and/or a mouse). The system bus 502 can be in communication with the processing unit 504 and the system memory 506. The additional memory devices 508 and 510, such as a hard disk drive, server, standalone database, or other non-volatile memory, can also be in communication with the system bus 502. The system bus 502 interconnects the processing unit 504, the memory devices 506-510, the communication interface 512, the display 516, and the input device 518. In some examples, the system bus 502 also interconnects an additional port (not shown), such as a universal serial bus (USB) port.

[0039] The processing unit 504 can be a computing device and can include an application-specific integrated circuit (ASIC). The processing unit 504 executes a set of instructions to implement the operations of examples disclosed herein. The processing unit can include a processing core.

[0040] The additional memory devices 506, 508, and 510 can store data, programs, instructions, database queries in text or compiled form, and any other information that may be needed to operate a computer. The memories 506, 508 and 510 can be implemented as computer-readable media (integrated or removable), such as a memory card, disk drive, compact disk (CD), or server accessible over a network. In certain examples, the memories 506, 508 and 510 can comprise text, images, video, and/or audio, portions of which can be available in formats comprehensible to human beings. Additionally or alternatively, the system 500 can access an external data source or query source through the communication interface 512, which can communicate with the system bus 502 and the communication link 514.

[0041] In operation, the system 500 can be used to implement one or more parts of a system in accordance with the present invention. Computer executable logic for implementing the diagnostic system resides on one or more of the system memory 506, and the memory devices 508 and 510 in accordance with certain examples. The processing unit 504 executes one or more computer executable instructions originating from the system memory 506 and the memory devices 508 and 510. The term "computer readable medium" as used herein refers to a medium that participates in providing instructions to the processing unit 504 for execution. This medium may be distributed across multiple discrete assemblies all operatively connected to a common processor or set of related processors.

[0042] Implementation of the techniques, blocks, steps, and means described above can be done in various ways. For example, these techniques, blocks, steps, and means can be implemented in hardware, software, or a combination thereof. For a hardware implementation, the processing units can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro- controllers, microprocessors, other electronic units designed to perform the functions described above, and/or a combination thereof.

[0043] Also, it is noted that the embodiments can be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged. A process is terminated when its operations are completed but could have additional steps not included in the figure. A process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

[0044] Furthermore, embodiments can be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and/or any combination thereof. When implemented in software, firmware, middleware, scripting language, and/or microcode, the program code or code segments to perform the necessary tasks can be stored in a machine- readable medium such as a storage medium. A code segment or machineexecutable instruction can represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or any combination of instructions, data structures, and/or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, and/or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, ticket passing, network transmission, etc.

[0045] For a firmware and/or software implementation, the methodologies can be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. Any machine-readable medium tangibly embodying instructions can be used in implementing the methodologies described herein. For example, software codes can be stored in a memory. Memory can be implemented within the processor or external to the processor. As used herein the term “memory” refers to any type of long term, short term, volatile, nonvolatile, or other storage medium and is not to be limited to any particular type of memory or number of memories, or type of media upon which memory is stored.

[0046] Moreover, as disclosed herein, the term "storage medium" can represent one or more memories for storing data, including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and/or other machine-readable mediums for storing information. The term "machine-readable medium" includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, and/or various other storage mediums capable of storing that contain or carry instruction(s) and/or data.

[0047] What have been described above are examples. It is, of course, not possible to describe every conceivable combination of components or methodologies, but one of ordinary skill in the art will recognize that many further combinations and permutations are possible. Accordingly, the disclosure is intended to embrace all such alterations, modifications, and variations that fall within the scope of this application, including the appended claims. As used herein, the term "includes" means includes but not limited to, the term "including" means including but not limited to. The term "based on" means based at least in part on.

Claims

What is claimed is:

1 . A system for guiding a procedure, the system comprising: an echograph system that provides an ultrasound image of a region of interest; a processor; a non-transitory computer readable medium storing instructions executable by the processor for providing an augmented representation of the image of the region of interest, the executable instructions comprising: a convolutional neural network that generates a predicted area and trajectory for a needle from the ultrasound image, the convolutional neural network being trained on a plurality of ultrasound images with corresponding labelled images having an area representing the needle indicated by a human expert using a loss function that, for a given ultrasound training image and the corresponded labelled image, penalizes mismatches between the predicted area generated from the given ultrasound training image and the area indicated by the human expert on the corresponding labelled image; and an output interface that provides the ultrasound image to an associated output device with the predicted area and trajectory for the needle indicated; and the output device.

2. The system of claim 1 , wherein the convolutional neural network further generates a predicted area for a landmark anatomical structure for the procedure from the ultrasound image, the convolutional neural network being trained on a plurality of ultrasound images with corresponding labelled images having an area representing the landmark anatomical structure indicated by a human expert using the loss function and the area representing the needle, the output interface providing the ultrasound image to the output device with each of the predicted area for the needle and the area representing the landmark anatomical structure indicated.

3. The system of claim 1 , wherein the output device is a display and the convolutional neural network is a first convolutional neural network, the system further comprising a second convolutional neural network that generates an upsampled representation of the ultrasound image, the output interface displaying the upsampled representation of the ultrasound image to the user with the predicted area for the needle highlighted at the display.

4. The system of claim 3, wherein the second convolutional neural network is trained with a plurality of pairs of images, each pair of images comprising a first image of an anatomical structure using a handheld ultrasound imager and a second image of the anatomical structure using a different imaging modality.

5. The system of claim 4, wherein the first image of the anatomical structure is acquired by imaging a first patient and the second image of the anatomical structure is acquired by imaging a second patient.

6. The system of claim 4, wherein the different imaging modality is a portable ultrasound imager having a resolution that is higher than a resolution of the handheld ultrasound imager.

7. The system of claim 3, wherein the second convolutional neural network is trained by optimizing a utility function comprising a first parameter representing a comparison of the upsampled representation to other images generated via the different imaging modality and a second parameter representing the similarity of a downsampled version of the upsampled representation to the ultrasound image.

8. The system of claim 3, the first convolutional neural network comprising a first set of a plurality of convolutional units, each comprising a plurality of convolutional layers, an activation layer, and a pooling layer, an input to each of the first set of convolutional units having a larger size and a smaller number of channels than the output of the convolutional unit, and a second set of convolutional units, each comprising a plurality of convolutional layers, an activation layer, and an upscaling layer, an input to each of the second set of convolutional units having a smaller size and a larger number of channels than the output of the convolutional unit.

9. The system of claim 1 , further comprising a region definition component that indicates all pixels inside of the area representing the needle based on an outline of the area provided by the human expert.

10. The system of claim 1 , wherein the output device is a display and the output interface is a user interface that displays the ultrasound image to a user with the predicted area for the needle highlighted at the display.

11. A method for guiding a procedure, the method comprising: receiving an ultrasound image of a region of interest; generating a predicted area and trajectory for a needle and a landmark anatomical structure from the ultrasound image from a convolutional neural network trained on a plurality of ultrasound images with corresponding labelled images having areas representing the needle and the landmark anatomical structure indicated by a human expert using a loss function that, for a given ultrasound training image and the corresponded labelled image, penalizes mismatches between the predicted areas generated from the given ultrasound training image and the areas indicated by the human expert on the corresponding labelled image; and displaying the ultrasound image to a user with the predicted area and trajectory for the needle and the predicted area for the landmark anatomical structure highlighted at an associated display.

12. The method of claim 11 , wherein the convolutional neural network is a first convolutional neural network, the method further comprising generating an upsampled representation of the ultrasound image at a second convolutional neural network, wherein displaying the ultrasound image to the user comprises displaying the upsampled representation of the ultrasound image with the predicted area for the needle and the landmark anatomical structure highlighted at the display.

13. The method of claim 12, wherein the second convolutional neural network is a generative component of a generative adversarial network trained on a first set of images, having a resolution substantially equal to a resolution of the received ultrasound image and a second set of images having a resolution greater than the resolution of the received ultrasound image.

14. The method of claim 11 , wherein receiving the ultrasound image of the region of interest comprising receiving the ultrasound image of the region of interest from a handheld ultrasound imager.

15. The method of claim 11 , further comprising generating the plurality of ultrasound images by allowing a user to outline the needle and the landmark anatomical structure on each of the plurality of ultrasound images and selecting all pixels within the outline as the areas indicated by the human expert.

16. A system for guiding a biopsy, the system comprising: an echograph system that provides an ultrasound image of a region of interest; a processor; a non-transitory computer readable medium storing instructions executable by the processor for providing an augmented representation of the image of the region of interest, the executable instructions comprising: a first convolutional neural network that generates a predicted area and trajectory for a biopsy needle from the ultrasound image, the convolutional neural network being trained on a plurality of ultrasound images with corresponding labelled images having an area representing the biopsy needle indicated by a human expert using a loss function that, for a given ultrasound training image and the corresponded labelled image, penalizes mismatches between the predicted area generated from the given ultrasound training image and the area indicated by the human expert on the corresponding labelled image; a second convolutional neural network that generates an upsampled representation of the ultrasound image; and a user interface that displays the upsampled representation of the ultrasound image to a user with the predicted area and trajectory for the biopsy needle highlighted at an associated display; and the display.

17. The system of claim 16, wherein the first convolutional neural network further generates a predicted area for a landmark anatomical structure for the biopsy from the ultrasound image, the first convolutional neural network being trained on a plurality of ultrasound images with corresponding labelled images having an area representing the landmark anatomical structure indicated by a human expert using the loss function and the area representing the biopsy needle, the user interface displaying the upsampled representation of the ultrasound image with each of the predicted area for the biopsy needle and the area representing the landmark anatomical structure highlighted.

18. The system of claim 16, the first convolutional neural network comprising a first set of a plurality of convolutional units, each comprising a plurality of convolutional layers, an activation layer, and a pooling layer, an input to each of the first set of convolutional units having a larger size and a smaller number of channels than the output of the convolutional unit, and a second set of convolutional units, each comprising a plurality of convolutional layers, an activation layer, and an upscaling layer, an input to each of the second set of convolutional units having a smaller size and a larger number of channels than the output of the convolutional unit.

19. The system of claim 16, wherein the second convolutional neural network is trained with a plurality of pairs of images, each pair of images comprising a first image of an anatomical structure from a first patient using a handheld ultrasound imager and a second image of the anatomical structure from a second patient using a portable ultrasound imager having a resolution that is higher than a resolution of the handheld ultrasound imager.

20. The system of claim 16, further comprising a region definition component that selects all pixels inside of the area representing the biopsy needle based on an outline of the area provided by the human expert.