WO2025054125A1

WO2025054125A1 - Systems and methods for image noise estimation and image noise compensation

Info

Publication number: WO2025054125A1
Application number: PCT/US2024/045018
Authority: WO
Inventors: Ramin Samadani; Jeffrey M. Dicarlo
Original assignee: Intuitive Surgical Operations Inc
Current assignee: Intuitive Surgical Operations Inc
Priority date: 2023-09-06
Filing date: 2024-09-03
Publication date: 2025-03-13
Anticipated expiration: 2026-03-06

Abstract

An image processing system may process a frame of captured image data at each image processing stage of a plurality of image processing stages of an image processing pipeline to generate a frame of processed image data. Based on the processing the frame of captured image data at each image processing stage of the plurality of image processing stages, the image processing system may determine a value of a variance component of noise for a pixel in the frame of processed image data. The image processing system may generate a frame of adjusted image data comprising a pixel having a signal value that is based on the processed image data and the value of the variance component of noise for the pixel in the frame of processed image data.

Description

SYSTEMS AND METHODS FOR IMAGE NOISE ESTIMATION AND IMAGE NOISE COMPENSATION

RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63/536,832, filed September 6, 2023, which is hereby incorporated by reference in its entirety.

BACKGROUND INFORMATION

[0002] During a minimally-invasive medical procedure, an imaging system (e.g., an endoscope) may be used to capture images of a scene illuminated with light from a light source. The images (e.g., a video stream) may be presented during the medical procedure to assist an operator in performing the medical procedure. The images may be visible light images, fluorescence images, or visible light images augmented with fluorescence images. Fluorescence images are generated based on detected fluorescence emitted by fluorophores present at the scene when the fluorophores are excited by fluorescence excitation light (e.g., near-infrared (NIR) light). The fluorescence images may be used, for example, to highlight certain portions of the scene, certain types of tissue, or tissue perfusion at the scene in a selected color (e.g., green).

[0003] However, captured images can be negatively affected by image noise, such as shot noise and read noise. Shot noise results from fluctuations over time in the number of photons that are incident on the sensor. Read noise arises from sensor components, such as amplifiers and circuits, which may add noise even in the absence of detected light. Image noise negatively affects visible light images, fluorescence images, and augmented images. Additionally, low fluorescent camera signals, even with ample illumination, suffer from low signal-to-noise ratio.

SUMMARY

[0004] The following description presents a simplified summary of one or more aspects of the systems and methods described herein. This summary is not an extensive overview of all contemplated aspects and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present one or more aspects of the systems and methods described herein as a prelude to the detailed description that is presented below.

[0005] An illustrative system may comprise a memory storing instructions and one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: processing a frame of captured image data at each image processing stage of a plurality of image processing stages of an image processing pipeline to generate a frame of processed image data; determining, based on the processing the frame of captured image data at each image processing stage of the plurality of image processing stages, a value of a variance component of noise for a pixel in the frame of processed image data; and generating a frame of adjusted image data comprising a pixel having a signal value that is based on the processed image data and the value of the variance component of noise for the pixel in the frame of processed image data.

[0006] An illustrative system may comprise a memory storing instructions and one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: processing, at a first image processing stage of an image processing pipeline, a pixel in a frame of captured image data; determining, at a first variance processing stage corresponding to the first image processing stage, a value of a variance component of noise for the pixel based on the processing the pixel at the first image processing stage; processing, after the processing the pixel at the first image processing stage, the pixel at a second image processing stage of the image processing pipeline; adjusting, at a second variance processing stage corresponding to the second image processing stage, the value of the variance component of noise for the pixel based on the processing the pixel at the second image processing stage; and adjusting, at a third image processing stage after the processing the pixel at the second image processing stage, a signal value of the pixel based on the value of the variance component of noise for the pixel.

[0007] An illustrative method may comprise: processing a frame of captured image data at each image processing stage of a plurality of image processing stages of an image processing pipeline to generate a frame of processed image data; determining, based on the processing the frame of captured image data at each image processing stage of the plurality of image processing stages, a value of a variance component of noise for a pixel in the frame of processed image data; and generating a frame of adjusted image data comprising a pixel having a signal value that is based on the processed image data and the value of the variance component of noise for the pixel in the frame of processed image data.

[0008] An illustrative method may comprise: processing, at a first image processing stage of an image processing pipeline, a pixel in a frame of captured image data; determining, at a first variance processing stage corresponding to the first image processing stage, a value of a variance component of noise for the pixel based on the processing the pixel at the first image processing stage; processing, after the processing the pixel at the first image processing stage, the pixel at a second image processing stage of the image processing pipeline; adjusting, at a second variance processing stage corresponding to the second image processing stage, the value of the variance component of noise for the pixel based on the processing the pixel at the second image processing stage; and adjusting, at a third image processing stage after the processing the pixel at the second image processing stage, a signal value of the pixel based on the value of the variance component of noise for the pixel.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings illustrate various embodiments and are a part of the specification. The illustrated embodiments are merely examples and do not limit the scope of the disclosure. Throughout the drawings, identical or similar reference numbers designate identical or similar elements.

[0010] FIG. 1 shows an illustrative configuration of an imaging system configured to capture visible light images and fluorescence images of a scene.

[0011] FIG. 2 shows a functional diagram of an illustrative image processing system that may estimate a value of a variance component of image noise for a pixel in a frame of image data.

[0012] FIG. 3 shows an illustrative image processing scheme for processing a frame of captured image data in an image processing pipeline.

[0013] FIG. 4A shows an illustrative image processing scheme for processing covariance terms within an image processing pipeline.

[0014] FIG. 4B shows an alternative illustrative image processing scheme for processing covariance terms within an image processing pipeline.

[0015] FIG. 5 shows a graph with a set of illustrative transfer functions for adjusting a signal value of a pixel. [0016] FIG. 6 shows an illustrative display image that includes a displayed confidence score indication for pixels in the display image.

[0017] FIG. 7 shows an illustrative method of estimating a value of a variance component of image noise in a frame of processed image data.

[0018] FIG. 8 shows another illustrative method of estimating a value of a variance component of image noise in a frame of processed image data.

[0019] FIG. 9 shows an illustrative computer-assisted surgical system that may be used in conjunction with the systems and methods described herein.

[0020] FIG. 10 shows a functional diagram of an illustrative computing device that may be specifically configured to perform one or more of the processes described herein.

DETAILED DESCRIPTION

[0021] Systems and methods for image noise estimation and image noise compensation are described herein. In some examples, an image processing system may process a frame of captured image data at each stage of a plurality of stages of an image processing pipeline to generate a frame of processed image data. Based on the processing the frame of captured image data at each stage of the plurality of stages of the image processing pipeline, the image processing system may determine a value of a variance component of noise for a pixel in the frame of processed image data. The image processing system may generate a frame of adjusted image data comprising a pixel having a pixel value that is based on the processed image data and the value of the variance component of noise for the pixel in the frame of processed image data.

[0022] For example, the image processing system may generate a frame of adjusted image data (e.g., a frame of adjusted fluorescence image data) by adjusting the signal value for the pixel in a frame of fluorescence image data based on the value of the variance component of noise for the pixel. In other examples, the image processing system may generate a frame of adjusted image data by adjusting the signal value for the pixel in a frame of fluorescence image data (e.g., a frame of adjusted fluorescence image data) based on a transfer function that is selected for the pixel based on the value of the variance component of noise for the pixel. In yet further examples, the frame of adjusted image data (e.g., a frame of adjusted visible light image data) is generated by applying a temporal filter based on the value of the variance component of noise for the pixel. In even further examples, the frame of adjusted image data (e.g., a frame of adjusted fluorescence image data) includes a confidence score indication that is indicative of a confidence score that is based on the value of the variance component of noise for the pixel. These and other examples of a frame of adjusted image data will be described below in more detail.

[0023] The systems and methods for image noise estimation and noise compensation described herein provide various benefits. For example, the systems and methods described herein enable estimation of the variance component of noise on a pixel-by-pixel basis for each frame of image data, and at each stage of an image processing pipeline. This per-pixel estimation of the variance component of noise provides a more accurate model of noise in an image as compared with conventional systems and methods. In some examples, the per-pixel estimation of the variance component of noise can be used to generate display images with reduced noise and improved sensitivity, and, in fluorescence images, more reliable fluorescence signals, even when fluorescence excitation light and/or fluorescence signals are low. The per-pixel estimation of the variance component of noise can also be used to generate display images that indicate a confidence score for the displayed content. These and other benefits of the systems and methods described herein will be made apparent in the description that follows.

[0024] Various examples of the systems and methods will be described in detail with reference to the figures. It will be understood that the examples described below are provided as non-limiting examples of how various novel and inventive principles may be applied in various situations. Additionally, it will be understood that other examples not explicitly described herein may also be captured by the scope of the claims set forth below. Systems and methods described herein may provide one or more benefits that will be described or made apparent below.

[0025] FIG. 1 shows an illustrative configuration of an imaging system 100 configured to capture visible light images and fluorescence images of a scene. In some examples, the scene includes an area associated with a subject on or within which a fluorescence- assisted medical procedure is being performed. The subject may be, for example, a body of a live human or animal, a human or animal cadaver, a portion of human or animal anatomy, tissue removed from human or animal anatomies, non-tissue work pieces, or a training model. In other examples, the scene may be a non-medical scene, such as a scene captured for calibration or operational assessment purposes.

[0026] Imaging system 100 includes an imaging device 102 and a controller 104. Imaging system 100 may include additional or alternative components as may serve a particular implementation, such as various optical and/or electrical signal transmission components (e.g., wires, lenses, optical fibers, choke circuits, waveguides, cables, etc.). As shown, imaging system 100 comprises a visible light imaging system integrated with a fluorescence imaging system. In some examples, a visible light imaging system and a fluorescence imaging system may be physically integrated into the same physical, optical, and/or electrical components (e.g., the same camera sensor, illumination channels, camera head, controller, etc.). In alternative examples, a standalone fluorescence imaging system may be inserted into an assistance port of a visible light imaging system. While imaging system 100 shown and described herein comprises a visible light imaging system integrated with a fluorescence imaging system, imaging system 100 may alternatively be implemented by a standalone visible light imaging system configured to capture only visible light images of the scene. Accordingly, components of imaging system 100 that function only to capture fluorescence images may be omitted. In further alternatives, imaging system 100 may be implemented by a standalone fluorescence imaging system configured to capture only fluorescence images of the scene. Accordingly, components of imaging system 100 that function only to capture visible light images may be omitted.

[0027] Imaging device 102 may be implemented by any suitable device configured to capture visible light images and fluorescence images of a scene. As shown, imaging device 102 includes a camera head 106 and a shaft 108 coupled to and extending away from camera head 106. Imaging device 102 may be manually handled and controlled (e.g., by a surgeon performing a surgical procedure on a subject). Alternatively, camera head 106 may be coupled to a manipulator arm of a computer-assisted surgical system and controlled using robotic and/or teleoperation technology. A distal end of shaft 108 may be positioned at or near the scene that is to be imaged by imaging device 102. For example, the distal end of shaft 108 may be inserted into a patient via a cannula (not shown). In some examples, imaging device 102 is implemented by an endoscope. As shown by arrow A in FIG. 1 , “distal” means situated near or toward the scene or region of interest (e.g., away from controller 104) and “proximal” means situated away from the scene or region of interest (e.g., near or toward controller 104).

[0028] Imaging device 102 captures images and outputs frames of captured image data 110 to controller 104. Imaging device 102 includes a visible light camera (not shown) configured to capture two-dimensional (2D) or three-dimensional (3D) visible light images of the scene and output frames of captured visible light image data 110-V representative of the visible light images. Imaging device 102 also includes a fluorescence camera (not shown) configured to capture fluorescence images of the scene and output frames of captured fluorescence image data 110-F representative of the fluorescence images. A field of view of imaging device 102 is represented by dashed lines 112.

[0029] The visible light camera and the fluorescence camera may each be implemented by any one or more suitable image sensors configured to detect (e.g., capture, collect, sense, or otherwise acquire) visible light and/or non-visible (e.g., NIR) light, such as a charge coupled device (CCD) image sensor, a complementary metal-oxide semiconductor (CMOS) image sensor, a hyperspectral camera, a multispectral camera, photodetectors based on time-correlated single photon counting (TCSPC) (e.g., a single photon counting detector, a photo multiplier tube (PMT), a single photon avalanche diode (SPAD) detector, etc.), photodetectors based on time-gating (e.g., intensified CCDs), time-of-flight sensors, streak cameras, and the like. In some examples, the visible light camera and the fluorescence camera are implemented by the same image sensor and signals generated by the image sensor are separated into a visible light channel and processed into visible light image data 110-V and a fluorescence channel and processed into fluorescence image data 110-F. In other examples, the visible light camera and the fluorescence camera are implemented by separate sensors. In some examples, the visible light camera and/or the fluorescence camera are positioned at or near the distal end of shaft 108. In alternative examples, the visible light camera and/or the fluorescence camera are positioned closer to the proximal end of shaft 108, inside camera head 106, or outside imaging device 102 (e.g., inside controller 104), and optics included in shaft 108 and/or camera head 106 convey captured light from the scene to the corresponding camera.

[0030] Controller 104 may be implemented by any suitable combination of hardware and/or software configured to control and/or interface with imaging device 102. For example, controller 104 may be at least partially implemented by a computing device included in a computer-assisted surgical system. Controller 104 may include a light source 114 and a camera control unit (CCU) 116. Controller 104 may include additional or alternative components as may serve a particular implementation. For example, controller 104 may include circuitry configured to provide power to components included in imaging device 102. In alternative examples, light source 114 and/or CCU 116 are included in imaging device 102 (e.g., in camera head 106). For example, light source 114 may be positioned at the distal end of shaft 108, closer to the proximal end of shaft 108, or inside camera head 106.

[0031] Light source 114 is configured to illuminate the scene with light (e.g., visible light and fluorescence excitation light). Light emitted by light source 114 (represented by light ray 118) travels by way of a light channel in shaft 108 (e.g., by way of one or more optical fibers, light guides, lenses, etc.) to a distal end of shaft 108, where the light exits to illuminate the scene. Thus, the light source for the scene is the distal end of imaging device 102. Visible light from light source 114 may include a continuous spectrum of light (e.g., white light) or one or more narrowband color components of light, such as a blue component, a green component, and/or a red component. Fluorescence excitation light from light source 114 may include one or more broadband spectra of light (e.g., NIR light) or may include one or more narrowband light components (e.g., narrowband NIR light components). Light source 114 may be implemented by any suitable device, such as a flash lamp, laser source, laser diode, light-emitting diode (LED), and the like. While light source 114 is shown to be a single device, light source 114 may alternatively include multiple light sources each configured to generate and emit differently configured light (e.g., visible light and fluorescence excitation light).

[0032] Visible light emitted from the distal end of shaft 108 is reflected by a surface 120 of the subject at the scene, and the reflected visible light is detected by the visible light camera of imaging device 102. The visible light camera (and/or other circuitry included in imaging device 102) converts the detected visible light into frames of captured visible light image data 110-V representative of one or more visible light images of the scene. The visible light images may include full color images or may be captured in one or more color channels of imaging system 100 (e.g., a blue color channel). Fluorescence excitation light emitted from the distal end of shaft 108 excites fluorophores 122 beneath surface 120, which when excited emit fluorescence that is detected by the fluorescence camera of imaging device 102. The fluorescence camera (and/or other circuitry included in imaging device 102) converts the detected fluorescence into frames of captured fluorescence image data 110-F representative of one or more fluorescence images of the scene. Imaging device 102 transmits the frames of captured visible light image data 110-V and the frames of captured fluorescence image data 110-F via a wired or wireless communication link to CCU 116.

[0033] CCU 116 may be configured to control (e.g., define, adjust, configure, set, etc.) operation of the visible light camera and fluorescence camera and is configured to receive and process the frames of captured visible light image data 110-V and the frames of captured fluorescence image data 110-F. For example, CCU 116 may packetize and/or format the captured visible light image data and the captured fluorescence image data 110- F. CCU 116 outputs the frames of visible light image data 110-V and the frames of fluorescence image data 110-F to an image processing system 124 for further processing. While CCU 116 is shown to be a single unit, CCU 116 may alternatively be implemented by multiple CCUs each configured to control distinct image streams (e.g., a 2D visible light image stream or a left visible light image stream and a right visible light stream, and a fluorescence image stream).

[0034] Image processing system 124 receives the frames of captured image data 110 (e.g., frames of visible light image data 110-V and frames of fluorescence image data 110- F) and processes the frames of captured image data 110 to generate frames of display image data 126 (e.g., frames of visible light display image data 126-V and frames of fluorescence display image data 126-F) for display by a display device 128 (e.g., a computer monitor, a projector, a tablet computer, or a television screen). The frames of display image data 126 may be in the form of one or more still images and/or video streams. As shown, image processing system 124 is included in imaging system 100 (e.g., in controller 104). In alternative examples, image processing system 124 is implemented by one or more computing devices and/or hardware external to imaging system 100, such as one or more computing devices and/or hardware included in a computer-assisted surgical system.

[0035] In some examples, image processing system 124 comprises an image processing pipeline comprising a plurality of image processing stages. Each image processing stage of the image processing pipeline performs an adjustment on frames of captured image data 110 to correct or adjust various image parameters, such as autoexposure, gain, and/or white balance. In frames of captured fluorescence image data 110- F, the image processing pipeline may also false-color fluorescing regions a select color (e.g., green, yellow, or blue) and/or selectively apply a gain to adjust (e.g., increase or decrease) the intensity of the fluorescing regions. In some examples, the image processing pipeline may generate a graphical overlay based on fluorescence image data and combine the graphical overlay with a frame of visible light image data 110-V to form an augmented image (e.g., a visible light image augmented with fluorescence image data).

[0036] The image processing pipeline may be implemented in image processing system 124 by any suitable combination of hardware and/or software configured to perform image processing on signals (e.g., image data). In some examples, the image processing pipeline is implemented in image processing system 124 as multiple discrete functional hardware components through which signals (e.g., image data) are passed sequentially for image processing in different image processing stages. In other examples, the image processing pipeline is implemented in software in which instructions for each image processing stage are executed by a processor in sequence. In further examples, the image processing pipeline is implemented in image processing system 124 as a combination of hardware and software.

[0037] In some examples, image processing system 124 performs a fluorescence signal adjustment process on frames of processed fluorescence image data 110-F to account for non-uniform distribution of fluorescence excitation light. Such non-uniform distributions of fluorescence excitation light may occur, for example, when a distal end of imaging device 102 changes position and/or orientation with respect to surface 120. Additionally, the intensity of the fluorescence excitation light may fall off with distance and/or angle from the light source (e.g., a distal end of imaging device 102). The non-uniform distributions of fluorescence excitation light result in corresponding variations in the intensity of the detected fluorescence signal because the intensity of emitted fluorescence is a function (e.g., proportional) to the intensity of incident fluorescence excitation light.

[0038] To account for such variations, the fluorescence signal adjustment process adjusts (e.g., normalizes) a portion of a frame of fluorescence image data 110-F with respect to a measure of fluorescence excitation light incident on surface 120 depicted by the portion of the frame of fluorescence image data 110-F (the “measure of incident fluorescence excitation light”). For example, the detected fluorescence signal for each pixel in the portion of the frame of fluorescence image data 110-F may be divided by the measure of incident fluorescence excitation light according to the following formula (1 ):

where ci is the detected fluorescence signal at pixel i, is the measure of incident fluorescence excitation light for pixel i, and qi is the adjusted fluorescence signal for pixel i. The measure of incident fluorescence excitation light may be determined or estimated in any suitable way, such as by using one or more sensors and/or an estimated depth of the surface depicted by the pixel i and an incident fluorescence excitation light model. Thus, the adjusted frame of fluorescence image data 110-F is substantially independent of any fluorescence signal variations attributed to the non-uniform distribution of fluorescence excitation light. Illustrative systems and methods for adjusting a frame of fluorescence image data with respect to a measure of incident fluorescence excitation light are described in U.S. Patent Application Publication No. 2022/0015857, published on January 20, 2022, which is incorporated herein by reference in its entirety. [0039] Frames of captured image data 110 have image noise that may affect the quality of frames of display image data 126. The image noise may have a variance component and a mean component (e.g., random bias) that fluctuate significantly overtime. The variance component of the image noise is believed to be caused by sensor noise. The mean component is believed to be caused by algorithmic channel separation errors and, in frames of fluorescence image data 110-F, by fluorescence excitation light leakage. In addition to image noise, weak fluorescence signals, even with ample fluorescence excitation light, suffer from low signal to noise ratio.

[0040] To compensate for the image noise, image processing system 124 may estimate image noise in frames of captured image data 110 and compensate for the estimated noise. For example, image processing system 124 may estimate the value of the variance component of the image noise of each pixel in a region of interest of a frame of captured image data 110 by propagating the value the variance component of the image noise for each pixel through each stage of the image processing pipeline. Illustrative methods of propagating the value of the variance component of the image noise through the image processing pipeline will be described below in more detail. Image processing system 124 may use the estimated value of the variance component for each pixel in the region of interest to generate a frame of display image data 126. In some examples, image processing system 124 may also estimate a value of the mean component of the image noise of the frame of captured image data 110 and use, in addition to the estimated value of the variance component of each pixel, the estimated value of the mean component of the image noise for the frame of captured imaged data 110 to generate the frame of display image data 126.

[0041] FIG. 2 shows a functional diagram of an illustrative image processing system 200 (“system 200”) that may perform any of the image noise estimation and image noise compensation operations described herein. System 200 may be included in, implemented by, or connected to an imaging system, an image processor, a surgical system, and/or any computing system described herein. For example, system 200 may be implemented, in whole or in part, by imaging system 100, image processing system 124, a computer- assisted surgical system, and/or a computing system communicatively coupled to an imaging system, an image processing system, and/or a computer-assisted surgical system. [0042] As shown, system 200 includes, without limitation, memory 202 and processor 204 selectively and communicatively coupled to one another. Memory 202 and processor 204 may each include or be implemented by hardware and/or software components (e.g., processors, memories, communication interfaces, instructions stored in memory for execution by the processors, etc.). For example, memory 202 and processor 204 may be implemented by any component in a computer-assisted surgical system. In some examples, memory 202 and processor 204 may be distributed between multiple devices and/or multiple locations as may serve a particular implementation.

[0043] Memory 202 may maintain (e.g., store) executable data used by processor 204 to perform any of the operations described herein. For example, memory 202 may store instructions 206 that may be executed by processor 204 to perform any of the operations described herein. Instructions 206 may be implemented by any suitable application, software, code, and/or other executable data instance. Memory 202 may also maintain any data received, generated, managed, used, and/or transmitted by processor 204.

[0044] Processor 204 may be configured to perform (e.g., execute instructions 206 stored in memory 202 to perform) various operations described herein. For example, processor 204 may process a frame of captured image data at each stage of a plurality of stages of an image processing pipeline to generate a frame of processed image data.

Processor 204 may determine, based on the processing the frame of captured image data at each stage of the plurality of stages of the image processing pipeline, a value of a variance component of noise for a pixel in the frame of processed image data. Processor 204 may generate a frame of adjusted image data comprising a pixel having a signal value that is based on the processed image data and the value of the variance component of noise for the pixel in the frame of processed image data. Additional illustrative operations that may be performed by processor 204 will be described herein. In the description that follows, any references to operations performed by system 200 may be understood to be performed by processor 204 of system 200.

[0045] FIG. 3 shows an illustrative image processing scheme 300 for processing a frame of captured image data 302 (labeled “Cap 302”) in an image processing pipeline 304.

Frame of captured image data 302 may be obtained from a camera (not shown) and may be a frame of captured visible light image data (e.g., a frame of captured visible light image data 110-V) or a frame of captured fluorescence image data (e.g., a frame of captured fluorescence image data 110-F). In other examples, frame of captured image data 302 may be a mixed signal (e.g., prior to separation into a visible light channel and a fluorescence channel), which may be separated within image processing pipeline 304.

[0046] Image processing pipeline 304 may be executed by system 200 and includes a plurality of image processing stages 306 (e.g., first image processing stage 306-1 to fourth image processing stage 306-4) for processing frame of captured image data 302 to generate a frame of processed image data 308 (labeled “Proc 308”). As shown, image processing pipeline 304 includes four image processing stages 306. However, image processing pipeline 304 may have any other suitable number of image processing stages 306 as may suit a particular implementation. At each image processing stage 306, system 200 processes frame of captured image data 302, such as by adjusting signal values of one or more pixels in the frame of captured image data 302 to adjust a parameter of the frame of captured image data 302.

[0047] For example, a pixel in the frame of captured image data 302 received from a camera has an initial signal value So. At first image processing stage 306-1 , the frame of captured image data 302 is processed so that the signal value of the pixel after first image processing stage 306-1 is Si, which may be the same or different than initial signal value So. At second image processing stage 306-2, the frame of captured image data 302 is processed so that the signal value of the pixel after second image processing stage 306-2 is S2, which may be the same or different than signal value Si. At third image processing stage 306-3, the frame of captured image data 302 is processed so that the signal value of the pixel after third image processing stage 306-3 is S3, which may be the same or different than signal value S2. At fourth image processing stage 306-4, the frame of captured image data 302 is processed so that the signal value of the pixel after fourth image processing stage 306-4 is S4, which may be the same or different than signal value S3. Adjustment of the signal value of the pixel at an image processing stage 306 may be performed in any suitable way using any image processing algorithm or technique. In some examples, image processing stages 306 adjust a gain, adjust a white balance, perform a gamma correction, perform autoexposure processing, perform a demosaicing process, and/or perform a dewarping process. Frame of captured image data 302, as processed at each image processing stage 306 of the plurality of stages of image processing pipeline 304 (e.g., image processing stages 306-1 to 306-4), is referred to herein as frame of processed image data 308.

[0048] At a fifth image processing stage 306-5, system 200 generates a frame of adjusted image data 310 (labeled “Adj 310”) based on frame of processed image data 308 and an estimated value of the variance component of image noise for the pixel in frame of processed image data 308 (the “estimated variance value”). For example, the signal value of the pixel in frame of processed image data 308 may be adjusted based on the estimated variance value for the pixel. Frame of processed image data 308, as processed at fifth image processing stage 306-5, is referred to herein as frame of adjusted image data 310. Frame of adjusted image data 310 may be a frame of adjusted fluorescence image data (e.g., where frame of captured image data 302 is a frame of captured fluorescence image data), a frame of adjusted visible light image data (e.g., where frame of captured image data 302 is a frame of captured visible light image data), or a frame of adjusted augmented image data (e.g., where frame of processed image data 308 is a frame of augmented image data).

[0049] The estimated variance value for the pixel is determined by propagating the value of the variance component of image noise (the “variance value”) through image processing pipeline 304. The variance value for the pixel in frame of captured image data 302 is propagated through image processing pipeline 304 by processing the variance value at each of a plurality of variance processing stages 314. Variance data 312 is representative of the variance value for each pixel within a region of interest of frame of captured image data 302. In some examples, the region of interest includes all pixels included in frame of captured image data 302. In other examples, the region of interest includes only a subset of all pixels included in frame of captured image data 302. For example, the region of interest may be a region of frame of captured image data 302 that is determined (e.g., based on image recognition) to be near or within a vicinity of a particular surgical instrument or a particular anatomical feature, a region designated by user input, a region of movement (e.g., containing moving objects such as surgical instruments and/or anatomical objects), a region within the fovea of a viewer (which may be determined by gaze or eye tracking), or a preset region (e.g., a center region of frame of captured image data 302).

[0050] Each variance processing stage 314 corresponds to a distinct image processing stage 306. For example, first variance processing stage 314-1 corresponds to first image processing stage 306-1 , second variance processing stage 314-2 corresponds to second image processing stage 306-2, third variance processing stage 314-3 corresponds to third image processing stage 306-3, and fourth variance processing stage 314-4 corresponds to fourth image processing stage 306-4.

[0051] System 200 processes variance data 312 to determine, at each variance processing stage 314, the variance value for the pixel. System 200 may determine the variance value for the pixel at each variance processing stage 314 based on the processing of the pixel at the corresponding image processing stage 306.

[0052] For example, the initial variance value for the pixel in frame of captured image data 302 is taken as Varo. In some examples, system 200 sets initial variance value Varo as a function of the mean signal value So for the pixel in frame of captured image data 302 based on the assumption that the variance component of the electrons detected by the camera sensor (which system 200 does not have direct access to) has a Poisson distribution (e.g., the mean is the same as the variance). The initial variance value Varo is a function of the mean signal value So for the pixel in frame of captured image data 302. That function also has as inputs the full well electron capacity, sensor read noise, and any included analog gain before digitization that provides captured image data 302.

[0053] At first variance processing stage 314-1 , system 200 processes initial variance value Varo using signal processing information Pi and outputs variance value Van. Signal processing information Pi is indicative of the processing that was performed on the initial signal value So for the pixel at first image processing stage 306-1 . For example, signal processing information Pi may be representative of signal adjustment coefficients and factors used to adjust initial signal value So. At second variance processing stage 314-2, system 200 processes variance value Van using signal processing information P2 and outputs variance value Varc. Signal processing information P2 is indicative of the processing that was performed on the signal value Si for the pixel at second image processing stage 306-2. At third variance processing stage 314-3, system 200 processes variance value Varc using signal processing information P3 and outputs variance value Vara. Signal processing information P3 is indicative of the processing that was performed on the signal value S2 for the pixel at third image processing stage 306-3. At fourth variance processing stage 314-4, system 200 processes variance value Vara using signal processing information P4 and outputs variance value Var4. Signal processing information P4 is indicative of the processing that was performed on the signal value S3 for the pixel at fourth image processing stage 306-4. Variance data 312, after processing at each variance processing stage 314, is referred to herein as processed variance data 316 (labeled “PVar”), and the variance value Var4 output from fourth variance processing stage 314-4 is the estimated variance value for the pixel in frame of processed image data 308.

[0054] Variance data 312 may be processed at each variance processing stage 314 by adjusting the variance value for the pixel based on the adjustment of the signal value for the pixel at the corresponding image processing stage 306. For example, assume that processing of captured image data 302 for a pixel i at second image processing stage 306- 2 is given by the following formula (2):

S_i2 = a_t x S_tl + bt (2) where SH is the signal value for pixel i input to second image processing stage 306-2, ai is a signal adjustment coefficient for pixel i, bi is a signal adjustment factor for pixel i, and Si2 is the signal value for pixel i output from second image processing stage 306-2. Processing of variance data 312 for pixel i at corresponding second variance processing stage 314-2 is given by the following formula (3):

Var_i2 = a- x Var_i (3) where Vam is the variance value for pixel i input to second variance processing stage 314- 2, ai is the signal adjustment factor for pixel i (e.g., signal processing information P2), and Vari2 is the variance value for pixel i output from second variance processing stage 314-2. Thus, the variance value for pixel i is adjusted using the same signal adjustment coefficient ai used to adjust the signal value for pixel i. Processing of variance data 312 for pixel i may proceed in like manner for each variance processing stage 314.

[0055] System 200 may process each pixel in the region of interest of frame of captured image data 302 in the manner shown in image processing scheme 300 so that processed variance data 316 includes an estimated variance value for each pixel in the region of interest of frame of processed image data 308. It will be understood that the signal processing information (e.g., signal adjustment coefficient ai and signal adjustment factor bi) are determined independently for each pixel at each image processing stage 306.

[0056] As mentioned, frame of adjusted image data 310 is generated at fifth image processing stage 306-5 based on frame of processed image data 308 and processed variance data 316. For example, system 200 may adjust, at fifth image processing stage 306-5, signal value S4 of a pixel in the frame of processed image data 308 based on the estimated variance value Var4 for the pixel in processed variance data 316. System 200 may adjust signal value S4 based on estimated variance value Var4 in any suitable way and using any suitable algorithm. Examples of generating frame of adjusted image data 310 at fifth image processing stage 306-5 will be described below in more detail. While FIG. 3 shows that image processing pipeline 304 terminates after fifth image processing stage 306-5 (e.g., frame of adjusted image data 310 is a frame of display image data for display by a display device), one or more subsequent image processing stages 306 may follow fifth image processing stage 306-5 to further process frame of adjusted image data 310, and a frame of display image data may be output by a subsequent image processing stage 306. In further examples, system 200 may adjust the signal value of a pixel in frame of captured image data 302 based on the estimated variance value for the pixel in variance data 312 at multiple locations within image processing pipeline 304 (e.g., at fifth image processing stage 306-5 and at one or more image processing stages 306 occurring prior to and/or after fifth image processing stage 306-5).

[0057] In some examples, the processing of captured image data 302 at a particular image processing stage 306 (e.g., a demosaicing stage) involves spatial processing of pixel signals, such as by mixing or blending signals from two or more pixels. If the pixels are statistically correlated with one another (e.g., are not independent), the corresponding variance processing stage 314 generates one or more space-varying covariances. As a result, the variance value for the pixel at the next particular variance processing stage 314 is determined based on the input variance value for the pixel (the variance value output from the particular variance processing stage 314) and the covariance terms associated with the particular variance processing stage 314. Processing of variance data 312 involving covariance terms will now be described with reference to FIGS. 4A and 4B.

[0058] FIG. 4A shows an illustrative image processing scheme 400A for processing covariance terms within an image processing pipeline. Image processing scheme 400A includes a first particular image processing stage 402-1 and a corresponding first particular variance processing stage 404-1 , followed by a second particular image processing stage 402-2 and a corresponding second particular variance processing stage 404-2. Scheme 400A may be included in or part of image processing pipeline 304, and first particular image processing stage 402-1 and second particular image processing stage 402-2 and first particular variance processing stage 404-1 and second particular variance processing stage 404-2 may be implemented by any image processing stages 306 and variance processing stages 314, respectively. In some examples, first particular image processing stage 402-1 is a demosaicing stage and second particular image processing stage 402-2 is a dewarping stage.

[0059] A signal value Sn for a pixel in a frame of captured image data is input to first particular image processing stage 402-1 , which processes signal value Sn with some spatial processing (e.g., by mixing and or blending signal value Sn with signal values from other pixels) and outputs a signal value Sn+1 for the pixel. Spatial processing information P_m+1, which is indicative of spatial processing performed on signal value Sn at first particular image processing stage 402-1 , is input to the corresponding first particular variance processing stage 404-1 . A variance value Var_n for the pixel is also input to the corresponding first particular variance processing stage 404-1 , which uses the variance value Var_n and spatial processing information P_m+1 to determine and output a variance value Var_n+i for the pixel and a set of covariance terms Cov_n+i for the pixel. The set of covariance terms Cov_n+1 for the pixel are associated with first particular variance processing stage 404-1 (e.g., determined based on inputs to first particular variance processing stage 404-1 , such as variance value Var_n and spatial processing information Pm+i).

[0060] The signal value Sn+1 for the pixel output from first particular image processing stage 402-1 is input to second particular image processing stage 402-2, which processes signal value Sn+i and outputs a signal value Sn+2 for the pixel. Spatial processing information Pm+2, which is indicative of spatial processing performed on signal value Sn+1 at second particular image processing stage 402-2, is input to second particular variance processing stage 404-2 along with variance value Var_n+i and set of covariance terms Covn+i. Second particular image processing stage 402-2 uses these inputs to determine and output variance value Var_n+2.

[0061] In image processing scheme 400A, system 200 determines, at first particular variance processing stage 404-1 , the covariance terms Cov_n+1 associated with first particular variance processing stage 404-1 and passes covariance terms Cov_n+1 to second particular variance processing stage 404-2. However, processing in this manner can easily consume too many processing resources and bandwidth of system 200, particularly when performed for a large region or full frame of pixels. For example, first particular image processing stage 402-1 may be a demosaicing stage at which system 200 blends the signals of four adjacent pixels (e.g., a set of red (R), green (G), green (G), and blue (B) pixels). Blending the signals of four pixels generates six different covariance terms Cov_n+1 for each pixel in the group in addition to the variance term Var_n+1 for each pixel. When processing a full frame of captured image data in this manner, system 200 passes a frame of variance data and six different frames of covariance data (a full frame of covariance data for each covariance term) from first particular variance processing stage 404-1 to second particular variance processing stage 404-2. This amount of data might consume too much bandwidth of the system bus.

[0062] FIG. 4B shows an alternative illustrative image processing scheme 400B for processing covariance terms within an image processing pipeline in a manner that prevents overloading the system bus. Image processing scheme 400B is similar to image processing scheme 400A except that, in image processing scheme 400B, covariance terms Cov_n+1 associated with first particular variance processing stage 404-1 are not determined at first particular variance processing stage 404-1 and/or are not passed from first particular variance processing stage 404-1 to second particular variance processing stage 404-2. In some examples, system 200 refrains from determining, at first particular variance processing stage 404-1 , covariance terms Cov_n+1 associated with first particular variance processing stage 404-1 . Rather, covariance terms Cov_n+1 associated with first particular variance processing stage 404-1 are determined at second particular variance processing stage 404-2 and only the information used to determine covariance terms Cov_n+1 associated with first particular variance processing stage 404-1 (e.g., variance value Var_n+1 and spatial processing information P_m+i) is passed from first particular variance processing stage 404-1 to second particular variance processing stage 404-2. The amount of spatial processing information Pm+i is small, in contrast to multiple frames of covariance terms Covn+1. With variance value Var_n+1 and spatial processing information P_m+1 input to second particular variance processing stage 404-2, system 200 determines, at second particular variance processing stage 404-2, covariance terms Cov_n+1 associated with first particular variance processing stage 404-1 . Passing spatial processing information Pm+i between variance processing stages 404 consumes far less bandwidth than passing multiple frames of covariance terms Cov_n+1 associated with first particular variance processing stage 404-1 . [0063] Referring again to FIG. 3, system 200 generates, at fifth image processing stage 306-5, frame of adjusted image data 310 based on frame of processed image data 308 and processed variance data 316. Illustrative examples of generating frame of adjusted image data 310 based on frame of processed image data 308 and frames of processed variance data 316 will now be described.

[0064] In some examples of generating frame of adjusted image data 310 based on processed variance data 316, frame of captured image data 302 and frame of processed image data 308 are frames of fluorescence image data and frame of processed image data 308 is adjusted (e.g., normalized) at fifth image processing stage 306-5 with a fluorescence signal adjustment process, as described above, to account for non-uniform spatial and/or temporal distribution of fluorescence excitation light. The processed image data 308, as adjusted at fifth image processing stage 306-5, is a frame of adjusted fluorescence image data. As explained above, a fluorescence signal adjustment process divides the fluorescence signal value for each pixel by a measure of incident fluorescence excitation light (see formula (1 )). The fluorescence signal adjustment process works well when both the incident fluorescence excitation light and the fluorescence signal are high. However, due to the image noise in a fluorescence image, simply dividing the detected fluorescence signal for a pixel in frame of processed image data 308 by the measure of incident fluorescence excitation light may amplify any errors in estimating the measure of incident fluorescence excitation light. Additionally, low fluorescence signals, even with ample illumination, suffer from a low signal-to-noise ratio. To compensate for these issues caused by image noise, the fluorescence signal adjustment process uses a shrinkage estimator that is based on processed variance data 316. Any suitable shrinkage estimator may be used as may serve a particular implementation. In some examples, the shrinkage estimator is also based on a value of a mean component of the image noise in frame of processed image data 308.

[0065] An illustrative example of performing a fluorescence signal adjustment process with a shrinkage estimator will now be described. At fifth image processing stage 306-5, system 200 may determine an adjusted fluorescence signal q_Si (e.g., signal S5 in FIG. 3) for a pixel i in frame of processed image data 308 in accordance with the following formula (4):

Qsi — ki "X. q_ui (4) where q_ui is a mean-adjusted normalized fluorescence signal for the pixel and ki is a shrinkage coefficient for pixel i. System 200 determines the mean-adjusted normalized fluorescence signal q_Ui in accordance with the following formula (5):

where ci is the signal value of the fluorescence signal for pixel i in frame of processed image data 308, f is a value of the frame mean component of image noise in frame of processed image data 308 (“frame mean component value pf”), and h is the measure of incident fluorescence excitation light on a surface depicted by pixel i. The frame mean component value pf may be determined in any suitable way. In some examples, the frame mean component value pf for a fluorescence image is estimated based on a corresponding frame of visible light image data, such as average incident light intensity for the frame and/or average camera gain for the frame. The measure of incident fluorescence excitation light li may be determined or estimated in any suitable way, such as based on a model of incident fluorescence excitation light, as described above. Formula (5) is a modification of formula (1 ) that includes a mean offset in the numerator to account for the random bias (mean component) of image noise in frame of processed image data 308. Shrinkage coefficient ki for pixel i may be determined based on signal value ci for pixel i, frame mean component value pf for frame of processed image data 308, and the variance value oi² (e.g., Var4 in FIG. 3) for pixel i in processed variance data 316 in accordance with the following formula (6):

[0066] System 200 may perform this fluorescence signal adjustment process for each pixel in the region of interest of frame of processed image data 308. The measure of incident fluorescence excitation light li and shrinkage coefficient ki are determined independently for each pixel (e.g., on a pixel-by-pixel basis) given that the pixel signal value a and variance value Oi² are obtained independently for each pixel. The frame mean component value f is determined on a frame (or region of interest) basis. By performing this fluorescence signal adjustment process at fifth image processing stage 306-5 using a shrinkage estimator, system 200 compensates for the variance component and the mean component of noise in frame of processed image data 308 and thus generates frame of adjusted image data 310 with reduced noise, improved sensitivity, and more reliable fluorescence signals, even when fluorescence excitation light and/or fluorescence signals are low. It will be recognized that other estimators may be used as may serve a particular implementation.

[0067] In other examples of generating frame of adjusted image data 310 based on processed variance data 316, frame of captured image data 302 and frame of processed image data 308 are frames of fluorescence image data and pixels in frame of processed image data 308 are adjusted at fifth image processing stage 306-5 based on per-pixel transfer functions. The processed image data 308, as adjusted at fifth image processing stage 306-5, is a frame of adjusted fluorescence image data. In some examples, the transfer function for a particular pixel is selected, from among a set of transfer functions each associated with a unique variance value, based on the estimated variance value for the pixel in frame of processed image data 308. In some examples, the transfer functions plot output signal values of a pixel as a function of input signal values. In alternative examples, the transfer functions plot a shrinkage coefficient as a function of input signal values. The shrinkage coefficient may be applied in any suitable way. For example, system 200 may multiply signal value S4 for the pixel in frame of processed image data 308 by the shrinkage coefficient for the pixel to obtain signal value Ss in frame of adjusted image data 310. In some examples, the transfer functions are configured to increasingly shrink or reduce the output signal value as the input signal value decreases, to reduce accentuating noise at low fluorescence signal levels.

[0068] FIG. 5 shows a graph 500 with a set of illustrative transfer functions 502 (e.g., transfer functions 502-1 to 502-4). Each transfer function 502 is associated with a unique variance value and plots an output normalized fluorescence signal value as a function of an input normalized fluorescence signal value. Referring to FIG. 3, at fifth image processing stage 306-5, system 200 may adjust the signal value of a pixel in frame of processed image data 308 based on the estimated variance value (as indicated by processed variance data 316) and a transfer function 502 corresponding with the estimated variance value. For example, assuming that the signal value S4 for a pixel in frame of processed image data 308 is 0.015 and system 200 determines that transfer function 502-3 corresponds to the estimated variance value (as obtained from processed variance data 316), system 200 may determine that the output signal value S5 for the pixel is 0.01. Accordingly, system 200 may adjust the signal value of the pixel from 0.015 to 0.01.

[0069] System 200 may identify and select the transfer function 502 corresponding to the estimated variance value (the “corresponding transfer function 502”) in any suitable way. Since the shrinkage estimator has a closed form formula, it may be implemented directly to the input pixels, instead of using lookup tables to implement the curves in FIG. 5. [0070] In some examples in which the estimated variance value for a pixel does not correspond to a predefined closed form formula that results in transfer function 502, system 200 may determine the output signal value (or shrinkage coefficient) by interpolation. For example, assuming that the signal value for a pixel in frame of processed image data 308 is 0.015 and system 200 determines that the estimated variance value lies between the variance values for transfer function 502-3 and transfer function 502-4, system 200 may determine, by interpolation, that the output signal value is 0.012. Accordingly, system 200 may adjust the signal value of the pixel from 0.015 to 0.012.

[0071] System 200 may perform this transfer function-based fluorescence adjustment process for each pixel in the region of interest of frame of processed image data 308. The transfer function for each pixel is determined independently (e.g., on a pixel-by-pixel basis) given that the estimated variance value is obtained independently for each pixel. By performing this this transfer function-based fluorescence adjustment process at fifth image processing stage 306-5, system 200 compensates for the variance component of noise in frame of processed image data 308 and thus generates frame of adjusted image data 310 with reduced noise, improved sensitivity, and more reliable fluorescence signals, even when fluorescence excitation light and/or fluorescence signals are low.

[0072] In other examples of generating frame of adjusted image data 310 based on processed variance data 316, frame of processed image data 308 is adjusted by denoising frame of processed image data 308 based on the value of the variance component of noise for the pixel. In some examples, system 200 denoises frame of processed image data 308 by applying a temporal filter at fifth image processing stage 306-5 to perform temporal noise filtering. In these examples, frame of captured image data 302 and frame of processed image data 308 may be visible light image data, fluorescence image data, or a combination of both (e.g., augmented image data). Temporal noise filtering is based on filtering the signal values of a current frame of processed image data 308 using estimated variance values (e.g., processed variance data 316) for the current frame of processed image data 308 and the signal values of one or more previously filtered frames of processed image data 308 (e.g., a prior frame of adjusted image data 310). Signal values from the previously filtered frame(s) of processed image data 308 may be combined at fifth image processing stage 306-5 with the signal values of the current frame of processed image data 308 to get a best estimate of the signal values with reduced noise. Over time, as more image frames are filtered at the temporal filter, the temporal filter may provide a higher quality image with less noise. Other filtering parameters may also be used in addition to estimated variance values, and other noise filters and denoisers may be applied at fifth image processing stage 306-5 in addition to, or in place of, a temporal noise filter.

[0073] In other examples of generating frame of adjusted image data 310 based on processed variance data 316, system 200 provides, in frame of adjusted image data 310, a confidence score indication for the pixel. The confidence score is a numerical value that is based on the estimated variance value for the pixel in frame of processed image data 308. The confidence score may be representative of a level of reliability of the displayed signal for the pixel. In some examples, the confidence score is high when the estimated variance value is low. In other examples, the confidence score is a number that is different from, but based on, the estimated variance value for the pixel. In some examples, the confidence score is a function of the estimated variance value normalized to a predefined scale. Any suitable scale may be used. For example, system 200 may assign a confidence score from 0 to 100 based on the estimated variance value, with the lowest confidence score (e.g., 0) representative of a highest reference variance value (or noisiest reference signal) and the highest confidence score (e.g., 100) representative of a lowest reference variance value (e.g., a noiseless or the least noisy reference signal). The highest reference variance value and the lowest reference variance value may be predetermined empirically, such as from a reference frame of image data. Alternatively, the highest reference variance value and the lowest reference variance value may be determined from frame of processed image data 308.

[0074] The confidence score indication for the pixel may have any suitable form. In some examples, the confidence score indication is a color assigned to the pixel based on the confidence score. System 200 may assign the color to the pixel based on a color scale that specifies a range of visible colors for different values of the confidence score. The color scale may be, for example, a continuous color scale that gradually varies (e.g., from red to blue) as the confidence score increases (or decreases). This form of the confidence score indication may be useful for visualization contexts in which a viewer may wish to quickly and easily visualize where fluorescence signals are detected, whether the fluorescence signals are low, and whether the fluorescence signals are reliable. In other examples, system 200 may assign the color to the pixel based on one or more threshold confidence scores. For example, pixels having a confidence score greater than or equal to a threshold confidence score may be assigned a first color (e.g., green) and pixels having a confidence score less than the threshold variance value may be assigned a second color (e.g., red). This form of the confidence score indication may be useful for comparison contexts, such as when a viewer is interested in comparing the fluorescence signals of different fluorescing regions of frame of adjusted image data 310, such as to determine tissue perfusion of a fluorescing region.

[0075] FIG. 6 shows an illustrative display image 600 that includes a displayed confidence score indication for pixels in display image 600. Display image 600 may be representative of frame of adjusted image data 310. In display image 600, pixels corresponding to a first fluorescing region 602-1 are false-colored based on the confidence score for pixels within first fluorescing region 602-1 and a color scale 604 that specifies colors (or shades of a color in a monochromatic image) for different values of the confidence score. Pixels corresponding to a second fluorescing region 602-2 are false- colored based on the confidence score for pixels within second fluorescing region 602-2 and color scale 604. Color scale 604 specifies a range of visible colors comprising, e.g., red (at a confidence score below a first threshold value), orange (at a confidence score above the first threshold value and below a second threshold value), yellow (at a confidence score above the second threshold value and below a third threshold value), green (at a confidence score above the third threshold value and below a fourth threshold value), and blue (at a confidence score above the fourth threshold value).

[0076] Additionally or alternatively to a color indication, the confidence score indication may include text or symbols that are indicative of the confidence score. The text or symbols may be provided in any suitable region of a display image. In some examples, the text or symbols are superimposed on a display image within the corresponding group of pixels. For example, a fluorescing region within a display image may include text superimposed within the region to indicate the confidence score. In other examples, the text or symbols are provided within a peripheral region of a display image (e.g., a menu region, a region that does not depict a captured image, a region outside the fovea of the viewer, etc.) and the text or symbol is tied to (e.g., associated with) its corresponding region of a display image. For example, the text or symbols may have the same display color as the pixels of the corresponding region, and/or may appear or flash when an indicator (e.g., a cursor, a real or virtual instrument, a finger on a touch screen, etc.) is positioned within the corresponding region of a display image.

[0077] In some examples, the confidence score indication may be toggled on and off by a user. For example, the confidence score indication may be toggled on and off by a user providing input with an indicator (e.g., a mouse, a stylus, a finger or hand, or a real or virtual surgical instrument). The input may include, for example, pressing a physical button on a computing device or display device, selecting a button presented on a displayed image, and/or performing a predefined action (e.g., a double-click, a right click, a long-press, hovering, a predefined gesture, etc.).

[0078] In other examples, the confidence score indication may be automatically toggled on and off by system 200. For example, system 200 may determine whether a movable object (e.g., a real or virtual instrument) is located within a region of a display image corresponding to the confidence score indication. In response to a determination that the movable object is located within the region of the display image corresponding to the confidence score indication, system 200 may turn on the indication of the confidence score (or maintain the indication in an on state if the indication is already on). On the other hand, in response to a determination that the movable object is not located within the region of the display image corresponding to the confidence score indication, system 200 may turn off the indication of the confidence score (or maintain the indication in an off state if the indication is already off).

[0079] In further examples, system 200 may automatically turn on or off the confidence score indication based on a threshold value. For example, system 200 may determine whether the confidence score is greater than a threshold confidence score. In response to a determination that the confidence score is greater than the threshold confidence score, system 200 may turn on (or off) the indication of the confidence score (or maintain the indication in an on (or off) state if the indication is already in the on (or off) state). On the other hand, in response to a determination that the confidence score is less than the threshold confidence score, system 200 may turn off (or on) the indication of the confidence score (or maintain the indication in an off (or on) state if the indication is already in the off (or on) state). Thus, system 200 may turn on the confidence score indication only when the confidence score exceeds (or falls below) a threshold value. In this way, a viewer can be alerted when a displayed image is noisy.

[0080] In some examples, the confidence score indication may be toggled on and off for only those pixels within the region of interest having a confidence score above (or below) a threshold value. For example, a user may wish to see the confidence score indication for only the pixels that have a relatively high (or low) confidence score. Accordingly, the user may provide input (e.g., select a real or virtual button or perform a predefined action) to view the confidence score for only the pixels having a confidence score above (or below) the threshold value. In some examples, the user can dynamically adjust the threshold value, such as by moving a slider representative of the threshold confidence score along a confidence score scale.

[0081] In some examples, the form of the confidence score indication may be different for different contexts or different tasks or use cases. For instance, in visualization contexts a viewer may wish to use the confidence score indication to visualize where, in frame of adjusted image data 310, fluorescence signals are present, where the fluorescence signals are low, and whether the fluorescence signals are reliable. Accordingly, system 200 may provide a confidence score indication by coloring the pixels in frame of adjusted image data 310 based on a continuous color scale. In this example, the confidence score is representative of an estimated level of confidence that the signal of the pixel in frame of adjusted image data 310 is accurately representative of the scene depicted by the pixel. This confidence score may be particularly useful for fluorescence images where the fluorescence signal (e.g., intensity) varies based on the amount of noise in the signal.

[0082] In other examples, a viewer may wish to compare the fluorescence signals from a first region of frame of adjusted image data 310 with the fluorescence signals from a second region of frame of adjusted image data 310. For instance, when performing a tissue resection operation, a surgeon may wish to compare the fluorescence signals at a first region within frame of adjusted image data 310 with the fluorescence signals at a second region within frame of adjusted image data 310 to determine whether the tissue is sufficiently perfused. Fluorescence signals (e.g., intensities) that are similar would, under normal conditions, indicate similar levels of perfusion. However, if the fluorescence signals in one of the regions are artificially high due to image noise, the fluorescence signals may not be reliable to determine tissue perfusion. [0083] For these comparison contexts, the confidence score for the pixel may be different than the confidence score for the visualization contexts described above. For example, system 200 may use a different formula or scale to determine confidence scores for comparison contexts than for visualization contexts. Additionally or alternatively, the confidence score indication for comparison contexts may be based on one or more threshold confidence scores, as described above with regard to FIG. 6. In some examples, the threshold confidence scores are weighted toward the lower confidence scores. For instance, on a confidence score scale from 0 to 100, a first (lowest) threshold confidence score may be set at 40, a second threshold confidence score may be set at 70, and a third (highest) threshold confidence score may be set at 90. Pixels having a confidence score below the first threshold confidence score may be colored a first color (e.g., violet), pixels having a confidence score above the first threshold confidence score but below the second threshold confidence score may be colored a second color (e.g., blue), pixels having a confidence score above the second threshold confidence score but below the third threshold confidence score may be colored a third color (e.g., green), and pixels having a confidence score above the third threshold confidence score may be colored a fourth color (e.g., yellow). In some examples, the threshold confidence score(s) may be configured or set by a user or may be automatically set based on a known or detected context or type of a surgical procedure.

[0084] In some examples of the comparison context, system 200 may determine the confidence score for a pixel based on a comparison of the estimated variance value for the pixel with a reference variance value. The reference variance value may be predefined (and may be user configurable), or the reference variance value may be the estimated variance value for a selected pixel or region of frame of adjusted image data 310. For example, a user may select two regions within frame of adjusted image data 310 for comparison. The user may select the regions in any suitable way, such as using an indicator (e.g., a mouse, cursor, stylus, finger, hand, real or virtual surgical instrument) to draw the regions or to select a point that system 200 uses, with image segmentation, to segment the selected regions. System 200 may determine, for each region of frame of adjusted image data 310, a difference between the estimated variance values for pixels in the different regions of frame of adjusted image data 310 and set the confidence score based on the difference. Pixels that have a confidence score (e.g., variance value difference) below the threshold difference may be colored a first color (e.g., red) and pixels that have a confidence score above the threshold difference may be colored a second color (e.g., green). In this way, a viewer can quickly and easily see if the fluorescence signals within two different fluorescing regions of frame of adjusted image data 310 are reliable for comparison.

[0085] System 200 may set the context for providing the confidence score indication based on user input (e.g., selecting a real or virtual button or performing a predefined action) selecting the desired context.

[0086] In some examples, system 200 provides the confidence score indication in addition to performing a fluorescence signal adjustment process to compensate for variations in incident fluorescence excitation light. For example, at fifth image processing stage 306-5, system 200 may adjust the fluorescence signal of a pixel using formula (1 ) or formula (5) described above and provide a confidence score indication in frame of adjusted image data 310. Alternatively, at fifth image processing stage 306-5, system 200 may adjust the fluorescence signal of a pixel using a transfer function that is selected based on the estimated variance value and provide a confidence score indication in frame of adjusted image data 310. In these examples, a viewer can use the confidence score indication to assess the reliability of the adjusted fluorescence signal.

[0087] Various modifications may be made to image processing scheme 300. For example, image processing scheme 300 shows that image processing and variance processing are performed in separate pipelines within image processing pipeline 304. In other examples, variance processing and image processing are performed together in the same pipeline. In other modifications, one or more pixels in frame of captured image data 302 may be adjusted based on the corresponding variance value for the one or more pixels at one or more intermediate image processing stages 306. For example, in image processing scheme 300, frame of captured image data 302 may be adjusted at third image processing stage 306-3 based on variance value Varc output from second variance processing stage 314-2. This may improve image processing at third image processing stage 306-3.

[0088] In other examples, frame of adjusted image data 310 is a frame of adjusted augmented image data generated based on a visible light image component and a fluorescence image component. The visible light image component may be adjusted in any way described herein (e.g., by applying a temporal filter or denoiser) based on the value of the variance component of noise for the pixel in the visible light image component. Additionally or alternatively, the fluorescence image component may be adjusted in any way described herein based on the value of the variance component of noise for the pixel in the fluorescence image component. [0089] FIG. 7 shows an illustrative method 700 of estimating a value of a variance component of noise in a frame of processed image data. While FIG. 7 shows operations according to one embodiment, other embodiments may omit, add to, reorder, and/or modify any of the operations shown in FIG. 7. One or more of the operations shown in FIG. 7 may be performed by system 200, by any components included therein, and/or by any implementation thereof. Operations of method 700 may be performed in any suitable way, including any way described herein.

[0090] At operation 702, a frame of captured image data is processed at each image processing stage of a plurality of image processing stages of an image processing pipeline to generate a frame of processed image data.

[0091] At operation 704, a value of a variance component of noise for a pixel in the frame of processed image data is determined based on the processing the frame of captured image data at each image processing stage of the plurality of image processing stages.

[0092] At operation 706, a frame of adjusted image data is generated. The frame of adjusted image data comprises a pixel having a signal value that is based on the processed image data and the value of the variance component of noise for the pixel in the frame of processed image data.

[0093] FIG. 8 shows another illustrative method 800 of estimating a value of a variance component of noise in a frame of processed image data. While FIG. 8 shows operations according to one embodiment, other embodiments may omit, add to, reorder, and/or modify any of the operations shown in FIG. 8. One or more of the operations shown in FIG. 8 may be performed by system 200, by any components included therein, and/or by any implementation thereof. Operations of method 800 may be performed in any suitable way, including any way described herein.

[0094] At operation 802, a pixel in a frame of captured image data is processed at a first image processing stage of an image processing pipeline.

[0095] At operation 804, a value of a variance component of noise for the pixel is determined, based on the processing the pixel at the first image processing stage, at a first variance processing stage corresponding to the first image processing stage.

[0096] At operation 806, the pixel is processed, after the processing the pixel at the first image processing stage, at a second image processing stage of the image processing pipeline. [0097] At operation 808, the value of the variance component of noise for the pixel is adjusted at a second variance processing stage corresponding to the second image processing stage. The value of the variance component of noise for the pixel is adjusted based on the processing the pixel at the second image processing stage.

[0098] At operation 810, a signal value of the pixel is adjusted at a third image processing stage after the processing the pixel at the second image processing stage. The signal value of the pixel is adjusted based on the value of the variance component of noise for the pixel.

[0099] The systems and methods described herein may be used in conjunction with a computer-assisted surgical system. FIG. 9 shows an illustrative computer-assisted surgical system 900 (“surgical system 900”) that may be used in conjunction with the systems and methods described herein. As described herein, system 200 may be implemented by surgical system 900, connected to surgical system 900, and/or otherwise used in conjunction with surgical system 900.

[0100] As shown, surgical system 900 includes a manipulating system 902, a user control system 904, and an auxiliary system 906 communicatively coupled one to another. Surgical system 900 may be utilized by a surgical team to perform a computer-assisted surgical procedure on a subject 908. As shown, the surgical team may include a surgeon 910-1 , an assistant 910-2, a nurse 910-3, and an anesthesiologist 910-4, all of whom may be collectively referred to as “surgical team members 910.” Additional or alternative surgical team members may be present during a surgical session as may serve a particular implementation.

[0101] While FIG. 9 illustrates an ongoing minimally invasive surgical procedure, it will be understood that surgical system 900 may similarly be used to perform open surgical procedures or other types of surgical procedures that may similarly benefit from the accuracy and convenience of surgical system 900. Additionally, it will be understood that the surgical session throughout which surgical system 900 may be employed may not only include an operative phase of a surgical procedure, as is illustrated in FIG. 9, but may also include preoperative, postoperative, and/or other suitable phases of the surgical procedure. A surgical procedure may include any procedure in which manual and/or instrumental techniques are used on a subject to investigate, diagnose, and/or treat a physical condition of the subject. Additionally, a surgical procedure may include any non-clinical procedure, e.g., a procedure that is not performed on a live subject, such as a calibration or testing procedure, a training procedure, and an experimental or research procedure. [0102] As shown in FIG. 9, manipulating system 902 includes a plurality of manipulator arms 912 (e.g., manipulator arms 912-1 through 912-4) to which a plurality of surgical instruments may be coupled. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool having tissue-interaction functions), medical tool, imaging device (e.g., an endoscope), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or the like that may be used for a computer-assisted surgical procedure on subject 908 (e.g., by being at least partially inserted into subject 908 and manipulated to perform a computer-assisted surgical procedure on subject 908). While manipulating system 902 is depicted and described herein as including four manipulator arms 912, manipulating system 902 may include only a single manipulator arm 912 or any other number of manipulator arms as may serve a particular implementation.

[0103] Manipulator arms 912 and/or surgical instruments attached to manipulator arms 912 may include one or more displacement transducers, orientational sensors, and/or positional sensors used to generate raw (i.e., uncorrected) kinematics information. One or more components of surgical system 900 may be configured to use the kinematics information to track (e.g., determine positions and orientations of) and/or control the surgical instruments.

[0104] User control system 904 is configured to facilitate control by surgeon 910-1 of manipulator arms 912 and surgical instruments attached to manipulator arms 912. For example, surgeon 910-1 may interact with user control system 904 to remotely move or manipulate manipulator arms 912 and the surgical instruments. To this end, user control system 904 provides surgeon 910-1 with images (e.g., high-definition 3D images, composite medical images, and/or fluorescence images) of a surgical area associated with subject 908 as captured by an imaging system (e.g., imaging system 100). In certain examples, user control system 904 includes a stereo viewer having two displays where stereoscopic images of a surgical area associated with subject 908 and generated by a stereoscopic imaging system may be viewed by surgeon 910-1. Surgeon 910-1 may utilize the images to perform one or more procedures with one or more surgical instruments attached to manipulator arms 912.

[0105] To facilitate control of surgical instruments, user control system 904 includes a set of master controls. The master controls may be manipulated by surgeon 910-1 to control movement of surgical instruments (e.g., by utilizing robotic and/or teleoperation technology). The master controls may be configured to detect a wide variety of hand, wrist, and finger movements by surgeon 910-1. In this manner, surgeon 910-1 may intuitively perform a procedure using one or more surgical instruments.

[0106] Auxiliary system 906 includes one or more computing devices configured to perform primary processing operations of surgical system 900. In such configurations, the one or more computing devices included in auxiliary system 906 may control and/or coordinate operations performed by various other components (e.g., manipulating system 902 and user control system 904) of surgical system 900. For example, a computing device included in user control system 904 may transmit instructions to manipulating system 902 by way of the one or more computing devices included in auxiliary system 906. As another example, auxiliary system 906 may receive, from manipulating system 902, and process image data (e.g., captured image data 110) representative of images captured by an imaging device (e.g., imaging device 102) attached to one of manipulator arms 912.

[0107] In some examples, auxiliary system 906 is configured to present visual content to surgical team members 910 who may not have access to the images provided to surgeon 910-1 at user control system 904. To this end, auxiliary system 906 may include a display monitor 914 configured to display one or more user interfaces, such as images (e.g., 2D images, composite medical images, and/or fluorescence images) of the surgical area, information associated with subject 908 and/or the surgical procedure, and/or any other visual content as may serve a particular implementation. For example, display monitor 914 may display images of the surgical area together with additional content (e.g., graphical content, contextual information, etc.) concurrently displayed with the images. In some embodiments, display monitor 914 is implemented by a touchscreen display with which surgical team members 910 may interact (e.g., by way of touch gestures) to provide user input to surgical system 900.

[0108] Manipulating system 902, user control system 904, and auxiliary system 906 may be communicatively coupled one to another in any suitable manner. For example, as shown in FIG. 9, manipulating system 902, user control system 904, and auxiliary system 906 are communicatively coupled by way of control lines 916, which may represent any wired or wireless communication link as may serve a particular implementation. To this end, manipulating system 902, user control system 904, and auxiliary system 906 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0109] The apparatuses, systems, and methods described herein have been described with reference to fluorescence. However, it will be appreciated that the systems and methods described herein are not limited to fluorescence but may be applied to any other type of luminescence, including but not limited to photoluminescence (e.g., phosphorescence, etc.), electroluminescence, chemiluminescence, mechanoluminescence, radioluminescence, and the like. Moreover, the scene may be illuminated with NIR light for purposes other than fluorescence imaging, such as spatial frequency domain imaging (SFDI) and other optical imaging techniques.

[0110] In some examples, a non-transitory computer-readable medium storing computer-readable instructions may be provided in accordance with the principles described herein. The instructions, when executed by a processor of a computing device, may direct the processor and/or computing device to perform one or more operations, including one or more of the operations described herein. Such instructions may be stored and/or transmitted using any of a variety of known computer-readable media.

[0111] A non-transitory computer-readable medium as referred to herein may include any non-transitory storage medium that participates in providing data (e.g., instructions) that may be read and/or executed by a computing device (e.g., by a processor of a computing device). For example, a non-transitory computer-readable medium may include, but is not limited to, any combination of non-volatile storage media and/or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, a solid-state drive, a magnetic storage device (e.g., a hard disk, a floppy disk, magnetic tape, etc.), ferroelectric random-access memory (RAM), and an optical disc (e.g., a compact disc, a digital video disc, a Blu-ray disc, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0112] FIG. 10 shows a functional diagram of an illustrative computing device 1000 that may be specifically configured to perform one or more of the processes described herein. Any of the systems, units, computing devices, and/or other components described herein may be implemented by computing device 1000.

[0113] As shown in FIG. 10, computing device 1000 may include a communication interface 1002, a processor 1004, a storage device 1006, and an input/output (I/O) module 1008 communicatively connected one to another via a communication infrastructure 1010. While an exemplary computing device 1000 is shown in FIG. 10, the components illustrated in FIG. 10 are not intended to be limiting. Additional or alternative components may be used in other embodiments. Components of computing device 1000 shown in FIG. 10 will now be described in additional detail. [0114] Communication interface 1002 may be configured to communicate with one or more computing devices. Examples of communication interface 1002 include, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio/video connection, and any other suitable interface.

[0115] Processor 1004 generally represents any type or form of processing unit capable of processing data and/or interpreting, executing, and/or directing execution of one or more of the instructions, processes, and/or operations described herein. Processor 1004 may perform operations by executing computer-executable instructions 1012 (e.g., an application, software, code, and/or other executable data instance) stored in storage device 1006.

[0116] Storage device 1006 may include one or more data storage media, devices, or configurations and may employ any type, form, and combination of data storage media and/or device. For example, storage device 1006 may include, but is not limited to, any combination of the non-volatile media and/or volatile media described herein. Electronic data, including data described herein, may be temporarily and/or permanently stored in storage device 1006. For example, data representative of computer-executable instructions 1012 configured to direct processor 1004 to perform any of the operations described herein may be stored within storage device 1006. In some examples, data may be arranged in one or more databases residing within storage device 1006.

[0117] I/O module 1008 may include one or more I/O modules configured to receive user input and provide user output. I/O module 1008 may include any hardware, firmware, software, or combination thereof supportive of input and output capabilities. For example, I/O module 1008 may include hardware and/or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., touchscreen display), a receiver (e.g., an RF or infrared receiver), motion sensors, and/or one or more input buttons.

[0118] I/O module 1008 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, I/O module 1008 is configured to provide graphical data to a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and/or any other graphical content as may serve a particular implementation. [0119] In the preceding description, various exemplary embodiments have been described with reference to the accompanying drawings. It will, however, be evident that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the claims that follow. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. The description and drawings are accordingly to be regarded in an illustrative rather than a restrictive sense.

[0120] As used herein, “first,” “second,” “third,” etc., are used as labels to distinguish features that they precede and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, an image processing pipeline may include a “first” stage and a “second” stage, but the terms “first” and “second” do not necessarily imply that the first stage precedes in time or position the second stage, or that the first stage is not preceded by another stage.

Claims

CLAIMS What is claimed is:

1. A system comprising: a memory storing instructions; and one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: processing a frame of captured image data at each image processing stage of a plurality of image processing stages of an image processing pipeline to generate a frame of processed image data; determining, based on the processing the frame of captured image data at each image processing stage of the plurality of image processing stages, a value of a variance component of noise for a pixel in the frame of processed image data; and generating a frame of adjusted image data comprising a pixel having a signal value that is based on the processed image data and the value of the variance component of noise for the pixel in the frame of processed image data.

2. The system of claim 1 , wherein the determining the value of the variance component of noise for the pixel in the frame of processed image data comprises determining the value of the variance component of noise for the pixel at each variance processing stage of a plurality of variance processing stages of the image processing pipeline.

3. The system of claim 2, wherein the determining the value of the variance component of noise for the pixel at each variance processing stage of the plurality of variance processing stages comprises adjusting, at each variance processing stage of the plurality of variance processing stages, a value of the variance component of noise for the pixel determined at a prior variance processing stage based on an adjustment, at a corresponding image processing stage, of a signal value for the pixel.

4. The system of claim 2, wherein the determining the value of the variance component of noise for the pixel at each variance processing stage of the plurality of variance processing stages comprises setting an initial value of the variance component of noise for the pixel based on a mean signal value for the pixel in the frame of captured image data.

5. The system of claim 2, wherein the determining the value of the variance component of noise for the pixel comprises: determining, at a first particular variance processing stage of the plurality of variance processing stages, the value of the variance component of noise for the pixel based on signal processing information for a first particular image processing stage corresponding to the first particular variance processing stage, wherein the signal processing information is indicative of processing of a signal value for the pixel at the first particular image processing stage; passing, to a second particular variance processing stage of the plurality of variance processing stages, the value of the variance component of noise for the pixel determined at the first particular variance processing stage and the signal processing information for the first particular image processing stage; determining, at the second particular variance processing stage and based on the signal processing information for the first particular image processing stage, a set of covariance terms associated with first particular variance processing stage for the pixel; and determining, at the second particular variance processing stage and based on the value of the variance component of noise for the pixel determined at the first particular variance processing stage and the set of covariance terms determined at the second particular variance processing stage, the value of the variance component of noise for the pixel.

6. The system of claim 5, wherein the determining the value of the variance component of noise for the pixel further comprises refraining from determining, at the first particular variance processing stage, the set of covariance terms associated with the first particular variance processing stage for the pixel.

7. The system of claim 5, wherein the first particular image processing stage comprises a demosaicing stage.

8. The system of claim 7, wherein the second particular image processing stage comprises a dewarping stage.

9. The system of claim 1 , wherein the frame of captured image data comprises a frame of captured fluorescence image data and the frame of adjusted image data comprises a frame of adjusted fluorescence image data.

10. The system of claim 9, wherein the generating the frame of adjusted image data comprises: determining, based on the value of the variance component of noise for the pixel in the frame of processed image data, a shrinkage coefficient; and adjusting a signal value of the pixel in the frame of processed image data based on the shrinkage coefficient.

11. The system of claim 10, wherein the generating the frame of adjusted image data comprises determining the shrinkage coefficient further based on a mean component of the noise for the frame of captured image data.

12. The system of claim 10, wherein the process further comprises: estimating a measure of fluorescence excitation light incident on a surface depicted by the pixel; and adjusting the signal value of the pixel further based on the measure of fluorescence excitation light incident on the surface depicted by the pixel.

13. The system of claim 12, wherein the estimating the measure of fluorescence excitation light incident on the surface depicted by the pixel is performed based on an estimated depth of the surface depicted by the pixel and an incident fluorescence excitation light model.

14. The system of claim 9, wherein the generating the frame of adjusted image data comprises adjusting a signal value of the pixel based on a transfer function associated with the value of the variance component of noise for the pixel.

15. The system of claim 9, wherein the frame of adjusted image data includes an indication of the value of the variance component of noise for the pixel.

16. The system of claim 15, wherein the indication of the value of the variance component comprises a color assigned to the pixel and that is selected based on a color scale that specifies a range of visible colors for different values of the variance component of noise for the pixel.

17. The system of claim 1 , wherein the frame of captured image data comprises a frame of captured visible light image data and the frame of adjusted image data comprises a frame of adjusted visible light image data.

18. The system of claim 1 , wherein the generating the frame of adjusted image data comprises denoising the frame of processed image data based on the value of the variance component of noise for the pixel.

19. The system of claim 18, wherein the denoising the frame of processed image data is performed with a temporal filter.

20. A system comprising: a memory storing instructions; and one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: processing, at a first image processing stage of an image processing pipeline, a pixel in a frame of captured image data; determining, at a first variance processing stage corresponding to the first image processing stage, a value of a variance component of noise for the pixel based on the processing the pixel at the first image processing stage; processing, after the processing the pixel at the first image processing stage, the pixel at a second image processing stage of the image processing pipeline; adjusting, at a second variance processing stage corresponding to the second image processing stage, the value of the variance component of noise for the pixel based on the processing the pixel at the second image processing stage; and adjusting, at a third image processing stage after the processing the pixel at the second image processing stage, a signal value of the pixel based on the value of the variance component of noise for the pixel.

21. The system of claim 20, wherein the process further comprises: processing, after the processing the pixel at the second image processing stage but before the processing the pixel at the third image processing stage, the pixel at a fourth image processing stage of the image processing pipeline; and adjusting, at a fourth variance processing stage corresponding to the fourth image processing stage, the value of the variance component of noise for the pixel based on the processing the pixel at the fourth image processing stage.

22. The system of claim 20, wherein the process further comprises setting an initial value of the variance component of noise for the pixel based on a mean signal value of the pixel of the frame of captured image data.

23. The system of claim 20, wherein: the process further comprises determining, at the second variance processing stage and based on the processing the pixel at the first image processing stage, a set of covariance terms associated with the first variance processing stage for the pixel; and the adjusting the value of the variance component of noise for the pixel at the second variance processing stage is further based on the set of covariance terms associated with the first variance processing stage for the pixel.

24. The system of claim 23, wherein the first image processing stage comprises a demosaicing stage.

25. The system of claim 23, wherein the second image processing stage comprises a dewarping stage.

26. The system of claim 20, wherein the frame of captured image data comprises a frame of captured fluorescence image data.

27. The system of claim 26, wherein the adjusting, at the third image processing stage, the signal value of the pixel based on the value of the variance component of noise for the pixel comprises: determining, based on the value of the variance component of noise for the pixel, a shrinkage coefficient; and adjusting the signal value of the pixel based on the shrinkage coefficient.

28. The system of claim 27, wherein: the process further comprises estimating a measure of fluorescence excitation light incident on a surface depicted by the pixel; and the adjusting, at the third image processing stage, the signal value of the pixel is further based on the measure of fluorescence excitation light incident on the surface depicted by the pixel.

29. The system of claim 26, wherein the adjusting, at the third image processing stage, the signal value of the pixel comprises adjusting the signal value of the pixel based on a transfer function associated with the value of the variance component of noise for the pixel.

30. The system of claim 26, wherein the adjusting the signal value of the pixel is based on a color scale that specifies a range of visible colors for different values of the variance component of noise for the pixel.

31. The system of claim 20, wherein the frame of captured image data comprises a frame of captured visible light image data.

32. The system of claim 20, wherein the adjusting the signal value of the pixel comprises denoising the pixel based on the value of the variance component of noise for the pixel.

33. The system of claim 32, wherein the denoising the pixel is performed with a temporal filter.

34. A method comprising: processing a frame of captured image data at each image processing stage of a plurality of image processing stages of an image processing pipeline to generate a frame of processed image data; determining, based on the processing the frame of captured image data at each image processing stage of the plurality of image processing stages, a value of a variance component of noise for a pixel in the frame of processed image data; and generating a frame of adjusted image data comprising a pixel having a signal value that is based on the processed image data and the value of the variance component of noise for the pixel in the frame of processed image data.

35. A method comprising: processing, at a first image processing stage of an image processing pipeline, a pixel in a frame of captured image data; determining, at a first variance processing stage corresponding to the first image processing stage, a value of a variance component of noise for the pixel based on the processing the pixel at the first image processing stage; processing, after the processing the pixel at the first image processing stage, the pixel at a second image processing stage of the image processing pipeline; adjusting, at a second variance processing stage corresponding to the second image processing stage, the value of the variance component of noise for the pixel based on the processing the pixel at the second image processing stage; and adjusting, at a third image processing stage after the processing the pixel at the second image processing stage, a signal value of the pixel based on the value of the variance component of noise for the pixel.