[go: up one dir, main page]

EP2679159A1 - Dispositif de compensation de distorsion d'image, dispositif d'imagerie médicale comprenant le dispositif de compensation et procédé de compensation de distorsion d'image - Google Patents

Dispositif de compensation de distorsion d'image, dispositif d'imagerie médicale comprenant le dispositif de compensation et procédé de compensation de distorsion d'image Download PDF

Info

Publication number
EP2679159A1
EP2679159A1 EP20130173835 EP13173835A EP2679159A1 EP 2679159 A1 EP2679159 A1 EP 2679159A1 EP 20130173835 EP20130173835 EP 20130173835 EP 13173835 A EP13173835 A EP 13173835A EP 2679159 A1 EP2679159 A1 EP 2679159A1
Authority
EP
European Patent Office
Prior art keywords
image
color
image distortion
safety goggles
display unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP20130173835
Other languages
German (de)
English (en)
Other versions
EP2679159B1 (fr
Inventor
Jung Teak Oh
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of EP2679159A1 publication Critical patent/EP2679159A1/fr
Application granted granted Critical
Publication of EP2679159B1 publication Critical patent/EP2679159B1/fr
Not-in-force legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0093—Detecting, measuring or recording by applying one single type of energy and measuring its conversion into another type of energy
    • A61B5/0095—Detecting, measuring or recording by applying one single type of energy and measuring its conversion into another type of energy by applying light and detecting acoustic waves, i.e. photoacoustic measurements
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
    • A61B6/46—Arrangements for interfacing with the operator or the patient
    • A61B6/461—Displaying means of special interest
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/46—Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient
    • A61B8/461—Displaying means of special interest
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61F—FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS
    • A61F9/00—Methods or devices for treatment of the eyes; Devices for putting in contact-lenses; Devices to correct squinting; Apparatus to guide the blind; Protective devices for the eyes, carried on the body or in the hand
    • A61F9/02—Goggles
    • A61F9/022—Use of special optical filters, e.g. multiple layers, filters for protection against laser light or light from nuclear explosions, screens with different filter properties on different parts of the screen; Rotating slit-discs
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/02—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the way in which colour is displayed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
    • A61B6/10—Safety means specially adapted therefor
    • A61B6/107—Protection against radiation, e.g. shielding
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10116—X-ray image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10132—Ultrasound image
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2320/00—Control of display operating conditions
    • G09G2320/02—Improving the quality of display appearance
    • G09G2320/0242—Compensation of deficiencies in the appearance of colours
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2380/00—Specific applications
    • G09G2380/08—Biomedical applications

Definitions

  • the present disclosure relates generally to image compensation techniques and more particularly to an image distortion compensation device for compensating image distortion caused by color-filtered safety goggles.
  • a medical imaging device acquires an image of a subject using transmission, absorption or reflection properties of ultrasonic waves, lasers, X-rays or the like with respect to the subject, and uses the image for diagnosis.
  • imaging devices include ultrasonic, photoacoustic, and X-ray imaging devices.
  • Some medical imaging devices require that safety goggles be worn to protect the eyes of both an inspector and a subject.
  • a laser having a nano-scaled short wavelength is irradiated during a procedure; thus it is typically necessary for all participants involved to wear safety goggles to protect their eyes.
  • safety goggles absorb light with a specific wavelength, image distortion occurs on an image passing through the safety goggles.
  • image distortion occurs on an image passing through the safety goggles.
  • Embodiments described herein provide an image distortion compensation technique which enables a user to view a medical image having minimal or eliminated image distortion while wearing color-filtered safety goggles, by correcting colors of a medical image displayed on a display unit using spectral characteristics of the safety goggles.
  • a color correction unit performs color correction to compensate the estimated image distortion with respect to an image to be displayed on a display unit.
  • the image distortion estimation unit may determine whether or not compensation of image distortion is possible through the color correction.
  • the image distortion estimation unit may determine that compensation of image distortion is impossible through the color correction when, among colors used for an image to be displayed on the display unit, a color not represented in an image to which light absorption effects of the safety goggles are applied is present.
  • the image distortion compensation device may further include a color mapping unit to substitute the color not represented in the image to which light absorption effects of the safety goggles, among the colors used for the image displayed on the display unit, by a color represented in the image to which light absorption effects of the safety goggles are applied.
  • the image distortion compensation device may further include a storage unit to store at least one spectral characteristic of the safety goggles.
  • the image distortion estimation unit may estimate image distortion using a selected one of more of the spectral characteristics stored in the storage unit.
  • the image distortion estimation unit may estimate coordinate change of primary colors by absorption effects of the safety goggles, and the coordinates of primary colors may include coordinates in a color space used for the display unit.
  • the color correction unit may control a synthesis ratio of primary colors to realize a specific color included in the image displayed on the display unit, based on the estimated coordinates of primary colors.
  • the image distortion compensation device may further include a spectrometer to measure the spectral characteristics of the safety goggles.
  • a medical imaging device includes the image distortion compensation device.
  • the medical imaging device may include a display unit, wherein the display unit displays a photoacoustic image or a photoacoustic/ultrasonic image of an object.
  • the display unit may display an image color-corrected by the color correction unit or an image color-mapped by the color mapping unit.
  • the medical imaging device may further include an input unit to input selection of a substituted color from the color mapping unit.
  • the medical imaging device may further include an input unit to input selection of at least one of spectral characteristics of the safety goggles stored in the storage unit.
  • Exemplary methods operable in the image distortion compensation device and medical imaging device are also disclosed.
  • distaltion refers to color distortion, contrast distortion, or other type of distortion, depending on the context of use.
  • ...unit or “module,” disclosed in the specification indicates a unit for performing at least one function or operation, and this may be implemented by hardware or a combination of hardware and software.
  • Ultrasonic imaging is widely used as a medical imaging method for diagnosing a subject.
  • photoacoustic imaging in which ultrasonic properties of an object (within a subject) is combined with photoacoustic properties thereof has been developed and utilized in a variety of diagnosis fields.
  • Photoacoustic imaging is a method in which an ultrasonic image having a high spatial resolution is combined with an optical image having a high contrast ratio suitable for imaging biological tissues.
  • a laser is irradiated to biological tissues, a short electromagnetic pulse of the laser is absorbed in the biological tissues and a momentary acoustic pressure is generated by thermo-elastic expansion in tissue sites acting as a generation source of an initial ultrasonic wave.
  • the ultrasonic waves thus formed reach the surface of the biological tissues with various delays and a photoacoustic image is obtained by imaging the same.
  • Ultrasonic imaging is an established medical imaging method which diagnoses lesions of the human body using ultrasonic waves.
  • An ultrasonic image may include a B-mode image to display a cross-sectional image of an object, an elasticity image showing elasticity information of the object, an M-mode image showing biological information of a specific part of the object, or a color doppler image to visualize the bloodstream in real-time.
  • photoacoustic/ultrasonic imaging in which photoacoustic imaging is combined with ultrasonic imaging has recently been developed and used.
  • FIG. 1 illustrates a system for diagnosis of a subject using photoacoustic imaging or photoacoustic/ultrasonic imaging according to an embodiment.
  • a photoacoustic imaging device 50 such as a photoacoustic imaging device or photoacoustic/ultrasonic imaging device, both an inspector 20 and the subject 30 need to wear safety goggles 10.
  • Safety goggles 10 protect the eyes from a laser irradiated from a laser irradiator 52 by blocking the irradiated laser and preventing the same from reaching the eyes. Accordingly, the safety goggles 10 are designed to absorb light in a specific wavelength range; particular goggles are selected depending on the wavelength of the irradiated laser. For example, when a laser corresponds to a visible ray region having a wavelength shorter than green, the safety goggles may be designed to absorb light with a wavelength shorter than 532 nm. When a laser corresponds to red and infrared regions, the safety goggles may be designed to absorb light with a wavelength of 600 nm or more.
  • Medical imaging device 50 displays a medical image through a display unit 51.
  • the display unit 51 may display a medical image using an RGB model, a CMYK model, or another suitable model. With the RGB model, a medical image is displayed through additive synthesis of primary colors, i.e., red (R), green (G) and blue (B). Color represented on the display unit 51 may be defined in various color spaces, for example, the CIE 1931 color space established by the Commission International de l'Eclairage (CIE).
  • CIE Commission International de l'Eclairage
  • FIG. 2 is an example of a chromaticity diagram showing color represented on a display unit 51 in the medical imaging device 50.
  • the color may be represented by various chromaticity diagrams, e.g., by a CIE 1931 chromaticity diagram for the case of CIE 1931 color space.
  • CIE 1931 color space is also called "CIE XYZ color space", where X, Y and Z represent trichromatic stimulus values which are similar to values indicating red, green and blue levels.
  • the concept of color is divided into two factors, i.e ., brightness and chromaticity.
  • the CIE XYZ color space may be designed such that the value Y represents brightness. Accordingly, chromaticity of the color may be represented by the values x and y calculated by the following Equation 1.
  • x X / X + Y + Z
  • y Y / X + Y + Z
  • the chromaticity diagram of the CIE 1931 color space drawn using x and y is the chromaticity diagram shown in FIG. 2 .
  • an outer curved boundary corresponds to monochromatic lights, and the wavelengths of the monochromatic lights are measured in nanometers.
  • the display unit may represent all colors represented in the CIE 1931 chromaticity diagram.
  • a color range that is represented is referred to as a color gamut.
  • the color gamut may correspond to a triangular inner space having R (0.675, 0.325), G (0.285, 0.595) and B (0.154, 0.068). These represent coordinates of red (R), green (G) and blue (B) as vertexes, which means that the display unit displays colors within the RGB triangle.
  • FIG. 3 is a block diagram illustrating a configuration of an image distortion compensation device, 100, according to one embodiment of the present invention.
  • Device 100 includes an image distortion estimation unit 110 to estimate image distortion caused by wearing of safety goggles 10, and a color correction unit 120 to at least preliminarily compensate the estimated image distortion.
  • Each of the units described herein such as units 110 and 120 can be embodied with at least one processor and a memory to execute the respective functions. Alternatively, a single processor and memory may be shared between some or all of the units of a described device to execute the respective functions by reading respective software programs from the memory and executing the programs on the processor.
  • the image distortion estimation unit 110 estimates image distortion which may occur when an image displayed on the display unit 51 passes through (i.e., "traverses") the safety goggles. Estimation is done on the basis of spectral characteristics of the safety goggles 10 and spectral characteristics of the display unit 51, both of which may be input by a designer or a user. When the image distortion compensation device 100 is incorporated in the medical imaging device 50, it may automatically obtain spectral characteristics of the display unit 51.
  • the type of safety goggles used depends on the wavelength of the laser generated in the medical imaging device; for a laser having a wavelength shorter than green, safety goggles blocking light of wavelength shorter than 532 nm are used. For a laser having a red or infrared region, the safety goggles are designed to block light of wavelength 600 nm or more. For convenience of description in explaining the following embodiments, a case in which the medical imaging device 50 generates a laser with a red region will be described.
  • FIG. 4A is a graph showing example spectral characteristics of a display unit
  • FIG. 4B is a graph showing example spectral characteristics of safety goggles.
  • the safety goggles having spectral characteristics of FIG. 4B are safety goggles which block or absorb light within a red region.
  • the display unit emits light within a visible region (about 380 nm to 700 nm) which corresponds to a light region visible to humans.
  • the spectral characteristics shown in FIG. 4A are those of a projection-type CRT display unit.
  • the graph of FIG. 4B shows an optical density indicating transmittance of light, illustrating spectral characteristics of the safety goggles.
  • the optical density is defined according to the following Equation 2.
  • OD log 10 I in / I out where OD represents optical density, I in represents intensity of incident light, and I out represents an intensity of a transmitted light.
  • Optical density which is also called absorbance, increases as light attenuation increases.
  • the optical density is considerably high in a region ranging from about 680 nm to about 710 nm.
  • the safety goggles having spectral characteristics of FIG. 4B have high absorbance with respect to light in the red region. Accordingly, when an inspector wears the safety goggles having spectral characteristics of FIG. 4B , the laser within a red region is blocked by the safety goggles and his eyes are thus protected, while at the same time, light within a red region emitted from the display unit is also blocked. That is, the image displayed on the display unit appears distorted in terms of color from the vantage point of the inspector.
  • CIE 1931 color space and the CIE 1931 chromaticity diagram will be described as examples in embodiments described below for convenience of description. In alternative embodiments, image distortion can be compensated using other color spaces.
  • a medical image is usually represented by a black-and-white image; this black-and-white image can be generated with an RGB-based color display unit through additive synthesis of the three primary colors (red, green, blue).
  • the black-and-white image renders gray through additive synthesis of three primary colors, and black and white is rendered by controlling brightness (luminance) of gray.
  • FIG. 5A is a chromaticity diagram illustrating a coordinate of gray on a CIE 1931 color space.
  • FIG. 5B is a chromaticity diagram showing change in coordinate as observed by a user, caused by wearing of the safety goggles.
  • the display unit when the coordinate "gr" on the chromaticity diagram renders gray, the display unit maintains a division ratio (a:b) of a BG straight line, and a division ratio (c:d) of an XR straight line and changes brightness to render a black-and-white image.
  • a division ratio is referred to as additive synthesis ratio.
  • the location of X is determined by extending a straight line drawn from the R point to the gray point gr.
  • the image distortion estimation unit 110 estimates image distortion shown in FIG. 5B using the spectral characteristics of the display unit 51 and spectral characteristics of the safety goggles 10. Specifically, when the display unit displaying the black-and-white image has spectral characteristics shown in FIG. 4B and the safety goggles to be worn have spectral characteristics shown in FIG. 4A , the image distortion of FIG. 5B may be estimated from the image displayed on the display unit using absorbance properties of the safety goggles.
  • the color correction unit 120 preliminarily performs color correction to compensate an estimated image distortion of the image to be displayed on the display unit, thereby preventing distortion from being observed while the image is viewed through the safety goggles 10.
  • FIG. 6 is a chromaticity diagram illustrating change in the coordinate gr, when a color correction unit 120 performs color correction.
  • the color correction unit 120 controls the additive synthesis ratio, based on the coordinate R' estimated in the image distortion estimation unit 110 to prevent image distortion from being experienced by the inspector wearing the safety goggles 10.
  • the color correction unit 120 changes the additive synthesis ratios to represent gray from a:b to a':b', and from c:d to c':d', with respect to the image represented on the display unit 51.
  • the additive synthesis ratios are controlled as described above, and a coordinate red is changed to R' through absorption of the safety goggles 10, the coordinate produced by additive synthesis is arranged at the same position (gr) as in FIG. 4A , indicating gray. In this manner, image distortion is not observed in the view through the safety goggles 10.
  • a line drawn from R' to gr is extended to intersect the BG line at location of X'.
  • the ratio a':b' is established relative to X' , gr and R'.
  • the ratio a':b' is established relative to X', B and G.
  • the image distortion estimation unit 110 estimates coordinate change of at least one of the three primary colors by applying absorption properties of the safety goggles 10 to the image displayed on the display unit 51.
  • the color correction unit 120 determines a ratio to create a specific color through additive synthesis from the changed coordinates of the three primary colors and applies the determined additive synthesis ratio to the image displayed on the display unit.
  • the specific color is gray.
  • the display unit 51 displays a black-and-white image, i.e., a grayscale (monochromatic) image, which has the appearance of a color image to the viewer wearing the safety goggles.
  • a grayscale (monochromatic) image which has the appearance of a color image to the viewer wearing the safety goggles.
  • equal intensity of R, G and B is applied to each pixel of the display, but the intensities of each pixel are allowed to differ in order to generate contrast within the image.
  • the actual image displayed on display unit 51 is changed from a grayscale image to a color image. This is done by applying, for a given pixel, a different amount of intensity to at least one of the R, G and B pixel elements.
  • the R pixel element of each pixel would be provided with higher intensity to compensate for the red wavelength filtering property of the safety goggles.
  • the compensated image displayed on display unit 51 would appear reddish.
  • the display unit 51 displays an image in which an additive synthesis ratio is controlled by the color correction unit 120, that is, a color-corrected image, and the color-corrected image does not exhibit distortion (or exhibits reduced distortion) from a vantage point of a viewer wearing the safety goggles.
  • the image distortion compensation device compensates distortion shown on the black-and-white image.
  • a medical image is displayed as a color image such as a color doppler image.
  • an embodiment in which distortion of a color image is compensated will be described.
  • FIG. 7A is a chromaticity diagram illustrating a coordinate of a specific color in an image displayed on the display unit.
  • FIG. 7B is a chromaticity diagram illustrating a coordinate changed due to absorption of light by the safety goggles.
  • the color having the coordinate k is represented by e:f and g:h, additive synthesis ratios.
  • the display unit is seen through the safety goggles absorbing light in a red region, the color having a coordinate k is seen as a color that differs from the color actually displayed on the display unit.
  • the image distortion estimation unit 110 estimates the image distortion shown in FIG. 7B using spectral characteristics of the display unit and spectral characteristics of the safety goggles. In an embodiment, by applying absorption properties of the safety goggles to the image displayed on the display unit, coordinate change of at least one of three primary colors is estimated.
  • the color correction unit 120 controls an additive synthesis ratio, based on the changed coordinates of three primary colors so that the color of light traversing the safety goggles becomes a color having the coordinate k.
  • FIG. 8 is a chromaticity diagram illustrating a change in coordinate k when color correction is performed by the color correction unit.
  • the color correction unit 120 adjusts additive synthesis ratios to e':f and g':h' and indicates the same on the display unit
  • color perceived through the safety goggles has the coordinate k
  • a user wearing the safety goggles views an intended color to be displayed on the display unit.
  • the actual image displayed by the display unit 51 is modified by increasing intensities applied to the red pixel elements in the current example of red filtering safety goggles. That is, the actual image displayed is redder than the original image.
  • the color correction unit 120 compensates image distortion through color correction by the color correction unit 120.
  • a region of light absorbed by the safety goggles is wide, there is a case in which compensation of image distortion through color correction is impossible, since a region of color which is represented in an image seen through the safety goggles is considerably narrow.
  • an image distortion compensation device useful for this case will be described below.
  • FIG. 9 is a block diagram illustrating an image distortion compensation device further including a color mapping unit.
  • FIG. 10 is a chromaticity diagram illustrating a coordinate of gray changed after passing through safety goggles.
  • an image distortion compensation device 100' may further include a color mapping unit 130 to substitute a color represented in an image seen through the safety goggles by another color.
  • the image distortion estimation unit 110 estimates image distortion which may be generated in an image traversing the safety goggles using spectral characteristics of the safety goggles and spectral characteristics of the display unit.
  • spectral characteristics of the safety goggles and spectral characteristics of the display unit.
  • the coordinate of color represented on the display unit falls within the triangle R'GB, corresponding to a range of color which may be represented by the image traversing the safety goggles.
  • the coordinate of red among the three primary colors is shifted to R" and the coordinate "gr" indicating gray is disposed in an outer region of the triangle R"GB.
  • the image distortion estimation unit 110 determines whether a color that the display unit intends to display is represented in the image traversing the safety goggles, when color correction is performed. In an embodiment, in a case where the image displayed on the display unit is a black-and-white image, the image distortion estimation unit 110 determines whether or not the coordinate "gr" of gray is present in the triangle R"GB created by the changed coordinates of the three primary colors.
  • the image distortion estimation unit 110 transmits a relevant command signal to the color correction unit 120 to allow the color correction unit 120 to perform color correction.
  • the image distortion estimation unit 110 transmits a relevant command signal to the color mapping unit 130.
  • the color mapping unit 130 substitutes a color not represented in the image traversing the safety goggles by another color which may be represented.
  • the color mapping unit 130 maps other colors on the image displayed on the display unit, controls brightness of the mapped color and represents the image.
  • the mapped other color is a color which may be represented in the image traversing the safety goggles.
  • the color mapping unit 130 maps a region corresponding to red in the image displayed on the display unit to another color which may be represented.
  • the image color-mapped in the color mapping unit 130 is displayed on the display unit 51 of the medical imaging device 50.
  • the image distortion compensation device 100' when the image distortion compensation device 100' determines that the image to be displayed on the display unit 51 contains a color not represented in the image passing through the safety goggles, it informs a user of the impossibility of color correction through the display unit 51. Also, image distortion compensation device 100' indicates what color is represented. In this case, although the color mapping unit 130 does not perform color mapping, a user anticipates image distortion and analyzes an image without difficulty or confusion.
  • the other (alternative) color mapped by the color mapping unit 130 is represented on the display unit 51 to inform the user of the alternative color.
  • the color mapping unit 130 may receive selection of the alternative color from the user after informing of the same through the display unit 51 and then map the selected color to the color mapping unit 130. In the two cases, the user is better able to analyze the image without difficulty or confusion, even though the image displayed on the display unit 51 represents a color different from that of the original image.
  • FIGS. 11A and 11B are block diagrams illustrating respective configurations of an image distortion compensation device, 100", which further includes a storage unit.
  • Device 100" includes a storage unit 140 to store spectral characteristics of the safety goggles.
  • the type of safety goggles is changed according to the wavelength region of a laser irradiated to the medical imaging device 50.
  • the safety goggles may be categorized according to spectral characteristics thereof and each pair of safety goggles has inherent spectral characteristics. Absorbance according to light wavelengths may be used as the spectral characteristic of the safety goggles; however, other properties can be alternatively or additionally accounted for in other embodiments of the present invention.
  • the storage unit 140 stores information on the safety goggles and spectral characteristics corresponding thereto.
  • the user is prompted to select a name of the safety goggles used. This eliminates an operation step of inputting spectral characteristics of the safety goggles whenever the medical imaging device 50 is used. User selection may be carried out through the input unit of the medical imaging device 50.
  • FIGS. 12A and 12B are block diagrams illustrating respective configurations of an image distortion compensation device, 100"', further including a spectrometer.
  • the image distortion compensation device 100" includes the storage unit 140 to store spectral characteristics of the safety goggles, whereas the image distortion compensation device 100"' includes a spectrometer 150 to measure spectral characteristics.
  • the spectrometer 150 measures spectral characteristics of the safety goggles, allows spectral characteristics of safety goggles to be automatically input to the image distortion estimation unit 110 and enables compensation of image distortion, without the need for the user to input spectral characteristics or a name of the safety goggles. Also, the spectrometer 150 measures spectral characteristics of the display unit and inputs the measurement results to the image distortion estimation unit 110.
  • FIG. 13 is a block diagram illustrating a configuration a medical imaging device, 200, according to one embodiment of the present invention.
  • Medical imaging device 200 includes an image production unit 210 to produce a medical image of an object, an image distortion compensation device 220 to compensate image distortion generated during wearing of the safety goggles, a display unit 230 to display the compensated image and an input unit 240 to receive user selection.
  • medical imaging device 200 In the normal use of medical imaging device 200, the user wears safety goggles to block light of a specific wavelength region in order to perceive compensated images of the object.
  • medical imaging device 200 include a photoacoustic imaging device emitting a laser with a short wavelength to obtain images of the object, and a photoacoustic/ultrasonic imaging device in which a photoacoustic imaging device is combined with an ultrasonic imaging device. Other types of imaging devices are also possible. Any medical imaging device may be applied to the embodiments of the present invention so long as its use requires, recommends or permits goggles absorbing light of a specific wavelength region be worn.
  • the image production unit 210 may produce an image of an object, produce a black-and-white image according to the application thereof and the diagnosis site of the object and/or produce a color image.
  • the display unit 230 displays an image produced by the image production unit 210, which is image compensated by the image distortion compensation device 100, and is realized with a CRT, LCD, LED, PDP or the like. Spectral characteristics of the display unit 230 depend on the type thereof.
  • the image distortion compensation device 220 compensates an image which is produced by the image production unit 210 and then displayed on the display unit 230. Thereby, it prevents the image from being perceived distorted when viewed using the safety goggles.
  • the image distortion compensation device 220 is the same as the image distortion compensation devices 100, 100', 100" or 100"' of the embodiments described in FIGS. 3 to 11 . Accordingly, the image distortion compensation device 220 includes the image distortion estimation unit 110 and the color correction unit 120 and further includes at least one of the color mapping unit 130, the storage unit 140 and the spectrometer 150.
  • the image distortion compensation device 220 may be mounted in a host device of the medical imaging device 200 and operated by a controller such as a CPU or MCU present in the host device.
  • the image compensated by the image distortion compensation device 220 is displayed on the display unit 230.
  • the impossibility and the alternative color are informed to the user through the display unit 230.
  • the input unit 240 may receive a selection of the alternative color to be mapped in the color mapping unit 130, or selection of the safety goggles may be input through the input unit 240, in a case where the image distortion compensation device 220 further includes the storage unit 140.
  • input unit 240 may receive a selection regarding wearing of safety goggles. According to the selection received by the input unit 240, when the user wears the safety goggles, a color-corrected image is displayed, and when the user does not wear the safety goggles, a non-color-corrected image is displayed.
  • FIG. 14 is a flowchart illustrating a method for compensating image distortion according to one embodiment of the present invention.
  • spectral characteristics of safety goggles and spectral characteristics of the display unit are input (410). These characteristics may be directly input from the user or a designer, or values measured by the spectrometer may be input.
  • the safety goggles and the display unit have inherent spectral characteristics; the spectral characteristics used for the safety goggles may be light absorbance according to wavelength.
  • estimated image distortion means an estimate of image distortion generated when the image displayed on the display unit traverses the safety goggles, i.e., distortion observable by a user wearing the glasses.
  • the image distortion may be generally estimated by calculating change in coordinate of at least one of the three primary colors caused by light absorption by the safety goggles and thus change in coordinate of a specific color.
  • color correction to compensate the estimated image distortion is performed (430). Specifically, an additive synthesis ratio is adjusted such that a specific color is created by combining three primary colors having changed coordinates.
  • FIG. 14 may be applied to compensation of black-and-white images as well as compensation of color images.
  • FIG. 15 is a flowchart illustrating a method for compensating image distortion applicable to a case in which color correction is impossible.
  • spectral characteristics of the safety goggles and spectral characteristics of the display unit are input (510), and image distortion is estimated using the input spectral characteristics (520).
  • whether or not a specific color is represented in the image traversing (passing through) the safety goggles is determined (530). Specifically, when the image displayed on the display unit is a black-and-white image, whether or not gray is represented in the image traversing the safety goggles is determined and when the image displayed on the display unit is a color image, whether or not a specific color not represented in the image traversing the safety goggles is present is determined.
  • the image displayed on the display unit is a black-and-white image
  • a gray coordinate is present in the color gamut created by the changed coordinates of three primary colors
  • representation of gray on the image traversing the safety goggles is determined to be possible
  • representation of gray on the image traversing the safety goggles is determined to be impossible.
  • a region corresponding to a specific color is mapped to another color (550).
  • the image displayed on the display unit is a black-and-white image
  • a color other than gray is mapped, brightness thereof is controlled, and intended information of the black-and-white image is displayed.
  • the image displayed on the display unit is a color image
  • a region corresponding to a specific color not represented in the image traversing the safety goggles is mapped to other color and the image is displayed on the display unit.
  • At least some embodiments of the present invention provide an image distortion compensation device which enables a user / inspector to perceive a medical image having reduced or eliminated image while wearing color-filtered safety goggles, by correcting colors of a medical image displayed on a display unit using spectral characteristics of the safety goggles.
  • Medical imaging devices including the same and method for compensating image distortion have also been disclosed.
  • the above-described methods according to the present invention can be implemented in hardware, firmware or as software or computer code that can be stored in a recording medium such as a CD ROM, an RAM, a floppy disk, a hard disk, or a magneto-optical disk or computer code downloaded over a network originally stored on a remote recording medium or a non-transitory machine readable medium and to be stored on a local recording medium, so that the methods described herein can be rendered in such software that is stored on the recording medium using a general purpose computer, or a special processor or in programmable or dedicated hardware, such as an ASIC or FPGA.
  • a recording medium such as a CD ROM, an RAM, a floppy disk, a hard disk, or a magneto-optical disk or computer code downloaded over a network originally stored on a remote recording medium or a non-transitory machine readable medium and to be stored on a local recording medium, so that the methods described herein can be rendered in such software that is stored on the recording medium using a
  • the computer, the processor, microprocessor controller or the programmable hardware include memory components, e.g., RAM, ROM, Flash, etc. that may store or receive software or computer code that when accessed and executed by the computer, processor or hardware implement the processing methods described herein.
  • memory components e.g., RAM, ROM, Flash, etc.
  • the execution of the code transforms the general purpose computer into a special purpose computer for executing the processing shown herein.

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Animal Behavior & Ethology (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Biophysics (AREA)
  • Surgery (AREA)
  • Pathology (AREA)
  • Molecular Biology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Optics & Photonics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Acoustics & Sound (AREA)
  • High Energy & Nuclear Physics (AREA)
  • Human Computer Interaction (AREA)
  • Ophthalmology & Optometry (AREA)
  • Vascular Medicine (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Magnetic Resonance Imaging Apparatus (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)
  • Control Of Indicators Other Than Cathode Ray Tubes (AREA)
  • Image Processing (AREA)
  • Endoscopes (AREA)
EP13173835.3A 2012-06-27 2013-06-26 Dispositif d'imagerie médicale comprenant un dispositif de compensation de distorsion d'image et procédé de compensation de distorsion d'image Not-in-force EP2679159B1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020120069189A KR101502782B1 (ko) 2012-06-27 2012-06-27 영상 왜곡 보정 장치, 이를 포함하는 의료 영상 장치 및 영상 왜곡 보정 방법

Publications (2)

Publication Number Publication Date
EP2679159A1 true EP2679159A1 (fr) 2014-01-01
EP2679159B1 EP2679159B1 (fr) 2017-11-29

Family

ID=48700361

Family Applications (1)

Application Number Title Priority Date Filing Date
EP13173835.3A Not-in-force EP2679159B1 (fr) 2012-06-27 2013-06-26 Dispositif d'imagerie médicale comprenant un dispositif de compensation de distorsion d'image et procédé de compensation de distorsion d'image

Country Status (4)

Country Link
US (1) US9378547B2 (fr)
EP (1) EP2679159B1 (fr)
KR (1) KR101502782B1 (fr)
CN (1) CN103514584B (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016065053A3 (fr) * 2014-10-21 2016-06-16 Qualcomm Incorporated Amélioration automatique d'une image affichée, basée sur un modèle de perception visuelle de l'utilisateur
EP3203308A3 (fr) * 2016-01-18 2017-11-29 Canon Kabushiki Kaisha Système d'affichage, lunettes et procédé de commande de système d'affichage

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5769751B2 (ja) * 2013-03-29 2015-08-26 キヤノン株式会社 画像処理装置、画像処理方法及びプログラム
US10438331B2 (en) * 2014-06-26 2019-10-08 Intel Corporation Distortion meshes against chromatic aberrations
KR20160059240A (ko) * 2014-11-18 2016-05-26 삼성전자주식회사 색 재현 영역을 표시하는 방법 및 장치
CN106155280A (zh) * 2015-03-30 2016-11-23 北京智谷睿拓技术服务有限公司 交互方法、交互装置以及用户设备
US20180144554A1 (en) 2016-11-18 2018-05-24 Eyedaptic, LLC Systems for augmented reality visual aids and tools
US20190012841A1 (en) * 2017-07-09 2019-01-10 Eyedaptic, Inc. Artificial intelligence enhanced system for adaptive control driven ar/vr visual aids
US10984508B2 (en) 2017-10-31 2021-04-20 Eyedaptic, Inc. Demonstration devices and methods for enhancement for low vision users and systems improvements
US11563885B2 (en) 2018-03-06 2023-01-24 Eyedaptic, Inc. Adaptive system for autonomous machine learning and control in wearable augmented reality and virtual reality visual aids
US11187906B2 (en) 2018-05-29 2021-11-30 Eyedaptic, Inc. Hybrid see through augmented reality systems and methods for low vision users
JP2022502798A (ja) 2018-09-24 2022-01-11 アイダプティック, インク.Eyedaptic, Inc. 電子視覚補助具における自律的ハンズフリー制御の改善
WO2020068721A1 (fr) 2018-09-24 2020-04-02 Ecolab Usa Inc. Méthodes et compositions pour l'enrichissement pré-extractif des minerais
DE102019102586A1 (de) * 2019-02-01 2020-08-06 tooz technologies GmbH Lichtleitanordnung, Abbildungsoptik, Head Mounted Display und Verfahren zum Verbessern der Abbildungsqualität einer Abbildungsoptik
US11263982B2 (en) * 2019-12-09 2022-03-01 Rockwell Collins, Inc. Advanced LEP display mode, architectures and methodologies for display compensation

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050024583A1 (en) * 2003-07-28 2005-02-03 Ceramoptec Industries, Inc. Laser safety contact lenses
WO2006073408A2 (fr) * 2005-01-07 2006-07-13 Harvie Mark R Systeme de protection laser actif
US20120075435A1 (en) * 2010-09-23 2012-03-29 Dolby Laboratories Licensing Corporation Method and System for 3D Display Calibration with Feedback Determined by A Camera Device

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000115558A (ja) * 1998-10-08 2000-04-21 Mitsubishi Electric Corp 色特性記述装置および色管理装置および画像変換装置ならびに色補正方法
US7184054B2 (en) * 2003-01-21 2007-02-27 Hewlett-Packard Development Company, L.P. Correction of a projected image based on a reflected image
KR100499149B1 (ko) * 2003-06-30 2005-07-04 삼성전자주식회사 디스플레이 주변 조명의 밝기 특성을 시각적으로 측정하는방법
KR100558330B1 (ko) 2003-10-08 2006-03-10 한국전자통신연구원 영상 시스템의 비네팅 효과를 보상하기 위한 방법 및 이를이용한 영상 장치
JP4446080B2 (ja) * 2003-10-27 2010-04-07 株式会社セガ 画像表示における歪補正機能を有する画像表示装置
JP4704238B2 (ja) * 2006-02-22 2011-06-15 オリンパスイメージング株式会社 電子撮像装置及び電子撮像方法
US7839414B2 (en) * 2007-07-30 2010-11-23 Motorola Mobility, Inc. Methods and devices for display color compensation
JP5147656B2 (ja) * 2008-11-20 2013-02-20 キヤノン株式会社 画像処理装置、画像処理方法、プログラム、及び記憶媒体
US8310502B2 (en) * 2009-06-02 2012-11-13 Hewlett-Packard Development Company, L.P. System and method for adjusting display input values
JP5457247B2 (ja) * 2010-03-26 2014-04-02 富士フイルム株式会社 電子内視鏡システム、電子内視鏡用のプロセッサ装置、及び電子内視鏡システムの作動方法
WO2012023922A1 (fr) * 2010-08-19 2012-02-23 Thomson Licensing Étalonnage et compensation de couleur pour systèmes d'affichage en 3d
JP5693246B2 (ja) * 2011-01-07 2015-04-01 キヤノン株式会社 立体映像表示装置
EP3553075B1 (fr) * 2012-01-23 2025-01-08 Washington University Systèmes et procédés d'imagerie par lunette

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050024583A1 (en) * 2003-07-28 2005-02-03 Ceramoptec Industries, Inc. Laser safety contact lenses
WO2006073408A2 (fr) * 2005-01-07 2006-07-13 Harvie Mark R Systeme de protection laser actif
US20120075435A1 (en) * 2010-09-23 2012-03-29 Dolby Laboratories Licensing Corporation Method and System for 3D Display Calibration with Feedback Determined by A Camera Device

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016065053A3 (fr) * 2014-10-21 2016-06-16 Qualcomm Incorporated Amélioration automatique d'une image affichée, basée sur un modèle de perception visuelle de l'utilisateur
EP3203308A3 (fr) * 2016-01-18 2017-11-29 Canon Kabushiki Kaisha Système d'affichage, lunettes et procédé de commande de système d'affichage
US10181307B2 (en) 2016-01-18 2019-01-15 Canon Kabushiki Kaisha Display system, eyewear, and method for controlling display system

Also Published As

Publication number Publication date
KR20140001487A (ko) 2014-01-07
US20140002475A1 (en) 2014-01-02
KR101502782B1 (ko) 2015-03-16
US9378547B2 (en) 2016-06-28
CN103514584A (zh) 2014-01-15
EP2679159B1 (fr) 2017-11-29
CN103514584B (zh) 2018-05-08

Similar Documents

Publication Publication Date Title
EP2679159B1 (fr) Dispositif d'imagerie médicale comprenant un dispositif de compensation de distorsion d'image et procédé de compensation de distorsion d'image
LU500127B1 (en) Enhanced augmented reality headset for medical imaging
US10231600B2 (en) Image processing apparatus
US11510599B2 (en) Endoscope system, processor device, and method of operating endoscope system for discriminating a region of an observation target
JP6128888B2 (ja) 画像処理装置、画像処理方法、及び画像処理プログラム
JP7289653B2 (ja) 制御装置、内視鏡撮像装置、制御方法、プログラムおよび内視鏡システム
US10080623B2 (en) Visible light projection device for surgery to project images on a patient
US10820786B2 (en) Endoscope system and method of driving endoscope system
US20210259553A1 (en) Computer systems and media for complementary color flashing for multichannel image presentation
JP5615041B2 (ja) 画像処理装置、画像処理方法
US20080069439A1 (en) System, method and medium performing color correction of display images
US9892512B2 (en) Medical image processing device, operation method therefor, and endoscope system
JP6629639B2 (ja) 内視鏡システム、プロセッサ装置、及び、内視鏡システムの作動方法
JP6533147B2 (ja) プロセッサ装置、内視鏡システム、及び画像処理方法
US7019745B2 (en) Three-dimensional image display device
EP1869875B1 (fr) Unite de conversion de couleur permettant de reduire le frangeage de couleur
JP2023160916A (ja) 内視鏡システム及びその作動方法
JP2016007355A (ja) 光源装置、内視鏡システム、光源装置の作動方法、及び内視鏡システムの作動方法
JP6029585B2 (ja) 視認性に優れたデジタル画像の合成
WO2018159082A1 (fr) Système d'endoscope, dispositif de processeur, et procédé de fonctionnement de système d'endoscope
WO2023192529A1 (fr) Mesure d'acuité visuelle dans un casque rv
NL2003807C2 (en) A method and system for visualizing monochromatic images in color hue.
JPWO2021079691A5 (fr)
JP7196196B2 (ja) 内視鏡システム
Fernandez-Maloigne et al. Spatio temporal characteristics of the human color perception for digital quality assessment

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

17P Request for examination filed

Effective date: 20140314

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

17Q First examination report despatched

Effective date: 20140429

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602013030021

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: A61B0006000000

Ipc: G06T0007000000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RIC1 Information provided on ipc code assigned before grant

Ipc: A61B 8/00 20060101ALI20170428BHEP

Ipc: G09G 5/02 20060101ALI20170428BHEP

Ipc: G06T 7/00 20170101AFI20170428BHEP

Ipc: A61B 5/00 20060101ALI20170428BHEP

Ipc: A61B 6/10 20060101ALI20170428BHEP

Ipc: A61B 6/00 20060101ALI20170428BHEP

Ipc: A61F 9/02 20060101ALI20170428BHEP

INTG Intention to grant announced

Effective date: 20170526

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

GRAL Information related to payment of fee for publishing/printing deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR3

GRAR Information related to intention to grant a patent recorded

Free format text: ORIGINAL CODE: EPIDOSNIGR71

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

INTC Intention to grant announced (deleted)
AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

INTG Intention to grant announced

Effective date: 20171020

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 951039

Country of ref document: AT

Kind code of ref document: T

Effective date: 20171215

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602013030021

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 951039

Country of ref document: AT

Kind code of ref document: T

Effective date: 20171129

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180228

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180301

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602013030021

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20180830

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20180626

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20180630

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180626

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180630

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180626

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180626

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180630

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180630

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180630

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180626

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20130626

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20171129

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171129

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180329

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20210521

Year of fee payment: 9

Ref country code: DE

Payment date: 20210520

Year of fee payment: 9

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602013030021

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MM

Effective date: 20220701

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220701

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230103