NL2027213B1

NL2027213B1 - Inspecting medicine objects based on hyperspectral imaging

Info

Publication number: NL2027213B1
Application number: NL2027213A
Authority: NL
Inventors: Rozeboom Tiemen; Johannes Pruimboom Dries; Giotis Ioannis
Original assignee: Ziuz Holding B V
Priority date: 2020-12-03
Filing date: 2020-12-23
Publication date: 2022-07-06

Abstract

A method of inspecting medicine objects is described, wherein the method comprises: capturing an image of a medicine object; capturing hyperspectral image data of the medicine object; selecting one or more hyperspectral image data parts from the hyperspectral image data based on medicine object localized in the image; determining one or more hyperspectral fingerprints based on the one or more hyperspectral image data parts respectively, a hyperspectral fingerprint being indicative of a spectral response of one or more chemical compounds in a medicine object; and, comparing one of the one or more hyperspectral fingerprints with a reference fingerprint.

Description

NL32863-Vi Inspecting medicine objects based on hyperspectral imaging Field of the invention The invention relates to inspecting medicine objects based on hyperspectral imaging, and, in particular, though not exclusively, to methods and systems for inspecting medicine objects based on hyperspectral imaging and a computer program product for executing such methods.

Background of the invention Patients are provided medicaments according to a prescription.

Especially, people with a chronic disease periodically need to take the same medicines over a long period of time.

Often patients need to take a combination of different medicaments, i.e. pills, tablets and/or capsules.

To facilitate the prescription for a patient, the medicaments may be packed into medicine packets, e.g. a transparent plastic pouches, blisters or bags, according to the prescription using an automated packaging system.

Incorrect packaging of a prescription may be result in the patient taking the wrong (combination of) medicaments or an incorrect dosage of medicaments, which may be harmful for the health of the patient.

To reduce the failure rate, medicine objects are checked by an inspection system which is configured to inspect medicine objects using an image processing system, wherein medicine obigcts may represent e.g. pills and/or tablets, capsules, ampules or packets, blisters or pouches comprising medicine objects.

An example of such inspection system is known from EP2951563, which includes a transporting system for guiding a string of medicine packets through an inspection chamber which includes an imaging system comprising an illumination system for illuminating packets and a multi-colour image sensor system that is capable of capturing color images of a top or bottom view of the medicine packets.

The medicine in the images may be analysed based on shape, dimensions, marks and/or colours.

The image sensor system may also be capable of capturing near infrared images for capturing back-light NIR images that may provide additional information on the shape and dimensions of the medicine.

This information may then be used to inspect if the content of the medicine packet is correct or not.

Although this know inspection system allows reliable inspection of medicine objects, certain circumstance may hinder accurate inspection resulting in for example false positives, i.e. the outcome predicted by the model incorrectly predicts a positive class.

For example, visual features of different medicaments in an image may be identical or at least very similar, making it very difficult to make a distinction.

Additionally, a medicine packet is a 3D object, comprising randomly arranged 3D objects in the form of pills, tablets and/or capsules, while the images created/used by the inspection system are actually 2D representations of a top or bottom view of the packet. Due to this fact, the images will include shading effects and visual effects caused by e.g. objects that lie partially over and/or next to each other and additionally, the images will include artefacts caused by light reflected by the packaging material. These effects and artefacts all contribute to the risk that the system may make an incorrect assessment.

To improve the inspection accuracy other inspection techniques may be considered. For example, it is known that hyperspectral imaging in the near infrared part of the spectrum can be used to inspect the chemical composition of medicine objects. For example, US2014/0319351 describes an inline inspection system for inspecting drugs, pills or tablets in a blister package, based on hyperspectral imaging in the NIR part of the electromagnetic spectrum. Hyperspectral imaging in the near infrared part of the spectrum is especially useful because different medicaments have different reflectance profiles according to their chemical composition. The inspection system is configured to illuminate pills in a blister package with light of a halogen lamp. A hyperspectral image sensor then detects fifteen response values for fifteen bands in the NIR spectrum. This way, for each pixel of the hyperspectral image sensor, multiple spectral response values are detected, which collectively define a spectral response of the object at that pixel location. This spectral response may then be compared to a reference in order to determine if the pills contain the correct composition.

Building an accurate high throughput inspection system for medicine objects including a hyperspectral imaging system as described above however is problematic for several reasons. Firstly, the NIR response of medicaments is a relatively weak signal because most medicaments largely consist of the same ingredients (coating, binder material, etc.) which often account for a large part of the mass of the pill. Therefore, a large number, e.g. a few hundred or more, spectral response values per pixel is required to distinguish different medicaments. In that case however, a hyperspectral image may comprise a considerable amount of data, e.g. more than 100 Mbyte per picture, that need to be analyzed in real-time. Additionally, the weak signal detected by the sensor will include photons that are scattered by the pharmaceutical compound in a tablet and photons that are specularly reflected by packaging material. The latter signal does not contain any information about the pharmaceutical compound and should be filtered out. Further, it is important that the medicine objects need to be illuminated by a homogeneous NIR radiation source, having an irradiance that is sufficiently high so that it penetrates the packet without damaging the packet and the tablets.

Hence, although hyperspectral imaging has been recognized as a valuable tool in to these problems, to date, we are not aware of any medication object inspection systems that includes this technology. Hence, there is a need in the art for improved methods and systems for inspecting medicine objects, in particular methods and systems for inspecting medicine packets based on hyperspectral imaging in the near infrared part of the electromagnetic spectrum, that allows accurate, real-time inspection of medicine packets.

Summary of the invention As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system." Functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.

The methods, systems, modules, functions and/or algorithms described with reference to the embodiments in this application may be realized in hardware, software, or a combination of hardware and software. The methods, systems, modules, functions and/or algorithms may be realized in a centralized fashion in at least one computing system, or in a distributed fashion where different elements are spread across several interconnected computing systems. Any kind of computing system or other apparatus adapted for carrying out the embodiments (or parts thereof) described in this application is suited. A typical implementation may comprise one or more digital circuits such as application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), and/or one or more processors (e.g., x86, x64, ARM, PIC, and/or any other suitable processor architecture) and associated supporting circuitry (e.g., storage, DRAM, FLASH, bus interface circuits, etc.). Each discrete ASIC, FPGA, processor, or other circuit may be referred to as “chip,” and multiple such circuits may be referred to as a “chipset.” In an implementation, the programmable logic devices may be provided with fast RAM, in particular block RAM (BRAM). Another implementation may comprise a non-transitory machine-readable (e.g., computer readable) medium (e.g., FLASH drive, optical disk, magnetic storage disk, or the like) having stored thereon one or more lines of code that, when executed by a machine, cause the machine to perform processes as described in this disclosure.

The flowcharts and block diagrams in the figures may represent architecture, functionality, and operation of possible implementations of the methods, systems and/or modules to various embodiments of the present invention. In this regard, each block in a flowchart or a block diagrams may represent a module, segment, or portion of code, which may be implemented as software, hardware or a combination of software and hardware.

It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware-based systems that performs the specified functions or acts, or combinations of special purpose hardware and computer instructions.

It is an aim of the embodiments in this application to provide an efficient and accurate inspection method for medicine packets that contain one or more medical objects, e.g. pills and/or capsules.

In particular, it is an aim of the embodiments in this application to use hyperspectral imaging in a medicine inspection system so that the system is able to distinguish medicine objects that appear the same to the human eye (for example same color and shape) and thus are not distinguishable by analysing image data in the visible spectrum of the medicine object. For accurate medicine object inspection systems, the ability to accurately distinguish medications based on the substances (composition) is very important, since a very large number of medications are not visually distinct (very often round, white tablets).

Technical advantages of hyperspectral imaging may include the high spectral resolution (>200 bands instead of the three conventional color bands with RGB multispectral imaging), which allows detection of differences in otherwise similar objects in the visible spectrum. Additionally, it allows recognizing different medications based on the non-visible part (the near infrared part) of the electromagnetic spectrum.

In an aspect, the invention may relate to a method for inspecting medicine objects comprising: ccapturing one or more single or multi-band images and hyperspectral image data of a medicine object; selecting one or more hyperspectral image data parts from the hyperspectral image data based on the medicine object localized in the one or more images; determining one or more hyperspectral fingerprints based on the one or more hyperspectral image data parts respectively, a hyperspectral fingerprint being indicative of a spectral response of one or more chemical compounds in a medicine object; and, comparing one of the one or more hyperspectral fingerprints with a reference fingerprint.

In an embodiment, the capturing of the hyperspectral image data may include exposing the medicine object to light having a continuous spectrum, preferably a continuous spectrum in the visible and/or near-infrared region of the electromagnetic spectrum.

In an embodiment, the hyperspectral data may include pixels, each pixel being 5 associated with a plurality of spectral values, preferably the plurality of spectral values including spectral values in the visible and/or the near-infrared region of the electromagnetic spectrum.

In an embodiment, the one or more single or multi-band images may include a 2D grid of pixels, each pixel being associated with one or a few spectral values, preferably a spectral value selected from one or more spectral values, e.g. RGB values and/or an IR value.

In an embodiment, the hyperspectral image data may include line-scan hyperspectral image data, the line-scan hyperspectral image data including lines of pixels.

In an embodiment, the method may further comprise: localizing one or more groups of pixels associated with one or more medicine objects in the image based on a segmentation algorithm.

In an embodiment, selecting one or more hyperspectral image data parts may include: mapping each of the one or more groups of pixels onto the pixels of the hyperspectral image data.

In an embodiment, prior to the selecting one or more hyperspectral image data parts, one or more of the following steps may be executed: removing background pixels (outliers) from the one or more hyperspectral image data using an algorithm, preferably a clustering algorithm; and, removing pixels that are contaminated with specular reflections and/or that are overexposed from the one or more hyperspectral image data.

In an embodiment, the determining one or more hyperspectral fingerprints may further comprise: reducing the dimension of the one or more hyperspectral image data parts, preferably based on a PCA methods; and, determining a fingerprint based on at least one of the one or more reduced hyperspectral image data parts.

In an embodiment, a camera system is used to capture the one or more single or multi-band images and hyperspectral image data, preferably the camera system including a multispectral camera and, optionally, a single or multi-band camera, such as a monochromatic or a color camera.

In an embodiment, the hyperspectral image data may be captured using a hyperspectral line scan camera, wherein during the capturing, the medicine object moves relative to the hyperspectral line scan camera, more preferably the medicine object moves through the field of view of the camera system.

In another aspect, the invention may relate to a module for controlling a medicine inspection apparatus comprising an camera system, the module comprising a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising: capturing one or more single or multi-band images and hyperspectral image data of a medicine object; selecting one or more hyperspectral image data parts from the hyperspectral image data based on the medicine object localized in the one or more images; determining one or more hyperspectral fingerprints based on the one or more hyperspectral image data parts respectively, a hyperspectral fingerprint being indicative of a spectral response of one or more chemical compounds in a medicine object; and, comparing one of the one or more hyperspectral fingerprints with a reference fingerprint.

In a further aspect, the invention may relate to a medicine object inspection apparatus comprising: a camera system, and, a computer readable storage medium having at least part of a program embodied therewith; and, a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising: controlling a camera system to capture one or more single or multi-band images of a medicine object and to capture hyperspectral image data of the medicine object; selecting one or more hyperspectral image data parts from the hyperspectral image data based on the medicine object localized in the one or more single or multi-band images; determining one or more hyperspectral fingerprints based on the one or more hyperspectral image data parts respectively, a hyperspectral fingerprint being indicative of a spectral response of one or more chemical compounds in a medicine object; and, comparing one of the one or more hyperspectral fingerprints with a reference fingerprint. in an embodiment, the hyperspectral data may be determined using a hyperspeciral camera which may be configured to detect the spectral response of an imaged area in the near-infrared (NIR) part of the spectrum.

In another embodiment, the hyperspectral camera may be configurad lc detect the spectral responses of an imaged area in both the visible and NIR part of the spectrum, In that case, the hyperspeciral camera may generale image data both in the visible range and in the NIE range. if the hyperspectral camera is configured to generate both NIR and visible speciral values for each pixel.

A separate mullispaciral camera, 2.g. an RGB or ROB/R camera is no longer needed.

In that cass, ong or more slices of spectral values at one or more wavelengths in the visible spectrum may be taken from the hyperspectal data stack, Hence, in this embodiment, a single or multi color image may be derived from the hyperspeciral Image data, Based on this color image medical obiects, e.g. pills, may be detected and located using standard Image processing algorithms. In an embodiment, the camera system may include a hyperspectral camera and a lamp for #luminating an imaging area of the hyperspeciral camera. In an embodiment, the lamp may includes a housing and an dlumination source. At one side, the housing may noiude an aperture allowing light to exit the housing and Hluminala a medicine object Typically, the dlumination source may be configured to generate light of a continuous spectrum such as a halogen lamp or the light. Such Hlumination sources generate a large amount of heal. Therefore, in some embodiments, the housing may include an cutie! which may be connected to a cooling system, e.g. an air cooling system, This way, a flow, e.g. an air flow, can be generated wherein heal is transported away from the aperture lowards the oullei. This way, it may be avoided that the heat produced by the Hlumination sources increases the temperature of its surroundings.

The invention may also relate to a computer program product comprising software code portions configured for, when run in the memory of a computer, executing the method steps according to any of process steps described above.

The invention will be further illustrated with reference to the attached drawings, which schematically will show embodiments according to the invention. It will be understood that the invention is not in any way restricted to these specific embodiments. Brief description of the drawings Fig. 1 illusirates a medicine object inspection system according to an embodiment of the invention: Fig. 2 iflusirates a medicine object inspection scheme based on hyperspectral imaging according to an embodiment of the invention; Fig. 3 depicts a flow diagram of a method for inspecting medicine packets according to an embodiment of the invention; Fig. 4 depicts a medicine object inspection apparatus according to an embodiment of the invention; Fig. 5 depicts a system for processing hyperspectral imaging data according to an embodiment of the invention; Fig. 6 depicts an example of an image of a medicine packet captured by a hyperspectral imaging system; Fig. 7A-7D depict images processed based on image processing methods according to the embodiments in this application;

Fig. 8A-8D depict images processed based on image processing methods according to the embodiments in this application; Fig. 9 and 10 show images of medicine pouches and fingerprints of medicine objects.

Detailed description Fig. 1 illusirates a medicine object inspection system according to an embodiment of the invention. In particular, the figure depicts an inspection system 104, comprising a transporting system 102 for transporting medicine objects 188 through an inspection grea configured to inspect the medicine objects bassd on an imaging system. The medicine obiects may represent e.g. pills and/or tablets, capsules, ampules or packets or pouches comprising medicine objects, which may be inspected based on an imaging system. in an embodiment, the imaging system may comprise one or more camera systems 114,116.

For exampig, in an embodiment, a first camera system 114 may comprise one oF more image sensors configured to capture images of the medicine objects based a {limited} number of color channels, For example, an image system may include RSB pixels for capturing an RGB color image or three images for each color channel. Additionally, the image system may include a further spectra channel, 8.¢. a channel in the near infrared {NIR) In a further embodiment, a second camera system 118 may comprise a hyperspectral camera system, in particular a hyperspectal camera that may be configured to perform hyperspectral imaging on medicine objects. Pharmaceutically active compounds in the medicine objects are responsive to near infrared radiation, i.e. radiation in the range between 900 and 1700 nm. This way, hyperspectral imaging may be a valuable tool for inspecting medicine objects, such as inspecting pharmaceutically active compounds in a medicine object. Hence, per pixel of the hyperspectal camera, a plurality of spectral values, preferably 100 or more speciral values, may be detected within a predelermined part of the elaciromagnstic spectrum, for example, the visible band between 400 nm and 800 nm andior the near infrared band, e.g. between 800 and 1700 nm. The NIS part of the EM spectrum is especially suitable to determine responses of pharmaceutically active compounds, Hence, each spectral value represents a spectral responses of an object, 2.9. a medicine, captured by the hyperspeciral imaging system.

During hyperspeciral imaging an chisct may be liluminated using an illumination source 122 that Is especially suitable for hyperspectral imaging. For hyperspectral applications the illumination source may be selected to have a continuous spectrum in the relevant parts of the spectrum, for example a continuous spectrum in the UV,

visible and/or near infrared {NIR} range. lllumination sources that are suitable for this purpose include incandescent light sources, such as halogen lamps, that are based on a high-temperature heated filament.

In another embodiment, the hyperspectal camera may be configured to delet the spectral response of an imaged area in both the visible and NIR part of the spectrum. In that case, the hyperspeciral camera may generate image data both in the visible range and in the NIS range. A separate multispectral camera, e.g. an RGB or RGEIR camera is not nesded if the hyperspeciral camera is configured to generate both NIR and visible spectra! values for pixels. In that case, one or more slices of spectral values at one or more wavelengths in the visible spectrum may be taken from the hyperspectrai data stack, Hence, in this embodiment, 3 single or multi-band Image may be derived from the hyperspectral image data. Based on this Image, groups of pixels (blobs) representing medical objects, ec. pills, may be detected and located using standard image processing algorithms.

A computer 118 may control the imaging system and the transport of the medicine objects. Further, the computer may comprise one or more image processing modules configured to process the image data generated by the imaging system so that medicine objects can be reliably inspected. The image processing module may be configured to execute the image processes as described with reference to the embodiments in this application.

Fig. 2 illustrates a scheme for inspecting medicine objects based on hyperspectral imaging according to an embodiment of the invention. In particular, the figure includes a scheme 200, including capturing an image, e.g. an RGB image, of a medicine pouch 201 comprising three pills, and localizing the pills in the image based on object detection and segmentation algorithms. This way groups. of pixels (blobs) can be localized that represent the medicine objects (step 202). Further, the medicine pouch may be imaged by a hyperspectral camera to create hyperspectral image data.

The hyperspectral camera may be implemented in different ways. In a embodiment, the camera may be a 2D camera capturing a 2D exposure area that includes the pouch. Alternatively, in an embodiment, the camera may be a 1D camera, i.e. a line scanner. Such a line scan camera may comprise a row of light-sensitive pixels, which constantly scan moving objects at a high line scan frequency. A two-dimensional image of an object can be generated with a line-scan camera if the object moves under the camera at a known speed. Data generated by a line scanner may be “stitched” together into a 2D image. The data acquired by the hyperspectral cameras may have the form of a “data cube” 204 having a third dimension representing spectral response at different parts of the spectrum and two other dimensions (in the x and y direction) representing the spatial axis and the time respectively as shown in the figure.

Then, based on the localized blobs in the color image, blobs in the hyperspectral image data representing the localized pills may be determined (step 205). Such hyperspectral blob may contain spectral values 206 for a localized medicine object. These values may represent a spectrum 208 of at a pixel location that is part of a medicine object. Based on the spectrum a fingerprint may be determined which can be compared with a reference fingerprint.

Fig. 3 depicts a flow diagram of a method for inspecting medicine objects according to an embodiment of the invention. The process may include z first step 300 of capturing one or more single or muiti-spactral images of the medicine packet. In an embodiment, the one or more images may be captured while exposing the medicine packet to light of one or more parts of the slechromagnetic spectrum, Here, a muli-speciral mage may be an image that has a limited number of color channels, eg. an RGB image or an IR image. In an embodiment, such images may be captured using and RGB camera or a BGB! camera wherein the | represents pixels forming an infrared channel.

In a further step 302, the method may include capturing hyperspectral image data of the medicine packet Here, a hyperspaciral pixel of the nyperspectal mage dats may comprise a plurality of spectral values representing the near-infrared spectral response of the medicine packet at that pixel location {as described above with reference to Fig. 2). In an smbodiment, during the capturing of the hyperspeciral image data the medicine packet may be exposed to light of a continuous spectrum in the visible and/or near-infrared INIR) part of the electromagnetic spectrum.

The process may further include determining one or more first biabs of first pixel groups, representing one or more medicine objects, e.g. pills and/or capsules, inthe ong of more single or multi-spectral Images (step 304). Then, one or more second blobs of second pixel groups may be selected from the hyperspeciral image dala based on the location of the ons or more first blobs In the one or more single or multi-speciral images (step 306}. In step 308 a hyperspectrsl fingering for one of the ons or more second pixel groups may be delermined, wherein a hyperspectral fingerprint may be indicative of a spectra! responses of one or more chemical compounds in the medicine object. Thereafter, the hyperspectral fingerprint may be compared with a reference fingsrprint io determing if the inspected medicine object can be identified as a medicine object according to the reference fingerprint (step 310).

Thus, in short, the method provides an efficient way of inspecting medicine objects based on capturing an image, such as color image, of one or more medicine objects and hyperspectral image data of the one or more medicine objects. Based on one or more medicine objects localized in the image, one or more hyperspectral image data parts from the hyperspectral image data may be selected. The one or more hyperspectral image data parts may be subsequently used for determining one or more hyperspectral fingerprints, wherein a hyperspectral fingerprint is indicative of a spectral response of one or more chemical compounds in a medicine object. These one or more hyperspectral fingerprints are used to determine if the one or more medicine objects can be identified based on reference fingerprints.

Fig. 4 depicts a medicine inspection apparatus comprising a hyperspectral imaging system according to an embodiment of the invention, In particular, the figure depicts an inspection system 400 comprising an imaging system 421 for imaging ong of more madicine objects 40244. The system may further comprise a transport structure 404 comprising a transporting path 406 for guiding one or more medicine objects through an inspection area of the maging system, The medicine objects may ncide pills, tabieis, capsules, ampules, elo. or a packet or pouch comprising such pills, tablets, capsules, ampules, etc. which are inspecisd based on Image dala generaled by ths imaging system. When the inspection system is in use, the medicine obiscts may De transported over the IS transport path to the Inspection area. In an embodiment, the medicine oljecis may be configured a3 a string of packets that can be unwound from a first (upstream) reel 408, guided through the inspection area and rewound around a second (downstream) reel 408. The movement of the reels may be controlled by a motor 412.

Depending on the implementation, the imaging system may comprise ong or more camera systems. For example, in an embodiment, the Imaging system may comprise a camera sysiom 414, 415 comprising one or more mulli-speciral image sensors which are configured to capture images of the packets, based on a (limited) number of color channels. For example, an image system may include RGB pixels for capturrng an RGD color Image or three images Tor each color channel, Additionally, the image system may include one or more further spectral channels, eg. a spectal channel in the near-infrared (NiR).

In another embodiment, the imaging system may comprise a hyperspectral camera system scoording to any of the smbodiments in this application. The hyperspeciral camera system may include a hyperspaciral camera 418 and a lamp 420 for Hluminating an imaging area of the hyperspeciral camera. In an embodiment, the lamp may include a housing 418 and an illumination source 423. At one side, the housing may include an aperture 421 allowing light to exit the housing and iuminate a medicing object. Typically, the fiumination source may be configured to generate Hight of a continuous spectrum such as a halogen lamp or the light. Typically, such lumination sources generate a large amount of heal Therefore, in some embodiment, the housing may include an outlet 425 which may be connected ic a cooling system 422, e.g. an air cooling system, This way, an flow, eg. an alr flow, can be generated wherein heal is ransporied away from the aperture lowards the oullei. This way, it may be avoided that the heat produced by the Hlumination sources increases the temperature of iis surroundings. The Inspection system may be controlled by a controler 424, e.g. a computer, that comprises different modules, e.g. software and/or hardware modules, configured to control the processes that are needed for inspecting the medicine objects.

In an embodiment, the hyperspectral camera may be configured to detect the spectral response of an imaged area in the near-infrared (NIR) pant of the spectrum, In some gmboadimsnts, the hyperspaciral camera may also be configured to detect the spectral response of an imaged area in the visible part of the spectrum. In that case, the hyperspectral camera may generate Image data both in the visible range and in the NIE range. In various embodiments, if the hyperspectral camera is configured to generate both NIS and visible spectral values for each pixel, a separate mullispaciral camera, e.g. an RGB or RGEBAR camera is no longer needed. In that case, one or more slices of speciral values at ong or more wavelengths in the visible spectrum may be laken from the hyperspectral data stack as e.g. described with reference io Fig. 2. Hence, in this embodiment, a single or mult band image may be derived from the hyperspeciral image data. Based on this Image medica! objects, eg. pills, may be detected and located using standard image processing algorithms.

This may be different, if the resolution of the hyperspectral camera is substantially smaller than the multispectral camera. Hence, per camera pixel, a plurality of spaciral values, preferably 100 or more spectral values, may be detected in the near-infrared band e.g between 800 and 1700 nm and/or the visible band. Hence, cach spectral value represents a spactral response of an obiect, e.g. a medicine obiect, that is imaged by the hyperspectral imaging system.

Pictures generated by the first and second camera system may be processed by an image processing module that is executed by the controller 424. For example, image data of the first camera system, e.g. 2D color pictures such as RGE color pictures, may be analyzed using an image processing algorithm which is configured to localize and recognize medicine objects in the picture based on features such as shape and/or color, Dimilarly, image data of the second camera system, eg. a 3D stack of Image data comprising spectral information on medicine objects, preferably near infrared spectral information, may be used io determine a fingerprint of a medicine object, which may be compared with reference fingerprinis In a database in order to derive information about the composition of the medicine obiecl The hyperspectral camera may be implemented in different ways. For example, in an embodiment, the camera may be a 2D imager. In another embodiment, the camera may be implemented as a line scanner. In case of a 2D imager, the camera may comprise a 2D grid of light sensitive pixels configured to generate 2D hyperspectral image data. The 2D hyperspectral image data may include pixels of the imaged area, wherein each pixel is associated with a plurality of spectral response values. In case of a line-scan camera, the camera may comprise a row of light-sensitive pixels, which scans an area at a high line scan frequency to produce 1D hyperspectral image data for each scan. A two-dimensional image of an object can be generated with a line-scan camera if the object moves under the camera at a known speed or if the camera moves over the object at a known speed. In that case, the 1D hyperspectral image data (a line of pixel data, wherein each pixel data includes a plurality of spectral values) that is generated by the line-scanner may be “stitched” together into 2D hyperspectral image data that include pixels of the imaged area, wherein each pixels is associated with a plurality of spectral response values. Thus, the data acquired by the hyperspectral cameras may have the form of a “data cube” having a third dimension representing spectral response at different parts of the spectrum and two other dimensions (in the x and y direction) representing the spatial axis and time, respectively.

In an embodiment, the hyperspectral camera may be configured to generate spectral values in at least the near infrared (NIR) range (wavelengths selected approximately between 900 nm and 1700 nm) of the electromagnetic spectrum. In other embodiments, the hyperspectral camera may be configured to generate spectral values both in the NIR range and in the visible range or only in the visible range. Further, a typical data acquisition of a line-scanner may correspond to a “line” of 600 to 1000 pixels with length approximately between 200 and 300 um each. The width of the pixel varies according to the field of view of the lens but in our case is approximately between 300 and 600 um. Every such spatial pixel may comprise more than 200 spectral values spread equidistantly in the 800 — 1700 nm bandwidth. It is submitted that this figure is merely a non-limiting example of a hyperspectral imaging system that may be used in a medicine inspection system according to the various embodiments described in this application.

The motor, e.g. a stepper motor, that drives the transport structure (e.g. a conveyor belt) may serve as the triggering mechanism for the camera. At each step of the motor the camera may be triggered to acquire a line of pixels. The conveyor belt may be controlled at a speed of 100-200 mm/sec, which would trigger the hyperspectral camera around 300 times per second, so the object is scanned with 300 fps. That means a maximum of 3.3 ms between the acquisition of two consecutive lines and therefore a maximum expasure time not longer than 3 ms, taking into account the time needed to transport the data.

The processing of the hyperspectral data may comprise a step of identifying in the hyperspectral image data, data that are related to specular reflections and overexposed areas (at the packet level) and removing the identified hyperspectral data. Then, in a further step hyperspectral fingerprint(s) (at the pill level) may be determined, wherein each detected medicine object (pill, capsule, tablet) may be represented by a blob on the x-y plane of the hyperspectral cube.

Overexposed pixels and/or pixels that are contaminated from specular reflections may be detected so that these values can excluded from the computation of hyperspectral fingerprints. The detection of pixel values that have been overexposed during acquisition may be based on threshold values. For example, in an embodiment, overexposure may be determined if the reflectance signal equals the maximum of the dynamic range of the sensor. These pixels may be filtered out of the raw data easily since their reflectance values are equal to the maximum of the dynamic range across all spectral bands.

Pixels that are contaminated by specular reflections, mostly reflect the light back to the camera like a mirror, rendering the underlying object invisible. Fig. 6 shows such reflections (white regions as e.g. indicated by references 602 and 604) on a hyperspectral scan of a pouch where the pill inside the pouch is not visible because of reflections of the pouch. The reflectance spectrum in those regions may be essentially equivalent to the spectral power distribution of the light source itself (SPD), which is equivalent to the reflection of the total amount of light emitted.

Known algorithms may be used to detect such regions. For example an target detection technique such as the Constrained Energy Minimization (CEM) technique may be used to detect such regions. CEM is a finite impulse response filter designed to maximize the response of a known target profile and at the same time suppress the response of the composite unknown background, thus matching only the known target spectra. The target spectra may be the SPD of the light source, which may be approximated based on the reflection of a white calibration target that has >95% reflectance grade across the whole spectrum. The composite unknown background may be expressed as a correlation or covariance matrix of all pixels on the x-y plane, giving the CEM detector the following mathematical formulation: Tepm = dR x d"R™ 1d where d is the light source of the target profile, x is the spectrum of a single pixel, and R is the composite background correlation or covariance matrix. Fig. 7A-7D schematically show the process of detection of specular reflections and overexposed pixels and the subsequent removal of these pixels from the hyperspectral image data as shown in Fig. 6. Here, in Fig.

7A specular reflections are detected based on a target detection technique as described above. Similarly, in Fig. 7B overexposed pixels may be determined based on a threshold value. Then, both the pixels affected by specular reflections and overexposure may be used to form a pixel mask as shown in Fig. 7C, identifying pixels (and associated spectral values) that should be removed from the spectral image data. Fig. 7D depicts the result wherein the pixel mask is applied to the hyperspectral image data. Based on these data hyperspectral fingerprints may be determined.

The extraction of a hyperspectral fingerprint of individual medicine objects inside a pouch may comprise a first step of localization of a medicine object, e.g. a pill. The image processing of conventional color pictures that precedes the hyperspectral processing may already provide a robust pill detection and segmentation. The contours of a detected blob representing a medicine object may be used to localize medicine objects inside the pouch. The resolution and the pixel size of an RGB image may be different compared to those of the hyperspectral image, so the contour coordinates need to be scaled so that it can be used to localize blobs of pixels in the hyperspectral data (hyperspectral blobs) representing medicine objects. The scaling coefficients may be constant for every pouch which results in a very fast computation of the coordinates of the tablet on the x-y plane of the hyperspectral image.

Then, outliers (background pixels} may be removed from in the hyperspectral blobs. The hyperspectral blobs may comprise background pixels because the mapping of coordinates from the RGB to the hyperspectral image may not be exact. Additionally, the position of a pouch or a medicine object in the pouch may change slightly when being transported from the color camera exposure area to the exposure area of the hyperspectral camera. In such cases using all the pixels designated by this mapping would result in some background pixels being taken into account in the computation of the medication fingerprint. In order to solve that problem designated pixels may be clustered in two groups according to their spectral characteristics. To this end, in an embodiment, a clustering algorithm such as a k-means clustering algorithm with k=2 clusters may be used for each blob separately. In an embodiment, the centroids of the two clusters may be defined as the spectral mean of the whole pouch, representing the background cluster and the center of mass of the mapped blob, representing the medicine objects. After execution of the clustering algorithm, the pixels assigned to the medication cluster may be used for all subsequent computations.

A further step relates to the de-noising and normalization of pixels in the hyperspectral blob. For the remaining valid pixels, the thermal noise of the camera may be subtracted. This may be realized based on the raw reflectance values. This noise is essentially the signal received by the sensor when the shutter of the camera is closed (complete absence of light}. To obtain a robust measurement of the noise, plurality of scans with the shutter closed may be taken and the values for each wavelength may be averaged. The thus obtained average noise profile may be subtracted from the reflectance of each individual pixel.

Subsequently spectral characteristics of the light source may be removed.

This is done to ensure that only the reflectance characteristics of the medicine objects are used in the determination of a fingerprint.

This may be realized by dividing the reflectance values of every pixel by the average reflectance of the aforementioned white calibration target.

For every pixel a logarithmic derivative may be computed to make the hyperspectral fingerprints invariant to the light intensity.

The logarithmic derivative of a spectrum p at the spectral band i can be computed as: ds; = 1 Piri +€ where € is a small positive constant that ensures that division by zero does not occur.

This form of derivative is called logarithmic because it uses the ratio between consecutive spectra instead of their difference.

The logarithmic derivative may accentuate small structural differences between nearly identical spectra.

The log-derivatives of the spectra may be smoothed with a filter, e.g. a Savitzky-Golay filter, that performs a piece-by-piece fitting of a polynomial function, e.g. second degree polynomial function to the input signal.

The mean of the smoothed logarithmic derivatives of all the valid pixels for each spectral bin may be computed, thus reducing the data to a single reflectance spectrum per medication and averaging out noise.

At this stage, a medication object may be represented by a vector of predetermined dimensions, e.g. 150 dimensions of more.

Each dimension may correspond to a different wavelength in the range 930 - 1630 nm and it may be possible that a number of wavelengths carry no significant discriminative power among different medicine objects.

Such redundant dimensions do not contribute anything to successfully matching medications and in fact they often reduce the performance of a matching algorithm In order to obtain the smallest number of dimensions carrying the maximum amount of discriminative information a dimensionality reduction algorithm such as a PCA dimensionality reduction algorithm may be used.

Such algorithm may be used to detect the non-linear structures in the original data and unfolds them to linearly separable projections.

In an embodiment, a cosine kernel may be used, which essentially means that the data is projected to a new feature space based on the matrix of pairwise cosine distances among the hyperspectral profiles in a reference set.

This step may require to define a set of reference pouches beforehand, as it is this set that is used to compute the Kernel PCA transformation.

The broader and more complete the set of reference pouches is the more robust the Kernel PCA model will be, especially for small numbers of reference patches.

After a certain number of pouches, the projections of the feature space “learned” by the

Kernel PCA algorithm hardly change, but that number is estimated at several hundred pouches. Fig. 5 depicts a method for processing hyperspectral image data according to an embodiment of the invention. Examples of images during the image processing are depicted in Fig. 8A-8D and Fig. 9 and Fig. 10. In particular, this figure depicts a method for processing hyperspectral image data based on the steps as described above. The method may include a step to capture an image of a medicine packet and localize one or more medicine objects in the image and to capture hyperspectral image data from the medicine packet (step 500). Then, a number of image processing steps may be applied to the hyperspectral data. These steps may include removal of background pixels (outliers) from the one or more hyperspectral image data parts using an algorithm, such as a clustering algorithm (step 502). Further, the method may comprise a step of removing pixels that are contaminated with specular reflections and/or that are overexposed from the one or more hyperspectral image data (step 504).

Fig. 8A depicts an example of a localized pill in a color image. Similarly, Fig. 8B depicts a hyperspectral image of the pill and Fig. 8C depicts an image in which pixels comprising specular reflections and overexposure are removed.

Then one or more hyperspectral image data parts may be determined by mapping the one or mare localized medicine objects in the image onto the hyperspectral image data (step 506). This step is illustrated by Fig. 8D which depicts the selection of a blob of pixels from the hyperspectral image data based on the pill that is localized in the color image. In a further step the dimension of the one or more hyperspectral image data parts may be reduced, preferably based on a PCA method (step 508).

Then, a fingerprint may be determined based on at least one of the one or more reduced hyperspectral image data parts (step 510). Fig. 9 and 10 depict examples of fingerprints of two pills of the same pharmaceutical composition, wherein the fingerprints are computed based on the data processing steps described with reference to the embodiments in this disclosure. These results show that the process provides reliable and reproducible results allowing accurate inspection of medicine objects.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and/or firmware.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

CONCLUSIONS

A method of inspecting drug objects comprising: capturing one or more single or multiband images and hyperspectral imaging data of a drug object; selecting one or more hyperspectral image data pieces from the hyperspectral image data based on the drug object located in the one or more images; determining one or more hyperspectral fingerprints based on the respective one or more hyperspectral imaging data parts, wherein a hyperspectral fingerprint is indicative of a spectral characteristic of one or more chemical compositions in a drug object; and, comparing one of the one or more hyperspectral fingerprints with a reference fingerprint.

The method of claim 1, wherein capturing the hyperspectral imaging data comprises exposing the drug object to light having a continuous spectrum, preferably a continuous spectrum in the visible and/or near infrared region of the electromagnetic spectrum.

The method of claims 1 or 2, wherein the hyperspectral data comprises pixels, each pixel associated with a plurality of spectral values, preferably wherein the plurality of spectral values comprise spectral values in the visible and/or near infrared region of the electromagnetic spectrum.

The method of claims 1 to 3, wherein the one or more single or multi-band images comprise a 2D grid of pixels, each pixel associated with one or more spectral values, preferably a spectral value selected from one or more spectral values, for example RGB values and/or an IR value.

The method of claims 1 to 4, wherein the hyperspectral image data comprises line scan hyperspectral image data, wherein the line scan hyperspectral image data comprises lines of pixels.

The method of claims 1 to 5, wherein the method further comprises: locating one or more groups of pixels associated with one or more drug objects in the image based on a segmentation algorithm.

The method of claim 6, wherein selecting one or more hyperspectral image data pieces comprises: mapping each of the one or more groups of pixels to the pixels of the hyperspectral image data.

The method of claims 1 to 7, wherein prior to selecting one or more hyperspectral image data portions, one or more of the following steps are performed:

removing background pixels (outliers) from the one or more hyperspectral image data using an algorithm, preferably a clustering algorithm; removing pixels that are contaminated with specular reflections and/or that are overexposed from the one or more hyperspectral image data.

The method of claims 1 to 8, wherein determining one or more hyperspectral fingerprints further comprises: decreasing the dimension of the one or more hyperspectral image data parts, preferably based on a PCA method; and, determining a fingerprint based on at least one of the one or more reduced hyperspectral image data parts.

The method of claims 1 to 9, wherein a camera system is used to capture the one or more single or multi-band images and hyperspectral image data, preferably the camera system comprises a multispectral camera and, optionally, a single or multi-band camera, such as a monochromatic or a color camera.

The method of claims 1 to 10, wherein the hyperspectral image data is captured using a hyperspectral line scan camera, wherein during recording the drug object moves relative to the hyperspectral line scan camera, more desirably the drug object moves through the field of view of the camera system.

A module for controlling a drug inspection device comprising a camera system, the module embodying a computer readable storage medium embodying computer readable program code, and a processor, preferably a microprocessor, coupled to the computer readable storage medium wherein in response to executing the computer readable program code, the processor is configured to perform executable operations comprising: capturing one or more single or multiband images and hyperspectral imaging data of a drug object; selecting one or more hyperspectral image data pieces from the hyperspectral image data based on the drug object located in the one or more images; determining one or more hyperspectral fingerprints based on the respective one or more hyperspectral imaging data parts, wherein a hyperspectral fingerprint is indicative of a spectral characteristic of one or more chemical compositions in a drug object; and, comparing one of the one or more hyperspectral fingerprints with a reference fingerprint.

13. A drug object inspection device comprising: a camera system,

a computer readable storage medium embodying at least a portion of a program; and, a computer-readable storage medium embodying computer-readable program code, and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein in response to execution of the computer-readable program code, the processor is configured to perform executable operations comprising: controlling a camera system to capture one or more single or multiband images of a drug object and capture hyperspectral imaging data of the drug object; selecting one or more hyperspectral imaging data pieces from the hyperspectral imaging data based on the drug object located in the one or more single or multiband images; determining one or more hyperspectral fingerprints based on the respective one or more hyperspectral imaging data parts, wherein a hyperspectral fingerprint is indicative of a spectral characteristic of one or more chemical compositions in a drug object; and, comparing one of the one or more hyperspectral fingerprints with a reference fingerprint.

A computer program product comprising portions of software code adapted, when executed in the memory of a computer, to perform the method steps of claims 1 to 11.