HK1211721B - Methods and systems for spoof detection for biometric authentication - Google Patents

Description

Method and system for spoofing detection for biometric authentication

Information about divisional applications

This application is a divisional application. The parent application is an invention patent application filed on July 2, 2013, with application number 201310276022.6, and titled “Method and System for Electronic Fraud Detection for Biometric Verification.”

Technical Field

The present invention relates to biometric authentication based on images of the eyes.

Background Art

It is often necessary to restrict access to a property or resource to specific individuals. Biometric recognition systems can be used to verify the identity of an individual to grant or deny access to a resource. For example, an iris scanner can be used by a biometric security system to identify an individual based on the unique structure in their iris.

Summary of the Invention

This specification describes technology related to biometric authentication based on eye images. Generally speaking, one aspect of the subject matter described herein can be embodied in a method comprising obtaining two or more images of a view of a subject including an eye, wherein the images collectively comprise multiple focal lengths. The method may further comprise determining a behavioral metric based at least on detected movement of the eye as it appears in the plurality of images. The behavioral metric may be a measure of deviation of the detected movement and timing from expected movement of the eye. The method may further comprise determining a spatial metric based at least on distances from a sensor to landmarks appearing in the plurality of images, each having a different corresponding focal length. The method may further comprise determining a reflectance metric based at least on detected changes in a surface glare or specular reflection pattern on a surface of the eye as it appears in the plurality of images, wherein the reflectance metric is a measure of changes in a glare or specular reflection patch on the surface of the eye. The method may further comprise determining a score based at least on the behavioral, spatial, and reflectance metrics. The method may further comprise rejecting or accepting one or more images based on the score.

In general, one aspect of the subject matter described in this specification can be embodied in a system comprising a sensor configured to capture two or more images of a view of a subject including an eye, wherein the images collectively comprise multiple focal lengths. The system can further comprise an illumination element that provides light stimulation in synchronization with the capture of the one or more images by the sensor. The system can further comprise means for determining a behavioral metric based at least on detected movement of the eye as it appears in the plurality of images. The behavioral metric is a measure of deviation of the detected movement and timing from expected movement of the eye. The system can further comprise a module configured to determine a spatial metric based at least on distances from the sensor to landmarks appearing in the plurality of images, each having a different corresponding focal length. The system can further comprise a module configured to determine a reflectance metric based at least on detected changes in a surface glare or specular reflection pattern on a surface of the eye as it appears in the plurality of images, wherein the reflectance metric is a measure of changes in a glare or specular reflection patch on the surface of the eye. The system may further include a module configured to determine a score based at least on the behavioral, spatial, and reflective metrics.The system may further include an interface configured to reject or accept one or more images based on the score.

In general, one aspect of the subject matter described in this specification can be embodied in a system comprising a data processing device and a memory coupled to the data processing device. The memory has stored thereon instructions that, when executed by the data processing device, cause the data processing device to perform operations comprising obtaining two or more images of a view of a subject including an eye, wherein the images collectively comprise multiple focal lengths. The operations may further comprise determining a behavioral metric based at least on detected movement of the eye as the eye appears in the plurality of images. The behavioral metric may be a measure of deviation of the detected movement and timing from expected movement of the eye. The operations may further comprise determining a spatial metric based at least on distances from a sensor to landmarks appearing in the plurality of images, each having a different corresponding focal length. The operations may further comprise determining a reflectance metric based at least on detected changes in a surface glare or specular reflection pattern on a surface of the eye as the eye appears in the plurality of images, wherein the reflectance metric is a measure of changes in a glare or specular reflection patch on the surface of the eye. The operations may further comprise determining a score based at least on the behavioral, spatial, and reflectance metrics. The operations may further include rejecting or accepting one or more images based on the scores.

In general, one aspect of the subject matter described herein can be embodied in a non-transitory computer-readable medium storing software comprising instructions executable by a processing device, the instructions, upon execution, causing the processing device to perform operations comprising obtaining two or more images of a view of a subject including an eye, wherein the images collectively comprise a plurality of focal lengths. The operations may further comprise determining a behavioral metric based at least on detected movement of the eye as the eye appears in the plurality of images. The behavioral metric may be a measure of deviation of the detected movement and timing from expected movement of the eye. The operations may further comprise determining a spatial metric based at least on distances from a sensor to landmarks appearing in the plurality of images, each having a different corresponding focal length. The operations may further comprise determining a reflectance metric based at least on detected changes in a surface glare or specular reflection pattern on a surface of the eye as the eye appears in the plurality of images, wherein the reflectance metric is a measure of changes in a glare or specular reflection patch on the surface of the eye. The operations may further comprise determining a score based at least on the behavioral, spatial, and reflectance metrics. The operations may further include rejecting or accepting one or more images based on the scores.

These and other embodiments may each optionally include one or more of the following features. Determining a behavioral metric may include determining the onset, duration, speed, or acceleration of pupil constriction in response to a light stimulus. The light stimulus may include a flashing light pulse. The light stimulus may include a change in the intensity of light output by a display. Determining the behavioral metric may include determining the onset, duration, or acceleration of a gaze shift in response to an external stimulus. The external stimulus may include a cue for instructing a user to direct their gaze. The external stimulus may include an object depicted in a display moving within the display. The spatial metric may be a measure of the subject's deviation from a two-dimensional plane. The spatial metric may be a measure of the subject's deviation from an expected three-dimensional shape. Determining the spatial metric may include determining a disparity of two or more landmarks appearing in a plurality of the images. Halftones may be detected in images captured using a reduced dynamic range, and the images may be rejected based at least in part on the halftones. Determining the behavioral metric may include detecting blood flow in the eye when the eye appears in the plurality of the images. Determining the score may include determining the score using a trained function approximator. The landmark may be a portion of a face depicted in the image. Determining the reflectance metric may include pulsing a flash of light to illuminate the subject while capturing one or more of the images, detecting the occurrence of glare on the eye from the flash in the image, and measuring a time difference between the pulse of the flash and the occurrence of corresponding glare on the eye in the image. Determining the reflectance metric may include pulsing a flash of light to illuminate the subject while capturing one or more of the images, and detecting a fine three-dimensional texture of the white of the eye by measuring the pattern uniformity of the glare on the eye from the flash in the image. A sensor setting controlling focus may be adjusted to a plurality of different settings during the capture of two or more of the images. The images captured at different focus settings may be compared to determine whether the images reflect their respective focus settings. A sensor setting controlling exposure may be adjusted to a plurality of different settings during the capture of two or more of the images. The images captured at different exposure settings may be compared to determine whether the images reflect their respective exposure settings. The sensor setting controlling the white balance may be adjusted to a plurality of different settings during the capture of two or more of the images. The images captured at different white balance settings may be compared to determine whether the images reflect their respective white balance settings.

Certain embodiments of the present invention may be implemented to achieve none, one, or more of the following advantages. Some embodiments may provide security by reliably authenticating individuals. Some embodiments may prevent spoofing eye-based biometric authentication systems using objects that are not real eyes.

The details of one or more embodiments of the present invention are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the present invention will be apparent from the description, drawings, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a diagram of the anatomy of the human eye.

2 is a diagram including an example image showing a portion of the vasculature of the white of the eye.

3 is a diagram of an example image segmented for analysis.

4 is a block diagram of an example security system configured to authenticate an individual based at least in part on one or more images of the white of the eye.

FIG5 is a block diagram of an example online environment.

6 is a flow diagram of an example process for authenticating an individual based on one or more images of the whites of the eyes, in which the eyes in the obtained images used for authentication are checked for realism.

7 is a flow diagram of an example process for determining a realism score for one or more images of an eye.

8A is a flow diagram of an example process for determining a behavioral metric based on pupil constriction in response to light stimulation.

8B is a flow diagram of an example process for determining a behavioral metric based on gaze shifts of the iris in response to an external stimulus.

9 shows an example of a computer device and a mobile computer device that can be used to implement the techniques described here.

DETAILED DESCRIPTION

Unique features of the visible vasculature in the whites of an individual's eyes can be used to identify or authenticate the individual. For example, an image of the whites of a user's eyes can be obtained and analyzed to compare the eye's features with a reference record in order to authenticate the user and grant or deny access to a resource. An adversary or intruder may attempt to spoof a security system using this authentication method by presenting something other than a real eye (e.g., an image of an authorized user's face or a plastic model of an authorized user's eye) to the security system's light sensor. Some spoofing attempts can be thwarted by configuring the security system to analyze the obtained image to distinguish between an image of a real eye and an image of a prop.

One or more realism metrics can be calculated that reflect properties that a real eye is expected to exhibit that may not be exhibited by certain spoofing attempts. For example, stimuli can be applied to the user during the image acquisition process, and the response of the eye depicted in the image can be quantified using metrics, compared to the expected response of a real eye to those stimuli. In some implementations, the acquired image can be examined at multiple focal lengths to determine whether the eye depicted in the image is three-dimensional (e.g., whether the eye has landmarks that appear to be positioned at a distance from the sensor that deviates from a single plane). In some implementations, metrics related to the reflectivity of the eye can be determined. A real eye has unique reflective properties due to its three-dimensional shape and its fine surface texture and moisture, which may not be exhibited by many spoofing attack props. For example, a flash device can be used to illuminate a subject during part of the image acquisition process, and the timing and quality of the reflection of the flash pulse on the subject's eye can be analyzed to determine whether it is indeed a real eye imaged in real time.

In some implementations, multiple realism metrics may be combined to determine a realism score or decision reflecting the likelihood that an image depicts a real eye, rather than, for example, a model image or a two-dimensional image of an eye. For example, a trained function approximator (e.g., a neural network) may be used to determine a realism score based on multiple realism metrics. The obtained image may then be accepted or rejected based on the realism score. In some implementations, a spoofing attempt may be reported when the realism score indicates that the image does not depict a real eye.

FIG1 is a diagram of the anatomy of a human eye 100. The diagram is a cross-section of the eye, with a magnified view 102 of the anatomy near the limbal boundary of the eye, which separates the colored iris 110 from the surrounding white of the eye. The white of the eye contains a complex vascular structure that is not only easily visible from the outside of the eye and scannable, but also unique and varies between individuals. Therefore, due primarily to the vasculature of the conjunctiva and episclera, these vascular structures of the white of the eye can be scanned and advantageously used as a biometric feature. This biometric feature can be used to authenticate a specific individual, or to identify an unknown individual.

The white of the eye has several layers. The sclera 120 is the eye's opaque, fibrous protective layer, containing collagen and elastic fibers. The sclera 120 is covered by the episclera 130, which has a significant number of blood vessels and veins running through and on it. The episclera 130 is covered by the bulbar conjunctiva 140, a thin, transparent membrane that interfaces with the eyelid 150 or, when the eyelid is open, with the environment. Blood vessels and veins pass through all of these layers of the white of the eye and can be detected in images of the eye. The eye also contains eyelashes 160, which can sometimes obscure portions of the white of the eye in an image.

FIG2 is a diagram including an example image 200 showing a portion of the vasculature of the white of the eye. This image 200 may be captured using a sensor (e.g., a camera) integrated into a computing device, such as a smartphone, tablet, television, laptop, or personal computer. For example, a user may be prompted, via a display or audio prompt, to look to the left when the image is captured, thereby exposing a larger area of the white of the eye to the right of the iris to the sensor's field of view. Similarly, the user may be prompted to look to the right, up, down, forward, etc., when the image is captured. The example image includes a view of the iris 220, with the pupil 210 at its center. The iris 220 extends to the limbus boundary 225 of the eye. The white of the eye 230 is outside the limbus boundary 225. The gross vasculature 240 of the white of the eye is visible in image 100. This vasculature 240 may be unique to an individual. In some embodiments, unique characteristics of the vasculature 240 may be used as a basis for identifying, verifying, or authenticating an individual user.

FIG3 is a diagram including an example image 300 showing portions of the vasculature of the whites of two eyes, the image being segmented for analysis. Captured image 310 can be obtained in a variety of ways. Captured image 310 can be pre-processed and segmented to isolate a region of interest within the image and enhance the view of the vasculature in the whites of the eyes. For example, the region of interest can be a tiled portion forming a grid covering some or all of the whites of the eyes. Portion 320 corresponding to the white of the right eye, to the left of the iris, can be isolated, for example, by identifying the limbus boundary and the edge of the eyelid. Similarly, portion 322 corresponding to the white of the left eye, to the left of the iris, can be isolated. Pre-processing can be used to enhance the view of the vasculature in this region, for example, by selecting a component color from the image data that maximizes contrast between the vasculature of the white of the eye and the surrounding white portions. In some embodiments, these portions of the image 320, 322 can be further segmented into tiles forming a grid 330, 332 that divides the exposed surface area of the white of the eye into smaller regions for analysis purposes. Features of the vasculature in these regions of interest can be used to identify, verify, or authenticate an individual.

FIG4 is a block diagram of an example security system 400 configured to authenticate an individual based at least in part on one or more images of the white of an eye 410. A user of security system 400 can present their eye 410 to a light sensor 420. In this manner, one or more images of the white of the eye 410 can be captured. Digital cameras, three-dimensional (3D) cameras, and light field sensors are examples of light sensors that can be used. Light sensor 420 can use a variety of technologies, such as a digital charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). In some implementations, the user can be prompted to perform certain gestures to expose portions of the white of the eye 410 and facilitate image acquisition via a message displayed on display 424. For example, the user can be prompted to orient their gaze so that the iris of their eye 410 turns left, right, upward to the left, and upward to the right. In some implementations (not shown), the user can be prompted to perform certain gestures via a message played through a speaker, via an indicator light (e.g., an LED), or not at all.

In some implementations, the sensor 420 can be configured to detect when the eye 410 is properly positioned in the sensor's field of view. Alternatively, software or hardware implemented on the computing device 430 can analyze one or more images generated by the light sensor 420 to determine whether the eye 410 is properly positioned. In some implementations, the user can manually indicate when the eye 410 is properly positioned through a user interface (e.g., a button, keyboard, keypad, touchpad, or touch screen).

The verification module 440 implemented on the computing device 430 can obtain one or more images of the white of the eye via the light sensor 420. In some implementations, the computing device 430 is integrated with or electronically coupled to the light sensor 420. In some implementations, the computing device 430 can communicate with the light sensor 420 through a wireless interface (e.g., an antenna).

Verification module 440 processes the image obtained via light sensor 420 to control access to security device 450. For example, verification module 440 may implement the verification process described with respect to FIG6. In some implementations, security device 450 may include an actuator 460 (e.g., a locking mechanism) that affects access control instructions from verification module 440.

The computing device can be integrated or interfaced with safety device 450 in a variety of ways. For example, safety device 450 can be a car, light sensor 420 can be a camera integrated into the car's steering wheel or dashboard, and computing device 430 can be integrated into the car and electrically connected to the camera and an ignition lock system acting as safety actuator 460. The user can present a view of the whites of their eyes to the camera in order to be authenticated as an authorized driver of the car and start the engine.

In some implementations, the security device 450 may be a real estate key box, the light sensor 420 may be a camera integrated with a user's mobile device (e.g., a smartphone or tablet device), and the processing of the verification module 440 may be performed in part by the user's mobile device and in part by a computing device integrated with the key box that controls the power locking mechanism. The two computing devices may communicate via a wireless interface. For example, a user (e.g., a real estate agent giving a property showing) may use the camera on their mobile device to obtain one or more images and submit data based on the images to the key box in order to be verified as an authorized user and permitted to use the keys stored in the key box.

In some embodiments, the security device 450 is a gate or door that controls access to a property. The light sensor 420 can be integrated into the door or gate or positioned on a wall or fence near the door or gate. The computing device 430 can be positioned near the light sensor 420 and the electric locking mechanism in the door or gate that acts as an actuator 460 and can communicate with the light sensor 420 and the electric locking mechanism via a wireless interface. In some embodiments, the security device 450 can be a rifle, and the light sensor 420 can be integrated with a scope attached to the rifle. The computing device 430 can be integrated into the stock of the rifle and can be electrically connected to the light sensor 420 and the trigger or firing pin locking mechanism that acts as an actuator 460. In some embodiments, the security device 450 can be a piece of rental equipment (e.g., a bicycle).

Computing device 430 may include processing device 432 (e.g., as described with respect to FIG. 9 ) and a machine-readable repository or database 434. In some implementations, the machine-readable repository may include flash memory. Machine-readable repository 434 may be used to store one or more reference records. The reference record may include data derived from one or more images of the whites of the eyes of registered or authorized users of security device 450. In some implementations, the reference record includes the entire reference image. In some implementations, the reference record includes features extracted from the reference image. In some implementations, the reference record includes encrypted features extracted from the reference image. In some implementations, the reference record includes an identification key encrypted from features extracted from the reference image. To establish a reference record for a new user, an enrollment or registration process may be performed. The enrollment process may include capturing one or more reference images of the whites of the eyes of the newly registered user. In some implementations, the enrollment process may be performed using light sensor 420 and processing device 430 of verification system 400.

5 is a block diagram showing an example of a network environment 500 in which the techniques described herein may be implemented. The network environment 500 includes computing devices 502, 504, 506, 508, 510 configured to communicate with a first server system 512 and/or a second server system 514 via a network 511. The computing devices 502, 504, 506, 508, 510 have respective users 522, 524, 526, 528, 530 associated therewith. The first and second server systems 512, 514 each include computing devices 516, 517 and machine-readable repositories or databases 518, 519. The example environment 500 may include thousands of websites, computing devices, and servers, which are not shown.

The network 511 may comprise a large computer network, examples of which include a local area network (LAN), a wide area network (WAN), a cellular network, or a combination thereof, connecting a plurality of mobile computing devices, fixed computing devices, and server systems. The networks included in the network 511 may provide communication in various modes or protocols, examples of which include transmission control protocol/Internet protocol (TCP/IP), global system for mobile communications (GSM) voice calling, short electronic message service (SMS), enhanced messaging service (EMS), or multimedia messaging service (MMS) messaging, Ethernet, code division multiple access (CDMA), time division multiple access (TDMA), personal digital cellular (PDC), wideband code division multiple access (WCDMA), CDMA2000, or general packet radio system (GPRS), among others. Communication may occur via a radio frequency transceiver. In addition, short-range communication may occur, for example, using Bluetooth, WiFi, or other such transceiver systems.

Computing devices 502, 504, 506, 508, 510 enable respective users 522, 524, 526, 528, 530 to access and view documents, such as web pages included in a website. For example, user 522 of computing device 502 may use a web browser to view the web page. The web page may be provided to computing device 502 by server system 512, server system 514, or another server system (not shown).

In example environment 500, computing devices 502, 504, 506 are illustrated as desktop computing devices, computing device 508 is illustrated as a laptop computing device 508, and computing device 510 is illustrated as a mobile computing device. However, it is noted that computing devices 502, 504, 506, 508, 510 may include, for example, a desktop computer, a laptop computer, a handheld computer, a television having one or more processors embedded therein and/or coupled thereto, a tablet computing device, a personal digital assistant (PDA), a cellular telephone, a network appliance, a camera, a smartphone, an Enhanced General Packet Radio Service (EGPRS) mobile phone, a media player, a navigation device, an electronic messaging device, a game console, or a combination of two or more of these data processing devices or other suitable data processing devices. In some implementations, the computing device may include being part of a motor vehicle (e.g., an automobile, an emergency vehicle (e.g., a fire truck, an ambulance), a bus).

Users interacting with computing devices 502, 504, 506, 508, 510 can interact with a secure transaction service 523, for example, hosted by server system 512, by authenticating themselves and issuing instructions or commands via network 511. Secure transactions may include, for example, e-commerce purchases, financial transactions (e.g., online banking transactions, credit or bank card transactions, loyalty points redemption), or online voting. The secure transaction service may include an authentication module 525 that coordinates the authentication of users from the secure server side of the transaction. In some implementations, the authentication module 525 may receive image data from a user device (e.g., computing device 502, 504, 506, 508, 510) that includes one or more images of the eyes of a user (e.g., user 522, 524, 526, 528, 530). The authentication module may then process the image data to authenticate the user by determining whether the image data matches a reference record of the recognized user's identity, previously established based on the image data collected during the registration session.

In some embodiments, a user who has submitted a service request may be redirected to a verification module 540 running on a separate server system 514. Verification module 540 may maintain a reference record of registered or enrolled users of secure transaction service 523 and may also include reference records of users of other secure transaction services. Verification module 540 may use encrypted network communications (e.g., using a public key encryption protocol) to establish secure sessions with various secure transaction services (e.g., secure transaction service 523) to indicate to the secure transaction services whether the user is verified as a registered or enrolled user. Much like verification module 525, verification module 540 may receive image data from a requesting user's computing device (e.g., computing devices 502, 504, 506, 508, 510) and may process the image data to authenticate the user. In some embodiments, verification module 540 may determine a authenticity score for an image received from a user and may accept or reject the image based on the authenticity score. When the image is rejected as a spoofing attempt to present something other than real human eyes, verification module 540 may send a network communication message to report the spoofing attempt to secure transaction service 523 or the relevant authorities.

The verification module 540 may be implemented as software, hardware, or a combination of software and hardware executing on a processing apparatus, such as one or more computing devices (such as the computer system illustrated in FIG. 9 ).

A user device (e.g., computing device 510) may include an authentication application 550. Authentication application 550 may facilitate authentication of the user as a registered or enrolled user identity for accessing secure services (e.g., secure transaction service 523) via network 511. For example, authentication application 550 may be a mobile application or another type of client application for interacting with a server-side authentication module (e.g., authentication module 540). Authentication application 550 may drive a sensor (e.g., a camera connected to or integrated with the user computing device) to capture one or more images of a user (e.g., user 530), including a view of the whites of the user's eyes. Authentication application 550 may prompt the user (e.g., via a display or speaker) to pose for image capture. For example, the user may be prompted to face the sensor and direct their gaze to the left or right, exposing a significant portion of the whites of their eyes to the sensor.

In some implementations, the verification application 550 transmits the captured image data to a verification module (e.g., verification module 525 or 540) on a remote server (e.g., server system 512 or 514) via the network 511. Collecting image data from the user can facilitate registration and establishing a reference record for the user. Collecting image data from the user can also facilitate verifying the user's identity against the reference record.

In some embodiments, additional processing of the image data may be performed by the verification application 550 for verification purposes, and the results of that processing may be transmitted to a verification module (e.g., verification module 525 or 540). In this way, verification functionality may be distributed between client-side and server-side processing in a manner suitable for a particular application. For example, in some embodiments, the verification application 550 determines a authenticity score for the captured image and rejects any image with a authenticity score that indicates a spoofing attack. If the authenticity score indicates a real eye, the image data may be transmitted to a server-side verification module (e.g., verification module 525 or 540) for further analysis based on the accepted image.

In some embodiments, the authentication application accesses a reference record of the user's identity and performs a full authentication process, then reports the results (eg, whether the user was accepted or rejected) to the server-side authentication module.

The authentication application 550 may be implemented as software, hardware, or a combination of software and hardware executing on a processing apparatus, such as one or more computing devices (such as the computer system illustrated in FIG. 9 ).

FIG6 is a flow chart of an example process 600 for verifying an individual based on one or more images of the whites of the eyes. A authenticity score is determined for the obtained image and used to accept or reject the image. When an image with a real eye is detected and accepted, the image is further analyzed to determine a match score by extracting features from the image and comparing them to a reference record. The user is then accepted or rejected based on the match score.

For example, process 600 may be implemented by verification module 440 in computing device 430 of FIG. 4 . In some embodiments, computing device 430 is a data processing device comprising one or more processors configured to perform the actions of process 600. For example, the data processing device may be a computing device (e.g., as illustrated in FIG. 9 ). In some embodiments, process 600 may be implemented in whole or in part by verification application 550, which is executed by a user computing device (e.g., computing device 510). For example, the user computing device may be a mobile computing device (e.g., mobile computing device 950 of FIG. 9 ). In some embodiments, process 600 may be implemented in whole or in part by verification module 540, which is executed by a user server system (e.g., server system 514). In some embodiments, server system 514 is a data processing device comprising one or more processors configured to perform the actions of process 600. For example, the data processing device may be a computing device (e.g., as illustrated in FIG. 9 ). In some implementations, the computer-readable medium may include instructions that, when executed by a computing device (eg, a computer system), cause the device to perform the actions of processor 600 .

One or more images of an eye are obtained 602. The images include a view of a portion of the eye's vasculature outside the limbus boundary of the eye. The obtained images may be monochrome or represented in various color spaces (e.g., RGB, SRGB, HSV, HSL, or YCbCr). In some implementations, the images may be obtained using a light sensor (e.g., a digital camera, a 3D camera, or a light field sensor). The sensor may be sensitive to light in various wavelength ranges. For example, the sensor may be sensitive to the visible spectrum of light. In some implementations, the sensor is paired with a flashlight or a flashlight that can be pulsed to illuminate an object in the sensor's view. The image capture may be synchronized with or time-locked to the pulsation of the flashlight. In some implementations, the sensor captures a series of images that can be used to track the movement of an object within the sensor's field of view. The sensor may include one or more settings that control image capture (e.g., focus, flash intensity, exposure, and white balance). The images may collectively include multiple focal lengths. For example, a series of images may be captured, each image being captured with a different focal length setting of the sensor, and/or some sensors (e.g., light field sensors) may capture images focused at multiple distances from the sensor. In some implementations, one or more images may be obtained 502 by being received via a network interface (e.g., a network interface of the server system 514).

A realism score for the one or more images can then be determined 604. In some implementations, image data elements (e.g., voxels, pixels, rays, or red, green, or blue channel values) are directly input to a trained function approximator, which outputs a realism score. The function approximator can be trained using data corresponding to training images of both real eyes and spoof props paired with ideal scores (e.g., real eyes are 1 and spoof props are 0). The function approximator or classifier models the mapping from input data (i.e., training image data or features) to output data (i.e., the resulting realism score or binary decision) using a set of model parameters. The model parameter values are selected using a training algorithm applied to the training data. For example, the function approximator can be based on a model such as linear regression, Volterra series, Wiener series, radial basis kernel function, kernel method, polynomial method, piecewise linear model, Bayesian classifier, k-nearest neighbor classifier, neural network, support vector machine, or chaotic function approximator. Other models are possible. In some implementations, the truthiness score may be binary.

In some implementations, a realism score is determined 604 based on one or more realism metrics, which in turn are determined based on the obtained image.Some examples of this process are described with respect to FIG.

For example, the authenticity score may be determined 604 by verification module 440 , verification application 550 , verification module 525 , or verification module 540 .

The realism score is checked 606 to determine whether the image likely contains a view of a real person's eyes.In some implementations, the realism score may be compared to a threshold.

If the authenticity score indicates a low likelihood of a real eye and, therefore, a high likelihood of a spoofing attack, then one or more images are rejected 608. In some embodiments, the spoofing attack can then be reported 610. In some embodiments, the spoofing attack is reported 610 through a display or speaker (e.g., with an alarm sound or flashing display). In some embodiments, the spoofing attack is reported 610 by transmitting one or more messages over a network using a network interface. The user can then reject 630 and not allow access to the secure device or service.

In some implementations (not shown), a check can be performed to verify that the obtained image was captured from a specific sensor and that the specific sensor has not been bypassed by the submission of spoofed image data. For example, during image capture, one or more sensor configuration settings can be adjusted to assume different settings during the capture of two or more of the images. These different settings are expected to be reflected in the obtained image data. If the image data changes between images with different settings, it can indicate that the sensor has been bypassed by a spoofing attack. For example, sensor configuration settings that control focus, exposure time, or white balance can be adjusted in this manner. If no corresponding change in the obtained image data is detected, the obtained image can be rejected 608.

If the authenticity score indicates a high likelihood that real human eyes are depicted in the image, then one or more images are accepted 616 and undergo further analysis to complete the verification process.

The one or more images may be segmented 620 to identify a region of interest that includes an optimal view of the vasculature in the white of the eye. In some embodiments, anatomical landmarks (e.g., the iris, its center and limbal border, the canthus, and the edge of the eyelid) may be identified in the one or more images. The region of interest within the image may be identified and selected based on its position relative to the identified anatomical landmarks. For example, the region of interest may be located to the left, right, above, or below the iris in the white of the eye. In some embodiments, the selected region of interest is tiled to form a grid that covers a larger portion of the white of the eye. In some embodiments, the selected region of interest of the image is discontinuous (e.g., adjacent regions may overlap, or there may be space between adjacent regions). The selected region of interest may correspond to a region of interest selected from a reference image (the data in the reference record is based on the reference image).

In some embodiments, the canthus is found by fitting curves to the selected portion of the eyelid on the sclera, and then extrapolating and finding the intersection of those curves. If an intersection point (the canthus) cannot be found due to the fact that the sclera is too close (e.g., due to gaze direction), a template from the same canthus region but from a photo of the opposite gaze direction can be derived and applied to the problematic canthus neighborhood in the image at hand, and the maximum correlation position can be marked as the canthus.

In some embodiments, the eyelid is found by an adaptive thresholding method that finds the white of the eye from the image, which borders the eyelid.The scleral mask itself can be corrected by morphological operations (e.g., convex hull) to remove aberrations.

In some embodiments, the limbal boundary is found from the scleral mask, which is where the sclera ends because it ends at the iris limbal boundary.

In some implementations, the iris center is found through a variety of methods. If the eye is bright in color, the center of the pupil can be found as the iris center. If the iris is too dark, the center of an ellipse fitted to the limbus boundary and its center is found, or it is determined as the focus of normal rays (i.e., lines perpendicular to the tangents to the limbus boundary) converging around the iris center, or a combination of these methods.

The image region may be pre-processed 622 to enhance the view of the vasculature within the image. In some embodiments, pre-processing 622 includes color image enhancement and contrast-limited adaptive histogram equalization (CLAHE), which enhances the contrast of the intensity image. CLAHE operates on small regions of the image, called tiles. The contrast of each tile is enhanced so that the output histogram approximately matches a histogram specified by a particular distribution (e.g., uniform, exponential, or Rayleigh). Adjacent tiles are then combined using bilinear interpolation to eliminate artifactual boundaries. In some embodiments, the image may be enhanced by selecting one of the red, green, or blue components that provides the best contrast between the blood vessels and the background. The green component may be preferred because it provides the best contrast between the blood vessels and the background.

In some embodiments, preprocessing 622 includes applying a multi-scale enhancement filtering scheme to enhance the intensity of the image, thereby facilitating the detection of vascular structures and subsequent feature extraction. The parameters of the filter can be determined empirically to take into account variations in the girth of the blood vessels. The algorithm used can have good curve sensitivity, good curve specificity, and suppress objects of other shapes. The algorithm can be based on the second-order derivative of the image. First, since the second-order derivative is sensitive to noise, the image segment is convolved with a Gaussian function. The parameter σ of the Gaussian function can correspond to the thickness of the blood vessel. Next, for each image data element, a Hessian matrix can be established, and the eigenvalues λ1 and λ2 can be calculated. In each Hessian matrix, the matrix ridge is defined as the point where the image has an extreme value in the curvature direction. The curvature direction is the eigenvector of the second-order derivative of the image, which corresponds to the maximum absolute eigenvalue λ. The sign of the eigenvalue determines whether it is a local minimum λ>0 or a maximum λ<0. The calculated eigenvalues are then used to filter the blood vessel lines using the following equation:

I_line(λ1,λ2)=|λ1|-|λ2| (if λl<0), and I_line(λ1,λ2)=0 (if λ1≥0)

The diameter of the blood vessels varies, but the algorithm assumes that the diameter is in the interval [d0, d1]. A Gaussian smoothing filter with a scale of [d0/4, d1/4] can be used. This filtering can be repeated N times based on the following smoothing scale:

σ1＝d0/4,σ2＝r*σ1,σ2＝r^2*σ1,...σ2＝r^(N-1)*σ1＝d1/4

This final output may be the maximum of the outputs from all individual filters at the N scales.

A feature of each image region is determined 624, which reflects the structure or characteristics of the vasculature visible in that region of the user's eye. In some implementations, a detail detection method can be used to extract features of the user's vasculature. An example of a detail detection process is described in U.S. Patent No. 7,327,860.

In some embodiments, features may be determined 624 in part by applying a set of filters corresponding to the texture characteristics of those image regions to the image regions. For example, features may be determined in part by applying a set of complex Gabor filters at various angles to the image. The parameters of the filters may be determined empirically to account for variations in the spacing, orientation, and girth of the vessels. The texture characteristics of the image may be measured as the amount of sharp visible vasculature in the region of interest. This quality may be determined as the ratio of the area of the sharp visible vasculature to the area of the region of interest. The phase of the Gabor-filtered image, when binarized using a threshold, may facilitate detection and revealing of sharp visible vasculature.

When the Gabor filter kernel is configured with sigma = 2.5 pixels, frequency = 6, and gamma = 1, the phase of the complex Gabor-filtered image reflects the vascular pattern at different angles. The choice of frequency can depend on the distance between vasculature, which in turn depends on the resolution and the distance between the image acquisition system and the subject. These parameters can be invariant for the image. For example, kernel parameters can be derived for an eye image captured at a distance of 6 to 12 centimeters from the eye using a specific sensor (e.g., the rear camera on a smartphone), and the segmented scleral region can be resized to a resolution of (e.g., 401 x 501 pixels) for analysis. It can be seen that the ocular surface vasculature is distributed in all directions across the white of the eye. For example, the Gabor kernel can be aligned at six different angles (angle = 0, 30, 60, 90, 120, and 150 degrees). The phase of the Gabor-filtered image can vary from -π to +π radians. Phase values above 0.25 and below -0.25 radians may correspond to vascular structures. To binarize the phase image using thresholding, all phase values above 0.25 or below -0.25 can be set to 1, and the remaining values set to 0. This can result in sharp vasculature structures in the corresponding phase image. This operation can be performed for images resulting from the application of all six Gabor kernels at different angles. All six binarized images can be added to reveal fine and clear vasculature structures. In some embodiments, a vector of elements of the binarized phase image can be used as a feature vector for comparing the image to a reference recording. In some embodiments, differences in texture features between image regions of interest can be used as feature vectors. The sum of all 1s in the binarized image region divided by the region of interest can reflect the extent of visible vasculature.

A match score is determined 626 based on the features and corresponding features from the reference record. The reference record may include data based at least in part on one or more reference images captured during the user's enrollment or registration process. In some implementations, the match score may be determined 626 as a distance (e.g., Euclidean distance, correlation coefficient, modified Hausdorff distance, Mahalanobis distance, Bregman divergence, cosine similarity, Kulbeck-Leibler distance, and Jensen-Shannon divergence) between a feature vector extracted from one or more obtained images and a feature vector from the reference record. In some implementations, the match score may be determined 626 by inputting the features extracted from one or more obtained images and the features from the reference record into a trained function approximator.

In some implementations, a quality-based fused match score is determined 626 based on the match scores of the multiple images of the same vasculature. In some implementations, the match scores of the multiple images are combined by adding the match scores together in a weighted linear combination with weights that depend on the quality scores determined for each of the multiple images, respectively. Other examples of techniques that can be used to combine the match scores of the multiple images based on their respective quality scores include hierarchical blending, sum rule, product rule, gated fusion, Dempster-Schafer combination, and stacked generalization, among others.

In some implementations, a match score is determined 626 by a verification module (eg, verification module 440 running on computing device 430).

The match score may be checked 628 to determine whether a match exists between the one or more acquired images and the reference record. For example, the match score may be compared to a threshold. A match may reflect a high likelihood that the user whose eyes are depicted in the one or more acquired images is the same individual associated with the reference record.

If there is no match, the user may be denied 630. As a result, the user may not be allowed to access the secure device or service (e.g., the secure device 450 or the secure transaction device 523). In some embodiments, the user may be notified of the denial 630 by a message displayed on a display or played through a speaker. In some embodiments, the denial may be effected by transmitting a message over a network reflecting the status of the denied user. For example, the verification module 540 may transmit a denial message to the secure transaction server 523 using the network interface of the server system 514 immediately after denying the user 530. The verification module 540 may also send the denial message to the user computing device 510 in this case.

If there is a match, the user may be accepted 632. As a result, the user may be granted access to the secure device or service (e.g., secure device 450 or secure transaction device 523). In some embodiments, the user may be notified of the acceptance 632 by a message displayed on a display or played through a speaker. In some embodiments, acceptance may be effected by transmitting a message reflecting the status of the user being accepted via a network. For example, upon accepting the user 530, the verification module 540 may transmit an acceptance message to the secure transaction server 523 using the network interface of the server system 514. The verification module 540 may also send the acceptance message to the user computing device 510 in this case.

7 is a flow diagram of an example process 700 for determining a realism score for one or more images of an eye.One or more realism metrics are determined 710 for the image, and a realism score is determined 730 based on the one or more realism metrics.

For example, process 700 may be implemented by verification module 440 in computing device 430 of FIG. 4 . In some embodiments, computing device 430 is a data processing device comprising one or more processors configured to perform the actions of process 700. For example, the data processing device may be a computing device (e.g., as illustrated in FIG. 9 ). In some embodiments, process 700 may be implemented in whole or in part by verification application 550, which is executed by a user computing device (e.g., computing device 510). For example, the user computing device may be a mobile computing device (e.g., mobile computing device 950 of FIG. 9 ). In some embodiments, process 700 may be implemented in whole or in part by verification module 540, which is executed by a user server system (e.g., server system 514). In some embodiments, server system 514 is a data processing device comprising one or more processors configured to perform the actions of process 700. For example, the data processing device may be a computing device (e.g., as illustrated in FIG. 9 ). In some implementations, the computer-readable medium may include instructions that, when executed by a computing device (eg, a computer system), cause the device to perform the actions of processor 700 .

Processor 700 begins 702 when one or more images are received for processing. For example, the one or more images may be encoded as a two-, three-, or four-dimensional array of data image elements (e.g., pixels, voxels, rays, or red, green, or blue channel values).

One or more realism metrics may then be determined 710 based on the one or more images. In this example, a behavior metric is determined 712 based on the detected movement of the eye as it appears in the multiple images. The behavior metric may be a measure of the deviation of the detected movement and timing from the expected movement of the eye.

In some implementations, light stimuli are applied to the subject while the image is being captured (e.g., a flashing light pulse, changing brightness of an LCD display). In response to these light stimuli, the pupil of a real eye is expected to constrict to adapt to the change in illumination. Furthermore, the pupil is expected to constrict in a specific manner over time, with a start time that depends on the user's reaction time, a duration of the constriction movement required to reach a new steady-state pupil diameter, an average velocity of the constriction, and a specific acceleration profile of the constriction motion. By examining a sequence of images captured before and after the onset of the light stimuli, one or more parameters of the detected motion can be determined and compared to one or more parameters of the expected motion. Substantial deviations from the expected motion in response to the light stimuli can indicate that the subject in the camera view is not a real eye and that a spoofing attack has occurred. An example of this implementation is described with respect to FIG8A.

In some implementations, behavioral metrics can be determined 712 by applying external stimuli to the subject during image capture (e.g., a cue instructing the user to direct their gaze or a display showing a moving object that the user follows with their eyes) and tracking the resulting gaze shifts. In response to these external stimuli, a real person's eyes are expected to move in a specific manner over time. Some parameters of the expected gaze shift motion may include a start time that depends on the user's reaction time, the duration of the gaze shift movement required to reach a new steady-state gaze direction, the average velocity, and a specific acceleration profile of the gaze shift motion. By examining a sequence of images captured before and after the onset of the external stimulus, one or more parameters of the detected motion can be determined and compared to one or more parameters of the expected motion. Substantial deviations from the expected motion in response to the external stimulus may indicate that the subject in the camera view is not a real person's eye and that a spoofing attack has occurred. An example of this implementation is described with respect to FIG. 8B .

In some implementations, determining 712 the behavioral metric may include detecting blood flow in the vasculature of the white of the eye (e.g., vasculature in the episclera). The sequence of images may be analyzed to detect changes in hue and visible width of veins and blood vessels in the white of the eye over time. The vasculature of a real eye is expected to exhibit regular changes in vessel width and hue corresponding to the user's pulses. Substantial deviations from the expected blood flow pattern may indicate that the subject in the camera view is not a real eye and that a spoofing attack has occurred.

For example, consider a section of vasculature between two branch points or sharp bends. The tubular body of that vessel changes shape and color as the heart pumps blood through it. In some implementations, 300 frames or images may be captured over a 10-second period. The image region may be registered, capture instance by capture instance. Blood flow can then be measured by comparing the physical dimensions (2D or 3D) of points of interest along the vessels over time, as well as the color of those vessels over time. In this way, changes consistent with a pulse can be detected. For example, a "pulse" signal is measured to see if it resembles a square wave, which would not be consistent with a natural circulatory system. If the measured "pulse" signal consists of spikes (both vessel dilation and appropriate color change) at regular intervals over time, within the normal range for a human user (perhaps even a specific user), then the input likely corresponds to a true pulse. The distance between the measured pulse signal and the expected pulse signal can be determined to assess the likelihood that the subject is a real person's eye, rather than a spoofing attack.

In some embodiments, the expected motion parameters are specific to a particular user and are determined during a registration session and stored as part of a reference record for the particular user. In some embodiments, the expected motion parameters are determined for a large population based on user data or offline research.

For example, the behavioral metrics may be determined 712 by a verification module or application (eg, verification module 440).

In this example, a spatial metric is determined 714 based on the distance from the sensor to a landmark appearing in multiple images, each having a different corresponding focal length. Focal length is the distance from the sensor to a perfectly focused point in its field of view. For certain reasons, the focal length can be adjusted for different images by adjusting the sensor's focus configuration settings. For example, a landmark (e.g., the iris, the corner of the eye, the nose, the ear, or a background object) can be identified and located in multiple images with different focal lengths. The landmark in a particular image is represented as having a degree of focus that depends on how far the object corresponding to the landmark is from the focal point in the sensor's field of view. The degree of focus is a measure of the degree to which the image of the landmark is blurred by optical effects in the optical sensor (e.g., due to diffraction and convolution due to the aperture shape). The degree of focus of a landmark in a particular image can be estimated by determining the high-frequency components of the image signal near the landmark. When the landmark is in focus, a higher high-frequency component is expected near it. When the landmark is out of focus, a lower high-frequency component is expected. By comparing the degree of focus of the landmark in the image at different focal lengths, the distance from the sensor to the landmark can be estimated. In some implementations, the distances of multiple landmarks from a sensor (e.g., a camera) are estimated to form a topological map of the subject in the sensor's view (consisting of a set of three-dimensional landmark positions). The positions of these landmarks in the space viewed by the camera can be compared to the model by determining a spatial metric reflecting the deviation from the model (e.g., the mean square error between the detected positions of one or more landmarks and the corresponding modeled positions of the one or more landmarks).

In some embodiments, the spatial metric is a measure of the deviation of a subject from a two-dimensional plane. One possible spoofing strategy is to present a two-dimensional image (e.g., a photograph) of the eyes of a registered user to the sensor. However, unlike the landmarks in and around a real eye, the locations of the landmarks in the two-dimensional image (e.g., eyes, nose, mouth, and ears) will appear in a two-dimensional plane. For example, the locations of multiple landmarks can be fitted to the closest two-dimensional plane, and the average distance of the landmarks from this fitted plane can be determined as the spatial metric. A high value of this spatial metric can indicate a three-dimensional subject and a high probability that the subject is a real eye, while a low value can indicate a high probability that the subject is a two-dimensional spoofing attack.

In some embodiments, a spatial metric is a measure of a subject's deviation from an expected three-dimensional shape. A three-dimensional model containing the positions of landmarks corresponding to the expected shape of a subject including a user's real eyes can be used to compare the detected landmark positions. In some embodiments, the relative positions of landmarks on a particular user's face can be determined during an enrollment session and used to generate a three-dimensional model that is stored as part of a reference record. In some embodiments, a three-dimensional model of a user population can be determined based on a collection of measurements or studies of many people. Various types of metrics can be used as spatial metrics to compare the detected landmark positions to the expected shape (e.g., Euclidean distance, correlation coefficient, modified Hausdorff distance, Mahalanobis distance, Bregman divergence, Kullback-Leibler distance, and Jensen-Shannon divergence).

In some implementations, determining 714 the spatial metric includes determining the parallax of two or more landmarks appearing in the multiple images. Parallax is the apparent displacement of an observed object due to a change in the observer's position. Multiple images captured from different angles on a subject may cause landmarks within the images to appear to move by different amounts due to differences in their distances from the sensor. This parallax effect can be measured and used as a spatial metric reflecting the three-dimensional nature of the subject in the sensor's view. If all landmarks in an image experience the same apparent displacement due to relative motion of the sensor—that is, if the differences in the landmarks' parallax effects are small—then the subject viewed by the camera has a higher probability of being a two-dimensional spoofing attack. In some implementations, the sensor moves around the subject during image capture to collect image data from different orientations relative to the subject. For example, a single camera may rotate or slide slightly, or multiple cameras at different locations may be used for image capture. In some implementations, the user is prompted to move in order to change the relative orientation of the subject and sensor. In some implementations, it is assumed that the sensor will naturally move relative to the subject. For example, where the sensor is a camera in a handheld user device (eg, a smartphone or tablet), the sensor may move naturally relative to the user's face due to involuntary tactile motion.

For example, the spatial metrics may be determined 714 by a verification module or application (eg, verification module 440).

In this example, a reflectance metric is determined 716 based on detected changes in surface glare or specular reflection patterns on the surface of the eye as the eye appears in multiple images. The reflectance metric can be a measurement of changes in the glare or specular reflection patch on the surface of the eye. When the illumination of the eye in the sensor's view changes, due to the relative motion of the eye and the light source or due to a dynamic light source (e.g., a flash, LCD screen, or other lighting element), the glare and specular reflection patterns visible on the eye are expected to change by appearing, disappearing, growing, shrinking, or moving. In some implementations, the illumination change is caused by a light stimulus (e.g., a flash pulse) or an external stimulus (e.g., a prompt instructing the user to change the direction of gaze) during image capture. For example, glare (including its boundaries) can be detected by thresholding a contrast-enhanced image to find the whitest point. By determining 716 a reflectance metric that measures the deviation of the detected change from the expected change, the detected change in the glare or specular reflection pattern on the eye in the image can be compared with the expected change in these patterns.

Look for changes in the area and shape of this glare. Also look at the perimeter to area ratio of the glare patch.

In some implementations, a flash may be pulsed while capturing one or more of the images, illuminating the subject. Glare from the flash can be detected on the eye as it appears in the image. The pulsation of the flash can be synchronized with image capture, allowing the time difference between the time the flash is pulsed and the time the corresponding glare appears in the image to be measured. A reflectance metric can be based on this time difference. Large deviations from the expected synchronization or time lock between the flash pulse and the onset of the corresponding glare or specular reflection can indicate a spoofing attack. For example, a replay attack uses pre-recorded video of the captured scene. Changes in glare from the pre-recorded video are unlikely to be time-locked to the real-time flash event during the current session. Another example is presenting a print image of the eye to the sensor. In this case, the glare may be spread across the print image in an unnaturally uniform manner or may change imperceptibly due to the lack of moisture on the viewed surface. If no corresponding glare or specular reflection is detected, the reflectance metric can be determined as an arbitrarily large number corresponding to poor synchronization or lack of time lock between the flash and the detected glare or specular reflection.

In some implementations, a change in illumination can be detected when the uniformity of the glare pattern changes as the intensity of the illumination increases, revealing a change in the uniformity of the glare pattern caused by a greater amount of fine three-dimensional texture in the white of the eye. For example, a flash may be emitted in pulses during one or more of the captured images, thereby illuminating the subject with a higher intensity. The fine three-dimensional texture of the white of the eye can be detected by measuring the uniformity of the pattern of glare on the eye in images before and after the start of the flash pulse. For example, the uniformity of the glare of the specular reflection pattern can be measured as the ratio of the perimeter to the area of the glare. The greater this number is than 2/R, the less circular and less uniform the glare is (R is the estimated radius of the glare diaphragm). In some implementations, a function approximator (e.g., a neural network) is trained to distinguish between specular reflection patterns recorded from real human eyes and synthetic eyes (e.g., 3D imprinted eyes) using a sensor with an illumination element (e.g., a flash).

For example, the reflectance metric may be determined 716 by a verification module or application (eg, verification module 440).

In some embodiments (not shown), additional truthiness metrics may be determined 710. For example, a metric may be determined that reflects the extent of saccadic motion of an eye in the sensor's view. The iris of the eye may be a landmark in the image sequence, allowing its position or orientation to be tracked. This sequence of positions or orientations may be analyzed to determine the extent of saccadic motion by filtering for motion at specific frequencies associated with normal saccadic motion.

In some implementations, a authenticity metric reflecting the degree of halftoning in a captured image may be determined 710. Halftoning is an artifact of digitally imprinted images that can be used in spoofing attacks, and therefore its presence may indicate a high likelihood of a spoofing attack. For example, one or more images may be captured using a reduced dynamic range of a sensor (e.g., a camera), enabling finer resolution of the intensity of detected light across the range of light intensity present in the captured image. In this way, intensity or color scales may be amplified to reveal more subtle changes in the level of the detected image signal. If the captured image were of a real human eye, the range of detected color or intensity values would be expected to vary continuously. In contrast, a spoofed image (e.g., a digital photograph presented to a sensor) may exhibit large, discontinuous jumps corresponding to halftoning. The degree of halftoning in an image can be measured in various ways (e.g., as the average or largest eigenvalue of a Hessian matrix evaluated in an image region, or as a high-frequency component of the image signal). In some implementations, images with a halftoning metric above a threshold are rejected. In some implementations, a histogram of gray shades in an image is generated, and the uniformity of the distribution between gray level bins (eg, 256 bins) is measured.

In some embodiments, the truthfulness measure is determined 710 in parallel. In some embodiments, the truthfulness measure is determined 710 continuously.

A realism score can then be determined 730 based on the one or more realism metrics.In some implementations, the realism score is determined by inputting the one or more realism metrics into a trained function approximator.

A function approximator can be trained using data corresponding to training images of real eyes and various spoofing attacks that have been correctly labeled to provide the desired output signal. The function approximator models the mapping from input data (i.e., training image realism metrics) to output data (i.e., realism scores) using a set of model parameters. The model parameter values are selected using a training algorithm applied to the training data. For example, the function approximator can be based on linear regression, Volterra series, Wiener series, radial basis kernel functions, kernel methods, polynomial methods, piecewise linear models, Bayesian classifiers, k-nearest neighbor classifiers, neural networks, support vector machines, or chaotic function approximators. In some implementations, the realism score can be binary.

For example, a authenticity score can be determined 730 by an authentication module or application (eg, authentication module 440 ) based on one or more authenticity metrics.

The resulting authenticity score may then be returned 740 and may be used in a variety of ways by a verification system, such as verification system 400. For example, the authenticity score may be used to accept or reject one or more images.

FIG8A is a flow diagram of an example process 800 for determining a behavioral metric based on pupil constriction in response to a light stimulus. One or more light stimuli are applied 810 to a scene viewed by a sensor (e.g., light sensor 420). For example, the light stimulus may include a flashing light pulse or a change in the brightness of a display (e.g., an LCD display). A sequence of images is captured 812 by the sensor before and after the onset of the light stimulus. For example, the image sequence may be captured at regularly spaced times (e.g., 10, 30, or 60 Hz) within a time interval (e.g., 2, 5, or 10 seconds) that includes the onset of the light stimulus.

In some implementations, the pupil and the diameter of the pupil, which are landmarked in each of the captured images, are determined 814 in each captured image. The diameter may be determined 814 relative to a starting diameter of the pupil measured in one or more images captured prior to the start of light stimulation.

The resulting sequence of pupil diameters measured in response to light stimulation may be analyzed to determine 816 one or more motion parameters of pupil constriction in response to the light stimulation. In some implementations, the motion parameters of pupil constriction may include the start time of the constriction motion relative to the start of the light stimulation. The start is the time delay between the start of the light stimulation and the start of the constriction motion. In some implementations, the motion parameters of pupil constriction may include the duration of the constriction motion. The duration is the length of time between the start of the constriction motion and the end of the constriction motion, when the pupil diameter reaches a new steady-state value (e.g., after which the diameter does not change within a minimum time interval). In some implementations, the motion parameters of pupil constriction may include the velocity of the pupil constriction. For example, the velocity may be determined as the difference in pupil diameter between two time points divided by the length of the time interval therebetween. In some implementations, the motion parameters of pupil constriction may include the acceleration of the pupil constriction in different time segments of the constriction cycle. For example, the acceleration may be determined as the difference in velocity between two time intervals.

A behavioral metric may be determined 818 as a distance between one or more determined motion parameters and one or more expected motion parameters. For example, the behavioral metric may include the difference between a detected start time and an expected start time of a real person's eye. For example, the behavioral metric may include the difference between a detected duration and an expected duration of pupil constriction of a real person's eye. In some implementations, a sequence of pupil diameters is compared to an expected sequence of pupil diameters by determining a distance between the two sequences (e.g., Euclidean distance, correlation coefficient, modified Hausdorff distance, Mahalanobis distance, Bregman divergence, Kulbeck-Leibler distance, and Jensen-Shannon divergence). In some implementations, a sequence of pupil constriction velocities of constriction motion is compared to an expected sequence of pupil constriction velocities by determining a distance between the two sequences of velocities. In some implementations, a sequence of pupil constriction accelerations of constriction motion is compared to an expected sequence of pupil constriction accelerations by determining a distance between the two sequences of accelerations.

For example, process 800 may be implemented by an authentication module or application (eg, authentication module 440 ) that controls a light sensor (eg, light sensor 420 ) and a lighting element.

FIG8B is a flow diagram of an example process 820 for determining a behavioral metric based on a shift in the gaze of the iris in response to an external stimulus. One or more external stimuli are applied 830 to a user viewed by a sensor (e.g., light sensor 420). For example, the external stimulus may include a cue instructing the user to direct their gaze during image capture (e.g., look right, look left, look up, look down, look forward). The cue may be visual, auditory, and/or tactile. In some implementations, the external stimulus may include an object that the user's eyes follow as it moves within a display.

A sequence of images is captured 832 by the sensor before and after the onset of the external stimulus. For example, the sequence of images may be captured at regularly spaced times (eg, 10, 30, or 60 Hz) within a time interval (eg, 2, 5, or 10 seconds) that includes the onset of the external stimulus.

In some implementations, the iris and the position or orientation of the iris that are landmarked in each of the captured images are determined in each captured image 834. The position may be determined 834 relative to a starting position of the iris measured in one or more images captured prior to the start of the external stimulus.

The resulting sequence of iris positions measured in response to the external stimulus may be analyzed to determine 836 one or more motion parameters of a gaze transition in response to the external stimulus. In some implementations, the motion parameters of the gaze transition may include a start time of the gaze transition motion relative to the start of the external stimulus. The start is the time delay between the start of the external stimulus and the start of the gaze transition motion. In some implementations, the motion parameters of the gaze transition may include a duration of the gaze transition motion. The duration is the length of time between the start of the gaze transition motion and the end of the gaze transition motion when the iris reaches a new steady-state position (e.g., after which the iris does not move for a minimum time interval). In some implementations, the motion parameters of the gaze transition may include a velocity of the gaze transition. For example, the velocity may be determined as the difference in iris position between two time points divided by the length of the time interval therebetween. In some implementations, the motion parameters of the gaze transition may include an acceleration of the gaze transition. For example, the acceleration may be determined as the difference in velocity between two time intervals.

The behavior metric may be determined as a distance 838 between one or more determined motion parameters and one or more expected motion parameters. For example, the behavior metric may include the difference between a detected start time and an expected start time of a real person's eye. For example, the behavior metric may include the difference between a detected duration and an expected duration of pupil constriction of a real person's eye. In some implementations, the sequence of iris positions is compared to the expected sequence of iris positions by determining a distance between the two sequences (e.g., Euclidean distance, correlation coefficient, modified Hausdorff distance, Mahalanobis distance, Bregman divergence, Kulbeck-Leibler distance, and Jensen-Shannon divergence). In some implementations, the sequence of transition velocities of a gaze transition motion is compared to the expected sequence of transition velocities by determining a distance between the two sequences of velocities. In some implementations, the sequence of gaze transition accelerations of a constriction motion is compared to the expected sequence of gaze transition accelerations by determining a distance between the two sequences of accelerations.

For example, process 820 may be implemented by an authentication module or application (eg, authentication module 440 ) that controls a light sensor (eg, light sensor 420 ) and a prompting device (eg, a display, a speaker, or a tactile feedback device).

FIG9 shows an example of a general-purpose computer device 900 and a general-purpose mobile computing device 950 that can be used with the techniques described herein. Computing device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are meant to be exemplary only and are not meant to limit the implementation of the present invention described and/or claimed in this document.

Computing device 900 includes a processor 902, memory 904, storage device 906, a high-speed interface 908 connected to memory 904 and a high-speed expansion port 910, and a low-speed interface 912 connected to a low-speed bus 914 and storage device 906. Each of components 902, 904, 906, 908, 910, and 912 is interconnected using various buses and can be mounted on a common motherboard or other suitable means. Processor 902 can process instructions for execution within computing device 900, including instructions stored in memory 904 or on storage device 906 to display graphical information of a GUI on an external input/output device (e.g., display 916 coupled to high-speed interface 908). In other embodiments, multiple processors and/or multiple buses can be used, along with multiple memories and memory types, as appropriate. Furthermore, multiple computing devices 900 can be connected, with each device providing a portion of the necessary operations (e.g., as a server bank, a cluster of blade servers, or a multi-processor system).

The memory 904 stores information within the computing device 900. In one embodiment, the memory 904 is one or more volatile memory units. In another embodiment, the memory 904 is one or more non-volatile memory units. The memory 904 may also be another form of computer-readable media, such as a magnetic or optical disk.

Storage device 906 can provide mass storage for computing device 900. In one embodiment, storage device 906 can be or contain a computer-readable medium, such as a floppy disk drive, a hard disk drive, an optical disk drive, or a magnetic tape drive, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. A computer program product can also contain instructions that, when executed, perform one or more methods, such as those described above. For example, the information carrier is a computer- or machine-readable medium, such as memory 904, storage device 906, or memory on processor 902.

High-speed controller 908 manages bandwidth-intensive operations of computing device 900, while low-speed controller 912 manages less bandwidth-intensive operations. This allocation of functionality is exemplary only. In one embodiment, high-speed controller 908 is coupled to memory 904, display 916 (e.g., via a graphics processor or accelerator), and to high-speed expansion ports 910, which can accept various expansion cards (not shown). In one embodiment, low-speed controller 912 is coupled to storage device 906 and low-speed expansion ports 914. The low-speed expansion ports, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input/output devices, such as a keyboard, pointing device, scanner, or network connection device, such as a switch or router, for example, via a network adapter.

As shown in the figure, computing device 900 can be implemented in a variety of different forms. For example, computing device 900 can be implemented as a standard server 920 or multiple times in a cluster of such servers. Computing device 900 can also be implemented as part of a rack-mounted server system 924. In addition, computing device 900 can be implemented in a personal computer such as laptop computer 922. Alternatively, components from computing device 900 can be combined with other components in a mobile device (not shown), such as device 950. Each of these devices can contain one or more of computing devices 900, 950, and the entire system can be composed of multiple computing devices 900, 950 communicating with each other.

Computing device 950 includes a processor 952, a memory 964, input/output devices such as a display 954, a communication interface 966, and a transceiver 968, among other components. Device 950 may also be provided with a storage device such as a micro drive or other device to provide additional storage. Each of components 950, 952, 964, 954, 966, and 968 are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other suitable manners.

Processor 952 can execute instructions within computing device 950, including instructions stored in memory 964. The processor can be implemented as a chipset including a single or multiple analog and digital processors. For example, the processor can provide coordination of other components of device 950, such as control of a user interface, applications executed by device 950, and wireless communications performed by device 950.

The processor 952 can communicate with the user through a control interface 958 and a display interface 956 coupled to a display 954. For example, the display 954 can be a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display or other appropriate display technology. The display interface 956 may include appropriate circuits for driving the display 954 to present graphics and other information to the user. The control interface 958 can receive commands from the user and convert them for submission to the processor 952. In addition, an external interface 962 in communication with the processor 952 can be provided to enable the device 950 to communicate with other devices in the vicinity. For example, the external interface 962 can provide wired communication in some embodiments, or wireless communication in other embodiments, and multiple interfaces can also be used.

Memory 964 stores information within computing device 950. Memory 964 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. Expansion memory 974 may also be provided and connected to device 950 via expansion interface 972, which may include, for example, a SIMM (Single In-line Memory Module) card interface. This expansion memory 974 may provide additional storage space for device 950 or may also store applications or other information for device 950. Specifically, expansion memory 974 may include instructions for performing or supplementing the processes described above and may also include security information. Thus, for example, expansion memory 974 may be provided as a security module for device 950 and may be programmed with instructions that permit secure use of device 950. Furthermore, secure applications and additional information may be provided via a SIMM card, such as placing identification information on the SIMM card in an uncontrolled manner.

For example, the memory may include flash memory and/or NVRAM memory, as discussed below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 964, the expansion memory 974, memory on the processor 952, or a propagated signal that can be received, for example, via the transceiver 968 or the external interface 962.

Device 950 can communicate wirelessly via communication interface 966, which may include digital signal processing circuitry, if necessary. Communication interface 966 can provide communication in various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among other protocols. This communication can occur, for example, via radio frequency transceiver 968. Additionally, short-range communication can occur, for example, using Bluetooth, WiFi, or other such transceivers (not shown). Additionally, a GPS (Global Positioning System) receiver module 970 can provide additional navigational and location-related wireless data to device 950, which can be used, as appropriate, by applications running on device 950.

Device 950 may also communicate auditorily using audio codec 960, which may receive verbal information from a user and convert it into usable digital information. Audio codec 960 may also produce audible sound for the user (e.g., via a speaker in a handset of device 950, for example). This sound may include sound from a voice phone call, recorded sound (e.g., voice messages, music files, etc.), and sound generated by applications operating on device 950.

As shown in the figure, computing device 950 can be implemented in a variety of different forms. For example, computing device 950 can be implemented as a cellular telephone 980. Computing device 950 can also be implemented as part of a smartphone 982, a personal digital assistant, or other similar mobile device.

Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementations in one or more computer programs executable and/or interpretable on a programmable system comprising at least one programmable processor coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device for special or general purposes.

These computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for a programmable processor and can be implemented in high-level procedural and/or object-oriented programming languages and/or in assembly/machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) that provides machine instructions and/or data to a programmable processor, including machine-readable media that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal that provides machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic input, voice, or tactile input.

The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form of digital data communication or any digital data communication medium (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

A computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Several embodiments have been described, however, it will be understood that various modifications can be made without departing from the spirit and scope of the invention.

Additionally, the logic flows depicted in the figures do not require the particular order or sequential sequence shown to achieve the desired results. Additionally, other steps can be provided, or steps can be eliminated from the described flows, and other components can be added to, or removed from, the described systems. Accordingly, other embodiments are within the scope of the following claims.

Claims (28)

1. A method for spoofing detection, comprising: A light sensor is used to capture two or more images of a subject, while the focal length of the light sensor is changed at different times and according to multiple configuration settings so that the multiple images have different corresponding focal lengths, and the two or more images of the subject contain a view of the subject's eyes; Detect the corresponding focus level of the landmarks in each of the plurality of images; The first metric is calculated by comparing the detected degree of focus with the corresponding focal length. The distance from the landmark to the sensor is calculated for each of two or more landmarks based on the corresponding focus of the landmarks in the plurality of images; A second metric is calculated based on the determination of whether the landmark lies in a single plane according to the corresponding distance; and The image may be rejected or accepted based at least on the first metric and the second metric. 2. The method of claim 1, wherein the specific focal length is the distance from the sensor to the focused point in the field of view of the sensor. 3. The method of claim 1, wherein detecting the corresponding focus of the image includes measuring the degree to which landmarks in the image are blurred. 4. The method of claim 3, wherein measuring the degree to which the landmark in the image is blurred includes identifying the amount of high-frequency components of the image near the landmark. 5. The method of claim 1, wherein the landmark is one of the iris, the corner of the eye, the nose, and the ear. 6. The method of claim 1, further comprising changing the white balance or exposure time of the light sensor at different times and according to a second plurality of configuration settings during the capture period, such that the plurality of images have different corresponding white balances or different corresponding exposure times. 7. The method of claim 6, wherein rejecting or accepting the image based at least on the first metric comprises: detecting a corresponding white balance or a corresponding exposure time for each of the plurality of images; calculating a second metric based on whether the detected corresponding white balance or corresponding exposure time corresponds to a determination of the second plurality of configuration settings; and rejecting or accepting the image based on the first metric and the second metric. 8. The method of claim 1, wherein rejecting or accepting the image based at least on the first metric comprises: providing the first metric and one or more second metrics to a trained function approximator; training the trained function approximator using a trained image and the first metric and the second metric applied to the trained image; and rejecting or accepting the image based on the output of the function approximator. 9. The method of claim 8, wherein the particular second metric is based on one of: a measurement of the deviation between a detected motion of the eye and a desired motion of the eye when the eye appears in the plurality of images; and a change in glare or specular reflection patterns on the plane of the eye when the eye appears in the plurality of images. 10. The method of claim 9, wherein the detected movement of the eye is a measurement of saccadic movements. 11. The method of claim 9, wherein the detected motion of the eye is a measurement of the initiation, duration, or acceleration of a gaze shift. 12. The method of claim 1, further comprising changing the illumination of the subject at different times and according to a second plurality of configuration settings during the capture such that the plurality of images have different corresponding illuminations. 13. The method of claim 12, wherein rejecting or accepting the image based at least on the first metric comprises: detecting a change in corresponding illumination for each of the plurality of images; calculating a second metric based on whether the detected change in illumination corresponds to a determination of a second plurality of configuration settings; and rejecting or accepting the image based on the first metric and the second metric. 14. A system for spoofing detection, comprising a data processing means programmed to perform operations including: A light sensor is used to capture two or more images of a subject, while the focal length of the light sensor is changed at different times and according to multiple configuration settings so that the multiple images have different corresponding focal lengths, and the two or more images of the subject contain a view of the subject's eyes; Detect the corresponding focus level of the landmarks in each of the plurality of images; The first metric is calculated by comparing the detected degree of focus with the corresponding focal length. The distance from the landmark to the sensor is calculated for each of two or more landmarks based on the corresponding focus of the landmarks in the plurality of images; A second metric is calculated based on the determination of whether the landmark lies in a single plane according to the corresponding distance; and The image may be rejected or accepted based at least on the first metric and the second metric. 15. The system of claim 14, wherein the specific focal length is the distance from the sensor to the focused point in the field of view of the sensor. 16. The system of claim 14, wherein detecting the corresponding focus of the image includes measuring the degree to which landmarks in the image are blurred. 17. The system of claim 16, wherein measuring the degree to which the landmark in the image is blurred includes identifying the amount of high-frequency components of the image near the landmark. 18. The system of claim 14, wherein the landmark is one of the iris, the corner of the eye, the nose, and the ear. 19. The system of claim 14, further comprising changing the white balance or exposure time of the light sensor at different times and according to a second plurality of configuration settings during the capture period, such that the plurality of images have different corresponding white balances or different corresponding exposure times. 20. The system of claim 19, wherein rejecting or accepting the image based at least on the first metric comprises: detecting a corresponding white balance or corresponding exposure time for each of the plurality of images; calculating a second metric based on whether the detected corresponding white balance or corresponding exposure time corresponds to a second plurality of configuration settings; and rejecting or accepting the image based on the first metric and the second metric. 21. The system of claim 14, wherein rejecting or accepting the image based at least on the first metric comprises: providing the first metric and one or more second metrics to a trained function approximator; training the trained function approximator using a trained image and the first metric and the second metric applied to the trained image; and rejecting or accepting the image based on the output of the function approximator. 22. The system of claim 21, wherein a particular second metric is based on one of: a measurement of the deviation between a detected motion of the eye and a desired motion of the eye when the eye appears in the plurality of images; and a change in glare or specular reflection patterns on the plane of the eye when the eye appears in the plurality of images. 23. The system of claim 22, wherein the detected movement of the eye is a measurement of saccadic movements. 24. The system of claim 22, wherein the detected motion of the eye is a measurement of the initiation, duration, or acceleration of a gaze shift. 25. The system of claim 14, further comprising changing the illumination of the subject at different times and according to a second plurality of configuration settings during the capture such that the plurality of images have different corresponding illuminations. 26. The system of claim 25, wherein rejecting or accepting the image based at least on the first metric comprises: detecting a change in corresponding illumination for each of the plurality of images; calculating a second metric based on whether the detected change in illumination corresponds to a determination of a second plurality of configuration settings; and rejecting or accepting the image based on the first metric and the second metric. 27. A non-volatile computer-readable storage medium storing a computer program that, when executed by a data processing apparatus, causes the data processing apparatus to perform operations, the operations including: A light sensor is used to capture two or more images of a subject, while the focal length of the light sensor is changed at different times and according to multiple configuration settings so that the multiple images have different corresponding focal lengths, and the two or more images of the subject contain a view of the subject's eyes; Detect the corresponding focus level of the landmarks in each of the plurality of images; The first metric is calculated by comparing the detected degree of focus with the corresponding focal length. The distance from the landmark to the sensor is calculated for each of two or more landmarks based on the corresponding focus of the landmarks in the plurality of images; A second metric is calculated based on the determination of whether the landmark lies in a single plane according to the corresponding distance; and The image may be rejected or accepted based at least on the first metric and the second metric. 28. The computer-readable storage medium of claim 27, wherein the specific focal length is the distance from the sensor to the focused point in the field of view of the sensor.