CN112101186A - Apparatus and method for vehicle driver identification and application thereof - Google Patents

Abstract

An apparatus for vehicle driver identification includes: a NIR LED illuminator configured to emit NIR light in a vehicle; a NIR light sensing unit configured to capture reflected NIR light; and an image control and processing unit configured To coordinate the NIR LED illuminator and the NIR light sensing unit, and to analyze the reflected NIR light to generate an image; a face detector, configured to determine the presence of a human face in the image, and to identify face regions; a facial feature extractor , is configured to analyze the face region to extract feature vectors representing the face region; the face feature dictionary is configured to store the existing feature vectors; the face retrieval system is configured to generate recognition results indicating that the feature vectors are related to any whether the similarity between the existing feature vectors is greater than a first threshold; and a user interface configured to display the recognition result.

Description

Apparatus and method for vehicle driver identification and application thereof

technical field

The present invention relates generally to artificial intelligence, and more particularly to an apparatus and method for in-cabin driver identification and applications thereof.

Background technique

The background description provided herein is for the purpose of generally presenting the context of the invention. Subject matter discussed in the Background of the Invention section should not be admitted to be prior art by virtue of its mention in the Background of the Invention section. Similarly, problems raised in the Background of the Invention or problems associated with the subject matter of the Background of the Invention should not be regarded as having been previously recognized in the prior art. The subject matter in the Background of the Invention section is merely indicative of various approaches, which may themselves be inventions.

Person recognition, which distinguishes individuals among a group of people, is a critical function in many applications. This identification requires the collection of biometrics from a set of target populations. A good collection of such biological information, and more specifically, a good statistical distribution of such collected data, requires that each data point have a statistically significant separation in its feature space.

In-cabin driver identification (driver ID) has recently attracted increasing attention in the automotive industry due to its potential to enable numerous smart or safety features for vehicles. For example, in-cabin driver identification can be used as a substitute for a vehicle key to provide a smooth keyless entry experience, or provide other trigger alerts to prevent any unauthorized entry. Additionally, in-cab driver recognition equips the vehicle with various custom settings based on driver preferences such as seat position, interior temperature, mirror angle, and more. More recently, in-cabin driver identification has also been used as an input to many in-cabin entertainment systems or interfaces. With knowledge of the driver ID, the vehicle's embedded system can provide customized entertainment features such as playing the driver's favorite album, personalizing route navigation, and providing the driver with a favorite news channel.

In the industry, there are two main types of person identification: invasive and contactless. Invasive methods often require direct measurement of biometrics, such as DNA fingerprints that are clearly associated with personal identification. Intrusive methods are superior in the accuracy of identification. However, these methods rely heavily on a well-controlled laboratory environment, expensive equipment, and long testing times. Furthermore, the nature of invasive methods that require direct contact between the system and the individual's body can lead to unpleasant and annoying experiences. Therefore, the time requirements and cost of the invasive approach make it impractical, if not impossible, if applied to most real-time applications, including settings in the vehicle cabin.

Contactless methods attempt to identify individuals based on indirect measurements of biometrics, such as footprints, handwriting, or facial recognition. Most contactless person recognition technologies do not require expensive equipment and highly-skilled professionals, greatly reducing costs.

Among all these contactless technologies, camera-based person recognition has attracted great interest from both academia and industry. Camera-based person recognition relies solely on a camera module and subsequent computational modules containing specific algorithms to distinguish individuals. These algorithms aim to extract salient features from input images and then make decisions based on similarities or dissimilarities in a previously learned dictionary of features. These features can be extracted from various features such as body pose, height, weight, skin and movement patterns, etc. The most robust and accurate camera-based recognition method widely adopted in the industry, especially in automotive cabin applications, is facial recognition technology.

In-cabin driver facial recognition is an emerging technology that provides a contactless and accurate solution to driver identification problems. In-cab driver facial recognition uses a camera to capture facial biometric information and compares it to a library of stored facial information to find the best match. As said, the system should include two modules: face feature library and face recognition module. The facial feature database can register new facial information or delete existing facial information. The face recognition module captures images through a camera, extracts features using a designed algorithm, and finds the best match in a library of pre-built facial features. The first process is called face registration, while the second is called face retrieval.

However, in-cabin facial recognition is fundamentally different from normal facial recognition in many ways. First, in-cabin camera-based facial recognition must be robust in all lighting conditions, including in pitch darkness. Unlike most face recognition setups in well-lit outdoor scenes or assuming well-controlled indoor lighting, in-cabin face recognition presents additional challenges in imaging and image quality. Even for under-illuminated outdoor identification applications, glare compensation can easily be added. However, in-cabin lighting is difficult because in-cabin devices are very sensitive to power consumption, which can cause severe heating due to their small size compared to common face recognition devices. In addition, the compensated light source for better imaging is visible to the human eye, creating an unpleasant experience for the vehicle driver. Strong light that is visible to the human eye and targeted at the driver's face is not only harmful to the human eye, but also distracting and can lead to traffic accidents.

In addition to the imaging challenges, in-cabin facial recognition is also unique in the way it operates using constrained computing resources. In-cab facial recognition, including face registration and retrieval, requires minimal computational resource consumption because all algorithms are run in embedded systems, the electronic control units (ECUs), which are equipped with only limited Calculate ability. Compared to general face recognition methods contained in powerful local servers or even online clouds with theoretically unlimited computing power, ECUs have few advantages in terms of scalability and real-time performance. In order to produce a practical in-cabin facial recognition system, the algorithm design for facial recognition should be very careful to reduce the amount of code computation.

Third, in-cabin facial recognition plays a very specific role in the vehicle, as it is an input to various modules and also needs to be well presented in the user interface. As mentioned earlier, the results of facial recognition should be connected to various functional modules and transmitted to vehicle displays such as embedded displays, head-up displays or dashboards. The complex connectivity of facial recognition systems to other modules further adds to the difficulty of designing such systems.

In-cab facial recognition is different compared to other recognition scenarios where the camera is located outside the car. For example, an off-board surveillance recognition system can capture only one or at most a few images for ID registration, while the inside of the cabin should achieve greater accuracy by capturing multiple images of the target's face from multiple angles. By capturing images from more angles, regardless of the driver's head pose, the system can gain a more complete picture of the driver's face, making recognition more robust.

Accordingly, there is a need in the art to address the above-mentioned deficiencies and deficiencies, but heretofore unaddressed.

SUMMARY OF THE INVENTION

The present invention relates to an apparatus and method for visualizing the underlying behavior of objects around a vehicle.

In one aspect of the invention, an apparatus for vehicle driver identification includes: a near infrared (NIR) light emitting diode (LED) illuminator configured to emit NIR light in a vehicle; a near infrared (NIR) light sensor a detection unit configured to capture reflected NIR light; an image control and processing unit configured to coordinate the NIR LED illuminator and the NIR light sensing unit and analyze the reflected NIR light captured by the NIR light sensing unit to generate an image; a face detector configured to determine the presence of a human face in the image and to identify a face region of the human face; a facial feature extractor configured to analyze the face region to extract feature vectors representing the face region; a face feature dictionary configured to store existing feature vectors; a face retrieval system configured to generate a recognition result indicating whether the similarity between the feature vector and any existing feature vector is greater than a first threshold; and a user interface, configured to display the recognition results.

In one embodiment, the NIR light sensing unit is a focal plane array (FPA) NIR light sensing unit.

In one embodiment, the NIR light sensing unit is covered with a filter having a passband between 825nm and 875nm. In another embodiment, the NIR light sensing unit is covered with a filter having a passband between 915nm and 965nm.

In one embodiment, the image control and processing unit is configured to control the duty cycle of the NIR LED illuminator, the analog gain of the NIR light sensing unit, the digital gain of the NIR light sensing unit, the exposure of the NIR light sensing unit One or more of time and the frame rate of the NIR light sensing unit to coordinate the NIR LED illuminator and the NIR light sensing unit.

In one embodiment, the image control and processing unit is configured to coordinate the NIR LED illuminator and the NIR light sensing unit to generate images with optimal imaging quality.

In one embodiment, the face detector is configured to employ a deep neural network (DNN) to determine the presence of a human face in the image and to identify the facial region of the human face. In one embodiment, the deep neural network is a multi-task convolutional neural network (MTCNN). In another embodiment, the deep neural network is a Fast Region-based Convolutional Neural Network (Fast R-CNN).

In one embodiment, the apparatus further includes a face alignment unit. The face alignment unit is configured to calibrate the face region to a calibrated face region associated with the driver's upright posture, wherein the facial feature extractor is configured to analyze the calibrated face region to extract a representative of the calibration The feature vector of the face region.

In one embodiment, the facial feature extractor is configured to employ one or more of backbone networks, local feature descriptors, clustering techniques, and dimensionality reduction techniques.

In one embodiment, the similarity is cosine similarity.

In another aspect of the present invention, a method for vehicle driver identification includes: emitting NIR light in a vehicle through a near-infrared (NIR) light emitting diode (LED) illuminator; and through a near-infrared (NIR) light sensing unit capturing reflected NIR light; coordinating the NIR LED illuminator and NIR light sensing unit by the image control and processing unit; analyzing the reflected NIR light captured by the NIR light sensing unit by the image control and processing unit to generate an image; Determine the presence of a face in the image; identify the face region of the face; analyze the face region to extract a feature vector representing the face region; determine the similarity between the feature vector and any existing feature vector in the face feature dictionary Whether it is greater than the first threshold value; when the similarity between the feature vector and the first existing feature vector in the face feature dictionary is greater than the first threshold value, generate the first existing feature vector indicating the identification associated with the first existing feature vector. a recognition result; and displaying a first recognition result; when the similarity between the feature vector and any existing feature vector in the face feature dictionary is not greater than a first threshold, generating an indication that the feature vector does not exist storing the second recognition result in the facial feature dictionary; displaying the second recognition result; and storing the facial feature in the facial feature dictionary.

In one embodiment, the NIR light sensing unit is a focal plane array (FPA) NIR light sensing unit.

In one embodiment, the NIR light sensing unit is covered with a filter having a passband between 825nm and 875nm. In another embodiment, the NIR light sensing unit is covered with a filter having a passband between 915nm and 965nm.

In one embodiment, the image control and processing unit controls the duty cycle of the NIR LED illuminator, the analog gain of the NIR light sensing unit, the digital gain of the NIR light sensing unit, the exposure time of the NIR light sensing unit, and the NIR One or more of the frame rates of the light sensing unit coordinate the NIR LED illuminator and the NIR light sensing unit.

In one embodiment, the image control and processing unit coordinates the NIR LED illuminator and the NIR light sensing unit to generate images with optimal imaging quality.

In one embodiment, the presence of a human face in the image is determined by employing a deep neural network (DNN) and the facial region of the human face is identified. In one embodiment, the deep neural network is a multi-task convolutional neural network (MTCNN). In one embodiment, the deep neural network is a Fast Region-based Convolutional Neural Network (Fast R-CNN).

In one embodiment, the method further comprises: calibrating the face region to a calibrated face region associated with the driver's upright posture, wherein the calibrated face region is analyzed to extract a face region representing the calibration eigenvectors of .

In one embodiment, the facial regions are analyzed by employing one or more of backbone networks, local feature descriptors, clustering techniques, and dimensionality reduction techniques to extract feature vectors representing the facial regions.

In one embodiment, the similarity is cosine similarity.

In another aspect, the invention relates to a non-transitory tangible computer-readable medium storing instructions that, when executed by one or more processors, cause a method of vehicle driver identification to be performed. The method includes: emitting NIR light in a vehicle via a near-infrared (NIR) light emitting diode (LED) illuminator; capturing the reflected NIR light via a near-infrared (NIR) light sensing unit; and coordinating the NIR LED lighting via an image control and processing unit and NIR light sensing unit; analyze reflected NIR light captured by the NIR light sensing unit by the image control and processing unit to generate an image; determine the presence of a human face in the image; identify the face area of the human face; analyze face region to extract feature vectors representing the face region; determine whether the similarity between the feature vector and any existing feature vector in the face feature dictionary is greater than a first threshold; when the feature vector and the face feature dictionary When the degree of similarity between the first existing feature vectors in is greater than a first threshold, generating a first recognition result indicating an identification associated with the first existing feature vector; and displaying the first recognition result; when the When the similarity between the feature vector and any existing feature vector in the face feature dictionary is not greater than the first threshold, generate a second recognition result indicating that the feature vector does not exist in the face feature dictionary; display the first recognition result. two recognition results; and storing the facial features in the facial feature dictionary.

These and other aspects of the present invention will become apparent from the following description of the preferred embodiments, taken in conjunction with the accompanying drawings, although changes and modifications therein may be effected without departing from the spirit and scope of the novel concepts of this disclosure.

Description of drawings

The drawings illustrate one or more embodiments of the invention, and together with the written description serve to explain the principles of the invention. The same reference numbers may be used throughout the drawings to refer to the same or like elements of the embodiments.

FIG. 1 schematically shows the overall architecture of a system for vehicle driver identification according to an embodiment of the present invention.

FIG. 2 schematically shows a flow chart of face registration using a system for vehicle driver identification according to an embodiment of the present invention.

FIG. 3 schematically shows a flow chart of face retrieval using a system for vehicle driver identification according to an embodiment of the present invention.

FIG. 4 schematically shows a flowchart of a method for vehicle driver identification according to an embodiment of the present invention.

FIG. 5 schematically shows a flowchart of a method for vehicle driver identification according to an embodiment of the present invention.

Detailed ways

The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the invention are shown. However, the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. The same reference numbers refer to the same elements throughout.

Terms used in this specification generally have their ordinary meanings in the art, in the context of the present invention and in the specific context in which each term is used. Certain terms used to describe the invention are discussed below or elsewhere in the specification to provide the practitioner with additional guidance regarding the description of the invention. Certain terms may be highlighted for convenience, such as using italics and/or quotation marks. The use of highlighting does not affect the scope and meaning of the term; the scope and meaning of the term in the same context is the same whether or not it is highlighted. It should be understood that the same thing can be described in more than one way. Accordingly, alternative language and synonyms may be used for any one or more of the terms discussed herein without any special meaning as to whether the term is set forth or discussed herein. Synonyms for certain terms are provided. The recitation of one or more synonyms does not exclude the use of other synonyms. Examples used anywhere in this specification, including examples of any terms discussed herein, are exemplary only and in no way limit the scope and meaning of the invention or any exemplary term. Likewise, the present invention is not limited to the various embodiments presented in this specification.

It will be understood that as used herein in the specification and throughout the claims that follow, the meanings of "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Also, it will be understood that when an element is referred to as being "on" another element, it can be directly on the other element or intervening elements may be present therebetween. In contrast, when an element is referred to as being "directly on" another element, there are no intervening elements present. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

It will be understood that, although the terms first, second, third, etc. may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be subject to these terms. These terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, Second layer or second part.

Furthermore, relative terms, such as "lower" or "bottom" and "upper" or "top," may be used herein to describe one element's relationship to another element as illustrated in the figures. It will be understood that relative terms are intended to encompass different orientations of the device in addition to the orientation depicted in the figures. For example, if the device in one of the figures is turned over, elements described as being on the "lower" side of the other elements would then be oriented on the "upper" side of the other elements. Thus, the exemplary term "lower" may include both "lower" and "upper" orientations, depending on the particular orientation of the drawings. Similarly, if the device in one of the figures is turned over, elements described as "below" or "beneath" other elements would then be oriented "above" the other elements. Thus, the exemplary terms "below" or "under" can encompass both an orientation of above and below.

It will be further understood that the terms "including", or "comprising", or "having", or "carrying", or "comprising", or "involving" etc. are open ended, meaning including but not limited to. When used in the present invention, they specify the presence of said features, regions, integers, steps, operations, elements and/or components, but do not preclude the presence or addition of one or more other features, regions, integers, steps, operations , elements, components and/or groups thereof.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will also be understood that terms such as those defined in commonly used dictionaries should be construed to have meanings consistent with their meanings in the relevant art and in the context of the invention, and not in an idealized or overly formal sense be construed unless expressly defined herein.

As used herein, at least one of the phrases A, B, and C should be construed as a logical (A or B or C) using a non-exclusive logical "or". As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

As used herein, the term module may include an application specific integrated circuit (ASIC), an electronic circuit,; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated or group) that executes code; Other suitable hardware components may provide the above-described functionality; or a combination of some or all of the above, such as in a system-on-a-chip, or may refer to portions thereof. The term module may include memory (shared, dedicated, or group) that stores code executed by a processor.

As used herein, the term chip or computer chip generally refers to a hardware electronic component, and may refer to or include a small electronic circuit unit, also known as an integrated circuit (IC), or a combination of electronic circuits or ICs.

As used herein, the term microcontroller unit or its acronym MCU generally refers to a small computer on a single IC chip that can execute programs for controlling other devices or machines. A microcontroller unit contains one or more CPUs (processor cores) as well as memory and programmable input/output (I/O) peripherals, and is typically designed for embedded applications.

As used herein, the term interface generally refers to a communication tool or device at an interaction point between components for performing wired or wireless data communication between components. In general, the interface can be available in both hardware and software, and can be a unidirectional or bidirectional interface. Examples of physical hardware interfaces may include electrical connectors, buses, ports, cables, terminals, and other I/O devices or components. The components in communication with the interface may be, for example, various components of a computer system or peripheral devices.

As used herein, the term code may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, and/or objects. A single (shared) processor may be used to execute some or all code from multiple modules. Additionally, some or all code from multiple modules may be stored by a single (shared) memory. A group of processors may also be used to execute some or all of the code from a single module. Additionally, a set of memories may be used to store some or all of the code from a single module.

The apparatus and methods will be described in the following detailed specification and illustrated in the accompanying drawings by means of various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as "elements"). These elements may be implemented using electronic hardware, computer software, or any combination thereof. Whether these elements are implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. For example, an element, or any portion of an element, or any combination of elements may be implemented as a "processing system" that includes one or more processors. Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, on-chip systems (SoCs), baseband processors, field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable Describes the various functions of the hardware. One or more processors in the processing system may execute software. Software shall be construed broadly to mean instructions, sets of instructions, codes, code segments, program code, programs, subroutines, software components, applications, software applications, software packages, routines, subroutines, objects, executables , threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

Accordingly, in one or more example embodiments, the described functions may be implemented in hardware, software, or any combination thereof. If implemented in software, the functions may be stored on or encoded on a computer-readable medium as one or more instructions or code. Computer readable media includes computer storage media. A storage medium can be any available medium that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM), read only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk memory, magnetic disk memory, other magnetic storage devices , combinations of the above types of computer-readable media, or any other medium that can be used to store computer-executable code in the form of computer-accessible instructions or data structures.

The following description is merely illustrative in nature and is in no way intended to limit the invention, its application or uses. The broad teachings of the present invention can be implemented in a variety of forms. Thus, although this disclosure includes specific examples, the true scope of this disclosure should not be so limited, as other modifications will become apparent after a study of the drawings, specification, and appended claims. For the sake of clarity, the same reference numbers will be used throughout the drawings to identify similar elements. It should be understood that one or more steps within a method may be executed in different order (or concurrently) without altering the principles of the present disclosure.

FIG. 1 schematically shows the overall architecture of a system for vehicle driver identification according to an embodiment of the present invention. The system 100 includes a near infrared (NIR) light emitting diode (LED) illuminator 102, a near infrared (NIR) light sensing unit 104, an image control and processing unit 106, a face detector 108, a facial feature extractor 110, a face Features dictionary 112 , face retrieval system 114 , user interface 116 , and one or more application interfaces 118 .

The NIR LED illuminator 102 emits light (ie, electromagnetic radiation) in the NIR spectrum. The NIR LED illuminator 102 is configured to synchronize with the NIR light sensing unit 104 and provide sufficient brightness so that the NIR light sensing unit 104 can capture details of the driver's face even at night. On the other hand, the NIR illuminator 102 is invisible to the driver because the NIR spectrum does not overlap with the human visible spectrum, thereby eliminating the potential for distraction and eye hazard. The human visual system only responds to the spectrum around 400nm to 700nm. In one embodiment, the NIR LED illuminator 102 emits light having a spectrum of 825 nm to 875 nm, aligned with the response spectrum of the NIR light sensing unit 104 . In another embodiment, the NIR LED illuminator 102 emits light having a spectrum of 915 nm to 965 nm, aligned with the response spectrum of the NIR light sensing unit 104 . In one embodiment, the NIR LED illuminator 102 is powered by pulse width modulation (PWM) with a configurable duty cycle. A larger duty cycle provides more benefits to the NIR light sensing unit 104, but at the expense of a higher risk of overheating. Since the limited module size leads to poor thermal conductivity, a balance needs to be struck between lighting quality and overheating. Likewise, the duty cycle should be synchronized with the NIR light sensing unit 104 so that its radiation is fully utilized by the NIR light sensing unit 104, resulting in the best sensing quality. This synchronization is achieved by the image control and processing unit 106, which will be described in detail below.

The light sensing unit 104 captures light originating from the NIR LED illuminator 102 or other light source, such as the sun, moon, or other illuminator that has a rich NIR component in its spectrum. The NIR light sensing unit 104 converts the illumination intensity into an electrical signal, more specifically, into a digital readout of the NIR light sensing unit 104 . NIR light sensing unit 104 captures light from NIR LED illuminator 102 or other light source. In one embodiment, the NIR light sensing unit 104 is designed to respond only to certain spectral bands. For example, the NIR light sensing unit 104 is designed to respond only to the spectral band of 825 nm to 875 nm. In another example, the NIR light sensing unit 104 is designed to respond only to the spectral band of 915 nm to 965 nm. This selective sensitivity to the spectrum can be achieved by applying a bandpass filter on top of the NIR light sensing unit 104 . It should be noted that the NIR light sensing unit does not rely on the NIR LED illuminator 102 alone. The spectrum of sunlight is much broader than that of the human visual system and is also strong in the NIR band. The NIR light sensing unit 104 should work well under good sunlight conditions, or even better. Nonetheless, the NIR LED illuminator 102 is still essential for low light or dark conditions when no natural NIR light source is present.

In one embodiment, the light sensing unit 104 is a camera covered with a NIR bandpass filter. In one embodiment, the NIR light sensing unit 104 is a focal plane array (FPA) NIR light sensing unit 104 . The FPA NIR light sensing unit 104 is an image sensing device consisting of an array (typically rectangular) of light sensing pixels at the focal plane of the lens. The FPA NIR light sensing unit 104 operates by detecting photons of specific wavelengths and then generating a charge, voltage or resistance related to the number of photons detected by each pixel. This charge, voltage, or resistance is then measured, digitized, and used to construct an image of the object, scene, or phenomenon that emits photons. The FPA NIR light sensing unit 104 can provide various properties to control its sensing behavior, including exposure time, digital gain, analog gain, gamma, and frame rate. These properties are critical for the image control and processing unit 106 to obtain optimal image quality, which is the basis for subsequent face registration or face retrieval.

The image control and processing unit 106 adjusts the behavior of both the NIR LED illuminator 102 and the NIR light sensing unit 104 and generates images. The image control and processing unit 106 is configured to adjust the on/off period of the NIR LED illuminator 102 and the light sensing shutter of the NIR light sensing unit 104 together to fully utilize the energy from the NIR LED illuminator 102 for better image quality. Also, the image control and processing unit 106 may analyze the statistics of the digital readout of the NIR light sensing unit 104 and send commands to both the NIR LED illuminator 102 and the NIR light sensing unit 104 . Some examples of properties of the NIR light sensing unit 104 that the image control and processing unit 106 may control include exposure time, analog gain, digital gain, gamma, and frame rate. Based on the commands sent by the image control and processing unit 106, the NIR LED illuminator 102 and the NIR light sensing unit can adjust their properties accordingly. The image control and processing unit 106 then generates images with the best imaging quality to maximize the likelihood of successful face registration and face retrieval.

In one embodiment, the image control and processing unit 106 is included in an electronic control unit (ECU) of the vehicle. Image control and processing unit 106 sends commands to NIR LED illuminator 102 and NIR light sensing unit 104 to coordinate them. More specifically, the image control and processing unit 106 coordinates the alignment of the duty cycle of the NIR LED illuminator 102 and the exposure time of the NIR light sensing unit 104 under low light conditions. In one embodiment, the image control and processing unit 106 may analyze the statistics of the digital readouts of the NIR light sensing unit 104 to assess lighting conditions. If the image control and processing unit 106 determines that the ambient lighting is strong enough, it will turn off the NIR LED illuminator 102, thereby significantly reducing power consumption and heat generation.

FIG. 2 schematically shows a flow chart of face registration using a system for vehicle driver identification according to an embodiment of the present invention. The face detector 108 is configured to determine whether a face region is present in the image 202 generated by the image control and processing unit 106 . If face detector 108 determines that a person's face region 204 is present in image 202 generated by image control and processing unit 106, face detector 108 locates the location of face region 204 using a bounding box. The face detector 108 may be implemented by a deep neural network (DNN) such as a multi-task convolutional neural network (MTCNN), a fast region-based convolutional neural network (Fast R-CNN), or any other deep neural network. A deep neural network is an artificial neural network with multiple layers between the input and output layers. Whether the relationship is linear or non-linear, DNN will find the correct mathematical operation to transform the input into the output. Each mathematical operation is treated as one layer, while complex deep neural networks have many layers. The network goes through the layers, calculating the probability of each output. Convolutional Neural Networks (CNN) are a class of deep neural networks most commonly used to analyze visual images.

In one embodiment, face detector 108 may be a software or hardware implementation in an ECU that may determine whether a person's face region 204 is present in image 202 generated by image control and processing unit 106 . As mentioned above, the image 202 has been optimized by the image control and processing unit 106 . Face detector 108 may crop face region 204 in image 202 . It should be noted that, for in-cabin driver registration, images of the driver's face from different perspectives can enhance the robustness of subsequent face retrieval, making subsequent face retrieval relatively unaffected by perspective changes. Capturing images of the driver's face from different perspectives can be achieved by displaying driver visual guidance on the embedded display, so that the driver can follow the visual guidance and move his head during the registration process until the driver's from different perspectives. A face image is captured.

In addition to bounding boxes, the face detector 108 may also output a series of key face points called landmarks 208 . These landmarks 208 are points such as the tip of the nose, the center of the eyes, and the corners of the mouth on a human face. These landmarks 208 are critical for subsequent feature generation. Since the distinction between faces is greatest in these salient regions, regions of the face near these landmarks 208 are given higher weights during subsequent face registration and face retrieval. More importantly, these landmarks 208 can be used for perspective correction of the driver's face. During face registration or face retrieval, it is likely that the driver's face is not perfectly perpendicular to the imaging plane. Therefore, perspective correction must be performed on the person's face region 204 before further processing.

Based on the landmarks 208, the face alignment unit 206 calibrates the posture of the driver's face to an upright posture. In one embodiment, the face alignment unit 206 aligns the pose of the driver's face using three-axis angles including yaw, pitch and roll to provide more information for subsequent face recognition. It should be noted that the face alignment unit 206 may be a separate unit from the face detector 108, but it may also be a component integrated in the face detector 108. It should also be noted that alignment by face alignment unit 206 based on landmarks 208 increases the accuracy of subsequent face recognition compared to other systems that only generate bounding boxes.

The face feature extractor 110 analyzes the aligned face regions 204 and extracts feature vectors 212 representing the face regions 204 . This process may also be referred to as face encoding. The facial feature extractor 110 may pass various neural networks, such as backbone networks (eg, MobileNets) or local feature descriptors (eg, SIFT) plus clustering (eg, K-means clustering) and its Subsequent dimensionality reduction techniques such as Bag of Words (BoW) are implemented. A backbone network is a part of a computer network that interconnects parts of the network and provides a path for information exchange between different LANs or subnets. Scale-invariant feature transform (SIFT) is a feature detection algorithm in computer vision for detecting and describing local features in images. K-means clustering is a vector quantization method originally derived from signal processing and popular in cluster analysis in data mining. K-means clustering aims to divide n observations into k clusters, where each observation belongs to the cluster with the nearest mean, serving as the prototype of the clustering. The Bag of Words (BoW) model is a simplified representation used in natural language processing and information retrieval. In this model, text (such as sentences or documents) is represented as a bag (multiset) of words in the text, ignoring grammar and even word order, but maintaining multiplicity. Bag of words (BoW) models have also been used in computer vision. It should be noted that for face registration, the system may need to capture multiple images of the same driver to better match.

In one embodiment, the facial feature extractor 110 is essentially an encoder of information for identifying each driver's facial region 204 . In one embodiment, the facial feature extractor 110 is included in the ECU. The facial feature extractor 110 outputs a high-dimensional feature vector 212 representing the detected driver's facial region 204 . This feature vector 212 should be a fair representation (mathematically speaking, a good discriminative distribution) of the detected driver's face region 204 . This means that, no matter how captured, the feature vectors 212 of the same person should be very close to each other in the feature space, and at the same time, the feature vectors 212 of two different people should be well separated, even if captured under the same conditions.

The facial feature dictionary 112 stores feature vectors for all registered drivers. Facial feature dictionary 112 can add new feature vectors (eg, feature vector 212). The facial feature dictionary 112 can also delete existing feature vectors. The facial feature dictionary 112 may add or delete feature vectors in response to a user's command (eg, a driver's command). Optionally, facial feature dictionary 112 may automatically add or delete feature vectors under certain circumstances (eg, add feature vector 212 after it is determined that it is a new feature vector). The facial feature dictionary 112 includes feature vectors 212 for different drivers, and a lookup table that associates each feature vector 212 with its corresponding driver.

Operating in the NIR spectrum reduces the negative impact of complex lighting on recognition performance, but presents more challenges when designing algorithms. First, NIR-based recognition ignores hue information, which helps build stronger discrimination. Second, a large-scale facial dataset needs to be collected to train the facial feature extractor 110 based on a deep convolutional neural network (CNN). Considering that almost all public face images are in the visible (VIS) spectrum and not in the NIR spectrum, deep CNNs may not generalize due to significant domain differences. Therefore, public VIS datasets (eg MS-Celeb-1M, VGGFace2) and private NIR images are used, and NIR spectra are oversampled to remove imbalances simultaneously. MS-Celeb-1M is a public VIS dataset for recognizing facial images and associating these images to corresponding entity keys in a knowledge base. VGGFace2 is another public VIS dataset that contains 3.31 million images of 91.31 million objects.

Constrained by computational resources and the size of CNN parameters, control over the trade-off between complexity (vs. time and space) and model capacity is critical. Intensive arithmetic operations consume a lot of power and cause overheating, while too many parameters can consume unbearable load times. Therefore, some light networks (eg Mobilenet, Shufflenet) with depthwise separable convolution (DW Conv) and pointwise convolution (PW Conv) can be employed. In addition, quantization and distillation techniques are employed to reduce the computational effort.

Compared to outdoor environments where true positive rate is critical, in-cabin scenarios focus more on safety and convenience, i.e. lower false negative rate and higher true rate. Therefore, a test dataset containing various difficult cases (e.g. glasses, lighting, head pose) is created, and extensive experiments are performed using said test dataset to find better solutions and find the best model. Models can only be deployed if they pass rigorous testing.

In order to engage the user (eg the driver) and improve the accuracy of the recognition, registration instructions can be displayed on the screen. More specifically, the user should be in several different poses so that feature vectors corresponding to different perspectives can be extracted and stored in the face feature dictionary 112 . Theoretically, the information loss caused by the projection from the three-dimensional (3D) real world to the two-dimensional (2D) face image can be partially eliminated in this way, thereby improving the face recognition performance in practice.

FIG. 3 schematically shows a flow chart of face retrieval using a system for vehicle driver identification according to an embodiment of the present invention. Similar to the flow chart of face registration shown in Figure 2, the face detector 108, the face alignment unit 206 and the face extractor function in the same way. If the face detector 108 determines that a person's face region 204 is present in the image 202 generated by the image control and processing unit 106 , the face detector 108 uses the bounding box to locate the face region 204 and outputs a series of landmarks 208 . Based on the landmarks 208, the face alignment unit 206 calibrates the posture of the driver's face to an upright posture. The face feature extractor 110 analyzes the aligned face regions 204 and extracts feature vectors 212 representing the face regions 204 .

The feature vector 212 generated by the facial feature extractor 110 will ultimately be used to initiate a query to the facial feature dictionary 112 . More specifically, the face retrieval system 114 attempts to find the most similar feature vector in the face feature dictionary 112 . There are many ways to quantify the similarity between two feature vectors and rank them accordingly. In one embodiment, the face retrieval system is the similarity comparator 316 . The similarity comparator 316 uses cosine similarity as a metric. For two N-dimensional feature vectors f ^q and fi representing the extracted feature vector 212 and the existing feature vectors in the face feature dictionary ¹¹² , their cosine similarity is measured by the following formula.

The ^numerator is the dot product of the two N-dimensional eigenvectors f ^q and fi , and the ^denominator is the product of the magnitudes of the two N-dimensional eigenvectors f ^q and fi . The feature vectors f ^q and f ⁱ are normalized to 1, so the denominator can be omitted, i.e. in this case the cosine similarity is essentially equal to the Euclidean distance. The similarity comparator 316 then compares the cosine similarity to a predetermined similarity threshold. If the cosine similarity is greater than the predetermined similarity threshold, the similarity comparator 316 generates a recognition result 318 . In other words, the similarity comparator 316 identifies the identity of the driver (eg, John Doe).

However, if the similarity comparator 316 does not find any matches in the face feature dictionary 112 (ie, no cosine similarity is greater than a predetermined similarity threshold), the similarity comparator 316 will output a recognition result 318 indicating that the driver has not registration, which in turn indicates the entry of a potentially illegal vehicle or the need for face registration.

As shown in FIGS. 1 and 3 , face retrieval system 114 (eg, similarity comparator 316 ) outputs recognition results 318 to user interface 116 . The user interface 116 then displays the identification results 318 to the driver (and passenger) accordingly. In one embodiment, user interface 116 is in-cabin display 320 . The in-cabin display 320 displays visual or audible feedback to the driver to inform the driver of the recognition result 318 . User interface 116 may be an in-cabin video or audio device including, but not limited to, an instrument panel display, an in-line display, a head-up display, and speakers. The recognition results 318 may be presented graphically or textually, indicating the failure or success of face retrieval. Also, the user interface 116 outputs guidance for the driver to switch between the face registration mode and the face retrieval mode. For security reasons, switching from retrieval mode to enrollment mode requires a second step of authentication, such as password, fingerprint, and key activation.

Face retrieval system 114 (eg, similarity comparator 316 ) may also output recognition results 318 to one or more application interfaces 118 . As shown in FIG. 3 , the one or more application interfaces 118 may include, but are not limited to, a personalized entertainment system 322 , a keyless entry system 324 , and a personalized seating system 326 . The one or more application interfaces 118 may also include an anti-theft system, a driving mode customization system, and a digitally authenticated payment system. All one or more of the application interfaces 118 previously required additional verification of identity, such as passwords, fingerprints, and key activation.

In another aspect of the present invention, the method for vehicle driver identification, as shown in FIG. 4 and FIG. 5 , includes the following steps: FIG. 4 and FIG. 5 together schematically show an embodiment according to the present invention A flowchart of a method for vehicle driver identification. The method may be implemented by the above-described system 100 for vehicle driver identification. It should be noted that this method may be implemented by other means. It should be noted that all or part of the steps according to the embodiments of the present invention may be implemented by hardware or a program instructing the relevant hardware.

At step 402, the NIR LED illuminator 102 emits NIR light in the vehicle.

At step 404, the NIR light sensing unit 104 captures the reflected NIR light.

At step 406, the image control and processing unit 106 coordinates the NIR LED illuminator and the NIR light sensing unit.

At step 408, the image control and processing unit 106 analyzes the reflected NIR light captured by the NIR light sensing unit to generate an image.

At step 410, the face detector 108 determines that a human face is present in the image.

At step 412, the face detector 108 identifies facial regions of the human face.

At step 414, the face detector 108 analyzes the face regions to extract feature vectors representing the face regions. Step 414 is followed by step 502 in FIG. 5 .

At step 502, the face retrieval system 114 determines whether the similarity between the feature vector and any existing feature vector in the face feature dictionary 112 is greater than a first threshold.

When the similarity between the feature vector and the first existing feature vector in the face feature dictionary 112 is greater than the first threshold, at step 504 the face retrieval system 114 generates a flag indicating association with the first existing feature vector the first recognition result. At step 512, the user interface 116 displays the first recognition result.

When the similarity between the feature vector and any existing feature vector in the face feature dictionary 112 is not greater than the first threshold, at step 506 the face retrieval system 114 generates an indication that the feature vector does not exist in the face feature dictionary 112 the second recognition result. At step 508, the user interface 116 displays the second recognition result. At step 510 , the facial feature dictionary 112 stores the facial features in the facial feature dictionary 112 .

Yet another aspect of the present invention provides a non-transitory tangible computer-readable medium storing instructions that, when executed by one or more processors, cause the above-disclosed method for vehicle driver identification to be performed. The computer-executable instructions or program code enables the above-disclosed apparatus or similar system to perform various operations in accordance with the above-disclosed methods. The storage medium or memory may include, but is not limited to, high-speed random access media or memory such as DRAM, SRAM, DDR RAM, or other random access solid state storage devices, and storage devices such as one or more magnetic disks, optical disks, flash memory, or Non-volatile storage devices for other non-volatile solid-state storage devices.

The foregoing description of the exemplary embodiments of the present invention has been presented for purposes of illustration and description only, and is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings.

The embodiments were chosen and described in order to explain the principles of the invention and its practical application to enable others skilled in the art to utilize the invention and various embodiments as are suited to the particular use contemplated. Alternative embodiments will become apparent to those skilled in the art to which the invention pertains without departing from the spirit and scope of the invention. Accordingly, the scope of the present invention is to be defined by the appended claims rather than the foregoing description and the exemplary embodiments described therein.

Several references, which may include patents, patent applications, and various publications, are cited and discussed in the specification of the present disclosure. The citation and/or discussion of such references is provided solely to clarify the description of the present disclosure and is not an admission that any such reference is "prior art" to the disclosure described herein. All references cited and discussed in this specification are hereby incorporated by reference in their entirety to the same extent as if each reference was incorporated by reference individually.

Claims (13)

1. A device for vehicle driver identification, comprising: a near-infrared NIR light-emitting diode LED illuminator configured to emit NIR light in the vehicle; a near-infrared NIR light sensing unit configured to capture reflected NIR light; an image control and processing unit configured to coordinate the NIR LED illuminator and the NIR light sensing unit and analyze the reflected NIR light captured by the NIR light sensing unit to generate an image; a face detector configured to determine the presence of a human face in the image, and to identify a facial region of the human face; a facial feature extractor configured to analyze the facial region to extract feature vectors representing the facial region; A face feature dictionary, configured to store existing feature vectors; a face retrieval system configured to generate a recognition result indicating whether the similarity between the feature vector and any of the existing feature vectors is greater than a first threshold; and A user interface configured to display the recognition result. 2. The apparatus of claim 1, wherein the image control and processing unit is configured to control the duty cycle of the NIR LED illuminator, the analog gain of the NIR light sensing unit, the NIR One or more of the digital gain of the light sensing unit, the exposure time of the NIR light sensing unit, and the frame rate of the NIR light sensing unit coordinate the NIR LED illuminator and the NIR light sensing unit . 3. The apparatus of claim 2, wherein the image control and processing unit is configured to coordinate the NIR LED illuminator and the NIR light sensing unit to generate an image with optimal imaging quality. 4. The apparatus of claim 1, further comprising: a face alignment unit configured to calibrate the face region to a calibrated face region associated with the driver's upright posture, wherein the facial feature extractor is configured to analyze the calibrated face regions to extract feature vectors representing the calibrated face regions. 5. The apparatus of claim 1, wherein the facial feature extractor is configured to employ one or more of backbone networks, local feature descriptors, clustering techniques, and dimensionality reduction techniques. 6. The apparatus of claim 1, wherein the similarity is a cosine similarity. 7. A method for vehicle driver identification, comprising: emitting NIR light in the vehicle through a near-infrared NIR light-emitting diode LED illuminator; The reflected NIR light is captured by a near-infrared NIR light sensing unit; Coordinating the NIR LED illuminator and the NIR light sensing unit through an image control and processing unit; analyzing, by an image control and processing unit, the reflected NIR light captured by the NIR light sensing unit to generate an image; determining that a human face exists in the image; identifying the facial region of the human face; analyzing the face region to extract feature vectors representing the face region; determining whether the similarity between the feature vector and any existing feature vector in the face feature dictionary is greater than a first threshold; and when the similarity between the feature vector and the first existing feature vector in the face feature dictionary is greater than the first threshold, generating a first recognition result indicative of an identity associated with the first existing feature vector; and displaying the first recognition result; when the similarity between the feature vector and any existing feature vector in the face feature dictionary is not greater than the first threshold, generating a second recognition result indicating that the feature vector does not exist in the facial feature library; displaying the second recognition result; and The facial features are stored in the facial feature dictionary. 8. The method of claim 7, wherein the image control and processing unit controls the duty cycle of the NIR LED illuminator, the analog gain of the NIR light sensing unit, the NIR light sensing One or more of the digital gain of the unit, the exposure time of the NIR light sensing unit, and the frame rate of the NIR light sensing unit coordinate the NIR LED illuminator and the NIR light sensing unit. 9. The method of claim 8, wherein the image control and processing unit coordinates the NIR LED illuminator and the NIR light sensing unit to generate images with optimal imaging quality. 10. The method of claim 7, further comprising: The face region is calibrated to a calibrated face region associated with the driver's upright posture, wherein the calibrated face region is analyzed to extract the feature vector representing the calibrated face region. 11. The method of claim 7, wherein the face region is analyzed to extract representations of the face region by employing one or more of backbone networks, local feature descriptors, clustering techniques, and dimensionality reduction techniques eigenvectors of . 12. The method of claim 7, wherein the similarity is a cosine similarity. 13. A non-transitory tangible computer-readable medium storing instructions that, when executed by one or more processors, cause a method for vehicle driver identification to be performed, the method comprising: emitting NIR light in the vehicle through a near-infrared NIR light-emitting diode LED illuminator; The reflected NIR light is captured by a near-infrared NIR light sensing unit; Coordinating the NIR LED illuminator and the NIR light sensing unit through an image control and processing unit; analyzing, by the image control and processing unit, the reflected NIR light captured by the NIR light sensing unit to generate an image; determining that a human face exists in the image; identifying the facial region of the human face; analyzing the face region to extract feature vectors representing the face region; determining whether the similarity between the feature vector and any existing feature vector in the face feature dictionary is greater than a first threshold; and when the similarity between the feature vector and the first existing feature vector in the face feature dictionary is greater than the first threshold, generating a first recognition result indicative of an identity associated with the first existing feature vector; and displaying the first recognition result; when the similarity between the feature vector and any existing feature vector in the face feature dictionary is not greater than the first threshold, generating a second recognition result indicating that the feature vector does not exist in the facial feature library; displaying the second recognition result; and The facial features are stored in the facial feature dictionary.

Images