WO2002032136A2

WO2002032136A2 - Device control via image-based recognition

Info

Publication number: WO2002032136A2
Application number: PCT/EP2001/011141
Authority: WO
Inventors: Miroslav Trajkovic; Yong Yan; Antonio Colmenarez; Srinivas Gutta
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2000-10-10
Filing date: 2001-09-26
Publication date: 2002-04-18
Also published as: KR20020081220A; JP2004511870A; WO2002032136A3; CN1395797A; EP1329105A2

Abstract

A video device is coupled to an appliance controller, via an image-based user recognition application. When a particular user's face is recognized in the provided image, or sequence of images, a controller accesses a database containing a user profile corresponding to the recognized user. This user profile contains the user's preferences for presentation material, such as preferences for particular types of entertainment at particular times of the day. Based on the information in the user profile, the controller activates one or more devices, such as a television set or stereo set, and selects material for presentation to the user that is likely to coincide with the user's desires at that time.

Description

Device control via image-based recognition

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to the field of consumer electronics, and in particular to a system and method for automated control of devices based on images from a video capture device in the vicinity of the controlled device.

2. Description of Related Art

Face recognition is commonly used for security purposes. In a manual security system, security badges containing facial photographs are used to control access to secured areas or secured material. In automated and semi-automated systems, face recognition software is used to similarly match a current image of a person, from, for example, a video camera, with a stored image. In conventional systems, the user identifies himself or herself, and the face recognition software compares the video image with one or more stored images of the identified person. U.S. patent 5,956, 482, "MULTIMEDIA INFORMATION SERVICE

ACCESS" of Agraharam et al, issued 21 September 1999, incorporated by reference herein, presents a security technique wherein a user requests access to an information service, the system takes a video snapshot of the user, and grants access to the information service only if the snapshot corresponds to an authorized user. U.S. patent 5,835,616, "FACE DETECTION USING TEMPLATES " of Lobo et al, issued 10 November 1998, incorporated by reference herein, presents a two step process for automatically finding a human face in a digitized image, and for confirming the existence of the face by examining facial features. The system of Lobo et al is particularly well suited for finding one or more faces within a camera's field of view, even though the view may not correspond to a typical facial snapshot.

In the field of consumer electronics, and in particular to the field of broadcast, cable, and satellite television, wherein a user has access to literally hundreds of choices at any given time, systems and methods are continually being developed to organize and present available content material to a user based on a characterization of the user's preferences. In order to effect this user-specific organization and presentation, the user typically identifies himself or herself to the control system, by providing a user name, or by presenting an identification device. Although user-identification is commonplace while using computer devices, or in particular environments, such as an office environment, user identification within a less structured environment, such as a home, may present problems. In an office environment, for example, employees can be expected to wear identification badges, or to purposefully engage biometric scanners, such as a facial recognition device, a retinal scanner, a fingerprint scanner, and so on. In a home environment, on the other hand, having to type in an identification sequence, or to carry an identification token, or to purposefully engage a recognition device, in order to activate or otherwise control an entertainment device would likely be viewed as an annoyance, or at least an inconvenience.

BRIEF SUMMARY OF THE INVENTION

It is an object of this invention to provide an unobtrusive means of identifying a user to facilitate the control in a home-automation system. It is a further object of this invention to provide an automated user-specific presentations of material based on a recognition of the user. It is a further object of this invention to provide automated television activation and channel selection, based on an automated recognition of a user. It is a further object of this invention to provide automated control of appliances, based on an automated recognition of a user.

These objects and others are achieved by providing a video device that is coupled to an appliance controller, via an image-processing application. When a particular user is recognized in an image, a controller accesses a database containing a user profile corresponding to the recognized user. This user profile contains the user's preferences for presentation material, such as preferences for particular types of entertainment at particular times of the day. Based on the information in the user profile, the controller activates one or more devices, such as a television set or stereo set, and selects or recommends material for presentation to the user that is likely to coincide with the user's desires at that time.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is explained in further detail, and by way of example, with reference to the accompanying drawings wherein:

FIG. 1 illustrates an example television controller with face recognition capabilities in accordance with this invention. FIG. 2 illustrates an example flow diagram of an appliance controller with face recognition in accordance with this invention.

Throughout the drawings, the same reference numerals indicate similar or corresponding features or functions.

DETAILED DESCRIPTION OF THE INVENTION

With the advent of home automation technologies, applications continue to be developed to facilitate the control of appliances, and in particular home entertainment systems. This facilitation of control includes, for example, automatically activating a device, providing recommendations for selecting from among a variety of options, limiting access to select material or controls, and so on.

Copending U.S. patent application, "PERSONALIZED NEWS RETRIEVAL SYSTEM", serial number 09/220,277, filed 12/23/98 for Jan H. Elenbaas, Tomas McGee, Nevenka Dimitrova, and Mark Simpson, Attorney Docket PHA 23,590 presents techniques for customizing the categorization and retrieval of information based on a user's preferences or viewing habits, and is incorporated by reference herein. In this copending application, a user is presented news items, or other content material, based on a user profile that is developed based on expressed preferences or implied preferences, based on viewing habits. Copending U.S. patent application, "HOME CONTROL SYSTEM WITH DISTRIBUTED, NETWORKED DEVICES", serial number 09/222,403, filed 12/29/98 for Doreen Y. Cheng, Attorney Docket PHA 23,605, incorporated by reference herein, presents techniques for controlling appliances, such as televisions, radios, and so on, based on the presence of a person in particular locales, such as rooms of a home. In this copending application, each user wears a locator device that notifies one or more controllers of the presence of the user within each locale. Based on a "profile" that is associated with each locator device, and thereby each user, the home control system activates and deactivates devices to comply with assumed desires of the identified user. When a user initially enters a room, the television or stereo may be activated, depending upon the user profile, and depending upon the time of day and other factors. The particular television channel or music selection is also selected based on the user profile, time of day, and so on. As the user traverses from one room to another, a television in the room that is entered is turned on and tuned to the station that the user was viewing in the prior room, and the television in the prior room is turned off, unless another locator device is detected in the prior room. Copending U.S. patent application "REMOTE CONTROL PROGRAM SELECTION BY GENRE", U.S. serial number 09/282,319, filed 03/31/99 for Karen Travato, Dan Pelletier, Paul Ran in, and Jacquelyn Martino, Attorney Docket PHA 23,644, incorporated by reference herein, presents techniques for programming the channel-up and channel-down keys of a remote control based on genre, and also based on a user profile for particular preferred channels.

In each of these aforementioned inventions, the system provides control, or recommendations, based on an identification of the current user or users of the device being controlled. As noted above, an 'active' method of user identification, such as the entry of a user name or code, or carrying an identification token, may not be consistent with a consumer's view of "convenience". In accordance with this invention, this user identification is provided in a 'passive' manner, based on a recognition of the presence of the user in the vicinity of the device being controlled. An image capture device, such as a video camera, captures an image of a scene in the vicinity of the device being controlled, and a user- recognition application, such as a face-recognition application, identifies the user from the image.

FIG. 1 illustrates an example device control system 100 with user-recognition capabilities in accordance with this invention. For ease of reference, a television system 130 is presented as the device being controlled, although any other user-controllable device, such as a stereo system, an Internet access device, and so on may be the device being controlled. An imaging device 110, such as a video camera, captures an image of objects within its field of view 115, and provides this image to a user-recognizer 120.

In a preferred embodiment, the user-recognizer 120 is a multi-stage face recognition device that searches the image for an area of the image having the general characteristics of a face, based on the location of flesh tones, the location of non-flesh tones corresponding to eye brows, demarcation lines corresponding to chins, nose, and so on (see the referenced US patent 5,835,616). If a face is detected, it is characterized for comparison with reference faces that are stored in a database associated with the face recognizer 120. This characterization of the face in the video image is preferably the same characterization process that is used to characterize the reference faces, and facilitates a comparison of faces based on characteristics, rather than an 'optical' match, thereby obviating the need to have two identical images (current face and reference face) in order to locate a match. In a preferred embodiment, the number of reference faces is relatively small, typically limited to the number of people in a household, or other small sized environment, thereby allowing the face recognition process to be effected quickly.

When the user-recognizer 120 determines that a face in the video image to correspond to a particular user, an identification of the user is communicated to a device controller 140. In the example embodiment of FIG. 1, a television is illustrated as the paradigm device 130 being controlled. The controller 140 includes a 'profile' corresponding to each recognizable user that preferably includes the user's preferences for the operation of the device 130, including preferences that are time dependent. These preferences may be explicitly input by each user, or may be based on prior experiences. For example, if a particular user always turns on the television at 6 p.m., and views a news program on channel 8, the profile will contain this 'preference'. Note that the term 'user preference' is used herein in the general sense. A child's profile may be defined, for example, based on the child's parents' preferences. These preferences may include a "preference" for only educational shows during particular hours, no access during other hours, no access to particular classes of material, a limit to the number of hours that a television can be viewed per day, and so on.

Any of a variety of machine learning techniques, common in the art, may also be used to develop correlations between each user and typical control activities, and thereafter imitate the execution of the control activities based on these correlations.

When a particular user is recognized, the controller 140 executes control options corresponding to the assumed or explicit preference of the user. In the aforementioned example, if the user is detected in the vicinity of the television at 6 p.m., the controller 140 turns on the television 130, and tunes it to channel 8. If the user generally turns on the television whenever the user enters the room, and generally leaves the television tuned to whatever channel is currently turned on until 6 p.m., the controller 140 will imitate this behavior by turning the television on when the user is detected, and tuning to channel 8 only at 6 p.m. These and other techniques for controlling a device's operation based on a recognition of a particular user will be evident to one of ordinary skill in the art, particularly in view of the aforementioned copending patent applications. Note that, for ease of reference and understanding, the term 'control' is used herein to include any and all techniques that facilitate control of a device, including the aforementioned recommendation or prioritization of options, automated and semi-automated activation of options, and so on.

As will be evident to one of ordinary skill in the art, other methods of user recognition may be utilized in lieu of, or in addition to, the face recognition techniques, above. For example, a gait recognition device can be used to identify a person based on the motion of a person in the image. A less complex recognition device may be based on the user's size. For example, the user profiles may be segregated into 'adult' and 'child' profiles, wherein some or all controls are based on whether the user is an adult or child based on the user's size. In a less sophisticated embodiment, the recognition may merely be the presence of a person in a particular location, such as a user's favorite chair, using for example, detection via an infrared image collector. Other image-based techniques, such as color distribution, edge distribution, and the like, may also be used to facilitate the identification of users. These and other embodiments may also be used to 'pre-process' the image information to facilitate a rapid identification from the video image. Copending U.S. patent application, "USING A

SECONDARY SENSOR FOR OPTIMIZED VIDEO COMMUNICATIONS", serial number 09/634,682, filed 8/8/00 for Michael Bakhmutsky, Attorney Docket US000158, incorporated by reference herein, presents techniques for finding warm items, such as humans, in a video image, to optimize the encoding of specific portions of an image, based on a lower-cost, lower-resolution infrared image that corresponds to the video image. This same principle can be used to provide a preliminary user identification, which is further modified or confirmed based on a further processing of a video image.

FIG. 2 illustrates an example flow diagram of a device control system with user recognition in accordance with this invention. Face recognition is used in FIG. 2 as the paradigm of image-based user recognition, for ease of understanding. At 210, the video image (from the imaging device 110 of FIG. 1) is scanned to determine the presence of a face in the image. This scanning may be continuous, periodic, event-driven, or otherwise initiated. For example, the scanning may be continuous until a recognized face is detected, then periodically scanned, for example, at intervals corresponding to the traditional 'program change' periods (in the U.S., programs generally change on half-hour or hour boundaries). Alternatively, the scanning may be coupled to a television, and activated at the beginning of each commercial break.

If, at 215, a face is detected in the image, the characteristics of the detected face are determined, at 220. If a user has been identified, via a prior image scan, the characteristics of the detected face are compared to the characteristics of the identified user, at 225. If the detected face does not correspond to the identified user, or if there is no currently identified user, a database of user faces 201 is searched, at 230, for a user face having the same characteristics of the detected face. If, at 235, a user is identified that corresponds to the detected face, the identification of the "current" user is set, at 240, and the device is controlled corresponding to this current user identification, at 260.

If, at 235, a user corresponding to the face is not identified, the current user ID is cleared. Alternatively, not illustrated, the new user's face characteristics can be added to the database of user faces 201, and a new user profile is created. Preferably, the new user profile is based on a default profile that can be created by a 'system administrator' of the device control system. As will be evident to one of ordinary skill in the art, this lack of user- identification at 235 can also be used to create the database of user faces 201, or to provide alternative images of existing identified users in the database 201. Also alternatively, if, at 235, a user corresponding to the face is not identified, the controller may default to a defined "guest" user profile.

If, at 215, a face is not detected in the image, the identification of a "current" user is cleared, at 250, and the device is controlled corresponding to the absence of a current user identification, at 260. In a multi-user environment, the identification of the "current" user comprises a set of identifications corresponding to each current user, and the process 210-260 is repeated for each face that is detected, or not detected, in each image. That is, for example, the decision block 215 results in a "yes" determination for each face in the image, then results in a "no" determination when there are no other faces in the image. Block 250 receives this "no" determination, and removes any identifications of users who have not been identified in the current image from the set of current users.

As discussed above, the control of the device, at 260, is dependent upon a database of user profiles 202. If a single user is identified, the control of the device corresponds directly to the determined user's profile. In a simple embodiment of this invention, if multiple users are identified, only unanimous preferences are effected. That is, for example, if each user generally turns on the television, or generally leaves the television in whatever state (on/off) it was in, when the user enters the room, the control block 260 is configured to turn on the television whenever the first user who generally turns on the television enters the room. If, however, another user, who normally turns off the television, subsequently enters the room, the control block 260 is configured to take no action until the other users are no longer in the room. In like manner, if the set of currently identified users have differing channel or genre selection preferences, the control block 260 takes no action until a coincidence of preferences occurs. In a more sophisticated embodiment, the database 202 includes multi-user profiles, such as the channels that are normally viewed when users A and B, but not C, are present, and so on.

As would be evident to one of ordinary skill in the art, the control block 260 and the user profile 202 may also operate using state-dependent rules. For example, if a particular user generally leaves and enters a room repeatedly, the rules may be structured to invoke a 'waiting period' before clearing the user's identification from the current set, at 250.

The foregoing merely illustrates the principles of the invention. It will thus be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and are thus within its spirit and scope. For example, the control system 100 may be configured to operate with other control systems, or within other control system architectures, as well. For example, the devices 110-140 may be interconnected via a home automation network and control protocol. Home automation standards include IEEE 1394, X-10, HAVi, HomeAPI, Jini, and the like. IEEE 1394 and X-10 are communication protocols; HAVi is a software architecture using IEEE 1394; Home API is an open industry specification that defines a standard set of software services and application programming interfaces which enable software applications to monitor and control home device; and Jini is a distributed software architecture (network) wherein clients see devices and services as objects. The television controller 140 may include, for example, a personal computer with an IEEE 1394 interface to a HAVi-compliant satellite television receiver, and the face recognizer 120 is a software application that is also located within the personal computer. As would also be evident to one of ordinary skill in the art, the control of the device may be via the control an ancillary device, such as controlling the channel being viewed on a television by controlling a cable or satellite receiver system, a VCR tuner, and the like. These and other system configuration and optimization features will be evident to one of ordinary skill in the art in view of this disclosure, and are included within the scope of the following claims.

Claims

CLAIMS:

1. A device control system (100) comprising: an imaging device (110) that is configured to collect an image within a field of view (115) of the imaging device (110), a user recognizer (120) that is configured to identify a user based on a recognition of characteristics of the user within the image, and a device controller (140) that is configured to facilitate control of a device (130), based on a profile of preferences (202) associated with the user.

2. The system (100) of claim 1, further including: a database that is configured to contain a plurality of reference characteristics, and wherein the user recognizer (120) identifies the user based on a comparison of characteristics within the image to the plurality of reference characteristics.

3. The system (100) of claim 2, wherein the database (201) is also configured to contain a plurality of profiles of preferences (202) that are associated with the plurality of reference characteristics.

4. The system (100) of claim 2, wherein the plurality of reference characteristics includes at least one of: a plurality of face characteristics, a plurality of gait characteristics, a plurality of infrared characteristics, a plurality of color characteristics, and a plurality of size characteristics.

5. The system (100) of claim 1, wherein the profile of preferences (202) includes time-dependent preferences.

6. The system (100) of claim 1 , wherein the device controller (140) effects the control of the device (130) via a home- automation network.

7. The system (100) of claim 1 , wherein the device (130) is a television receiver, and the profile of preferences (202) includes preferred channels at specified times.

8. The system ( 100) of claim 1 , wherein the control of the device (130) includes at least one of: an activation of the device (130), and a tuning of the device (130).

9. The system (100) of claim 1, wherein the user recognizer (120) is further configured to identify at least one other user, based on a recognition of characteristics of the at least one user within the image, and the device controller (140) is further configured to effect the control of the device (130) based on preferences associated with the at least one other user.

10. A method for controlling a device (130), comprising: recognizing (220) characteristics within an image, identifying a user based on a comparison (235) of the characteristics with a plurality of user characteristics (201), and facilitating control (260) of the device (130) based on one or more preferences associated with the user.

11. The method of claim 10, further including selecting the one or more preferences associated with the user from a plurality of preferences (202) that correspond to the plurality of user characteristics.

12. The method of claim 11 , wherein the device (130) is a television receiver, and the one or more preferences includes preferred channels at specified times.

13. The method of claim 10, wherein facilitating control (260) of the device (130) includes at least one of: activating the device (130), recommending one or more control settings of the device (130), presetting a control setting in a control device for the device (130), and tuning the device (130).

14. The method of claim 10, further including: recognizing other characteristics within the image, identifying an other user based on a comparison of the other characteristics with the plurality of user characteristics (201), and further facilitating control of the device (130) based on one or more preferences associated with both the user and the other user.

15. A television control system (100) comprising: a video camera (110) that provides an image, a face recognition device (120), operably coupled to the video camera (110), that identifies a user based on the image, a television controller (140), operably coupled to the face recognition device (120), that facilitates a control of the television (130) based on preferences associated with the user.

16. The television control system (100) of claim 15, wherein the control of the television (130) includes at least one of: activating the television (130), recommending one or more settings of the television (130), presetting options of the television (130), and selecting a source of material for presentation on the television (130).

17. The television control system (100) of claim 15, wherein the user recognition device is further configured to recognize multiple users based on the image, and the television controller (140) is further configured to effect the control of the television (130) based on preferences associated with the multiple users.